• home
• products
• learn
• free games
• company

Tutorials -> Java Beginner -> Tutorial 0

Tutorial 2: Use Your Own

• Author: Brian A. Ree

  
  

  Files/Resources for this Tutorial
  Project Files (PyCharm)
  Make Your Own Neural Network

  
  

  1: The MNIST Dataset of Handwritten Numbers

  There is a collection of images of handwritten numbers used by artificial intelligence researchers as a popular set to test their latest code. The dataset is called the MNIST database of handwritten digits, and is available from the respectable neural network researcher Yann LeCun's site http://yann.lecun.com/exdb/mnist/.

  
  

  The format of the MNIST database isn't the easiest to work with, so others have helpfully created data files of a simpler format, such as this one http://pireddie.com/project/mnist-in-csv/. You can research online and find different MNIST databases and conversions quite easily for more information and format options. Before we can do anything with the data, like training a neural network with it, we need to load the data with some python code.

  
  

  # load the mnist training data CSV file into a list
  training_data_file = open("mnist_dataset/mnist_train_100.csv", 'r')
  training_data_list = training_data_file.readlines()
  training_data_file.close()
  

  
  
  

  This code is straight forward it takes a CSV file path and loads in all the rows in one call. The file handle is then closed and we're ready to process our data. Let's take a look at the code to parse the CSV data.

  
  

   # go through all records in the training data set
  for record in training_data_list:
      # split the record by the',' commas
      all_values = record.split(',')
  
      # scale and shift the inputs
      inputs = (numpy.asfarray(all_values[1:])/ 255.0 * 0.99) + 0.01
  
      # create the target output values (all 0.01, except the desired label which is 0.99)
      targets = numpy.zeros(output_nodes) + 0.01
  
      # all_values[0] is the target label for this record
      targets[int(all_values[0])] = 0.99
      n.train(inputs, targets)
  
      pass
  

  
  
  

  2: Preparing the MNIST Training Data

  We added special checks to our class to enforce certain sizes on our data. We need to reshape our input data so that our values stay within the comfort zone of the network node activation method. We have to scale the input color values from the larger range 0 to 255 to the much smaller range 0.01 - 1.0. The low end value is purposely larger than zero to prevent the zero value from artificially killing some of the node's output. The following python code scales the input.

  
  

  # split the record by the',' commas
  all_values = record.split(',')
      
  # scale and shift the inputs
  inputs = (numpy.asfarray(all_values[1:])/ 255.0 * 0.99) + 0.01
  

  
  
  

  The output of, print(inputs), confirms we have the correct values. We are asking the network to cassify the image and assign it the correct label. The label options are the numbers zero to nine. That means we will have an output layer of ten nodes, one of the possible outcomes, or labels. The following python code constructs the target matrix.

  
  

  onodes = 10
  
  # create the target output values (all 0.01, except the desired label which is 0.99)
  targets = numpy.zeros(output_nodes) + 0.01
      
  # all_values[0] is the target label for this record
  targets[int(all_values[0])] = 0.99
  

  
  
  

  We now worked out how to prepare the inputs for training and querying, and the output for training too. The reason we chose 784 input nodes is because the input images have 28x28 pixes or 784 individual pixels which make up the handwritten number image.

  
  
  
  

  3: Testing the Network

  Let's test how well our training worked we do this via a comparison with a second dataset the validation dataset. Loading the validation data is similar to loading the training data.

  
  

  # load the mnist test data CSV file into a list
  test_data_file = open("mnist_dataset/mnist_test_10.csv", 'r')
  test_data_list = test_data_file.readlines()
  test_data_file.close()
  

  
  
  

  We process the validation data in the same way we processed the training data. Let's see how the validation code looks.

  
  

  # scorecard for how well the network performs, initially empty
  scorecard = []
  
  # go through all the records in the test data set
  for record in test_data_list:
      # split the record by the',' commas
      all_values = record.split(',')
  
      # correct answer is first value
      correct_label = int(all_values[0])
      print(correct_label, "correct label")
  
      # scale and shift the inputs
      inputs = (numpy.asfarray(all_values[1:])/ 255.0 * 0.99) + 0.01
  
      # query the network
      outputs = n.query(inputs)
  
      # the index of the highest value corresponds to the label
      label = numpy.argmax(outputs)
      print(label, "network's answer")
  
      # append correct or incorrect to list
      if (label == correct_label):
          # network's answer matches correct answer, add 1 to scorecard
          scorecard.append(1)
      else:
          # network's answer doesn't match correct answer, add 0 to scorecard
          scorecard.append(0)
          pass
      
      pass
  

  
  
  

  The code is very similar to the training data code except here we get our output value by calling the query method on our trained neural network. The answer is pulled out of an array of output values, the highest value marks the chosen label. The results are stored in our local scorecard variable, this will tell us how many correct matches we get. It's kind of amazing what we were able to accomplish with just a three layer neural network, we can match handwritten numbers! Amazing!! We get about a 60% accuracy with just 100 rows in our training set, not too bad. See if you can beat that with more data. Let's take a look at our final code.

  
  

  # python notebook for Make Your Own Neural Network #code for a 3­layer neural network,
  # and code for learning the MNIST dataset #(c) Tariq Rashid, 2016 #license is GPLv2
  
  import numpy
  import scipy.special
  
  class MyNn:
  
      # Initialize a simple 3 layer neural network.
      def __init__(self, inputnodes, hiddennodes, outputnodes, learningrate):
          # Set the number of nodes in each layer.
          self.inodes = inputnodes
          self.hnodes = hiddennodes
          self.onodes = outputnodes
  
          # if len(self.inodes) != len(self.hnodes) or len(self.inodes) != len(self.onodes):
          #    print("Error: You must provide nodes lists of the same size.")
          #    print("Unexpected results may occur.")
          # eif
  
          # Set the learning rate.
          self.lr = learningrate
  
          if self.lr > 1.0:
              print("Error: You must provide a learning rate that is less than or equal to one.")
              print("Unexpected results may occur.")
          # eif
  
          # Set the link weight matrices, wih and who.
          self.wih = numpy.random.normal(0.0, pow(self.hnodes, -0.5), (self.hnodes, self.inodes))
          self.who = numpy.random.normal(0.0, pow(self.onodes, -0.5), (self.onodes, self.hnodes))
  
          # Set the activation function for our neurons.
          self.activation_function = lambda x: scipy.special.expit(x)
      # edef
  
      def train(self, inputs_list, answers_list):
          # Convert inputs to a two dimensional matrix
          inputs = numpy.array(inputs_list, ndmin=2).T
          answers = numpy.array(answers_list, ndmin=2).T
  
          # Calculate the signals from the input layer to the hidden layer
          hidden_inputs = numpy.dot(self.wih, inputs)
  
          # Calculate the signals from the hidden layer to the output layer
          hidden_outputs = self.activation_function(hidden_inputs)
  
          # Calculate signals into final output layer
          final_inputs = numpy.dot(self.who, hidden_outputs)
  
          # Calculate the signals emerging from the final output layer
          final_outputs = self.activation_function(final_inputs)
  
          # Output layer error is the (answer - guess)
          output_errors = answers - final_outputs
  
          # Hidden layer error is the output_errors, split by weights, recombined at hidden nodes
          hidden_errors = numpy.dot(self.who.T, output_errors)
  
          # Update the weights for the links between the hidden and output layers
          self.who += self.lr * numpy.dot((output_errors * final_outputs * (1.0 - final_outputs)), numpy.transpose(hidden_outputs))
  
          # Update the weights for the links between the input and hidden layers
          self.wih += self.lr * numpy.dot((hidden_errors * hidden_outputs * (1.0 - hidden_outputs)), numpy.transpose(inputs))
      # edef
  
      def query(self, inputs_list):
          # Convert inputs to a two dimensional matrix
          inputs = numpy.array(inputs_list, ndmin=2).T
  
          # Calculate the signals from the input layer to the hidden layer
          hidden_inputs = numpy.dot(self.wih, inputs)
  
          # Calculate the signals from the hidden layer to the output layer
          hidden_outputs = self.activation_function(hidden_inputs)
  
          # Calculate signals into final output layer
          final_inputs = numpy.dot(self.who, hidden_outputs)
  
          # Calculate the signals emerging from the final output layer
          final_outputs = self.activation_function(final_inputs)
          return final_outputs
      # edef
  
  # eclassvbh
  
  # number of input, hidden and output nodes
  input_nodes = 784
  hidden_nodes = 100
  output_nodes = 10
  
  # learning rate is 0.3
  learning_rate = 0.3
  
  # create instance of neural network
  n = MyNn(input_nodes, hidden_nodes, output_nodes, learning_rate)
  
  # load the mnist training data CSV file into a list
  training_data_file = open("mnist_dataset/mnist_train_100.csv", 'r')
  training_data_list = training_data_file.readlines()
  training_data_file.close()
  
  # train the neural network
  # go through all records in the training data set
  for record in training_data_list:
      # split the record by the',' commas
      all_values = record.split(',')
  
      # scale and shift the inputs
      inputs = (numpy.asfarray(all_values[1:])/ 255.0 * 0.99) + 0.01
  
      # create the target output values (all 0.01, except the desired label which is 0.99)
      targets = numpy.zeros(output_nodes) + 0.01
  
      # all_values[0] is the target label for this record
      targets[int(all_values[0])] = 0.99
      n.train(inputs, targets)
      pass
  
  # load the mnist test data CSV file into a list
  test_data_file = open("mnist_dataset/mnist_test_10.csv", 'r')
  test_data_list = test_data_file.readlines()
  test_data_file.close()
  
  # test the neural network
  # scorecard for how well the network performs, initially empty
  scorecard = []
  
  # go through all the records in the test data set
  for record in test_data_list:
      # split the record by the',' commas
      all_values = record.split(',')
  
      # correct answer is first value
      correct_label = int(all_values[0])
      print(correct_label, "correct label")
  
      # scale and shift the inputs
      inputs = (numpy.asfarray(all_values[1:])/ 255.0 * 0.99) + 0.01
  
      # query the network
      outputs = n.query(inputs)
  
      # the index of the highest value corresponds to the label
      label = numpy.argmax(outputs)
      print(label, "network's answer")
  
      # append correct or incorrect to list
      if (label == correct_label):
          # network's answer matches correct answer, add 1 to scorecard
          scorecard.append(1)
      else:
          # network's answer doesn't match correct answer, add 0 to scorecard
          scorecard.append(0)
          pass
      pass