﻿ Middlemind Games

## Tutorial 2: Use Your Own

• ### Author: Brian A. Ree

#### 1: The MNIST Dataset of Handwritten Numbers

##### The format of the MNIST database isn't the easiest to work with, so others have helpfully created data files of a simpler format, such as this one http://pireddie.com/project/mnist-in-csv/. You can research online and find different MNIST databases and conversions quite easily for more information and format options. Before we can do anything with the data, like training a neural network with it, we need to load the data with some python code.

```# load the mnist training data CSV file into a list
training_data_file = open("mnist_dataset/mnist_train_100.csv", 'r')
training_data_file.close()
```

##### This code is straight forward it takes a CSV file path and loads in all the rows in one call. The file handle is then closed and we're ready to process our data. Let's take a look at the code to parse the CSV data.

``` # go through all records in the training data set
for record in training_data_list:
# split the record by the',' commas
all_values = record.split(',')

# scale and shift the inputs
inputs = (numpy.asfarray(all_values[1:])/ 255.0 * 0.99) + 0.01

# create the target output values (all 0.01, except the desired label which is 0.99)
targets = numpy.zeros(output_nodes) + 0.01

# all_values is the target label for this record
targets[int(all_values)] = 0.99
n.train(inputs, targets)

pass
```

#### 2: Preparing the MNIST Training Data

##### We added special checks to our class to enforce certain sizes on our data. We need to reshape our input data so that our values stay within the comfort zone of the network node activation method. We have to scale the input color values from the larger range 0 to 255 to the much smaller range 0.01 - 1.0. The low end value is purposely larger than zero to prevent the zero value from artificially killing some of the node's output. The following python code scales the input.

```# split the record by the',' commas
all_values = record.split(',')

# scale and shift the inputs
inputs = (numpy.asfarray(all_values[1:])/ 255.0 * 0.99) + 0.01
```

##### The output of, print(inputs), confirms we have the correct values. We are asking the network to cassify the image and assign it the correct label. The label options are the numbers zero to nine. That means we will have an output layer of ten nodes, one of the possible outcomes, or labels. The following python code constructs the target matrix.

```onodes = 10

# create the target output values (all 0.01, except the desired label which is 0.99)
targets = numpy.zeros(output_nodes) + 0.01

# all_values is the target label for this record
targets[int(all_values)] = 0.99
```

#### 3: Testing the Network

##### Let's test how well our training worked we do this via a comparison with a second dataset the validation dataset. Loading the validation data is similar to loading the training data.

```# load the mnist test data CSV file into a list
test_data_file = open("mnist_dataset/mnist_test_10.csv", 'r')
test_data_file.close()
```

##### We process the validation data in the same way we processed the training data. Let's see how the validation code looks.

```# scorecard for how well the network performs, initially empty
scorecard = []

# go through all the records in the test data set
for record in test_data_list:
# split the record by the',' commas
all_values = record.split(',')

# correct answer is first value
correct_label = int(all_values)
print(correct_label, "correct label")

# scale and shift the inputs
inputs = (numpy.asfarray(all_values[1:])/ 255.0 * 0.99) + 0.01

# query the network
outputs = n.query(inputs)

# the index of the highest value corresponds to the label
label = numpy.argmax(outputs)

# append correct or incorrect to list
if (label == correct_label):
scorecard.append(1)
else:
scorecard.append(0)
pass

pass
```

##### The code is very similar to the training data code except here we get our output value by calling the query method on our trained neural network. The answer is pulled out of an array of output values, the highest value marks the chosen label. The results are stored in our local scorecard variable, this will tell us how many correct matches we get. It's kind of amazing what we were able to accomplish with just a three layer neural network, we can match handwritten numbers! Amazing!! We get about a 60% accuracy with just 100 rows in our training set, not too bad. See if you can beat that with more data. Let's take a look at our final code.

```# python notebook for Make Your Own Neural Network #code for a 3­layer neural network,
# and code for learning the MNIST dataset #(c) Tariq Rashid, 2016 #license is GPLv2

import numpy
import scipy.special

class MyNn:

# Initialize a simple 3 layer neural network.
def __init__(self, inputnodes, hiddennodes, outputnodes, learningrate):
# Set the number of nodes in each layer.
self.inodes = inputnodes
self.hnodes = hiddennodes
self.onodes = outputnodes

# if len(self.inodes) != len(self.hnodes) or len(self.inodes) != len(self.onodes):
#    print("Error: You must provide nodes lists of the same size.")
#    print("Unexpected results may occur.")
# eif

# Set the learning rate.
self.lr = learningrate

if self.lr > 1.0:
print("Error: You must provide a learning rate that is less than or equal to one.")
print("Unexpected results may occur.")
# eif

# Set the link weight matrices, wih and who.
self.wih = numpy.random.normal(0.0, pow(self.hnodes, -0.5), (self.hnodes, self.inodes))
self.who = numpy.random.normal(0.0, pow(self.onodes, -0.5), (self.onodes, self.hnodes))

# Set the activation function for our neurons.
self.activation_function = lambda x: scipy.special.expit(x)
# edef

# Convert inputs to a two dimensional matrix
inputs = numpy.array(inputs_list, ndmin=2).T

# Calculate the signals from the input layer to the hidden layer
hidden_inputs = numpy.dot(self.wih, inputs)

# Calculate the signals from the hidden layer to the output layer
hidden_outputs = self.activation_function(hidden_inputs)

# Calculate signals into final output layer
final_inputs = numpy.dot(self.who, hidden_outputs)

# Calculate the signals emerging from the final output layer
final_outputs = self.activation_function(final_inputs)

# Output layer error is the (answer - guess)

# Hidden layer error is the output_errors, split by weights, recombined at hidden nodes
hidden_errors = numpy.dot(self.who.T, output_errors)

# Update the weights for the links between the hidden and output layers
self.who += self.lr * numpy.dot((output_errors * final_outputs * (1.0 - final_outputs)), numpy.transpose(hidden_outputs))

# Update the weights for the links between the input and hidden layers
self.wih += self.lr * numpy.dot((hidden_errors * hidden_outputs * (1.0 - hidden_outputs)), numpy.transpose(inputs))
# edef

def query(self, inputs_list):
# Convert inputs to a two dimensional matrix
inputs = numpy.array(inputs_list, ndmin=2).T

# Calculate the signals from the input layer to the hidden layer
hidden_inputs = numpy.dot(self.wih, inputs)

# Calculate the signals from the hidden layer to the output layer
hidden_outputs = self.activation_function(hidden_inputs)

# Calculate signals into final output layer
final_inputs = numpy.dot(self.who, hidden_outputs)

# Calculate the signals emerging from the final output layer
final_outputs = self.activation_function(final_inputs)
return final_outputs
# edef

# eclassvbh

# number of input, hidden and output nodes
input_nodes = 784
hidden_nodes = 100
output_nodes = 10

# learning rate is 0.3
learning_rate = 0.3

# create instance of neural network
n = MyNn(input_nodes, hidden_nodes, output_nodes, learning_rate)

# load the mnist training data CSV file into a list
training_data_file = open("mnist_dataset/mnist_train_100.csv", 'r')
training_data_file.close()

# train the neural network
# go through all records in the training data set
for record in training_data_list:
# split the record by the',' commas
all_values = record.split(',')

# scale and shift the inputs
inputs = (numpy.asfarray(all_values[1:])/ 255.0 * 0.99) + 0.01

# create the target output values (all 0.01, except the desired label which is 0.99)
targets = numpy.zeros(output_nodes) + 0.01

# all_values is the target label for this record
targets[int(all_values)] = 0.99
n.train(inputs, targets)
pass

# load the mnist test data CSV file into a list
test_data_file = open("mnist_dataset/mnist_test_10.csv", 'r')
test_data_file.close()

# test the neural network
# scorecard for how well the network performs, initially empty
scorecard = []

# go through all the records in the test data set
for record in test_data_list:
# split the record by the',' commas
all_values = record.split(',')

# correct answer is first value
correct_label = int(all_values)
print(correct_label, "correct label")

# scale and shift the inputs
inputs = (numpy.asfarray(all_values[1:])/ 255.0 * 0.99) + 0.01

# query the network
outputs = n.query(inputs)

# the index of the highest value corresponds to the label
label = numpy.argmax(outputs)