Self-Driving Car Engineer Nanodegree¶
Deep Learning¶
Project: Build a Traffic Sign Recognition Classifier¶
In this notebook, a template is provided for you to implement your functionality in stages which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission, if necessary. Sections that begin with 'Implementation' in the header indicate where you should begin your implementation for your project. Note that some sections of implementation are optional, and will be marked with 'Optional' in the header.
In addition to implementing code, there will be questions that you must answer which relate to the project and your implementation. Each section where you will answer a question is preceded by a 'Question' header. Carefully read each question and provide thorough answers in the following text boxes that begin with 'Answer:'. Your project submission will be evaluated based on your answers to each of the questions and the implementation you provide.
Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.
Step 0: Load The Data¶
# Load pickled data
import pickle
# TODO: Fill this in based on where you saved the training and testing data
training_file = "train.p"
testing_file = "test.p"
with open(training_file, mode='rb') as f:
train = pickle.load(f)
with open(testing_file, mode='rb') as f:
test = pickle.load(f)
X_train, y_train = train['features'], train['labels']
X_test, y_test = test['features'], test['labels']
assert(len(X_train) == len(y_train))
assert(len(X_test) == len(y_test))
print("completed")
Step 1: Dataset Summary & Exploration¶
The pickled data is a dictionary with 4 key/value pairs:
'features'is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).'labels'is a 2D array containing the label/class id of the traffic sign. The filesignnames.csvcontains id -> name mappings for each id.'sizes'is a list containing tuples, (width, height) representing the the original width and height the image.'coords'is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES
Complete the basic data summary below.
### Replace each question mark with the appropriate value.
import numpy as np
#Number of training examples
n_train = len(X_train)
#Number of testing examples.
n_test = len(X_test)
#the shape of an traffic sign image
image_shape = X_train[0].shape
#unique classes/labels there are in the dataset.
n_classes = np.bincount(y_test).nonzero()[0].size
print("Number of training examples =", n_train)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)
Visualize the German Traffic Signs Dataset using the pickled file(s). This is open ended, suggestions include: plotting traffic sign images, plotting the count of each sign, etc.
The Matplotlib examples and gallery pages are a great resource for doing visualizations in Python.
NOTE: It's recommended you start with something simple first. If you wish to do more, come back to it after you've completed the rest of the sections.
### Data exploration visualization goes here.
### Feel free to use as many code cells as needed.
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import random
%matplotlib inline
# find and display a picture for every type of sign
for i in range(0,43):
for j in range(len(y_train)):
if (i == y_train[j]):
image = X_train[j].squeeze()
plt.figure(i+1,figsize=(1,1))
plt.title(str(y_train[j]))
plt.imshow(image)
break
else:
print("no sign of this number found")
Step 2: Design and Test a Model Architecture¶
Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the German Traffic Sign Dataset.
There are various aspects to consider when thinking about this problem:
- Neural network architecture
- Play around preprocessing techniques (normalization, rgb to grayscale, etc)
- Number of examples per label (some have more than others).
- Generate fake data.
Here is an example of a published baseline model on this problem. It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.
NOTE: The LeNet-5 implementation shown in the classroom at the end of the CNN lesson is a solid starting point. You'll have to change the number of classes and possibly the preprocessing, but aside from that it's plug and play!
Implementation¶
Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.
### Preprocess the data here.
### Feel free to use as many code cells as needed.
from sklearn.utils import shuffle
from sklearn import preprocessing
import matplotlib.pyplot as plt
import cv2
%matplotlib inline
# normalize the data using CLAHE. This works for color images.
for i in range(len(X_train)):
img = X_train[i]
lab= cv2.cvtColor(img, cv2.COLOR_BGR2LAB)
l, a, b = cv2.split(lab)
clahe = cv2.createCLAHE(clipLimit=5.0, tileGridSize=(8,8))
cl = clahe.apply(l)
limg = cv2.merge((cl,a,b))
X_train[i] = cv2.cvtColor(limg, cv2.COLOR_LAB2BGR)
for i in range(len(X_test)):
img = X_test[i]
lab= cv2.cvtColor(img, cv2.COLOR_BGR2LAB)
l, a, b = cv2.split(lab)
clahe = cv2.createCLAHE(clipLimit=5.0, tileGridSize=(8,8))
cl = clahe.apply(l)
limg = cv2.merge((cl,a,b))
X_test[i] = cv2.cvtColor(limg, cv2.COLOR_LAB2BGR)
for k in range(0,43):
for l in range(len(y_train)):
if (k == y_train[l]):
image = X_train[l].squeeze()
plt.figure(k+1,figsize=(1,1))
plt.title(str(y_train[l]))
plt.imshow(image)
break
else:
print("no sign of this number found")
#shuffle the training data
X_train, y_train = shuffle(X_train, y_train)
Question 1¶
Describe how you preprocessed the data. Why did you choose that technique?
Answer: For data preprocessing, I normalized the data using a CLAHE transform, because several of the images were dim. I then shuffled the data.
### Generate data additional data (OPTIONAL!)
### and split the data into training/validation/testing sets here.
### Feel free to use as many code cells as needed.
from sklearn.utils import shuffle
from sklearn.model_selection import train_test_split
print ("original length = ", len(X_train))
X_train, X_validation, y_train, y_validation = train_test_split(X_train, y_train, test_size=0.2, random_state=0)
print("x train len= ", len(X_train))
print("x validation len= ", len(X_validation))
print("y train len= ", len(y_train))
print("y validation len= ", len(y_validation))
EPOCHS = 13
BATCH_SIZE = 128
Question 2¶
Describe how you set up the training, validation and testing data for your model. Optional: If you generated additional data, how did you generate the data? Why did you generate the data? What are the differences in the new dataset (with generated data) from the original dataset?
Answer: As a rule of thumb, 80% of the data should be training, 20% should be validation. I split the data to those percentages.
### Define your architecture here.
### Feel free to use as many code cells as needed.
from tensorflow.contrib.layers import flatten
def LeNet(x, dropout_prob):
# Hyperparameters
mu = 0.0
sigma = 0.1
# Layer 1: Convolutional. Input = 32x32x1. Output = 28x28x6.
conv1_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 3, 6), mean = mu, stddev = sigma))
conv1_b = tf.Variable(tf.zeros(6))
conv1 = tf.nn.conv2d(x, conv1_W, strides=[1, 1, 1, 1], padding='VALID') + conv1_b
# Activation.
conv1 = tf.nn.relu(conv1)
# Pooling. Input = 28x28x6. Output = 14x14x6.
conv1 = tf.nn.max_pool(conv1, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')
# Layer 2: Convolutional. Output = 10x10x16.
conv2_W = tf.Variable(tf.truncated_normal(shape=(5, 5, 6, 16), mean = mu, stddev = sigma))
conv2_b = tf.Variable(tf.zeros(16))
conv2 = tf.nn.conv2d(conv1, conv2_W, strides=[1, 1, 1, 1], padding='VALID') + conv2_b
# Activation.
conv2 = tf.nn.relu(conv2)
# Pooling. Input = 10x10x16. Output = 5x5x16.
conv2 = tf.nn.max_pool(conv2, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='VALID')
# Flatten. Input = 5x5x16. Output = 400.
fc0 = flatten(conv2)
# Layer 3: Fully Connected. Input = 400. Output = 120.
fc1_W = tf.Variable(tf.truncated_normal(shape=(400, 120), mean = mu, stddev = sigma))
fc1_b = tf.Variable(tf.zeros(120))
fc1 = tf.matmul(fc0, fc1_W) + fc1_b
# Activation.
fc1 = tf.nn.relu(fc1)
fc1 = tf.nn.dropout(fc1, dropout_prob)
# Layer 4: Fully Connected. Input = 120. Output = 84.
fc2_W = tf.Variable(tf.truncated_normal(shape=(120, 84), mean = mu, stddev = sigma))
fc2_b = tf.Variable(tf.zeros(84))
fc2 = tf.matmul(fc1, fc2_W) + fc2_b
# Activation.
fc2 = tf.nn.relu(fc2)
fc2 = tf.nn.dropout(fc2, dropout_prob)
# Layer 5: Fully Connected. Input = 84. Output = 43.
fc3_W = tf.Variable(tf.truncated_normal(shape=(84, 43), mean = mu, stddev = sigma))
fc3_b = tf.Variable(tf.zeros(43))
logits = tf.matmul(fc2, fc3_W) + fc3_b
return logits
Question 3¶
What does your final architecture look like? (Type of model, layers, sizes, connectivity, etc.) For reference on how to build a deep neural network using TensorFlow, see Deep Neural Network in TensorFlow from the classroom.
**Answer:
Traffic signs have several particular features. A model architecture will need to discover sign shapes, colors, and features on the signs. The architecture needs to have several layers to be able to correctly identify these signs.
I used the LeNet architecture from the lab as a starting point for this project. It takes a 32 x 32 x 3 channel as input. I did not use grayscaled images because color is helpful for classification. The model is sequential and contains 5 layers. Each layer is able to increasingly identify more complexity in the image. For example, the first layer might identify lines, while the last should be able to identify full objects.
Layer 1: Convolution Layer. The first portion of this layer takes the 32 x 32 x 3 image as an input and has output of size 28 x 28 with a filter depth of 6. This layer is useful for identifying image characteristics. I used a stride of 1 for my filter, which is slower, but more accurate.
The next portion of the layer is pooling. This groups features together and decreases the size of the output, while keeping depth the same. It uses a stride of 2, thus reducing the output to 14 x 14 x 6. Alternatively, I could have used subsampling in the convolutional layer. Pooling, however, worked fine for this project.
The final step of the layer uses rectified linear output (RELU) activation, which sets negative outputs to zero. The relu has the advantage of not vanishing the gradient.
Layer 2: Convolutional Layer. This layer is similar to Layer 1 and takes the first layer as input. The convolutional portion outputs 10 x 10 x 16. The pooling portion reduces the image to 5 x 5 x 16. Relu is also used in this layer. The image is then flattened to size of 400.
3 fully connected layers. A fully connected layer is similar to convolutional, but it is fully connected to all of the activations of the previous layer. There is no pooling for these layers. For layer 3 and 4, I added dropout in order to account for overfitting. Layer 3 takes 400 as input and outputs 120. Layer for inputs 120 and outputs 84. Layer 5 inputs 84 and outputs 43, which is the number of sign classes..
### Train your model here.
### Feel free to use as many code cells as needed.
#one hot encoding
import tensorflow as tf
x = tf.placeholder(tf.float32, (None, 32, 32, 3))
y = tf.placeholder(tf.int32, (None))
one_hot_y = tf.one_hot(y, 43)
#training pipeline
rate = 0.001
dropout = 0.94
logits = LeNet(x,dropout)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits, one_hot_y)
loss_operation = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdamOptimizer(learning_rate = rate)
training_operation = optimizer.minimize(loss_operation)
#evaluate the model
correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))
accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
saver = tf.train.Saver()
def evaluate(X_data, y_data):
num_examples = len(X_data)
total_accuracy = 0
sess = tf.get_default_session()
for offset in range(0, num_examples, BATCH_SIZE):
batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]
accuracy = sess.run(accuracy_operation, feed_dict={x: batch_x, y: batch_y})
total_accuracy += (accuracy * len(batch_x))
return total_accuracy / num_examples
#Train the model
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
num_examples = len(X_train)
print("Training...")
print()
for i in range(EPOCHS):
X_train, y_train = shuffle(X_train, y_train)
for offset in range(0, num_examples, BATCH_SIZE):
end = offset + BATCH_SIZE
batch_x, batch_y = X_train[offset:end], y_train[offset:end]
sess.run(training_operation, feed_dict={x: batch_x, y: batch_y})
validation_accuracy = evaluate(X_validation, y_validation)
print("EPOCH {} ...".format(i+1))
print("Validation Accuracy = {:.3f}".format(validation_accuracy))
print()
saver.save(sess, 'lenet')
print("Model saved")
#evaluate model
with tf.Session() as sess:
saver.restore(sess, tf.train.latest_checkpoint('.'))
test_accuracy = evaluate(X_test, y_test)
print("Test Accuracy = {:.3f}".format(test_accuracy))
Question 4¶
How did you train your model? (Type of optimizer, batch size, epochs, hyperparameters, etc.)
**Answer:
** I chose a convolutional layer because it has worked historically well with similar labs. In retrospect from further lessons, I should have mixed the activations and possibly used softplus in some layers. Relu has the advantage of non-vanishing gradients, but doesn't constrain the output. In future projects, I will attempt other activations, such as ELU, which gives a faster activation.
** With images, convolutional layer will filter over and connect spatial regions. The filter will eventually learn features of the images (edges, orientations, colors, etc..). Knowing this information, I believe the convolutional layer is a good solution.
I used Adam gradient descent optimizer. Because this is a classification, not regression problem, cross entropy is used instead of mean squared error for the loss. When compared to other optimizers, Adam has the advantages of bias correction, while being able to account for local minima. From what I read at http://sebastianruder.com/optimizing-gradient-descent/index.html#whichoptimizertochoose, Adam seems to outperform similar algorithms.
**In tuning hyperparameters, I used a process of trial and error. I would adjust a single parameter, then see how the network performed. I changed the hyperparameters until I was satisfied. I ended with a batch size of 128, 13 epochs, and hyperparameters of mu = 0.0, sigma = 0.1
To evaluate my model, I ran the program several times, tweaking the parameters until I was satisfied with the validation accuracy. I then ran the program on the test set.**
Question 5¶
What approach did you take in coming up with a solution to this problem? It may have been a process of trial and error, in which case, outline the steps you took to get to the final solution and why you chose those steps. Perhaps your solution involved an already well known implementation or architecture. In this case, discuss why you think this is suitable for the current problem.
**Answer: Because I am still new to neural networks, I did not try to create a new network from scratch. Instead, I relied on the LeNet architecture and evaluated whether or not it was a valid solution to the problem. Because classifying letters is similar to classifying traffic signs, I decided it was a valid architecture.
My first step was to import and normalize the images. Matplotlib worked well for visualizing the data. I decided to loop through each type of sign. I noticed that the images were darkened. Therefore, I used CLAHE to adjust the brightness. Before proceeding, I needed to verify that the correct optimizer, layer architecture and activation was used.
There are several types of optimizers for the project. Most stochastic gradient descent optimizers would work well. I chose Adam becuase it has bias correctiona and can account for local minima. Bias correction is needed because the signs are not always centered.
The architecture used (which I thoroughly described in other questions) progressively identifies features of the signs. Several layers are needed because signs have several layers of features (lines, shapes, color, etc...) I verified through testing that the LeNet architecture also works on signs.
For all layers, I used relu activation. Other activations will tend to vanish gradients. This could be detrimental to my network. Relu seemed to be suitable for the task.
I used a lot of trial and error in coming up with a solution. I changed the parameters one by one. The only parameters that seemed to help were changing the epochs and sometimes lowering sigma to 0.8. However, sigma did not produce consistent results. The initial validation accuracy ranges for 0.3 to 0.7. For lower validation accuracies, more epochs proved helpful**
Step 3: Test a Model on New Images¶
Take several pictures of traffic signs that you find on the web or around you (at least five), and run them through your classifier on your computer to produce example results. The classifier might not recognize some local signs but it could prove interesting nonetheless.
You may find signnames.csv useful as it contains mappings from the class id (integer) to the actual sign name.
Implementation¶
Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.
### Load the images and plot them here.
### Feel free to use as many code cells as needed.
import matplotlib.image as mpimg
%matplotlib inline
#image = 'stop.JPG'
image=mpimg.imread('stop.JPG')
plt.figure(1, figsize=(1,1))
plt.imshow(image)
X_test2 = []
y_test2 = []
X_test2.append(image)
y_test2.append(14)
image=mpimg.imread('stopsign2.jpg')
plt.figure(2, figsize=(1,1))
plt.imshow(image)
X_test2.append(image)
y_test2.append(14)
image=mpimg.imread('speedlimit50.jpg')
plt.figure(3, figsize=(1,1))
plt.imshow(image)
X_test2.append(image)
y_test2.append(2)
image=mpimg.imread('SpLim30.jpg')
plt.figure(4, figsize=(1,1))
plt.imshow(image)
X_test2.append(image)
y_test2.append(1)
image=mpimg.imread('pedestrian.jpg')
plt.figure(27, figsize=(1,1))
plt.imshow(image)
X_test2.append(image)
y_test2.append(1)
image=mpimg.imread('noentry.jpg')
plt.figure(6, figsize=(1,1))
plt.imshow(image)
X_test2.append(image)
y_test2.append(17)
image=mpimg.imread('childrencrossing.jpg')
plt.figure(28, figsize=(1,1))
plt.imshow(image)
X_test2.append(image)
y_test2.append(1)
with tf.Session() as sess:
saver.restore(sess, tf.train.latest_checkpoint('.'))
test_accuracy = evaluate(X_test2, y_test2)
print("Test Accuracy = {:.3f}".format(test_accuracy))
Question 6¶
Choose five candidate images of traffic signs and provide them in the report. Are there any particular qualities of the image(s) that might make classification difficult? It could be helpful to plot the images in the notebook.
Answer: The largest changes in the images is that they may not be completely centered, and they are american street signs. Because of this, I expect a low accuracy.
### Run the predictions here.
### Feel free to use as many code cells as needed.
image=mpimg.imread('stopsign2.jpg')
plt.figure(2, figsize=(1,1))
plt.imshow(image)
X_test2.append(image)
y_test2.append(14)
image=mpimg.imread('speedlimit50.jpg')
plt.figure(3, figsize=(1,1))
plt.imshow(image)
X_test2.append(image)
y_test2.append(2)
image=mpimg.imread('SpLim30.jpg')
plt.figure(4, figsize=(1,1))
plt.imshow(image)
X_test2.append(image)
y_test2.append(1)
image=mpimg.imread('noentry.jpg')
plt.figure(6, figsize=(1,1))
plt.imshow(image)
X_test2.append(image)
y_test2.append(17)
image=mpimg.imread('childrencrossing.jpg')
plt.figure(28, figsize=(1,1))
plt.imshow(image)
X_test2.append(image)
y_test2.append(1)
Question 7¶
Is your model able to perform equally well on captured pictures when compared to testing on the dataset? The simplest way to do this check the accuracy of the predictions. For example, if the model predicted 1 out of 5 signs correctly, it's 20% accurate.
NOTE: You could check the accuracy manually by using signnames.csv (same directory). This file has a mapping from the class id (0-42) to the corresponding sign name. So, you could take the class id the model outputs, lookup the name in signnames.csv and see if it matches the sign from the image.
**Answer: My model is not able to accurately predict American signs. The model is only 33 percent accurate, although I achieved results as high as 50%.
### Visualize the softmax probabilities here.
### Feel free to use as many code cells as needed.
with tf.Session() as sess:
saver.restore(sess, tf.train.latest_checkpoint('.'))
softmax = tf.nn.softmax(logits)
output = sess.run(softmax, feed_dict={x: X_test[0:10]})
prob = sess.run(tf.nn.top_k(tf.constant(output), k=3))
print("output =: ", prob)
test_accuracy = evaluate(X_test2, y_test2)
print("Test Accuracy = {:.3f}".format(test_accuracy))
# I manually loaded the images into the dataset.
Question 8¶
Use the model's softmax probabilities to visualize the certainty of its predictions, tf.nn.top_k could prove helpful here. Which predictions is the model certain of? Uncertain? If the model was incorrect in its initial prediction, does the correct prediction appear in the top k? (k should be 5 at most)
tf.nn.top_k will return the values and indices (class ids) of the top k predictions. So if k=3, for each sign, it'll return the 3 largest probabilities (out of a possible 43) and the correspoding class ids.
Take this numpy array as an example:
# (5, 6) array
a = np.array([[ 0.24879643, 0.07032244, 0.12641572, 0.34763842, 0.07893497,
0.12789202],
[ 0.28086119, 0.27569815, 0.08594638, 0.0178669 , 0.18063401,
0.15899337],
[ 0.26076848, 0.23664738, 0.08020603, 0.07001922, 0.1134371 ,
0.23892179],
[ 0.11943333, 0.29198961, 0.02605103, 0.26234032, 0.1351348 ,
0.16505091],
[ 0.09561176, 0.34396535, 0.0643941 , 0.16240774, 0.24206137,
0.09155967]])Running it through sess.run(tf.nn.top_k(tf.constant(a), k=3)) produces:
TopKV2(values=array([[ 0.34763842, 0.24879643, 0.12789202],
[ 0.28086119, 0.27569815, 0.18063401],
[ 0.26076848, 0.23892179, 0.23664738],
[ 0.29198961, 0.26234032, 0.16505091],
[ 0.34396535, 0.24206137, 0.16240774]]), indices=array([[3, 0, 5],
[0, 1, 4],
[0, 5, 1],
[1, 3, 5],
[1, 4, 3]], dtype=int32))Looking just at the first row we get [ 0.34763842, 0.24879643, 0.12789202], you can confirm these are the 3 largest probabilities in a. You'll also notice [3, 0, 5] are the corresponding indices.
Answer: [16, 9, 3] are the indeces with probabilities of [ 9.99999881e-01, 6.28015258e-08, 1.70343156e-10], although I noticed a few entries with probability of 1
Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.
sess.run(tf.nn.top_k(tf.constant(a), k=3))