In this exercise, we will use Tensorflow for classification.
To begin, install tensorflow on your machine. One way to do so is using Anaconda. Another is to install pip, and then type in the command line 'pip install tensorflow'.
import tensorflow as tf
hello = tf.constant('Hello, TensorFlow!')
sess = tf.Session()
print(sess.run(hello))
Basic Operations¶
We now execute some basic operations using the tensorflow library.
In a basic constant operation, the value returned by the constructor represents the output of the Constant op.
a = tf.constant(2)
b = tf.constant(3)
Let us launch the default graph.
with tf.Session() as sess:
print("a: %i" % sess.run(a), "b: %i" % sess.run(b))
print("Addition with constants: %i" % sess.run(a+b))
print("Multiplication with constants: %i" % sess.run(a*b))
In basic Operations with variable graph input, the value returned by the constructor represents the output of the Variable op (define as input when running session). This is done so using a placeholder.
a = tf.placeholder(tf.int16)
b = tf.placeholder(tf.int16)
We can define the addition and multiplication operations as follows.
add = tf.add(a, b)
mul = tf.multiply(a, b)
Let us launch the default graph.
with tf.Session() as sess:
# Run every operation with variable input
print("Addition with variables: %i" % sess.run(add, feed_dict={a: 2, b: 3}))
print("Multiplication with variables: %i" % sess.run(mul, feed_dict={a: 2, b: 3}))
Let us now look at matrix multiplication.
Create a Constant op that produces a 1x2 matrix. The op is added as a node to the default graph.
matrix1 = tf.constant([[3., 3.]])
Create another Constant that produces a 2x1 matrix.
matrix2 = tf.constant([[2.],[2.]])
Create a Matmul op that takes 'matrix1' and 'matrix2' as inputs. The returned value, 'product', represents the result of the matrix multiplication.
product = tf.matmul(matrix1, matrix2)
To run the matmul op we call the session 'run()' method, passing 'product' which represents the output of the matmul op. This indicates to the call that we want to get the output of the matmul op back. All inputs needed by the op are run automatically by the session. They typically are run in parallel. The call 'run(product)' thus causes the execution of threes ops in the graph: the two constants and matmul. The output of the op is returned in 'result' as a numpy ndarray object.
with tf.Session() as sess:
result = sess.run(product)
print(result)
Linear Regression¶
Let us now perform linear regression using tensorflow.
import tensorflow as tf
import numpy
import matplotlib.pyplot as plt
rng = numpy.random
Define the parameters. 'learning_rate' sets the rate of convergence of the gradient descent optimizer. 'training_epochs' sets the number of iterations, where an 'epoch' is an iteration. 'display_step' sets the frequency of epochs to print the current status of the algorithm.
learning_rate = 0.01
training_epochs = 1000
display_step = 50
The training data is given as below.
train_X = numpy.asarray([3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167,
7.042,10.791,5.313,7.997,5.654,9.27,3.1])
train_Y = numpy.asarray([1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221,
2.827,3.465,1.65,2.904,2.42,2.94,1.3])
n_samples = train_X.shape[0]
The input graph is created and weights for the model are set.
X = tf.placeholder("float")
Y = tf.placeholder("float")
W = tf.Variable(rng.randn(), name="weight")
b = tf.Variable(rng.randn(), name="bias")
The linear model is constructed.
pred = tf.add(tf.multiply(X, W), b)
The mean squared error and the optimizer are defined.
cost = tf.reduce_sum(tf.pow(pred-Y, 2))/(2*n_samples)
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
Initialize the variables (i.e. assign their default value).
init = tf.global_variables_initializer()
Start training.
with tf.Session() as sess:
sess.run(init)
# Fit all training data
for epoch in range(training_epochs):
for (x, y) in zip(train_X, train_Y):
sess.run(optimizer, feed_dict={X: x, Y: y})
#Display logs per epoch step
if (epoch+1) % display_step == 0:
c = sess.run(cost, feed_dict={X: train_X, Y:train_Y})
print("Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(c), \
"W=", sess.run(W), "b=", sess.run(b))
print("Optimization Finished.")
training_cost = sess.run(cost, feed_dict={X: train_X, Y: train_Y})
print("Training cost=", training_cost, "W=", sess.run(W), "b=", sess.run(b), '\n')
#Graphic display
plt.plot(train_X, train_Y, 'ro', label='Original data')
plt.plot(train_X, sess.run(W) * train_X + sess.run(b), label='Fitted line')
plt.legend()
plt.show()
Q1. Observe the dependence of the training cost (error) on the learning rate.¶
With all other parameters fixed, we now vary the learning rate to take three values in $\{0.0001,0.01,0.8\}$. Draw a plot with y-axis representing the training cost and the x-axis representing the iteration (0 through 1000). There must be three curves in this plot, each representing a choice of learning rate.
#----------------- Your code here ---------------#
#------------------------------------------------#
As we can observe, very small or very large learning rates aren't preferable since even convergence is not guaranteed.
Logistics Regression¶
Let us now perform logistics regression using tensorflow. We will now be using the MNIST database of handwritten digits.
The data can be imported
import tensorflow as tf
from matplotlib import pyplot as plt
import numpy as np
# Import MINST data
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/", one_hot=True)
Let us see visualize a part of the data.
def gen_image(arr):
two_d = (np.reshape(arr, (28, 28)) * 255).astype(np.uint8)
plt.imshow(two_d, interpolation='nearest')
return plt
# Get a batch of two random images and show in a pop-up window.
batch_xs, batch_ys = mnist.test.next_batch(20)
for i in range(20):
gen_image(batch_xs[i]).show()
print("True classification:",list(batch_ys[i]).index(1))
Then, we set the parameters similar to the linear regression exercise.
# Parameters
learning_rate = 0.01
training_epochs = 100
batch_size = 100
display_step = 1
# tf Graph Input
x = tf.placeholder(tf.float32, [None, 784]) # mnist data image of shape 28*28=784
y = tf.placeholder(tf.float32, [None, 10]) # 0-9 digits recognition => 10 classes
# Set model weights
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
# Construct model
pred = tf.nn.softmax(tf.matmul(x, W) + b) # Softmax
# Minimize error using cross entropy
cost = tf.reduce_mean(-tf.reduce_sum(y*tf.log(pred), reduction_indices=1))
# Gradient Descent
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)
# Initialize the variables (i.e. assign their default value)
init = tf.global_variables_initializer()
Start training.
with tf.Session() as sess:
sess.run(init)
# Training cycle
for epoch in range(training_epochs):
avg_cost = 0.
total_batch = int(mnist.train.num_examples/batch_size)
# Loop over all batches
for i in range(total_batch):
batch_xs, batch_ys = mnist.train.next_batch(batch_size)
# Fit training using batch data
_, c = sess.run([optimizer, cost], feed_dict={x: batch_xs,
y: batch_ys})
# Compute average loss
avg_cost += c / total_batch
# Display logs per epoch step
if (epoch+1) % display_step == 0:
print("Epoch:", '%04d' % (epoch+1), "cost=", "{:.9f}".format(avg_cost))
prediction=tf.argmax(pred,1)
print("Optimization Finished.")
# Test model
correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
# Calculate accuracy for 3000 examples
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print("Accuracy:", accuracy.eval({x: mnist.test.images[:3000], y: mnist.test.labels[:3000]}))
predicted_labels = list(prediction.eval({x: mnist.test.images[:3000]}))
images = mnist.test.images[:3000]
for i in range(10):
gen_image(images[i]).show()
print("Predicted classification:",predicted_labels[i])
Q2. Observe the dependence of the training cost (error) on the learning rate.¶
With all other parameters fixed, we now vary the learning rate to take three values in $\{0.0001,0.01,1\}$. Draw a plot with y-axis representing the training cost and the x-axis representing the iteration (0 through 1000). There must be three curves in this plot, each representing a choice of learning rate.
#----------------- Your code here ---------------#
#------------------------------------------------#
