This chapter describes basic operations in TensorFlow.
- Basic Python operations (assignment, branch & loop statement, library import)
- ‘With’ statement in Python
- NumPy , a common library for scientific computation, important for TensorFlow
- Vectors & Matrices operations (matrix addition & subtraction, matrix multiplication with vectors & matrices, matrix transpose, etc., Quiz: )
- Derivatives of functions , derivatives of multivariable functions (Quiz: )
- Linear regression;
- Gradient descent that searches local minima of a function
TensorFlow can be simply regarded as a library of scientific calculation (like Numpy in Python). Here we calculate and as our first example.
import tensorflow as tf tf.enable_eager_execution() a = tf.constant(1) b = tf.constant(1) c = tf.add(a, b) # The expression c = a + b is equivalent. print(c) A = tf.constant([[1, 2], [3, 4]]) B = tf.constant([[5, 6], [7, 8]]) C = tf.matmul(A, B) print(C)
tf.Tensor(2, shape=(), dtype=int32) tf.Tensor( [[19 22] [43 50]], shape=(2, 2), dtype=int32)
The code above declares four tensors named
B. It also invokes two operations
tf.matmul() which respectively do addition and matrix multiplication on tensors. Operation results are immediately stored in the tensors
C. Shape and dtype are two major attributes of a tensor. Here
c are scalars with null shape and int32 dtype, while
C are 2-by-2 matrices with
(2, 2) shape and int32 dtype.
In machine learning, it’s common to differentiate functions. TensorFlow provides us with the powerful Automatic Differentiation Mechanism for differentiation. The following codes show how to utilize
tf.GradientTape() to get the slope of at .
import tensorflow as tf tf.enable_eager_execution() x = tf.get_variable('x', shape=, initializer=tf.constant_initializer(3.)) with tf.GradientTape() as tape: # All steps in the context of tf.GradientTape() are recorded for differentiation. y = tf.square(x) y_grad = tape.gradient(y, x) # Differentiate y with respect to x. print([y.numpy(), y_grad.numpy()])
[array([9.], dtype=float32), array([6.], dtype=float32)]
x is a variable initialized to 3, declared by
tf.get_variable(). Like common tensors, variables also have shape and dtype attributes, but require an initialization. We can assign an initializer to
tf.get_variable() by setting the
Initializer parameter. Here we use
tf.constant_initializer(3.) to initialize the variable
3. with a float32 dtype. . An important difference between variables and common tensors is that a function can be differentiated by variables, not by tensors, using the automatic differentiation mechanism by default. Therefore variables are usually used as parameters defined in machine learning models.
tf.GraidentTape() is a recorder of automatic differentiation which records all variables and steps of calculation automatically. In the previous example, the variable
x and the calculation step
y = tf.square(x) are recorded automatically, thus the derivative of the tensor
y with respect to
x can be obtained through
y_grad = tape.gradient(y, x).
In machine learning, calculating the derivatives of a multivariable function, a vector or a matrix is a more common case, which is a piece cake for TensorFlow. The following codes show how to utilize
tf.GradientTape() to differentiate with respect to and at .
X = tf.constant([[1., 2.], [3., 4.]]) y = tf.constant([[1.], [2.]]) w = tf.get_variable('w', shape=[2, 1], initializer=tf.constant_initializer([[1.], [2.]])) b = tf.get_variable('b', shape=, initializer=tf.constant_initializer([1.])) with tf.GradientTape() as tape: L = 0.5 * tf.reduce_sum(tf.square(tf.matmul(X, w) + b - y)) w_grad, b_grad = tape.gradient(L, [w, b]) # Differentiate L with respect to w and b. print([L.numpy(), w_grad.numpy(), b_grad.numpy()])
[62.5, array([[35.], [50.]], dtype=float32), array([15.], dtype=float32)]
Here the operation
tf.square() squares every element in the input tensor without altering its shape. The operation
tf.reduce_sum() outputs the sum of all elements in the input tensor with a null shape (the dimensions of the summation can be indicated by the
axis parameter, while all elements are summed up if not specified). TensorFlow contains a large number of tensor operation APIs including mathematical operations, tensor shape operations (like
tf.reshape()), slicing and concatenation (like
tf.concat()), etc. You can heck TensorFlow official API documentation  for further information.
As we can see from the output, TensorFlow helps us figure out that
A Basic Example: Linear Regression¶
Let’s consider a practical problem. The house prices of a city between 2013 and 2017 are given by the following table:
Now we want to do linear regression on the given data, i.e. using the linear model to fit the data, where
b are unknown parameters.
First, we define and normalize the data.
import numpy as np X_raw = np.array([2013, 2014, 2015, 2016, 2017], dtype=np.float32) y_raw = np.array([12000, 14000, 15000, 16500, 17500], dtype=np.float32) X = (X_raw - X_raw.min()) / (X_raw.max() - X_raw.min()) y = (y_raw - y_raw.min()) / (y_raw.max() - y_raw.min())
Then, we use gradient descent to evaluate these two parameters
b in the linear model .
Recalling from the fundamentals of machine learning, for searching local minima of a multivariable function , we use gradient descent which taking the following steps:
Initialize the argument to and have
Iterate the following steps repeatedly till the convergence criteria is met:
- Find the gradient of the function with respect to the parameter
- Update the parameter where is the learning rate (like the step size of the gradient descent)
Next we focus on how to implement gradient descent in order to solve the linear regression .
The implementation of machine learning models is not a patent of TensorFlow. In fact, even most common scientific calculators or tools can solve simple models. Here, we use Numpy, a general library for scientific computation, to implement gradient descent. Numpy supports multidimensional arrays to represent vectors, matrices and tensors with more dimensions. Meanwhile, it also supports lots of operations on multidimensional arrays (e.g.
np.dot() calculates the inner products and
np.sum() adds up all the elements). In this way Numpy is somewhat like MATLAB. In the following codes, we evaluate the partial derivatives of loss function with respect to the parameters
b manually , and then iterate by gradient descent to acquire the value of
a, b = 0, 0 num_epoch = 10000 learning_rate = 1e-3 for e in range(num_epoch): # Calculate the gradient of the loss function with respect to arguments (model parameters) manually. y_pred = a * X + b grad_a, grad_b = (y_pred - y).dot(X), (y_pred - y).sum() # Update parameters. a, b = a - learning_rate * grad_a, b - learning_rate * grad_b print(a, b)
However, you may have noticed that there are several pain points using common libraries for scientific computation to implement machine learning models:
- It’s often inevitable to differentiate functions manually. Simple ones may be fine, however the more complex ones (especially commonly appeared in deep learning models) are another story. Manual differentiation may be painful, even infeasible in the latter cases.
- It’s also often inevitable to update parameters based on the gradients manually. Manual update is still easy here because the gradient descent is a rather basic method while it’s not going to be easy anymore if we apply a more complex approach to update parameters (like Adam or Adagrad).
However, the appearance of TensorFlow eliminates these pain points to a large extent, granting users convenience for implementing machine learning models.
The Eager Execution Mode of TensorFlow  have very similar operations as the above-mentioned Numpy. In addition, it also provides us with a series of critical functions for deep learning such as faster operation speed (need support from GPU), automatic differentiation and optimizers, etc. We will show how to do linear regression using Tensorflow. You may notice that its code structure is similar to the one of Numpy. Here we delegates TensorFlow to do two important jobs:
tape.gradient(ys, xs)to get the gradients automatically;
optimizer.apply_gradients(grads_and_vars)to update parameters automatically.
X = tf.constant(X) y = tf.constant(y) a = tf.get_variable('a', dtype=tf.float32, shape=, initializer=tf.zeros_initializer) b = tf.get_variable('b', dtype=tf.float32, shape=, initializer=tf.zeros_initializer) variables = [a, b] num_epoch = 10000 optimizer = tf.train.GradientDescentOptimizer(learning_rate=1e-3) for e in range(num_epoch): # Use tf.GradientTape() to record the gradient info of the loss function with tf.GradientTape() as tape: y_pred = a * X + b loss = 0.5 * tf.reduce_sum(tf.square(y_pred - y)) # TensorFlow calculates the gradients of the loss function with respect to each argument (model paramter) automatically. grads = tape.gradient(loss, variables) # TensorFlow updates parameters automatically based on gradients. optimizer.apply_gradients(grads_and_vars=zip(grads, variables))
Here, we use the aforementioned approach to calculate the partial derivatives of the loss function with respect to each parameter, while we also use
tf.train.GradientDescentOptimizer(learning_rate=1e-3) to declare an optimizer for graident descent with a learning rate of 1e-3. The optimizer can help us update parameters based on the result of differentiation in order to minimize a specific loss function by calling its
Note that, for calling
optimizer.apply_gradients() to update model parameters, we need to provide it with parameters
grads_and_vars, i.e. the variables to be updated (like
variables in the aforementioned codes). To be specific, a Python list has to be passed, whose every element is a (partial derivative with respect to a variable, this variable) pair. For instance,
[(grad_w, w), (grad_b, b)] is passed here. By executing
grads = tape.gradient(loss, variables) we get partial derivatives of the loss function with respect to each variable recorded in
grads = [grad_w, grad_b]. Then we use
zip() in Python to pair the elements in
grads = [grad_w, grad_b] and
vars = [w, b] together respectively so as to get the required parameters.
In practice, we usually build much more complex models rather the linear model
y_pred = tf.matmul(X, w) + b here which can be simply written in a single line. Therefore, we often write a model class and call it by
y_pred = model(X) when needed. The following chapter elaborates writing model classes.
|||We can add a decimal point after an integer to make it become a floating point number in Python. E.g. |
|||Mainly refer to Tensor Transformations and Math. Note that the tensor operation API of TensorFlow is very similar to Numpy, thus one can get started on TensorFlow rather quickly if knowing about the latter.|
|||In fact there is an analytic solution for the linear regression. We use gradient descent here just for showing you how TensorFlow works.|
|||The loss function here is the mean square error whose partial derivatives with respect to |
|||The opposite of Eager Execution is Graph Execution that TensorFlow adopts before version 1.8 in Mar 2018. This handbook is mainly written for Eager Execution aiming at fast iterative development, however the basic usage of Graph Execution is also attached in the appendices in case of reference.|