Autoencoder Tutorial
Topic: AutoEncoder
An Autoencoder neural network is a type of unsupervised learning algorithm that applies backpropagation and setting the target values to be equal to the inputs that means; y(i) = x(i)
Ref:http://ufldl.stanford.edu/tutorial/unsupervised/Autoencoders/ https://www.deeplearningbook.org/contents/autoencoders.html
The model is actually try to learn a function hW,b(X) $\approx$ X which means that the fuction is trying to learn an approximatly identity function. The model is a neural network with 3 layers. The input layer size is the size of X and output layer also have same size while the hidden layer are of varing size. The number of neurons in the hidden layer may be less or more than the input layer. Lets assume the number of neuron in the hidden layer is 50 and the output layer is 100. So the function is force to learn compressed representation of the input data. If the input features were each independent of one another, this compression and subsequent reconstructioin would be very difficult task. However if there is some sort of structure exist for example correlation between the features then this stucture can be learned and leverage when forcing the input the network’s bottleneck.
There are many application of autoencoder such as:
- Dimension reduction - Extract hidden pattern (information) in the data - Recomendation system for the missing values - Data denoising. We are going to do a hands on exercise for better understanding.
There are different types of autoencoder: - Undercomplete autoencoder - Sparse autoencoder - Denoising autoencoder - Contractive autoencoder
### Undercomplete autoencoder ### Complete autoencoder
# Building an autoencoder # loading important libraries import tensorflow as tf from keras.layers import Input, Dense from keras.models import Model from keras import regularizers from keras.callbacks import TensorBoard import numpy as np import pandas as pd import matplotlib.pyplot as plt Using TensorFlow backend. # encoding dimenison encoding_dim = 32 input_img = Input(shape=(784,)) # encoded is the encoding representation of the input encoded = Dense(encoding_dim,activation="relu")(input_img) # decoded is the decoding representation of the encoded image decoded = Dense(784, activation="sigmoid")(encoded) # This model map an input to its reconstruction autoencoder = Model(input_img,decoded) # only encoded repsentation encoded = Model(input_img,encoded) # Next is the decoded model encoded_input = Input(shape=(encoding_dim,)) decoder_layer = autoencoder.layers[-1] decoder = Model(encoded_input, decoder_layer(encoded_input)) # Compile the model autoencoder.compile(loss="binary_crossentropy",optimizer="adadelta") Let’s prepare our datasets
from keras.datasets import mnist (x_train, _), (x_test,_) = mnist.load_data() print(x_train.shape) print(x_test.shape) (60000, 28, 28) (10000, 28, 28) # Normalized the values between 0 to 1 x_train = x_train.astype('float32') / 255. x_test = x_test.astype('float32') / 255. x_train.shape (60000, 28, 28) # flatten the shape x_train = x_train.reshape(x_train.shape[0],np.prod(x_train.shape[1:])) x_test = x_test.reshape(x_test.shape[0],np.prod(x_test.shape[1:])) np.prod(x_train.shape[1:]) 784 print(x_train.shape) print(x_test.shape) (60000, 784) (10000, 784) # training autoencoder.fit(x_train, x_train,epochs=100, batch_size=256, shuffle=True, validation_data=(x_test, x_test),callbacks=[TensorBoard(log_dir='autoencoderLog')]) Train on 60000 samples, validate on 10000 samples Epoch 1/100 60000/60000 [==============================] - 2s 37us/step - loss: 0.3702 - val_loss: 0.2719 Epoch 2/100 60000/60000 [==============================] - 2s 33us/step - loss: 0.2648 - val_loss: 0.2543 Epoch 3/100 60000/60000 [==============================] - 2s 32us/step - loss: 0.2438 - val_loss: 0.2312 Epoch 4/100 60000/60000 [==============================] - 2s 32us/step - loss: 0.2230 - val_loss: 0.2130 Epoch 5/100 60000/60000 [==============================] - 2s 32us/step - loss: 0.2081 - val_loss: 0.2006 Epoch 6/100 60000/60000 [==============================] - 2s 32us/step - loss: 0.1973 - val_loss: 0.1912 Epoch 7/100 60000/60000 [==============================] - 2s 33us/step - loss: 0.1886 - val_loss: 0.1835 Epoch 8/100 60000/60000 [==============================] - 2s 33us/step - loss: 0.1816 - val_loss: 0.1771 Epoch 9/100 60000/60000 [==============================] - 2s 34us/step - loss: 0.1756 - val_loss: 0.1715 Epoch 10/100 60000/60000 [==============================] - 3s 46us/step - loss: 0.1704 - val_loss: 0.1667 Epoch 11/100 60000/60000 [==============================] - 2s 33us/step - loss: 0.1658 - val_loss: 0.1623 Epoch 12/100 60000/60000 [==============================] - 2s 35us/step - loss: 0.1616 - val_loss: 0.1582 Epoch 13/100 60000/60000 [==============================] - 2s 36us/step - loss: 0.1579 - val_loss: 0.1548 Epoch 14/100 60000/60000 [==============================] - 2s 32us/step - loss: 0.1544 - val_loss: 0.1514 Epoch 15/100 60000/60000 [==============================] - 2s 34us/step - loss: 0.1512 - val_loss: 0.1482 Epoch 16/100 60000/60000 [==============================] - 2s 32us/step - loss: 0.1482 - val_loss: 0.1453 Epoch 17/100 60000/60000 [==============================] - 2s 32us/step - loss: 0.1455 - val_loss: 0.1426 Epoch 18/100 60000/60000 [==============================] - 2s 32us/step - loss: 0.1429 - val_loss: 0.1401 Epoch 19/100 60000/60000 [==============================] - 2s 32us/step - loss: 0.1404 - val_loss: 0.1378 Epoch 20/100 60000/60000 [==============================] - 2s 34us/step - loss: 0.1381 - val_loss: 0.1355 Epoch 21/100 60000/60000 [==============================] - 2s 34us/step - loss: 0.1360 - val_loss: 0.1333 Epoch 22/100 60000/60000 [==============================] - 2s 32us/step - loss: 0.1339 - val_loss: 0.1313 Epoch 23/100 60000/60000 [==============================] - 2s 33us/step - loss: 0.1319 - val_loss: 0.1294 Epoch 24/100 60000/60000 [==============================] - 2s 32us/step - loss: 0.1300 - val_loss: 0.1275 Epoch 25/100 60000/60000 [==============================] - 2s 34us/step - loss: 0.1282 - val_loss: 0.1258 Epoch 26/100 60000/60000 [==============================] - 2s 34us/step - loss: 0.1265 - val_loss: 0.1241 Epoch 27/100 60000/60000 [==============================] - 3s 49us/step - loss: 0.1249 - val_loss: 0.1225 Epoch 28/100 60000/60000 [==============================] - 2s 35us/step - loss: 0.1234 - val_loss: 0.1211 Epoch 29/100 60000/60000 [==============================] - 2s 33us/step - loss: 0.1219 - val_loss: 0.1196 Epoch 30/100 60000/60000 [==============================] - 2s 34us/step - loss: 0.1206 - val_loss: 0.1183 Epoch 31/100 60000/60000 [==============================] - 2s 34us/step - loss: 0.1193 - val_loss: 0.1171 Epoch 32/100 60000/60000 [==============================] - 2s 33us/step - loss: 0.1182 - val_loss: 0.1160 Epoch 33/100 60000/60000 [==============================] - 2s 33us/step - loss: 0.1171 - val_loss: 0.1150 Epoch 34/100 60000/60000 [==============================] - 2s 31us/step - loss: 0.1161 - val_loss: 0.1141 Epoch 35/100 60000/60000 [==============================] - 2s 30us/step - loss: 0.1152 - val_loss: 0.1132 Epoch 36/100 60000/60000 [==============================] - 2s 31us/step - loss: 0.1144 - val_loss: 0.1123 Epoch 37/100 60000/60000 [==============================] - 2s 31us/step - loss: 0.1136 - val_loss: 0.1116 Epoch 38/100 60000/60000 [==============================] - 2s 31us/step - loss: 0.1129 - val_loss: 0.1109 Epoch 39/100 60000/60000 [==============================] - 2s 30us/step - loss: 0.1122 - val_loss: 0.1103 Epoch 40/100 60000/60000 [==============================] - 2s 30us/step - loss: 0.1116 - val_loss: 0.1097 Epoch 41/100 60000/60000 [==============================] - 2s 31us/step - loss: 0.1110 - val_loss: 0.1091 Epoch 42/100 60000/60000 [==============================] - 2s 31us/step - loss: 0.1105 - val_loss: 0.1086 Epoch 43/100 60000/60000 [==============================] - 3s 42us/step - loss: 0.1100 - val_loss: 0.1082 Epoch 44/100 60000/60000 [==============================] - 2s 33us/step - loss: 0.1096 - val_loss: 0.1077 Epoch 45/100 60000/60000 [==============================] - 2s 33us/step - loss: 0.1092 - val_loss: 0.1073 Epoch 46/100 60000/60000 [==============================] - 2s 41us/step - loss: 0.1088 - val_loss: 0.1069 Epoch 47/100 60000/60000 [==============================] - 3s 48us/step - loss: 0.1084 - val_loss: 0.1066 Epoch 48/100 60000/60000 [==============================] - 3s 57us/step - loss: 0.1080 - val_loss: 0.1062 Epoch 49/100 60000/60000 [==============================] - 3s 45us/step - loss: 0.1077 - val_loss: 0.1059 Epoch 50/100 60000/60000 [==============================] - 2s 41us/step - loss: 0.1074 - val_loss: 0.1056 Epoch 51/100 60000/60000 [==============================] - 2s 38us/step - loss: 0.1071 - val_loss: 0.1053 Epoch 52/100 60000/60000 [==============================] - 2s 41us/step - loss: 0.1068 - val_loss: 0.1050 Epoch 53/100 60000/60000 [==============================] - 2s 37us/step - loss: 0.1065 - val_loss: 0.1047 Epoch 54/100 60000/60000 [==============================] - 2s 36us/step - loss: 0.1063 - val_loss: 0.1045 Epoch 55/100 60000/60000 [==============================] - 2s 39us/step - loss: 0.1060 - val_loss: 0.1043 Epoch 56/100 60000/60000 [==============================] - 3s 46us/step - loss: 0.1058 - val_loss: 0.1041 Epoch 57/100 60000/60000 [==============================] - 3s 52us/step - loss: 0.1056 - val_loss: 0.1038 Epoch 58/100 60000/60000 [==============================] - 3s 48us/step - loss: 0.1054 - val_loss: 0.1036 Epoch 59/100 60000/60000 [==============================] - 3s 45us/step - loss: 0.1052 - val_loss: 0.1034 Epoch 60/100 60000/60000 [==============================] - 3s 53us/step - loss: 0.1050 - val_loss: 0.1032 Epoch 61/100 60000/60000 [==============================] - 2s 33us/step - loss: 0.1048 - val_loss: 0.1030 Epoch 62/100 60000/60000 [==============================] - 3s 42us/step - loss: 0.1046 - val_loss: 0.1028 Epoch 63/100 60000/60000 [==============================] - 2s 35us/step - loss: 0.1044 - val_loss: 0.1027 Epoch 64/100 60000/60000 [==============================] - 2s 41us/step - loss: 0.1042 - val_loss: 0.1025 Epoch 65/100 60000/60000 [==============================] - 2s 35us/step - loss: 0.1041 - val_loss: 0.1024 Epoch 66/100 60000/60000 [==============================] - 2s 38us/step - loss: 0.1039 - val_loss: 0.1022 Epoch 67/100 60000/60000 [==============================] - 2s 36us/step - loss: 0.1038 - val_loss: 0.1020 Epoch 68/100 60000/60000 [==============================] - 2s 35us/step - loss: 0.1036 - val_loss: 0.1019 Epoch 69/100 60000/60000 [==============================] - 2s 33us/step - loss: 0.1035 - val_loss: 0.1018 Epoch 70/100 60000/60000 [==============================] - 2s 34us/step - loss: 0.1033 - val_loss: 0.1017 Epoch 71/100 60000/60000 [==============================] - 2s 33us/step - loss: 0.1032 - val_loss: 0.1015 Epoch 72/100 60000/60000 [==============================] - 2s 33us/step - loss: 0.1031 - val_loss: 0.1014 Epoch 73/100 60000/60000 [==============================] - 2s 35us/step - loss: 0.1030 - val_loss: 0.1013 Epoch 74/100 60000/60000 [==============================] - 3s 55us/step - loss: 0.1028 - val_loss: 0.1012 Epoch 75/100 60000/60000 [==============================] - 2s 35us/step - loss: 0.1027 - val_loss: 0.1010 Epoch 76/100 60000/60000 [==============================] - 3s 45us/step - loss: 0.1026 - val_loss: 0.1009 Epoch 77/100 60000/60000 [==============================] - 2s 36us/step - loss: 0.1025 - val_loss: 0.1008 Epoch 78/100 60000/60000 [==============================] - 2s 38us/step - loss: 0.1023 - val_loss: 0.1007 Epoch 79/100 60000/60000 [==============================] - 2s 37us/step - loss: 0.1022 - val_loss: 0.1006 Epoch 80/100 60000/60000 [==============================] - 3s 49us/step - loss: 0.1021 - val_loss: 0.1004 Epoch 81/100 60000/60000 [==============================] - 2s 40us/step - loss: 0.1020 - val_loss: 0.1003 Epoch 82/100 60000/60000 [==============================] - 3s 52us/step - loss: 0.1019 - val_loss: 0.1002 Epoch 83/100 60000/60000 [==============================] - 3s 49us/step - loss: 0.1017 - val_loss: 0.1001 Epoch 84/100 60000/60000 [==============================] - 2s 32us/step - loss: 0.1016 - val_loss: 0.1000 Epoch 85/100 60000/60000 [==============================] - 3s 48us/step - loss: 0.1015 - val_loss: 0.0999 Epoch 86/100 60000/60000 [==============================] - 2s 35us/step - loss: 0.1014 - val_loss: 0.0998 Epoch 87/100 60000/60000 [==============================] - 2s 36us/step - loss: 0.1013 - val_loss: 0.0996 Epoch 88/100 60000/60000 [==============================] - 2s 41us/step - loss: 0.1012 - val_loss: 0.0995 Epoch 89/100 60000/60000 [==============================] - 2s 37us/step - loss: 0.1010 - val_loss: 0.0994 Epoch 90/100 60000/60000 [==============================] - 2s 33us/step - loss: 0.1009 - val_loss: 0.0993 Epoch 91/100 60000/60000 [==============================] - 2s 37us/step - loss: 0.1008 - val_loss: 0.0992 Epoch 92/100 60000/60000 [==============================] - 2s 39us/step - loss: 0.1007 - val_loss: 0.0991 Epoch 93/100 60000/60000 [==============================] - 2s 37us/step - loss: 0.1006 - val_loss: 0.0990 Epoch 94/100 60000/60000 [==============================] - 2s 33us/step - loss: 0.1005 - val_loss: 0.0989 Epoch 95/100 60000/60000 [==============================] - 2s 35us/step - loss: 0.1004 - val_loss: 0.0988 Epoch 96/100 60000/60000 [==============================] - 2s 35us/step - loss: 0.1003 - val_loss: 0.0986 Epoch 97/100 60000/60000 [==============================] - 2s 33us/step - loss: 0.1002 - val_loss: 0.0985 Epoch 98/100 60000/60000 [==============================] - 2s 32us/step - loss: 0.1000 - val_loss: 0.0984 Epoch 99/100 60000/60000 [==============================] - 2s 32us/step - loss: 0.0999 - val_loss: 0.0984 Epoch 100/100 60000/60000 [==============================] - 2s 33us/step - loss: 0.0998 - val_loss: 0.0982 <keras.callbacks.History at 0x134fde208> print(autoencoder.summary()) _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_3 (InputLayer) (None, 784) 0 _________________________________________________________________ dense_3 (Dense) (None, 32) 25120 _________________________________________________________________ dense_4 (Dense) (None, 784) 25872 ================================================================= Total params: 50,992 Trainable params: 50,992 Non-trainable params: 0 _________________________________________________________________ None encoded_imgs = encoded.predict(x_test) decoded_imgs = decoder.predict(encoded_imgs) Lets plot with matplotlib
n =10 plt.figure(figsize=(20, 4)) for i in range(n): # original image ax = plt.subplot(2, n, i + 1) plt.imshow(x_test[i].reshape(28,28)) plt.gray() ax.get_xaxis().set_visible(False) ax.get_yaxis().set_visible(False) # Reconstructed image ax = plt.subplot(2, n, i + 1 + n) plt.imshow(decoded_imgs[i].reshape(28,28)) plt.gray() ax.get_xaxis().set_visible(False) ax.get_yaxis().set_visible(False) plt.show() 
This post is licensed under CC BY 4.0 by the author.