Add Exercises
This commit is contained in:
tsb1995
parent
6d4765582f
commit
48718490d9
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
1622
Ch1/cleanup.ipynb
1622
Ch1/cleanup.ipynb
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,41 @@
|
|||
Country,Air pollution,Dwellings without basic facilities,Educational attainment,Employees working very long hours,Employment rate,Feeling safe walking alone at night,Homicide rate,Household net adjusted disposable income,Household net wealth,Housing expenditure,Labour market insecurity,Life expectancy,Life satisfaction,Long-term unemployment rate,Personal earnings,Quality of support network,Rooms per person,Self-reported health,Stakeholder engagement for developing regulations,Student skills,Time devoted to leisure and personal care,Voter turnout,Water quality,Years in education,GDP per capita
|
||||
South Africa,22.0,37.0,73.0,18.12,43.0,36.1,13.7,,,18.0,,57.5,4.7,16.46,,88.0,,,,,14.92,73.0,67.0,,6193.170999999999
|
||||
Colombia,10.0,23.9,54.0,26.56,67.0,44.4,24.5,,,17.0,,76.2,6.3,0.79,,89.0,1.2,,1.4,410.0,,53.0,75.0,14.1,6744.007
|
||||
Brazil,10.0,6.7,49.0,7.13,61.0,35.6,26.7,,,,,74.8,6.4,,,90.0,,,2.2,395.0,,79.0,73.0,16.2,8955.65
|
||||
Turkey,20.0,8.0,39.0,32.64,52.0,59.8,1.4,,,20.0,12.5,78.0,5.5,2.39,,86.0,1.0,69.0,1.5,425.0,14.79,86.0,65.0,18.3,9683.565
|
||||
Mexico,16.0,25.5,38.0,28.7,61.0,41.8,18.1,,,20.0,5.5,75.4,6.5,0.07,15314.0,81.0,1.0,66.0,3.2,416.0,,63.0,68.0,15.2,10405.789
|
||||
Russia,15.0,14.8,94.0,0.14,70.0,52.8,9.6,,,18.0,,71.8,5.8,1.59,,89.0,0.9,43.0,,492.0,,68.0,55.0,16.2,11305.118
|
||||
Chile,16.0,9.4,65.0,9.72,63.0,47.9,4.2,,100967.0,18.0,8.7,79.9,6.5,,25879.0,85.0,1.2,57.0,1.3,443.0,,47.0,71.0,17.5,15854.652
|
||||
Poland,22.0,3.0,92.0,5.95,66.0,67.3,0.7,19814.0,210991.0,22.0,5.7,78.0,6.1,1.52,27046.0,86.0,1.1,58.0,2.6,504.0,14.42,55.0,82.0,17.6,15988.035
|
||||
Hungary,19.0,4.7,84.0,3.03,68.0,56.3,1.0,,104458.0,19.0,4.7,76.2,5.6,1.72,22576.0,86.0,1.2,60.0,1.2,474.0,,70.0,77.0,16.4,18535.147
|
||||
Latvia,11.0,13.9,88.0,1.27,70.0,62.4,4.8,16275.0,70160.0,23.0,9.6,74.7,5.9,3.35,23683.0,86.0,1.2,47.0,2.2,487.0,13.83,59.0,79.0,18.0,19104.841
|
||||
Lithuania,14.0,13.6,93.0,0.54,70.0,55.9,3.4,21660.0,,19.0,,74.8,5.9,2.69,24287.0,88.0,1.5,43.0,2.4,475.0,,51.0,81.0,18.4,20354.997
|
||||
Slovak Republic,21.0,1.2,91.0,4.14,66.0,63.5,0.8,20474.0,,23.0,9.9,77.3,6.2,4.78,24328.0,91.0,1.1,66.0,3.0,463.0,,60.0,85.0,15.8,20494.627
|
||||
Greece,18.0,0.5,73.0,6.42,53.0,60.0,0.8,17700.0,150134.0,23.0,29.8,81.5,5.4,15.65,26064.0,80.0,1.2,74.0,1.8,458.0,,64.0,69.0,19.0,20845.315
|
||||
Portugal,10.0,1.0,48.0,8.27,68.0,73.4,1.0,21203.0,232666.0,21.0,10.0,81.2,5.4,4.43,25367.0,88.0,1.7,48.0,1.5,497.0,,56.0,86.0,16.9,23731.127
|
||||
Czech Republic,20.0,0.7,94.0,5.65,74.0,72.3,0.5,21453.0,,24.0,3.1,79.1,6.7,1.04,25372.0,91.0,1.4,60.0,1.6,491.0,,61.0,87.0,17.9,24569.057999999997
|
||||
Estonia,8.0,7.0,89.0,2.42,74.0,69.0,3.1,19697.0,159373.0,17.0,3.8,77.8,5.7,1.92,24336.0,92.0,1.6,53.0,2.7,524.0,14.9,64.0,84.0,17.7,24802.77
|
||||
Slovenia,16.0,0.4,88.0,4.39,69.0,86.1,0.6,20820.0,203044.0,18.0,5.8,81.3,5.9,3.17,34933.0,92.0,1.5,64.0,2.5,509.0,14.75,53.0,90.0,18.3,27452.322999999997
|
||||
Spain,11.0,0.1,59.0,4.01,62.0,82.1,0.6,23999.0,373548.0,21.0,23.1,83.4,6.3,7.66,38507.0,93.0,1.9,72.0,1.8,491.0,15.93,70.0,72.0,17.9,30734.119
|
||||
Korea,28.0,2.5,88.0,,67.0,66.6,1.0,21882.0,285980.0,15.0,2.6,82.4,5.9,0.05,35191.0,78.0,1.5,33.0,2.9,519.0,14.7,77.0,76.0,17.3,31246.041
|
||||
Italy,18.0,0.7,61.0,4.11,58.0,58.4,0.6,26588.0,279889.0,23.0,12.3,83.3,6.0,6.59,36658.0,92.0,1.4,71.0,2.5,485.0,16.47,73.0,71.0,16.6,33431.25
|
||||
United Kingdom,11.0,0.3,81.0,12.15,75.0,77.7,0.2,28715.0,548392.0,26.0,4.5,81.2,6.8,1.13,43732.0,94.0,1.9,69.0,3.1,500.0,14.92,69.0,84.0,17.5,40391.839
|
||||
New Zealand,5.0,,79.0,15.11,77.0,65.7,1.3,,388514.0,26.0,4.7,81.7,7.3,0.74,40043.0,96.0,2.4,88.0,2.5,506.0,14.87,80.0,89.0,17.7,42084.399000000005
|
||||
France,13.0,0.5,78.0,7.67,65.0,70.5,0.5,31304.0,280653.0,21.0,7.6,82.4,6.5,4.0,43755.0,90.0,1.8,66.0,2.1,496.0,16.36,75.0,81.0,16.5,42643.949
|
||||
Japan,14.0,6.4,,,75.0,72.5,0.2,29798.0,305878.0,22.0,1.4,84.1,5.9,1.03,40863.0,89.0,1.9,36.0,1.4,529.0,,53.0,87.0,16.4,43043.422
|
||||
Israel,21.0,,87.0,15.45,69.0,69.8,1.8,,,,4.2,82.5,7.2,0.49,35067.0,88.0,1.2,84.0,2.5,472.0,,72.0,67.0,15.6,44474.065
|
||||
Belgium,15.0,1.9,77.0,4.75,63.0,70.1,1.0,30364.0,386006.0,21.0,3.7,81.5,6.9,3.54,49675.0,91.0,2.2,74.0,2.0,503.0,15.7,89.0,84.0,19.3,45979.691
|
||||
Canada,7.0,0.2,91.0,3.69,73.0,82.2,1.3,30854.0,423849.0,22.0,6.0,81.9,7.4,0.77,47622.0,93.0,2.6,88.0,2.9,523.0,14.56,68.0,91.0,17.3,47931.46
|
||||
Germany,14.0,0.2,87.0,4.26,75.0,72.5,0.5,34294.0,259667.0,20.0,2.7,81.1,7.0,1.57,47585.0,90.0,1.8,65.0,1.8,508.0,15.62,76.0,91.0,18.1,47992.322
|
||||
Finland,6.0,0.5,88.0,3.81,70.0,85.1,1.3,29943.0,200827.0,23.0,3.9,81.5,7.6,2.13,42964.0,95.0,1.9,70.0,2.2,523.0,15.17,67.0,95.0,19.8,50774.198
|
||||
Austria,16.0,0.9,85.0,6.66,72.0,80.6,0.5,33541.0,308325.0,21.0,3.5,81.7,7.1,1.84,50349.0,92.0,1.6,70.0,1.3,492.0,14.55,80.0,92.0,17.0,51330.462
|
||||
Sweden,6.0,0.0,83.0,1.07,77.0,75.6,0.9,31287.0,,19.0,3.2,82.4,7.3,1.12,42393.0,91.0,1.7,75.0,2.0,496.0,15.18,86.0,96.0,19.3,51892.079000000005
|
||||
Australia,5.0,,81.0,13.04,73.0,63.5,1.1,32759.0,427064.0,20.0,5.4,82.5,7.3,1.31,49126.0,95.0,,85.0,2.7,502.0,14.35,91.0,93.0,21.0,52952.268
|
||||
Netherlands,14.0,0.1,78.0,0.42,76.0,82.0,0.6,29333.0,157824.0,19.0,4.8,81.6,7.4,1.97,52877.0,91.0,1.9,76.0,2.6,508.0,,82.0,93.0,18.7,53873.37
|
||||
Denmark,9.0,0.5,81.0,2.34,74.0,83.5,0.6,29606.0,118637.0,23.0,4.2,80.9,7.6,1.31,51466.0,95.0,1.9,71.0,2.0,504.0,15.87,86.0,95.0,19.5,61732.569
|
||||
Iceland,3.0,0.0,77.0,15.06,86.0,86.0,0.5,,,24.0,0.7,82.3,7.5,0.26,61787.0,98.0,1.6,76.0,2.1,481.0,,79.0,99.0,19.0,66601.87700000001
|
||||
United States,10.0,0.1,91.0,11.09,70.0,73.9,5.5,45284.0,632100.0,19.0,7.7,78.6,6.9,0.66,60558.0,91.0,2.4,88.0,3.1,488.0,14.44,65.0,83.0,17.2,67426.835
|
||||
Norway,5.0,0.0,82.0,2.93,74.0,90.1,0.4,35725.0,228936.0,17.0,,82.5,7.6,0.66,51212.0,94.0,2.1,77.0,2.2,504.0,15.56,78.0,98.0,18.3,78333.22200000001
|
||||
Ireland,7.0,1.0,82.0,5.25,67.0,75.9,0.7,25310.0,217130.0,20.0,7.8,81.8,7.0,3.23,47653.0,95.0,2.1,83.0,1.3,509.0,,65.0,85.0,18.1,80264.835
|
||||
Switzerland,15.0,0.1,88.0,0.37,80.0,85.3,0.6,37466.0,,22.0,,83.7,7.5,1.82,62283.0,93.0,1.9,78.0,2.3,506.0,,49.0,95.0,17.5,86673.50099999999
|
||||
Luxembourg,12.0,0.5,77.0,3.82,66.0,75.8,0.6,39264.0,769053.0,21.0,1.7,82.8,6.9,2.35,63062.0,93.0,1.9,69.0,1.7,483.0,,91.0,84.0,15.1,116727.211
|
||||
|
|
|
@ -480,7 +480,7 @@
|
|||
"\n",
|
||||
"validator = GridSearchCV(keras_clf,\n",
|
||||
" param_grid={'n_neurons': [250, 275, 300, 325, 350],\n",
|
||||
" 'n_hidden' : [4,5, 6],\n",
|
||||
" 'n_hidden' : [4, 5, 6],\n",
|
||||
" # epochs is avail for tuning even when not\n",
|
||||
" # an argument to model building function\n",
|
||||
" 'epochs': [25, 30]},\n",
|
||||
|
|
|
|||
|
|
@ -297,18 +297,18 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"ename": "NameError",
|
||||
"evalue": "name 'model' is not defined",
|
||||
"evalue": "name 'keras' is not defined",
|
||||
"output_type": "error",
|
||||
"traceback": [
|
||||
"\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
|
||||
"\u001b[1;31mNameError\u001b[0m Traceback (most recent call last)",
|
||||
"\u001b[1;32m<ipython-input-9-956a5442dc95>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m 13\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 14\u001b[0m \u001b[1;31m# Fit the model with callbacks\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 15\u001b[1;33m history = model.fit(X_train, y_train, epochs=20,\n\u001b[0m\u001b[0;32m 16\u001b[0m \u001b[0mvalidation_data\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX_val\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0my_val\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 17\u001b[0m callbacks=[checkpoint_cb, \n",
|
||||
"\u001b[1;31mNameError\u001b[0m: name 'model' is not defined"
|
||||
"\u001b[1;32m<ipython-input-1-568908560bfe>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m 2\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m 3\u001b[0m \u001b[1;31m# Model Checkpoint callback incase of crash\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 4\u001b[1;33m checkpoint_cb = keras.callbacks.ModelCheckpoint('simple_mlp.h5',\n\u001b[0m\u001b[0;32m 5\u001b[0m save_best_only=True)\n\u001b[0;32m 6\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
|
||||
"\u001b[1;31mNameError\u001b[0m: name 'keras' is not defined"
|
||||
]
|
||||
}
|
||||
],
|
||||
|
|
@ -338,13 +338,13 @@
|
|||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"Reusing TensorBoard on port 6006 (pid 13744), started 0:24:54 ago. (Use '!kill 13744' to kill it.)"
|
||||
"Reusing TensorBoard on port 6006 (pid 13744), started 5 days, 11:55:57 ago. (Use '!kill 13744' to kill it.)"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
|
|
@ -354,11 +354,11 @@
|
|||
"data": {
|
||||
"text/html": [
|
||||
"\n",
|
||||
" <iframe id=\"tensorboard-frame-bfe354e751a4b5f4\" width=\"100%\" height=\"800\" frameborder=\"0\">\n",
|
||||
" <iframe id=\"tensorboard-frame-fad4fe9b3ae5c234\" width=\"100%\" height=\"800\" frameborder=\"0\">\n",
|
||||
" </iframe>\n",
|
||||
" <script>\n",
|
||||
" (function() {\n",
|
||||
" const frame = document.getElementById(\"tensorboard-frame-bfe354e751a4b5f4\");\n",
|
||||
" const frame = document.getElementById(\"tensorboard-frame-fad4fe9b3ae5c234\");\n",
|
||||
" const url = new URL(\"/\", window.location);\n",
|
||||
" url.port = 6006;\n",
|
||||
" frame.src = url;\n",
|
||||
|
|
@ -480,7 +480,7 @@
|
|||
"\n",
|
||||
"validator = GridSearchCV(keras_clf,\n",
|
||||
" param_grid={'n_neurons': [250, 275, 300, 325, 350],\n",
|
||||
" 'n_hidden' : [4,5, 6],\n",
|
||||
" 'n_hidden' : [4, 5, 6],\n",
|
||||
" # epochs is avail for tuning even when not\n",
|
||||
" # an argument to model building function\n",
|
||||
" 'epochs': [25, 30]},\n",
|
||||
|
|
@ -604,7 +604,7 @@
|
|||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.4"
|
||||
"version": "3.7.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,959 @@
|
|||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"**Exercise 8**\n",
|
||||
"\n",
|
||||
"Train a Deep Neural Network on CIFAR10 image dataset"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stderr",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Using TensorFlow backend.\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"import tensorflow as tf\n",
|
||||
"import numpy as np\n",
|
||||
"import keras\n",
|
||||
"from keras.datasets import cifar10\n",
|
||||
"from keras.models import Sequential\n",
|
||||
"from keras.layers import Dense, Dropout, Flatten, Activation, BatchNormalization, AlphaDropout\n",
|
||||
"from sklearn.model_selection import train_test_split\n",
|
||||
"import os"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class config:\n",
|
||||
" def __init__(self, n_classes = 10, n_epochs = 5, n_hidden = 20, \n",
|
||||
" n_neurons = 100, kernel_initializer = \"he_normal\", \n",
|
||||
" optimizer = keras.optimizers.Nadam(), \n",
|
||||
" activation='elu'):\n",
|
||||
" self.n_classes = n_classes\n",
|
||||
" self.n_epochs = n_epochs\n",
|
||||
" self.n_hidden = n_hidden\n",
|
||||
" self.n_neurons = n_neurons\n",
|
||||
" self.kernel_initializer = kernel_initializer\n",
|
||||
" self.activation = activation\n",
|
||||
" self.optimizer = optimizer\n",
|
||||
" self.input_shape = [32, 32, 3]\n",
|
||||
" self.loss = 'sparse_categorical_crossentropy'\n",
|
||||
" self.metrics = ['accuracy']\n",
|
||||
" \n",
|
||||
"cfg = config()\n",
|
||||
"\n",
|
||||
"# Note that our n_epochs is low (only 5), this is to keep this exercise to a reasonable time frame\n",
|
||||
"# To achieve better accuracy you can raise the epochs to a large armount\n",
|
||||
"# and let our Early Stopping callback do its work"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"x_train shape: (50000, 32, 32, 3)\n",
|
||||
"50000 train samples\n",
|
||||
"10000 test samples\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# The data, split between train and test sets:\n",
|
||||
"(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
|
||||
"print('x_train shape:', x_train.shape)\n",
|
||||
"print(x_train.shape[0], 'train samples')\n",
|
||||
"print(x_test.shape[0], 'test samples')\n",
|
||||
"\n",
|
||||
"X_train = x_train.astype('float32')\n",
|
||||
"X_test = x_test.astype('float32')\n",
|
||||
"X_train /= 255\n",
|
||||
"X_test /= 255\n",
|
||||
"\n",
|
||||
"# Split X_train into X_train and X_val\n",
|
||||
"X_train, X_val, y_train, y_val = train_test_split(\n",
|
||||
" x_train, y_train, test_size=0.1)\n",
|
||||
"\n",
|
||||
"pixel_means = X_train.mean(axis=0, keepdims=True)\n",
|
||||
"pixel_stds = X_train.std(axis=0, keepdims=True)\n",
|
||||
"X_train_scaled = (X_train - pixel_means) / pixel_stds\n",
|
||||
"X_val_scaled = (X_val - pixel_means) / pixel_stds\n",
|
||||
"X_test_scaled = (X_test - pixel_means) / pixel_stds"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Setup logdir for TensorBoard\n",
|
||||
"root_logdir = os.path.join(os.curdir, 'my_logs')\n",
|
||||
"\n",
|
||||
"# Setup function to get directory for logging our current run\n",
|
||||
"def get_run_logdir():\n",
|
||||
" import time\n",
|
||||
" run_id = time.strftime('run_%Y_%m_%d_%H_%M_%S')\n",
|
||||
" return os.path.join(root_logdir, run_id)\n",
|
||||
"\n",
|
||||
"run_logdir = get_run_logdir() "
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Model Checkpoint callback incase of crash\n",
|
||||
"checkpoint_cb = keras.callbacks.ModelCheckpoint('simple_mlp.h5',\n",
|
||||
" save_best_only=True)\n",
|
||||
"\n",
|
||||
"# Early Stopping callback\n",
|
||||
"early_stopping_cb = keras.callbacks.EarlyStopping(patience=5,\n",
|
||||
" restore_best_weights=True)\n",
|
||||
"\n",
|
||||
"# TensorBoard callback\n",
|
||||
"tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"WARNING:tensorflow:From C:\\Users\\TSB\\Miniconda3\\envs\\geron_env\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:74: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
|
||||
"\n",
|
||||
"WARNING:tensorflow:From C:\\Users\\TSB\\Miniconda3\\envs\\geron_env\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.\n",
|
||||
"\n",
|
||||
"WARNING:tensorflow:From C:\\Users\\TSB\\Miniconda3\\envs\\geron_env\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:4185: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.\n",
|
||||
"\n",
|
||||
"WARNING:tensorflow:From C:\\Users\\TSB\\Miniconda3\\envs\\geron_env\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.\n",
|
||||
"\n",
|
||||
"WARNING:tensorflow:From C:\\Users\\TSB\\Miniconda3\\envs\\geron_env\\lib\\site-packages\\keras\\optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
|
||||
"\n",
|
||||
"WARNING:tensorflow:From C:\\Users\\TSB\\Miniconda3\\envs\\geron_env\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:3341: The name tf.log is deprecated. Please use tf.math.log instead.\n",
|
||||
"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"flatten_1 (Flatten) (None, 3072) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_1 (Dense) (None, 100) 307300 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_2 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_3 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_4 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_5 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_6 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_7 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_8 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_9 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_10 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_11 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_12 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_13 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_14 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_15 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_16 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_17 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_18 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_19 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_20 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_21 (Dense) (None, 10) 1010 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 500,210\n",
|
||||
"Trainable params: 500,210\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Create baseline model with He Initialization, elu activation, and Nadam optimization\n",
|
||||
"\n",
|
||||
"model = Sequential()\n",
|
||||
"model.add(Flatten(input_shape=cfg.input_shape))\n",
|
||||
"for layer in range(cfg.n_hidden):\n",
|
||||
" model.add(Dense(cfg.n_neurons, activation=cfg.activation, \n",
|
||||
" kernel_initializer=cfg.kernel_initializer))\n",
|
||||
"model.add(Dense(10, activation='softmax'))\n",
|
||||
"model.compile(loss=cfg.loss, optimizer=cfg.optimizer, metrics=cfg.metrics)\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"WARNING:tensorflow:From C:\\Users\\TSB\\Miniconda3\\envs\\geron_env\\lib\\site-packages\\tensorflow_core\\python\\ops\\math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
|
||||
"Instructions for updating:\n",
|
||||
"Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
|
||||
"WARNING:tensorflow:From C:\\Users\\TSB\\Miniconda3\\envs\\geron_env\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.\n",
|
||||
"\n",
|
||||
"WARNING:tensorflow:From C:\\Users\\TSB\\Miniconda3\\envs\\geron_env\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.\n",
|
||||
"\n",
|
||||
"WARNING:tensorflow:From C:\\Users\\TSB\\Miniconda3\\envs\\geron_env\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:2741: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.\n",
|
||||
"\n",
|
||||
"Train on 45000 samples, validate on 5000 samples\n",
|
||||
"WARNING:tensorflow:From C:\\Users\\TSB\\Miniconda3\\envs\\geron_env\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.\n",
|
||||
"\n",
|
||||
"WARNING:tensorflow:From C:\\Users\\TSB\\Miniconda3\\envs\\geron_env\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
|
||||
"\n",
|
||||
"WARNING:tensorflow:From C:\\Users\\TSB\\Miniconda3\\envs\\geron_env\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.\n",
|
||||
"\n",
|
||||
"WARNING:tensorflow:From C:\\Users\\TSB\\Miniconda3\\envs\\geron_env\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.\n",
|
||||
"\n",
|
||||
"WARNING:tensorflow:From C:\\Users\\TSB\\Miniconda3\\envs\\geron_env\\lib\\site-packages\\keras\\backend\\tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.\n",
|
||||
"\n",
|
||||
"WARNING:tensorflow:From C:\\Users\\TSB\\Miniconda3\\envs\\geron_env\\lib\\site-packages\\keras\\callbacks.py:850: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.\n",
|
||||
"\n",
|
||||
"WARNING:tensorflow:From C:\\Users\\TSB\\Miniconda3\\envs\\geron_env\\lib\\site-packages\\keras\\callbacks.py:853: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.\n",
|
||||
"\n",
|
||||
"Epoch 1/5\n",
|
||||
"45000/45000 [==============================] - 76s 2ms/step - loss: 1.9980 - acc: 0.2697 - val_loss: 2.0319 - val_acc: 0.2578\n",
|
||||
"WARNING:tensorflow:From C:\\Users\\TSB\\Miniconda3\\envs\\geron_env\\lib\\site-packages\\keras\\callbacks.py:995: The name tf.Summary is deprecated. Please use tf.compat.v1.Summary instead.\n",
|
||||
"\n",
|
||||
"Epoch 2/5\n",
|
||||
"45000/45000 [==============================] - 68s 2ms/step - loss: 1.8190 - acc: 0.3308 - val_loss: 1.8712 - val_acc: 0.3270\n",
|
||||
"Epoch 3/5\n",
|
||||
"45000/45000 [==============================] - 73s 2ms/step - loss: 1.7708 - acc: 0.3539 - val_loss: 1.8496 - val_acc: 0.3278\n",
|
||||
"Epoch 4/5\n",
|
||||
"45000/45000 [==============================] - 61s 1ms/step - loss: 1.7449 - acc: 0.3714 - val_loss: 1.7296 - val_acc: 0.3866\n",
|
||||
"Epoch 5/5\n",
|
||||
"45000/45000 [==============================] - 41s 919us/step - loss: 1.7015 - acc: 0.3888 - val_loss: 1.7038 - val_acc: 0.3828\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.History at 0x1e301821f08>"
|
||||
]
|
||||
},
|
||||
"execution_count": 7,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.fit(X_train_scaled, y_train, epochs=cfg.n_epochs, \n",
|
||||
" validation_data = (X_val_scaled, y_val),\n",
|
||||
" callbacks = [checkpoint_cb, \n",
|
||||
" early_stopping_cb, \n",
|
||||
" tensorboard_cb])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"Launching TensorBoard..."
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Load TensorBoard to port 6006\n",
|
||||
"\n",
|
||||
"%load_ext tensorboard\n",
|
||||
"%tensorboard --logdir=./my_logs --port=6006"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Create same model, this time adding Batch Normalization before each hidden layer\n",
|
||||
"\n",
|
||||
"model = Sequential()\n",
|
||||
"model.add(Flatten(input_shape=cfg.input_shape))\n",
|
||||
"for layer in range(cfg.n_hidden):\n",
|
||||
" model.add(Dense(cfg.n_neurons, activation=cfg.activation, \n",
|
||||
" kernel_initializer=cfg.kernel_initializer))\n",
|
||||
" model.add(BatchNormalization())\n",
|
||||
"model.add(Dense(10, activation='softmax'))\n",
|
||||
"model.compile(loss=cfg.loss, optimizer=cfg.optimizer, metrics=cfg.metrics)\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"run_logdir = get_run_logdir() \n",
|
||||
"\n",
|
||||
"# TensorBoard callback\n",
|
||||
"tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"model.fit(X_train_scaled, y_train, epochs=cfg.n_epochs, \n",
|
||||
" validation_data = (X_val_scaled, y_val),\n",
|
||||
" callbacks = [checkpoint_cb, \n",
|
||||
" early_stopping_cb, \n",
|
||||
" tensorboard_cb])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Load TensorBoard to port 6006\n",
|
||||
"\n",
|
||||
"%load_ext tensorboard\n",
|
||||
"%tensorboard --logdir=./my_logs --port=6006"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"run_logdir = get_run_logdir() \n",
|
||||
"\n",
|
||||
"# TensorBoard callback\n",
|
||||
"tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)\n",
|
||||
"\n",
|
||||
"# Prepare configuration for a Selu based model (needs mean of 0 and standard deviation of 1)\n",
|
||||
"cfg.kernel_initializer='lecun_normal'\n",
|
||||
"cfg.activation='selu'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential_3\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"flatten_3 (Flatten) (None, 3072) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_43 (Dense) (None, 100) 307300 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_44 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_45 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_46 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_47 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_48 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_49 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_50 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_51 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_52 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_53 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_54 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_55 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_56 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_57 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_58 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_59 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_60 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_61 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_62 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_63 (Dense) (None, 10) 1010 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 500,210\n",
|
||||
"Trainable params: 500,210\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Construct Selu model (note that there are no Batch Normalization layers this time)\n",
|
||||
"model = Sequential()\n",
|
||||
"model.add(Flatten(input_shape=cfg.input_shape))\n",
|
||||
"for layer in range(cfg.n_hidden):\n",
|
||||
" model.add(Dense(cfg.n_neurons, activation=cfg.activation))\n",
|
||||
"model.add(Dense(10, activation='softmax'))\n",
|
||||
"model.compile(loss=cfg.loss, optimizer=cfg.optimizer, metrics=cfg.metrics)\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Train on 45000 samples, validate on 5000 samples\n",
|
||||
"Epoch 1/5\n",
|
||||
"45000/45000 [==============================] - 40s 883us/step - loss: 2.1985 - accuracy: 0.2347 - val_loss: 1.8834 - val_accuracy: 0.2900\n",
|
||||
"Epoch 2/5\n",
|
||||
"45000/45000 [==============================] - 42s 923us/step - loss: 1.8693 - accuracy: 0.2965 - val_loss: 1.8639 - val_accuracy: 0.2964\n",
|
||||
"Epoch 3/5\n",
|
||||
"45000/45000 [==============================] - 39s 876us/step - loss: 1.8027 - accuracy: 0.3231 - val_loss: 1.7998 - val_accuracy: 0.3196\n",
|
||||
"Epoch 4/5\n",
|
||||
"45000/45000 [==============================] - 39s 859us/step - loss: 1.7604 - accuracy: 0.3451 - val_loss: 1.8129 - val_accuracy: 0.3088\n",
|
||||
"Epoch 5/5\n",
|
||||
"45000/45000 [==============================] - 38s 852us/step - loss: 1.7392 - accuracy: 0.3527 - val_loss: 1.6917 - val_accuracy: 0.3714\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.callbacks.History at 0x1629adb5c48>"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.fit(X_train_scaled, y_train, epochs=cfg.n_epochs, \n",
|
||||
" validation_data = (X_val_scaled, y_val),\n",
|
||||
" callbacks = [checkpoint_cb, \n",
|
||||
" early_stopping_cb, \n",
|
||||
" tensorboard_cb])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"run_logdir = get_run_logdir() \n",
|
||||
"\n",
|
||||
"# TensorBoard callback\n",
|
||||
"tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)\n",
|
||||
"\n",
|
||||
"# Prepare configuration for a Selu based model (needs mean of 0 and standard deviation of 1)\n",
|
||||
"cfg.kernel_initializer='lecun_normal'\n",
|
||||
"cfg.activation='selu'"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential_4\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"flatten_4 (Flatten) (None, 3072) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_1 (AlphaDropou (None, 3072) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_64 (Dense) (None, 100) 307300 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_2 (AlphaDropou (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_65 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_3 (AlphaDropou (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_66 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_4 (AlphaDropou (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_67 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_5 (AlphaDropou (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_68 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_6 (AlphaDropou (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_69 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_7 (AlphaDropou (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_70 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_8 (AlphaDropou (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_71 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_9 (AlphaDropou (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_72 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_10 (AlphaDropo (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_73 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_11 (AlphaDropo (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_74 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_12 (AlphaDropo (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_75 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_13 (AlphaDropo (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_76 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_14 (AlphaDropo (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_77 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_15 (AlphaDropo (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_78 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_16 (AlphaDropo (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_79 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_17 (AlphaDropo (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_80 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_18 (AlphaDropo (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_81 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_19 (AlphaDropo (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_82 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"alpha_dropout_20 (AlphaDropo (None, 100) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_83 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_84 (Dense) (None, 10) 1010 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 500,210\n",
|
||||
"Trainable params: 500,210\n",
|
||||
"Non-trainable params: 0\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Construct Selu model (note that there are no Batch Normalization layers this time)\n",
|
||||
"# Now with alpha dropout\n",
|
||||
"model = Sequential()\n",
|
||||
"model.add(Flatten(input_shape=cfg.input_shape))\n",
|
||||
"for layer in range(cfg.n_hidden):\n",
|
||||
" model.add(AlphaDropout(rate=0.2))\n",
|
||||
" model.add(Dense(cfg.n_neurons, activation=cfg.activation, kernel_initializer=cfg.kernel_initializer))\n",
|
||||
"model.add(Dense(10, activation='softmax'))\n",
|
||||
"model.compile(loss=cfg.loss, optimizer=cfg.optimizer, metrics=cfg.metrics)\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 18,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Train on 45000 samples, validate on 5000 samples\n",
|
||||
"Epoch 1/5\n",
|
||||
"45000/45000 [==============================] - 57s 1ms/step - loss: 2.2238 - accuracy: 0.1515 - val_loss: 92.7995 - val_accuracy: 0.1574\n",
|
||||
"Epoch 2/5\n",
|
||||
"45000/45000 [==============================] - 54s 1ms/step - loss: 2.1047 - accuracy: 0.1786 - val_loss: 35.0895 - val_accuracy: 0.1534\n",
|
||||
"Epoch 3/5\n",
|
||||
"45000/45000 [==============================] - 58s 1ms/step - loss: 2.0622 - accuracy: 0.1984 - val_loss: 26.8686 - val_accuracy: 0.1080\n",
|
||||
"Epoch 4/5\n",
|
||||
"45000/45000 [==============================] - 58s 1ms/step - loss: 2.0472 - accuracy: 0.2090 - val_loss: 12.6033 - val_accuracy: 0.1708\n",
|
||||
"Epoch 5/5\n",
|
||||
"45000/45000 [==============================] - 58s 1ms/step - loss: 2.0519 - accuracy: 0.2057 - val_loss: 17.4287 - val_accuracy: 0.1782\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<keras.callbacks.callbacks.History at 0x1631fa52d48>"
|
||||
]
|
||||
},
|
||||
"execution_count": 18,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"model.fit(X_train_scaled, y_train, epochs=cfg.n_epochs, \n",
|
||||
" validation_data = (X_val_scaled, y_val),\n",
|
||||
" callbacks = [checkpoint_cb, \n",
|
||||
" early_stopping_cb, \n",
|
||||
" tensorboard_cb])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 19,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"The tensorboard extension is already loaded. To reload it, use:\n",
|
||||
" %reload_ext tensorboard\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"Reusing TensorBoard on port 6006 (pid 11252), started 0:13:58 ago. (Use '!kill 11252' to kill it.)"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/html": [
|
||||
"\n",
|
||||
" <iframe id=\"tensorboard-frame-50d3e8cf3006bcb9\" width=\"100%\" height=\"800\" frameborder=\"0\">\n",
|
||||
" </iframe>\n",
|
||||
" <script>\n",
|
||||
" (function() {\n",
|
||||
" const frame = document.getElementById(\"tensorboard-frame-50d3e8cf3006bcb9\");\n",
|
||||
" const url = new URL(\"/\", window.location);\n",
|
||||
" url.port = 6006;\n",
|
||||
" frame.src = url;\n",
|
||||
" })();\n",
|
||||
" </script>\n",
|
||||
" "
|
||||
],
|
||||
"text/plain": [
|
||||
"<IPython.core.display.HTML object>"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"# Load TensorBoard to port 6006\n",
|
||||
"\n",
|
||||
"%load_ext tensorboard\n",
|
||||
"%tensorboard --logdir=./my_logs --port=6006"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 20,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Implement 1 Cycle Scheduling\n",
|
||||
"# Thanks to https://github.com/ageron/handson-ml2/blob/master/11_training_deep_neural_networks.ipynb\n",
|
||||
"\n",
|
||||
"batch_size = 128\n",
|
||||
"K = keras.backend\n",
|
||||
"\n",
|
||||
"class OneCycleScheduler(keras.callbacks.Callback):\n",
|
||||
" def __init__(self, iterations, max_rate, start_rate=None,\n",
|
||||
" last_iterations=None, last_rate=None):\n",
|
||||
" self.iterations = iterations\n",
|
||||
" self.max_rate = max_rate\n",
|
||||
" self.start_rate = start_rate or max_rate / 10\n",
|
||||
" self.last_iterations = last_iterations or iterations // 10 + 1\n",
|
||||
" self.half_iteration = (iterations - self.last_iterations) // 2\n",
|
||||
" self.last_rate = last_rate or self.start_rate / 1000\n",
|
||||
" self.iteration = 0\n",
|
||||
" def _interpolate(self, iter1, iter2, rate1, rate2):\n",
|
||||
" return ((rate2 - rate1) * (self.iteration - iter1)\n",
|
||||
" / (iter2 - iter1) + rate1)\n",
|
||||
" def on_batch_begin(self, batch, logs):\n",
|
||||
" if self.iteration < self.half_iteration:\n",
|
||||
" rate = self._interpolate(0, self.half_iteration, self.start_rate, self.max_rate)\n",
|
||||
" elif self.iteration < 2 * self.half_iteration:\n",
|
||||
" rate = self._interpolate(self.half_iteration, 2 * self.half_iteration,\n",
|
||||
" self.max_rate, self.start_rate)\n",
|
||||
" else:\n",
|
||||
" rate = self._interpolate(2 * self.half_iteration, self.iterations,\n",
|
||||
" self.start_rate, self.last_rate)\n",
|
||||
" rate = max(rate, self.last_rate)\n",
|
||||
" self.iteration += 1\n",
|
||||
" K.set_value(self.model.optimizer.lr, rate)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 21,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"run_logdir = get_run_logdir() \n",
|
||||
"\n",
|
||||
"# TensorBoard callback\n",
|
||||
"tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 22,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Model: \"sequential_5\"\n",
|
||||
"_________________________________________________________________\n",
|
||||
"Layer (type) Output Shape Param # \n",
|
||||
"=================================================================\n",
|
||||
"flatten_5 (Flatten) (None, 3072) 0 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_85 (Dense) (None, 100) 307300 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_21 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_86 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_22 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_87 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_23 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_88 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_24 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_89 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_25 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_90 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_26 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_91 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_27 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_92 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_28 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_93 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_29 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_94 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_30 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_95 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_31 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_96 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_32 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_97 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_33 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_98 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_34 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_99 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_35 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_100 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_36 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_101 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_37 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_102 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_38 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_103 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_39 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_104 (Dense) (None, 100) 10100 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"batch_normalization_40 (Batc (None, 100) 400 \n",
|
||||
"_________________________________________________________________\n",
|
||||
"dense_105 (Dense) (None, 10) 1010 \n",
|
||||
"=================================================================\n",
|
||||
"Total params: 508,210\n",
|
||||
"Trainable params: 504,210\n",
|
||||
"Non-trainable params: 4,000\n",
|
||||
"_________________________________________________________________\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"cfg.kernel_initializer = \"he_normal\"\n",
|
||||
"cfg.activation='elu'\n",
|
||||
"\n",
|
||||
"model = Sequential()\n",
|
||||
"model.add(Flatten(input_shape=cfg.input_shape))\n",
|
||||
"for layer in range(cfg.n_hidden):\n",
|
||||
" model.add(Dense(cfg.n_neurons, activation=cfg.activation, \n",
|
||||
" kernel_initializer=cfg.kernel_initializer))\n",
|
||||
" model.add(BatchNormalization())\n",
|
||||
"model.add(Dense(10, activation='softmax'))\n",
|
||||
"model.compile(loss=cfg.loss, optimizer=cfg.optimizer, metrics=cfg.metrics)\n",
|
||||
"model.summary()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 23,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Train on 45000 samples, validate on 5000 samples\n",
|
||||
"Epoch 1/5\n",
|
||||
"45000/45000 [==============================] - 39s 867us/step - loss: 1.9622 - accuracy: 0.2782 - val_loss: 4.5324 - val_accuracy: 0.2428\n",
|
||||
"Epoch 2/5\n",
|
||||
"45000/45000 [==============================] - 31s 680us/step - loss: 1.7844 - accuracy: 0.3499 - val_loss: 9.2643 - val_accuracy: 0.3186\n",
|
||||
"Epoch 3/5\n",
|
||||
"45000/45000 [==============================] - 31s 690us/step - loss: 1.6917 - accuracy: 0.3895 - val_loss: 3.3850 - val_accuracy: 0.4072\n",
|
||||
"Epoch 4/5\n",
|
||||
"45000/45000 [==============================] - 25s 563us/step - loss: 1.5552 - accuracy: 0.4431 - val_loss: 1.7749 - val_accuracy: 0.4264\n",
|
||||
"Epoch 5/5\n",
|
||||
"45000/45000 [==============================] - 28s 625us/step - loss: 1.4320 - accuracy: 0.4866 - val_loss: 1.4649 - val_accuracy: 0.4712\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"onecycle = OneCycleScheduler(len(X_train) // batch_size * cfg.n_epochs, max_rate=0.05)\n",
|
||||
"history = model.fit(X_train_scaled, y_train, epochs=cfg.n_epochs, batch_size=batch_size,\n",
|
||||
" validation_data=(X_val_scaled, y_val),\n",
|
||||
" callbacks=[onecycle])"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.7.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue