There have been many strides and advances made in machine learning over the last decade. For the first time, we have just witnessed an intelligent computer program beat a professional player at the game of Go (read more). Machine learning has been successfully applied to a wide range of fields and industries. It goes without saying that the finance world is one of its many early-adopters.
Today I’m going to show you how you can set up your own GPU-based deep learning environment on Windows using Keras with a Tensorflow backend. At the end, we’ll also train a simple convolutional neural network to recognize handwritten characters!
What is Machine Learning?
Without going into too much detail, machine learning is a type of artificial intelligence (AI) that allows computers to learn from vast volumes of data without being explicitly programmed. Machine learning algorithms are able to model the patterns and trends found within the data which can then be used to classify, forecast, and detect anomalies (ex. credit card fraud). Some popular machine learning algorithms are:
- Artificial neural networks
- Clustering (k-NN)
- Decision trees
- Support vector machines
What is Deep Learning?
Deep learning is a branch within machine learning that uses specific algorithms to model high level abstractions in data. Generally, it is associated with neural networks with multiple hidden layers and more neurons. Until recently, training neural networks with more than 1 hidden layer had been very difficult (vanishing gradient problem) and time-consuming. Some applications of deep learning include natural language processing (automatic translation, sentiment analysis), image classification, and time-series forecasting.
Keras and Tensorflow
There have been many machine learning libraries published, but for today’s article we will be talking about Keras and Tensorflow.
Made by researchers and developers at Google, Tensorflow is an “open-source software library for machine intelligence”. Tensorflow supports computing using one or many CPUs or GPUs via CUDA. GPU performance is dramatically faster than CPU, sometimes by factors of 30 times and more! So if you have a GPU that supports CUDA, now is the time to take advantage of it! To check if your GPU supports CUDA, see list of supported GPUs. You can also find more information about Tensorflow from their website.
As the developers at Keras explains it, Keras is a “high-level neural networks library, written in Python and capable of running on top of either TensorFlow or Theano”. Keras makes building neural networks extremely easy, and with just a few lines of code anyone can get started with their first neural network. Written in Python, Keras is modular, minimalist, and yet still very powerful. Keras documentation is also very clear and concise, making the learning curve very smooth.
Setting Up Keras with Tensorflow
We will be setting up a Keras (using Tensorflow as the backend) machine learning environment today on Windows 8, 8.1 and 10. The reason I chose the Windows operating system is because like most of you, I prefer to use Windows as my daily driver and I was not able to find a good tutorial for Windows. At the same time, the installation process is already simple enough for those on Linux/Mac.
To follow along with this step-by-step guide, you’ll need to meet the following requirements:
- Windows 8.1 or 10 64-bit
- NVIDIA Graphics Card with CUDA support (this is only if you want GPU-accelerated computing, see list of supported GPUs)
If you’re all ready, let’s get started!
Download and Install CUDA Toolkit 8.0 and cuDNN v5.1
CUDA Toolkit and cuDNN are required for GPU computing. If you do not want to enable GPU computing, or your current GPU does not support CUDA, you can skip this step.
First, visit https://developer.nvidia.com/cuda-downloads and grab download CUDA Toolkit 8.0. You will need to register an account with Nvidia’s Accelerated Computing Developer Program. Select Windows as your operation system, choose the version of Windows you’re on (8.1 or 10), and select your installer type.
Run the installer (you may have to run it as adminstrator) and follow the on-screen prompts. Remember the location that it was installed at, you will need it for the next step (default is “C:\Program Files\NVIDIA GPU Computing Toolkit”).
Next, we’ll download the cuDNN files from https://developer.nvidia.com/rdp/cudnn-download. Make sure to select the correct operating system.
Extract the files from the zip and you should see a single directory named “cuda”, inside you will find a few more directories: “bin”, “include”, and “lib” (there is another directory inside “lib” called “x64”). Each directory contains a file: “cudnn64_5.dll”, “cudnn.h”, and “cudnn.lib”. Locate the path that CUDA Toolkit was installed at and go into “CUDA\v8.0” (ex. “C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0”).
Copy and paste “cudnn64_5.dll” into “\bin”, “cudnn.h” into “\include”, and “cudnn.lib” into “\lib\x64”.
You’ve now got CUDA all set up!
Anaconda – Python 3.6
Now we’re going to install our Python environment. I prefer Anaconda because it bundles all the common packages that you’ll need in a single convenient download. Keras requires h5py, HDF5, NumPy, PyYAML, and SciPy, which luckily Anaconda comes with.
Visit https://www.continuum.io/downloads, and download the latest version of Anaconda for Windows. At the time of this writing, it is Anaconda 4.3.0 Python 3.6 Version. Again, launch the installer and follow the prompts (keep the Advanced Options checked).
Create and Activate Conda Environment
Once Anaconda has been successfully installed, launch the Anaconda Prompt with administrator privileges and type in the following command:
conda create -n KerasEnv python=3.5 anaconda
NOTE: Make sure that the python version is set to 3.5. Tensorflow currently only supports python 3.5 on Windows.
Feel free to replace KerasEnv with a name of your choosing. Press y and then enter to continue and Anaconda will create a new virtual environment with all the included packages for you at “C:\ProgramData\Anaconda3\envs\KerasEnv. To activate the environment, use the command:
Replace KerasEnv with the name you gave your environment when it was created. Your prompt should now display:
To deactivate it, simply enter:
To delete your environment, enter the following:
conda remove -n KerasEnv --all
Again, replace KerasEnv with the name of your environment.
Install Keras and Tensorflow with Pip
Make sure the environment is activated before proceeding with the next steps.
Pip is packaged with Anaconda so we can directly install Keras and Tensorflow using pip install. Let’s start with Tensorflow.
We will use pip to install Tensorflow. Although conda can be used to install Tensorflow, only the CPU version is available.
To install the GPU version of Tensorflow, use the following command:
pip3 install --upgrade tensorflow-gpu
If you do not have a GPU that supports CUDA, you can install the CPU-only version of Tensorflow using the command below:
pip3 install --upgrade tensorflow
Once completed, we can go ahead and install Keras using the following simple command:
pip install keras
At this point, Keras and Tensorflow has been successfully installed on your machine! The next couple of steps will go through how to set up PyCharm as our IDE and training an example neural network to make sure everything is working.
NOTE: TensorFlow requires MSVCP140.DLL, which may not be installed on your system. If, when you import tensorflow as tf, you see an error about No module named “_pywrap_tensorflow” and/or DLL load failed, check whether MSVCP140.DLL is in your %PATH% and, if not, you should install the Visual C++ 2015 redistributable (x64 version).
March 21, 2017: With the release of Keras 2.0, you can easily update by uninstalling your current Keras package through the command
pip uninstall keras
then re-installing with the command above.
Setting Up PyCharm
For our IDE, we will be using PyCharm Community (it’s free!), so let’s go ahead and download the Windows version from https://www.jetbrains.com/pycharm/download/#section=windows.
Once the download is completed, launch the installer and follow the prompts to install it.
Now, let’s launch PyCharm and create a new project. You can save your project in any location you’d like. For the Interpreter, we’re going to select our conda environment we made earlier. Click the gear on the right side and select “Add Local”, find the location of the conda environment you created earlier and select the “python.exe” file. The full path might look like the following:
It may take a few minutes for PyCharm to update the interpreter afterwards.
Once that’s done and the PyCharm has launch, we can create a new python file. To create a new file, simply go to File -> New… -> Python File. Let’s name it “main.py”. It’s time to run our first neural network!
Running Our First Neural Network!
Keras offers an entire directory of example models made to demonstrate the features of the library which you can find here. You should be able to run any one of them now on your new Keras environment! I’ll use “mnist_cnn.py” as an example, which you can find here. This particular example builds and trains a convolutional neural network (CNN) on the MNIST dataset. The MNIST dataset is a very popular dataset comprised on images of handwritten characters. We will use a CNN to find features in the dataset which allows the CNN to recognize the characters. Since CNNs have wide applications in image and video recognition, it is the perfect tool to solve this type of problem.
The code is below:
'''Trains a simple convnet on the MNIST dataset. Gets to 99.25% test accuracy after 12 epochs (there is still a lot of margin for parameter tuning). 16 seconds per epoch on a GRID K520 GPU. ''' from __future__ import print_function import numpy as np np.random.seed(1337) # for reproducibility from keras.datasets import mnist from keras.models import Sequential from keras.layers import Dense, Dropout, Activation, Flatten from keras.layers import Convolution2D, MaxPooling2D from keras.utils import np_utils from keras import backend as K batch_size = 128 nb_classes = 10 nb_epoch = 12 # input image dimensions img_rows, img_cols = 28, 28 # number of convolutional filters to use nb_filters = 32 # size of pooling area for max pooling pool_size = (2, 2) # convolution kernel size kernel_size = (3, 3) # the data, shuffled and split between train and test sets (X_train, y_train), (X_test, y_test) = mnist.load_data() if K.image_dim_ordering() == 'th': X_train = X_train.reshape(X_train.shape, 1, img_rows, img_cols) X_test = X_test.reshape(X_test.shape, 1, img_rows, img_cols) input_shape = (1, img_rows, img_cols) else: X_train = X_train.reshape(X_train.shape, img_rows, img_cols, 1) X_test = X_test.reshape(X_test.shape, img_rows, img_cols, 1) input_shape = (img_rows, img_cols, 1) X_train = X_train.astype('float32') X_test = X_test.astype('float32') X_train /= 255 X_test /= 255 print('X_train shape:', X_train.shape) print(X_train.shape, 'train samples') print(X_test.shape, 'test samples') # convert class vectors to binary class matrices Y_train = np_utils.to_categorical(y_train, nb_classes) Y_test = np_utils.to_categorical(y_test, nb_classes) model = Sequential() model.add(Convolution2D(nb_filters, kernel_size, kernel_size, border_mode='valid', input_shape=input_shape)) model.add(Activation('relu')) model.add(Convolution2D(nb_filters, kernel_size, kernel_size)) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=pool_size)) model.add(Dropout(0.25)) model.add(Flatten()) model.add(Dense(128)) model.add(Activation('relu')) model.add(Dropout(0.5)) model.add(Dense(nb_classes)) model.add(Activation('softmax')) model.compile(loss='categorical_crossentropy', optimizer='adadelta', metrics=['accuracy']) model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch, verbose=1, validation_data=(X_test, Y_test)) score = model.evaluate(X_test, Y_test, verbose=0) print('Test score:', score) print('Test accuracy:', score)
Copy the code to the “main.py” file you just created in your PyCharm project and save it. Open the terminal by clicking the “Terminal” button on the bottom left of the IDE. You will notice that the terminal has automatically activated the environment you set up earlier.
To run the code, we simply run the command “python main.py” in our terminal and watch the magic happen!
If everything is working correctly, we’ll see the network train a total of 12 epochs and finally reaching an accuracy of 98.9%! Feel free to test any of the other models from Keras Examples.
I strongly encourage you to go through Keras documentation and try to make your own models. If you enjoyed this post and want to see more like this or if you have any feedback, be sure to let me know in the comments.
Until next time!
PS. For anyone interested in reading more about machine learning or deep learning, I strongly suggest Jason Brownlee’s website/blog at http://machinelearningmastery.com/. I have personally found Jason’s practical style of teaching to be incredibly effective.