Set Up Your Own Keras with Tensorflow GPU Deep Learning Environment on Windows 8.1 and 10

There have been many strides and advances made in machine learning over the last decade. For the first time, we have just witnessed an intelligent computer program beat a professional player at the game of Go (read more). Machine learning has been successfully applied to a wide range of fields and industries. It goes without saying that the finance world is one of its many early-adopters.

Today I’m going to show you how you can set up your own GPU-based deep learning environment on Windows using Keras with a Tensorflow backend. At the end, we’ll also train a simple convolutional neural network to recognize handwritten characters!

What is Machine Learning?

Without going into too much detail, machine learning is a type of artificial intelligence (AI) that allows computers to learn from vast volumes of data without being explicitly programmed. Machine learning algorithms are able to model the patterns and trends found within the data which can then be used to classify, forecast, and detect anomalies (ex. credit card fraud). Some popular machine learning algorithms are:

  • Artificial neural networks
  • Clustering (k-NN)
  • Decision trees
  • Support vector machines

What is Deep Learning?

Deep learning is a branch within machine learning that uses specific algorithms to model high level abstractions in data. Generally, it is associated with neural networks with multiple hidden layers and more neurons. Until recently, training neural networks with more than 1 hidden layer had been very difficult (vanishing gradient problem) and time-consuming. Some applications of deep learning include natural language processing (automatic translation, sentiment analysis), image classification, and time-series forecasting.

Keras and Tensorflow

There have been many machine learning libraries published, but for today’s article we will be talking about Keras and Tensorflow.

Tensorflow

Made by researchers and developers at Google, Tensorflow is an “open-source software library for machine intelligence”. Tensorflow supports computing using one or many CPUs or GPUs via CUDA. GPU performance is dramatically faster than CPU, sometimes by factors of 30 times and more! So if you have a GPU that supports CUDA, now is the time to take advantage of it! To check if your GPU supports CUDA, see list of supported GPUs. You can also find more information about Tensorflow from their website.

Keras

As the developers at Keras explains it, Keras is a “high-level neural networks library, written in Python and capable of running on top of either TensorFlow or Theano”. Keras makes building neural networks extremely easy, and with just a few lines of code anyone can get started with their first neural network. Written in Python, Keras is modular, minimalist, and yet still very powerful. Keras documentation is also very clear and concise, making the learning curve very smooth.

Setting Up Keras with Tensorflow

We will be setting up a Keras (using Tensorflow as the backend) machine learning environment today on Windows 8, 8.1 and 10. The reason I chose the Windows operating system is because like most of you, I prefer to use Windows as my daily driver and I was not able to find a good tutorial for Windows. At the same time, the installation process is already simple enough for those on Linux/Mac.

Requirements

To follow along with this step-by-step guide, you’ll need to meet the following requirements:

  • Windows 8.1 or 10 64-bit
  • NVIDIA Graphics Card with CUDA support (this is only if you want GPU-accelerated computing, see list of supported GPUs)

If you’re all ready, let’s get started!

Download and Install CUDA Toolkit 8.0 and cuDNN v5.1

CUDA Toolkit and cuDNN are required for GPU computing. If you do not want to enable GPU computing, or your current GPU does not support CUDA, you can skip this step.

First, visit https://developer.nvidia.com/cuda-downloads and grab download CUDA Toolkit 8.0. You will need to register an account with Nvidia’s Accelerated Computing Developer Program. Select Windows as your operation system, choose the version of Windows you’re on (8.1 or 10), and select your installer type.

CUDA Toolkit Download Page

Run the installer (you may have to run it as adminstrator) and follow the on-screen prompts. Remember the location that it was installed at, you will need it for the next step (default is “C:\Program Files\NVIDIA GPU Computing Toolkit”).

Next, we’ll download the cuDNN files from https://developer.nvidia.com/rdp/cudnn-download. Make sure to select the correct operating system.

Extract the files from the zip and you should see a single directory named “cuda”, inside you will find a few more directories: “bin”, “include”, and “lib” (there is another directory inside “lib” called “x64”). Each directory contains a file: “cudnn64_5.dll”, “cudnn.h”, and “cudnn.lib”. Locate the path that CUDA Toolkit was installed at and go into “CUDA\v8.0” (ex. “C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0”).

Copy and paste “cudnn64_5.dll” into “\bin”, “cudnn.h” into “\include”, and “cudnn.lib” into “\lib\x64”.

You’ve now got CUDA all set up!

Anaconda – Python 3.6

Now we’re going to install our Python environment. I prefer Anaconda because it bundles all the common packages that you’ll need in a single convenient download. Keras requires h5py, HDF5, NumPy, PyYAML, and SciPy, which luckily Anaconda comes with.

Visit https://www.continuum.io/downloads, and download the latest version of Anaconda for Windows. At the time of this writing, it is Anaconda 4.3.0 Python 3.6 Version. Again, launch the installer and follow the prompts (keep the Advanced Options checked).

Anaconda Python Installer Advanced Options

Create and Activate Conda Environment

Once Anaconda has been successfully installed, launch the Anaconda Prompt with administrator privileges and type in the following command:

conda create -n KerasEnv python=3.5 anaconda

NOTE: Make sure that the python version is set to 3.5. Tensorflow currently only supports python 3.5 on Windows.

Feel free to replace KerasEnv with a name of your choosing. Press y and then enter to continue and Anaconda will create a new virtual environment with all the included packages for you at “C:\ProgramData\Anaconda3\envs\KerasEnv. To activate the environment, use the command:

activate KerasEnv

Replace KerasEnv with the name you gave your environment when it was created. Your prompt should now display:

(KerasEnv) C:\WINDOWS\system32>_

To deactivate it, simply enter:

deactivate

To delete your environment, enter the following:

conda remove -n KerasEnv --all

Again, replace KerasEnv with the name of your environment.

Install Keras and Tensorflow with Pip

Make sure the environment is activated before proceeding with the next steps.

Pip is packaged with Anaconda so we can directly install Keras and Tensorflow using pip install. Let’s start with Tensorflow.

We will use pip to install Tensorflow. Although conda can be used to install Tensorflow, only the CPU version is available.

To install the GPU version of Tensorflow, use the following command:

pip3 install --upgrade tensorflow-gpu

If you do not have a GPU that supports CUDA, you can install the CPU-only version of Tensorflow using the command below:

pip3 install --upgrade tensorflow

Once completed, we can go ahead and install Keras using the following simple command:

pip install keras

At this point, Keras and Tensorflow has been successfully installed on your machine! The next couple of steps will go through how to set up PyCharm as our IDE and training an example neural network to make sure everything is working.

NOTE: TensorFlow requires MSVCP140.DLL, which may not be installed on your system. If, when you import tensorflow as tf, you see an error about No module named “_pywrap_tensorflow” and/or DLL load failed, check whether MSVCP140.DLL is in your %PATH% and, if not, you should install the Visual C++ 2015 redistributable (x64 version).

March 21, 2017: With the release of Keras 2.0, you can easily update by uninstalling your current Keras package through the command

pip uninstall keras

then re-installing with the command above.

Setting Up PyCharm

For our IDE, we will be using PyCharm Community (it’s free!), so let’s go ahead and download the Windows version from https://www.jetbrains.com/pycharm/download/#section=windows.

Once the download is completed, launch the installer and follow the prompts to install it.

Now, let’s launch PyCharm and create a new project. You can save your project in any location you’d like. For the Interpreter, we’re going to select our conda environment we made earlier. Click the gear on the right side and select “Add Local”, find the location of the conda environment you created earlier and select the “python.exe” file. The full path might look like the following:

C:\ProgramData\Anaconda3\envs\KerasEnv\python.exe

It may take a few minutes for PyCharm to update the interpreter afterwards.

PyCharm Splash

PyCharm New Project Window

Once that’s done and the PyCharm has launch, we can create a new python file. To create a new file, simply go to File -> New… -> Python File. Let’s name it “main.py”. It’s time to run our first neural network!

Running Our First Neural Network!

Keras offers an entire directory of example models made to demonstrate the features of the library which you can find here. You should be able to run any one of them now on your new Keras environment! I’ll use “mnist_cnn.py” as an example, which you can find here. This particular example builds and trains a convolutional neural network (CNN) on the MNIST dataset. The MNIST dataset is a very popular dataset comprised on images of handwritten characters. We will use a CNN to find features in the dataset which allows the CNN to recognize the characters. Since CNNs have wide applications in image and video recognition, it is the perfect tool to solve this type of problem.

Some samples of the MNIST dataset

The code is below:

'''Trains a simple convnet on the MNIST dataset.

Gets to 99.25% test accuracy after 12 epochs
(there is still a lot of margin for parameter tuning).
16 seconds per epoch on a GRID K520 GPU.
'''

from __future__ import print_function
import numpy as np
np.random.seed(1337)  # for reproducibility

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers import Convolution2D, MaxPooling2D
from keras.utils import np_utils
from keras import backend as K

batch_size = 128
nb_classes = 10
nb_epoch = 12

# input image dimensions
img_rows, img_cols = 28, 28
# number of convolutional filters to use
nb_filters = 32
# size of pooling area for max pooling
pool_size = (2, 2)
# convolution kernel size
kernel_size = (3, 3)

# the data, shuffled and split between train and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()

if K.image_dim_ordering() == 'th':
    X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)
    X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)
    input_shape = (1, img_rows, img_cols)
else:
    X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 1)
    X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 1)
    input_shape = (img_rows, img_cols, 1)

X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
print('X_train shape:', X_train.shape)
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')

# convert class vectors to binary class matrices
Y_train = np_utils.to_categorical(y_train, nb_classes)
Y_test = np_utils.to_categorical(y_test, nb_classes)

model = Sequential()

model.add(Convolution2D(nb_filters, kernel_size[0], kernel_size[1],
                        border_mode='valid',
                        input_shape=input_shape))
model.add(Activation('relu'))
model.add(Convolution2D(nb_filters, kernel_size[0], kernel_size[1]))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=pool_size))
model.add(Dropout(0.25))

model.add(Flatten())
model.add(Dense(128))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy',
              optimizer='adadelta',
              metrics=['accuracy'])

model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=nb_epoch,
          verbose=1, validation_data=(X_test, Y_test))
score = model.evaluate(X_test, Y_test, verbose=0)
print('Test score:', score[0])
print('Test accuracy:', score[1])

Copy the code to the “main.py” file you just created in your PyCharm project and save it. Open the terminal by clicking the “Terminal” button on the bottom left of the IDE. You will notice that the terminal has automatically activated the environment you set up earlier.

(C:\ProgramData\Anaconda3\envs\KerasEnv) C:\Users\Clark\Documents\KerasExample>

To run the code, we simply run the command “python main.py” in our terminal and watch the magic happen!

python main.py

Executing main.py

Training Keras CNN on MNIST

If everything is working correctly, we’ll see the network train a total of 12 epochs and finally reaching an accuracy of 98.9%! Feel free to test any of the other models from Keras Examples.

I strongly encourage you to go through Keras documentation and try to make your own models. If you enjoyed this post and want to see more like this or if you have any feedback, be sure to let me know in the comments.

Until next time!

PS. For anyone interested in reading more about machine learning or deep learning, I strongly suggest Jason Brownlee’s website/blog at http://machinelearningmastery.com/. I have personally found Jason’s practical style of teaching to be incredibly effective.

Clearing the Air on Automated Trading in Forex

It has come to my attention recently that opinions on automated trading, especially in the retail Forex market, tend to consist of mixed opinions which usually results in heated discussions within the trading community. I think the main cause of these disagreements is due to the fact that everybody seems to have their own definition on what automated trading means. In today’s post, I will attempt to explain the situation as best as I can from my personal and professional perspective. At the end I will offer my own opinions on the matter.

Let’s start by first clearing up some common terms that are thrown around whenever there is a discussion over automated trading in the Forex market. This will help start everybody off on the right page so we are able to have more clarity in our dialogue.

Algorithmic trading is simply mechanical trading following a set of procedures, rules or when objective conditions are met. Algorithmic trading DOES NOT necessarily mean automated trading, since it is simply following a set of rules; you can follow and trade these rules manually or program it in code. With that being said, the automation of algorithmic trading systems hold many benefits to traders as it allows one to be able to backtest as well as optimise the system rules and parameters using historical market data.

Automated trading is the process of automating certain parts or all logic of a trading system, generally using software. Automated trading could mean something as small as setting a take profit and stop loss after a trader manually opens a new position, or it could mean automating an entire trading system that can look for entries, calculate position sizes, manage risk, exit positions, and other features required by the operator/trader. Without becoming too philosophical, discretionary trading systems simply cannot be automated.

An Expert Advisor (EA) is a proprietary file format used to program and execute automated trading supported only by the MetaTrader 4 (MT4) and MetaTrader 5 (MT5) trading platforms. Programming a trading system as an EA will allow a trader to backtest/optimise the system through MT4 and MT5’s Strategy Tester as well as execute it real-time in live markets. I see many traders using the term EA and automated trading interchangeably, which can lead to some miscommunication.

High-frequency trading (HFT), as the name suggests, is trading at high speeds that leverages high-frequency market data. Hundreds or even thousands of orders can be placed within a time span of a few seconds, with each position seeking relatively small profits. A HFT strategy is algorithmic in nature and can be considered a sub-category of algorithmic trading. HFT is almost always automated due to data processing and execution requirements.

A trading system is a set of rules or conditions that traders can use to make trading and investment decisions. These rules and/or conditions can be subjective (discretionary) and objective (mechanical). A discretionary trading system will leverage the trader’s experience, knowledge, and intuition to make trading decisions. On the other hand, a mechanical trading system will have a set of hard and quantifiable rules, and with a little effort can be written as an algorithm. “Trading system”, “trading model”, and “trading strategy” are generally terms that can be used interchangeably.

Before I dive into my opinions, I encourage you to take what I have to say with a grain of salt. My experiences and opinions will differ with yours and that is perfectly fine! Everybody is entitled to their own opinions and I hope that by sharing my perspectives we can get the ball rolling on healthy and productive discussions.

Now that we have gotten the basic terms out of the way, we can shed some light on automated trading in the Forex market. There is perhaps a general misconception in the retail Forex community that automated trading is an easy way out. What I mean by this is that I see many individuals who seem confident in that all it takes to be profitable is to purchase premade trading “robots” (usually in the form of EAs), set it on auto-pilot and watch their trading account collect profits. In reality, what ends up happening is that they spend hundreds if not thousands of their hard-earned money on EAs from various online vendors only to have it unexpectedly blow up their trading accounts; some not even making it past demo accounts. In order to keep this article short, I will forgo explaining the reasons why these commercial EAs do not perform as marketed; but if there is interest I will make sure to make another post explaining why commercial EAs fail.

This is NOT automated trading. This is just an example of how mis-informed the trading community can be.

Real automated trading can take years to understand on top of having a strong trading foundation; it requires technical skills as well as a shift in problem solving. To be an automated trader a strong mathematics, statistics, and programming background is required, alongside market and trading experience; some machine learning basics can go a long way as well. Like any business venture, it takes dedication to truly become successful. A trader needs to familiarize themselves with each step and process involved. Only then can an automated trader plan, design, prototype, backtest/optimise, execute, and manage their own automated trading systems (ATS) on their own. Only by being in control of each step along the way, will an automated trader have any real chance of becoming consistently profitable in the long run.

To summarize, automated trading done correctly can potentially yield a healthy return on investment, but effort and dedication is a must; a lazy approach will not work. Sure, there are times when an individual gets lucky and makes a quick and temporary profit trading a commercial EA, but more often than not the results are lackluster at best. I believe automated trading as a whole has been given a negative connotation because of the results of falsely marketed and over-hyped commercial EAs. Because of this, I personally refrain from associating the terms “Expert Advisor”, or “EA”, in the majority of my discussions with peers when it comes to algorithmic or automated trading. My purpose with this article is to shed some light on this hotly debated subject and hopefully offer some hope to aspiring automated traders!

With all this being said, what are your opinions on automated trading in the retail Forex market? Do you have any experiences you’d like to share? Let’s talk about them!