GPU computing

Nowadays GPU computing is a standard approach in deep learning. Even low end graphics cards can give a significant performance boost in arithmetic operations. In this post I will check how much faster you can train neural network with GPU computing.

GPU performance

First, I used Caffe framework for deep learning. It comes with several known benchmarks which allows to test some of the functionality and checks how your hardware performs.

On classical MNIST LeNet example I had almost 5x better times when using GPU:

CPU vs. GPU on Caffe

Next, I tried cxxnet.There were some problems during compilation with CUDA support turned on. The error message during build.sh was:

nvcc -c -o updater_gpu.o --use_fast_math -g -O3 -ccbin g++  -Xcompiler "-DMSHADOW_FORCE_STREAM -Wall -g -O3 -I./mshadow/  -fPIC -msse3 -funroll-loops -Wno-unused-parameter -Wno-unknown-pragmas -DMSHADOW_USE_CBLAS=1 -DMSHADOW_USE_MKL=0 -DMSHADOW_DIST_PS=0 -DCXXNET_USE_OPENCV=1 -DCXXNET_USE_OPENCV_DECODER=1 " src/updater/updater_impl.cu
src/updater/./sgd_updater-inl.hpp(18): error: expected an identifier

1 error detected in the compilation of "/tmp/tmpxft_000021a9_00000000-6_updater_impl.cpp1.ii".
make: *** [updater_gpu.o] Error 2

I had to remove std:: prefix in line 18 of src/updater/sgd_updater-inl.hpp file:

if (std::isnan(a)) return 0.0f;

After fixing it the build was successful. I tried once again an example based on MNIST dataset (training convolution neural net). That’s cxxnet/example/MNIST/MNIST_CONV.conf configuration file.

This time the performance on GPU was over x20 better when compared to standard CPU (with BLAS).

CPU vs. GPU cxxnet

Libraries

It’s worth mentioning some other factors that can influence overall performance. When building various deep learning frameworks you can often choose which libraries it will used. Usually there are ATLAS, BLAS/OpenBLAS or MKL options. It can matter a lot when you work without GPU computing.

In my experiments it turned out that BLAS/OpenBLAS had much better performance than standard ATLAS option. In fact OpenBLAS Wikipedia page says that it has similar performance to Intel proprietary MKL, which I have not tested.

  • In Caffe scenario BLAS compilation was about 1.5 times faster than ATLAS (1765 s vs. 1161 s)
  • In cxxnet the difference was smaller: 806 s vs 907 s (again BLAS faster)

Hardware

The tests were executed on Intel(R) Xeon(R) CPU 5140 @ 2.33GHz system (2x dual core) with 10GB RAM. The GPU used was GeForce GTX 460 card (bought used for about 50-60 USD).

References and useful links

I mentioned earlier Caffe and cxxnet deep learning frameworks. The documentation and tutorials are helpful and I didn’t have much problems when configuring it and running. There were many missing dependencies during compilation, but everything could be installed from standard Ubuntu repository.

Installing Nvidia CUDA library on Ubuntu was straight forward.

There is a great blog written by Tim Dettmers about deep learning especially from hardware perspective (this post for example). Seems to be very helpful if you consider investing in your hardware.

House of Cards

It’s been over a month since House of Cards season 3 release, so we can check now in which countries this Netflix series attracted most people. I will also try to compare popularity of some actors and directors which are related to the latests episodes.

It’s not a big secret that most of the page views come from the English Wikipedia. Probably many non-English users also visit English page when they want to check the most recent updates about the movie. But still it looks quite differently when we compare the popularity (page views / total page views for given language) instead of page views:

House of Cards popularity on Wikipedia

On Italian Wikipedia this article was most popular, than we have Portuguese and Chinese followed by English, Spanish and Dutch. It seems that House of Cards is very popular also in countries where Netflix is still not available.

Frank and Claire Underwood

Kevin Spacey and Robin Wright shared some popularity patterns, but there are  differences among languages. For example on Portuguese Wikipedia Kevin Spacey gained more attention, whereas on most of the languages they both had similar number of page views (or Robin Wright was slightly more popular):

Kevin Spacey and Robn Wright page views on Wikipedia after releasing season 3 of House of Cards

Zoe Barns still alive (in Germany)

What really surprised me was the fact that Kate Mara (she played journalist Zoe Barnes, murdered in season 2) still follows the typical House of Cards pattern. Most noticeable on German Wikipedia:

Kate Mara popularity after realsing Seson 3 of House of Cards

This could be coincidence, but maybe people preferred to watch season 2 again before enjoying season 3.

Pussy Riot

Pussy Riot members appeared in one of the episodes and speak up to the Russian president visiting the White House. This had a follow-up in Wikipedia traffic in many languages, but not in Russian:

Pussy Riot popularity on Wikipedia after releasing seson 3 of House of Cards

Directors

In the latest House of Cards season two episodes were directed by Agnieszka Holland. I checked popularity of others directors and it seems that only James Foley had a higher number of page views. I didn’t included Robin Wright here, who also directed two episodes, because as a main actress she had a much higher number of readers on Wikipedia than directors:

House of Cards directors