Machine Learning

Getting Started

DeepLearning.TV YouTube playlist -- good starter!

Tuts

UFLDL Stanford (Deep Learning) Tutorial

Principles of training multi-layer neural network using backpropagation <-- Great visual guide!

Courses

Neural Networks for Machine Learning — Geoffrey Hinton, UToronto

- Coursera course
- Vids (on YouTube)
- same, better organized
- Intro vid for course
- Hinton's homepage
- Bayesian Nets Tutorial -- helpful for later parts of Hinton

Deep learning at Oxford 2015 (Nando de Freitas)

Notes for Andrew Ng's Coursera course.

Hugo Larochelle: Neural networks class - Université de Sherbrooke

Karpathy -- CS231n

Boltzmann Machines

Boltzmann Distribution connects the probability of a system being in a certain state with the energy of that state: $P(s)\propto e^{{-kE_{s}}}$ where $E_{s}$ is the energy for that state.

So we can write: $P(s)={\frac {1}{Z}}e^{{-kE_{s}}}$ where $Z$ is the PARTITION FUNCTION (energy summed over all possible states). Reversing we can write $E_{s}=-{\frac {1}{k}}ln(ZP(s))$

IIUC there is a wide class of systems that settle down into a Boltzmann distribution.

Each (binary) vector of unit-outputs creates a corresponding Energy state. Let's assume these energy states form a Boltzmann distribution.

We can calculate the Probability of a given unit's state being ON just by knowing the system's energy gap between that unit being ON and OFF: $p_{{\text{i=on}}}={\frac {1}{1+\exp(-{\frac {\Delta E_{i}}{T}})}}$ where $\Delta E_{i}=E_{{\text{i=off}}}-E_{{\text{i=on}}}$ (working here)

---

Think: Hopfield Net, but each neuron stochastically has state 0 or 1. Prob(state <- 1) = sigma(-z) where z = preactivation.

Boltzmann distribution: Wikipedia Susskind lecture

Hopfield to Boltzmann http://haohanw.blogspot.co.uk/2015/01/boltzmann-machine.html

Hinton's Lecture, then:

https://en.wikipedia.org/wiki/Boltzmann_machine

http://www.scholarpedia.org/article/Boltzmann_machine

Hinton (2010) -- A Practical Guide to Training Restricted Boltzmann Machines

https://www.researchgate.net/publication/242509302_Learning_and_relearning_in_Boltzmann_machines

Papers

Applying Deep Learning To Enhance Momentum Trading Strategies In Stocks

- Hinton, Salakhutdinov (2006) -- Reducing the Dimensionality of Data with Neural Networks

Books

Nielsen -- Neural Networks and Deep Learning <-- online book

http://www.deeplearningbook.org/

https://page.mi.fu-berlin.de/rojas/neural/ <-- Online book

S/W

http://playground.tensorflow.org

TensorFlow in IPython YouTube (5 vids)

SwiftNet <-- My own back propagating NN (in Swift)

Misc

Oxford AI/Trader

Links: https://github.com/memo/ai-resources

http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ <-- Great article!

http://karpathy.github.io/2015/05/21/rnn-effectiveness/

https://www.youtube.com/watch?v=gfPUWwBkXZY <-- Hopfield vid

http://www.gitxiv.com/ <-- Amazing projects here!