Implement Two-Layer Perceptron Algorithm

Question

Train a two-layer perceptron. You will find the Wikipedia article and these lecture slides useful.

Use a sigmoid activation function for both the hidden and output layers, $$\sigma(x) = \frac{1}{1 + e^{-x}}$$

Your function should take in a list or array of input/output pairs and an integer specifying how many training passes through the input set to make. Both the input and output will be vectors of any convenient representation in your language. You should output or otherwise initialize a classifier function or object.

Initialize the weights randomly and to prevent overflow/underflow, normalize the weight vectors after each update.

You should handle input data vectors up to 1000 dimensions and at least 10 output dimensions. The classifier function should return a vector of the activation levels of each of the outputs.

You may assume access to implementations of all of the BLAS routines.

The MNIST database has a very large training and test set you might find useful for testing purposes.

EDIT: Example dataset.

Input and output vectors (input).

Input  Output
[0,0]   [0]
[0,1]   [1]
[1,0]   [1]
[1,1]   [0]

Your program should produce a classifier that takes in a vector of two elements and returns a vector of one element that has similar behavior as the input.

Answer 1

I'm going to cheat here, hence community wiki: instead of logistic sigmoid, I'll use the hyperbolic tangent and instead of backprop, I'll use ELM to train the perceptron. Here's the version for XOR, in Python, using BLAS via SciPy.

import numpy as np
from scipy.linalg import pinv2

n_hidden = 10  # hyperparameter

# Training set
X_train = np.array([[0,0], [0,1], [1,0], [1,1]])
y_train = [0, 1, 1, 0]

# set hidden layer parameters randomly
w = np.random.randn(X_train.shape[1], n_hidden)
b = np.random.randn(n_hidden)

# compute hidden layer activation
H = np.tanh(np.dot(X_train, w) + b)

# fit output layer parameters
beta = np.dot(pinv2(H), np.atleast_2d(y_train).T)

# prediction and evaluation
pred_train = (np.dot(H, beta) > .5).ravel()
print "Training set accuracy:", np.mean(pred_train == y_train)

I previously published a digits classification script using this algorithm. After removing comments, this is 11 lines of code @ 425 characters.