# Implement Two-Layer Perceptron Algorithm

Train a two-layer perceptron. You will find the Wikipedia article and these lecture slides useful.

Use a sigmoid activation function for both the hidden and output layers, $$\sigma(x) = \frac{1}{1 + e^{-x}}$$

Your function should take in a list or array of input/output pairs and an integer specifying how many training passes through the input set to make. Both the input and output will be vectors of any convenient representation in your language. You should output or otherwise initialize a classifier function or object.

Initialize the weights randomly and to prevent overflow/underflow, normalize the weight vectors after each update.

You should handle input data vectors up to 1000 dimensions and at least 10 output dimensions. The classifier function should return a vector of the activation levels of each of the outputs.

You may assume access to implementations of all of the BLAS routines.

The MNIST database has a very large training and test set you might find useful for testing purposes.

EDIT: Example dataset.

Input and output vectors (input).

Input  Output
[0,0]   
[0,1]   
[1,0]   
[1,1]   


Your program should produce a classifier that takes in a vector of two elements and returns a vector of one element that has similar behavior as the input.

• Which learning algorithm? What about examples or testcases? (We should just search the MNIST? :-/) – Eelvex Mar 25 '11 at 4:24
• The backpropagation algorithm. Minimize the MSE of the classifier output against the training data by gradient descent. I admit MNIST database is a bit (okay, very) large, but you should be able to solve a version of the XOR problem. – Hoa Long Tam Mar 25 '11 at 4:43
• This seems far too open ended to be a code-golf problem. It would be very hard to compare solutions in terms of source-code character count when they could be doing very different things. I would suggest narrowing this down to a particular training set, and and quantifiable, and testable measure of success on a given test set. Input and output formats should be rigidly defined, and hence enable testability. This would level the field significantly, and better allow code size comparisons to be meaningful. – MtnViewMark Mar 25 '11 at 6:33

I'm going to cheat here, hence community wiki: instead of logistic sigmoid, I'll use the hyperbolic tangent and instead of backprop, I'll use ELM to train the perceptron. Here's the version for XOR, in Python, using BLAS via SciPy.

import numpy as np
from scipy.linalg import pinv2

n_hidden = 10  # hyperparameter

# Training set
X_train = np.array([[0,0], [0,1], [1,0], [1,1]])
y_train = [0, 1, 1, 0]

# set hidden layer parameters randomly
w = np.random.randn(X_train.shape, n_hidden)
b = np.random.randn(n_hidden)

# compute hidden layer activation
H = np.tanh(np.dot(X_train, w) + b)

# fit output layer parameters
beta = np.dot(pinv2(H), np.atleast_2d(y_train).T)

# prediction and evaluation
pred_train = (np.dot(H, beta) > .5).ravel()
print "Training set accuracy:", np.mean(pred_train == y_train)


I previously published a digits classification script using this algorithm. After removing comments, this is 11 lines of code @ 425 characters.