axel's dev corner

When I had to implement some backprop-exercise for a computational intelligence course, I realized how much fun blocks in Ruby really are. Ok, cutting away the theory, the basic beaviour of my 3-layer feed foreward net is as follows:

The inputs i0..n are combined with an extra 1 to a vector ai. That is multiplied with the input weight matrix wi, and each vector component fed to a tanh activation function. The resulting hidden activation vector ah is in the same manner processed using the output weight matrix wo. The result can be binarized using the mapping { [-1..0],(0..1] } -> { 0,1 }.

In order to achieve a fluent ruby implementation, I added an index_collect! method to my vector class. It simply yields the indices 0..n-1 and collects the new component values. For convenience, it also returns itself, so that  I can use the temp var y.

# evaluate input
def eval(x)
  # inputs
  @ai=Vector.from_array(x); @ai.push 1.0
  # hidden activation
  y=@ah.index_collect!{ |i| @activation.f(@wi[i]*@ai) }
  # output layer
  y=@ao.index_collect!{ |i| @activation.f(@wo[i]*y) }
end

In the back-propagation step, the error on the outputs is fed backwards through the net, where at each neuron the derivative of the function is applied. The so gathered values at the neurons are used to update the weight matrix, using a small nuber as learining stepwidth. Moving alog the derivative, aka gradient decent, moves the weigths towards a (local) error minimum. A nice graphical explanation of the standard backpropagation algorithm can be found here.

To simlify the calculation loops, I added an index-value collection to the vector class (iv_collect), and the matrix as well (iiv_collect). So the formulars can be written as is. The output derivative od is calulated as s(o-v)' = s'(c)*(v-o), i.e ao.iv_collect of  @activation.df(v)*(v-y[i]). The hidden derivative has to be calculated as sum over od and wo, multiplied by s'(ah). The weight update for the output layer is calculated as wo_j,k = wo_j,k- nu od_jah_k, so I justiiv_collect! it, same goes for wi:

# online backpropagation  
def backprop_online(x,y)
  eval(x)
  # backpropagation to output layer, output deltas s(o-v)' = s'(v)*v-o
  od = @ao.iv_collect { |i,v|  @activation.df(v) * (v-y[i])  }
  # backpropagation to hidden layer, hidden deltas 
  hd = @ah.iv_collect { |i,v|  od.sum_iv{ |j,o| o*@wo[j][i] } * @activation.df(v) }
  # update output weights  
  @wo.iiv_collect! { |j,k,wo_jk| wo_jk-@nu*od[j]*@ah[k] }
  # update input weights
  @wi.iiv_collect! { |j,k,wi_jk| wi_jk-@nu*hd[j]*@ai[k] }      
end

Yes, we are already done with backprop.

Time enough to add a fancy FX gui:

The whole code is now part of APRL, you can download it here or view the rdocs, if you like