axel's dev corner

When I had to implement some backprop-exercise for a computational intelligence course, I realized how much fun blocks in Ruby really are. Ok, cutting away the theory, the basic beaviour of my 3-layer feed foreward net is as follows:

The inputs i0..n are combined with an extra 1 to a vector ai. That is multiplied with the input weight matrix wi, and each vector component fed to a tanh activation function. The resulting hidden activation vector ah is in the same manner processed using the output weight matrix wo. The result can be binarized using the mapping { [-1..0],(0..1] } -> { 0,1 }.

In order to achieve a fluent ruby implementation, I added an index_collect! method to my vector class. It simply yields the indices 0..n-1 and collects the new component values. For convenience, it also returns itself, so that  I can use the temp var y.

# evaluate input
def eval(x)
  # inputs
  @ai=Vector.from_array(x); @ai.push 1.0
  # hidden activation
  y=@ah.index_collect!{ |i| @activation.f(@wi[i]*@ai) }
  # output layer
  y=@ao.index_collect!{ |i| @activation.f(@wo[i]*y) }
end

In the back-propagation step, the error on the outputs is fed backwards through the net, where at each neuron the derivative of the function is applied. The so gathered values at the neurons are used to update the weight matrix, using a small nuber as learining stepwidth. Moving alog the derivative, aka gradient decent, moves the weigths towards a (local) error minimum. A nice graphical explanation of the standard backpropagation algorithm can be found here.

To simlify the calculation loops, I added an index-value collection to the vector class (iv_collect), and the matrix as well (iiv_collect). So the formulars can be written as is. The output derivative od is calulated as s(o-v)' = s'(c)*(v-o), i.e ao.iv_collect of  @activation.df(v)*(v-y[i]). The hidden derivative has to be calculated as sum over od and wo, multiplied by s'(ah). The weight update for the output layer is calculated as wo_j,k = wo_j,k- nu od_jah_k, so I justiiv_collect! it, same goes for wi:

# online backpropagation  
def backprop_online(x,y)
  eval(x)
  # backpropagation to output layer, output deltas s(o-v)' = s'(v)*v-o
  od = @ao.iv_collect { |i,v|  @activation.df(v) * (v-y[i])  }
  # backpropagation to hidden layer, hidden deltas 
  hd = @ah.iv_collect { |i,v|  od.sum_iv{ |j,o| o*@wo[j][i] } * @activation.df(v) }
  # update output weights  
  @wo.iiv_collect! { |j,k,wo_jk| wo_jk-@nu*od[j]*@ah[k] }
  # update input weights
  @wi.iiv_collect! { |j,k,wi_jk| wi_jk-@nu*hd[j]*@ai[k] }      
end

Yes, we are already done with backprop.

Time enough to add a fancy FX gui:

The whole code is now part of APRL, you can download it here or view the rdocs, if you like

Motivation

Unit Testing is a natural way of maintaining a class library. I actually switched to writing the test cases before the actual implementation, or both simultaneously, quickly after writing the first test. If you don't believe me that unit testing is natural and inevitable, just do not do it and keep count of the problems you could have avoided in, say, a 5-year maintenance of some project involving more than two programmers.

Ruby=short

Allow me to quote "Whenever I find a new feature in ruby it tends to be drastically easier than I expected and it makes me happy" [glu.ttono.us] Well, I could not have said it any better. In Ruby, your test case looks like this

# this makes it a runnable test
require 'test/unit'
# sample test case for class Klass
require 'klass'
class TestOfKlass
  def test_some_name # must start with test_
     k = Klass.new('gin')
     assert(k.name=='gin')
  end
  def test_sanity # for demonstration
     assert_equal(1,1)
  end
  def test_bud # and another one
     assert(true) # true true ..
  end
end

More more more...

However, you'll typically want to have a large test suit for a project assuring you that "all works as expected" assuming the above was called 'test_case_1.rb' and you got 3 more you could write

require 'test/unit/testsuite'
require 'test_case_1'
require 'test_case_2'
require 'test_case_3'
require 'test_case_4'

Loaded suite test_all
Started
...........................
Finished in 12.4 seconds.

24 tests, 119 assertions, 0 failures, 0 errors

Ok, I might rephrase "Whenever I want to do something reasonable in Ruby, it turns out to be drastically easy."

That's it, goodbye with a large thank-you to Nathaniel Talbott, author of Test::Unit