Dear all,
There has been a lot of interest recently in performing deep learning
without backprop - papers by Geoff Hinton and Barak Pearlmutter in
particular showing how supervised and self-supervised learning can be
achieved solely with forward message propagation. We wanted to mention
here our own work dating back to 2019, in which we realised real-time
reinforcement learning in a closed-loop setting via the forward
propagation of error signals. Naturally forward propagation has the
attraction of biological feasibility, but from our point of view the
more important point is that error signals are then defined in the space
of sensory inputs rather than motor outputs, which makes a lot more
sense in the context of a self-contained autonomous agent. Rather than
the reward functions of Q learning, we use reflex circuits to generate
the errors: the goal of sensory learning is to predict disturbances
before they occur, thereby silencing the reflex.
To see the system in operation, here is a real robot learning a simple
line-following task, where the system initially uses a light sensor to
drive a reflex steering action, and learning discovers predictive
information from vision which pre-empts this reflex:
https://sigmoid.social/@berndporr/109534507823054808
Because the algorithm works for deep networks, arbitrarily complex
predictive cues can in principle be discovered.
Source code and paper:
https://github.com/glasgowneuro/feedforward_closedloop_learning
We would be highly interested to hear views on this approach.
/Bernd Porr and Paul Miller
--
http://www.berndporr.me.uk
http://www.attys.tech
+44 (0)7840 340069
