I think a better question is how can HTM computation be parallelized? For example, can we map a whole region to a single GPU card? What functions in a region can be isolated as kernels? What could be running as a (CUDA) thread? How can these threads be partitioned into blocks and grids?
On Fri, Feb 6, 2015 at 6:22 AM, Fergal Byrne <[email protected]> wrote: > Hi Rich, > > Thanks for restarting this discussion. > > I started a project to reimplement HTM in Elixir in December 2013, but > then switched to Clojure for mainly non-technical reasons. One of the > "thought leaders" in the Elixir community is interested in being the > lynchpin of the project to bring HTM to Elixir, so I'll be making some > announcements on that in the near future (I have some code archaeology to > perform first!). > > Your intuitions are correct about both the scalability and simplicity of > using Elixir/Erlang to do HTM. In my initial tests, I was able to spring up > 250k neuron processes in 20-25ms in Elixir on a laptop. > > On the general point of distributed HTM, Michael is correct to identify > the granularity at which things can be split up, and Tim is on the money > about the kernel of the issue - state (synapses, in particular). In a > typical NuPIC-sized region, we have 2048 cols x 32 cells = 64k total cells, > with in the neighbourhood of 1-300m synapses. The number of "messages" > passed between these neurons (and their "state") is thus very large > compared with the number of input and out messages between regions. It > makes sense to have the processing for a contiguous "patch" of neurons such > as this all contained within a single (OS level) process, and to have > patches communicate using SDRs. > > Matt is correct when he describes the importance of this in the context of > Temporal Pooling and hierarchy. With a multi-layer architecture for a > single region, and a hierarchy of regions, we will very quickly hit the > skids if we continue with a single-threaded, monolithic design for HTM. On > the other hand, Matt is also correct that, once solved, we can take > advantage of distributed processing to build HTM systems as large and > powerful as we like. > > Within a patch, I think the jury is very much out on the performance of > message-passing versus sparse vectors (as used in NuPIC). Due to sparseness > both in space and time in real world data, it's not clear that > message-passing (or some equivalent, functional reactive scheme) would not > outperform the use of big sparse arrays. > > Regards, > > Fergal Byrne > > On Thu, Feb 5, 2015 at 7:07 PM, Rich Morin <[email protected]> wrote: > >> On Feb 5, 2015, at 05:18, Kevin Archie <[email protected]> wrote: >> > https://github.com/nupic-community/comportex >> > >> > (I have no connection to the project, I’m just aware of it.) >> >> The Clojure ports are certainly worth a look, if only to see how they >> decompose the problem. Although scalability is a motivation, my real >> interest has to do with seeing how Elixir (including Erlang and OTP) >> can be used to simplify the model. That is, can I model things like >> neurons, columns, and regions using lightweight processes, leaving >> the communication and management to OTP. >> >> -r >> >> -- >> http://www.cfcl.com/rdm Rich Morin [email protected] >> http://www.cfcl.com/rdm/resume San Bruno, CA, USA +1 650-873-7841 >> >> Software system design, development, and documentation >> >> >> >> > > > -- > > Fergal Byrne, Brenter IT > > http://inbits.com - Better Living through Thoughtful Technology > http://ie.linkedin.com/in/fergbyrne/ - https://github.com/fergalbyrne > > Founder of Clortex: HTM in Clojure - > https://github.com/nupic-community/clortex > > Author, Real Machine Intelligence with Clortex and NuPIC > Read for free or buy the book at https://leanpub.com/realsmartmachines > > Speaking on Clortex and HTM/CLA at euroClojure Krakow, June 2014: > http://euroclojure.com/2014/ > and at LambdaJam Chicago, July 2014: http://www.lambdajam.com > > e:[email protected] t:+353 83 4214179 > Join the quest for Machine Intelligence at http://numenta.org > Formerly of Adnet [email protected] http://www.adnet.ie >
