Hi Ritchie, Unfortunately, I can't help you as to why your swarm uses up so much memory, but I have swarmed over data files with sizes of 8MB without problems (it only takes some time, as you would expect). I have 6GB of RAM. According to the resources monitor the swarming process uses about 320MB initially and grows to a total of 460MB.
However, I don't think file size matters too much when swarming. What is more relevant is the number of fields in your file and the two things are not necessarily connected (a field can have any byte size, so a large file may not indicate a large number of fields). But I am not too confident about this. Let's see what the rest of the community has to say about it ;). A workaround would for now would be to limit the number of lines you swarm over (in the search_def.json), but that is equivalent to swarming over a smaller file. Manuel ________________________________________ From: nupic [[email protected]] on behalf of Ritchie Lee [[email protected]] Sent: 09 April 2014 22:07 To: [email protected] Subject: [nupic-discuss] Swarming process using a large amount of memory Hi friends of NuPIC! I have been running swarms on csv data files that are around 3 megabytes in size, and I have found that it is using about >6 gigabytes of RAM during the process. If I run swarms on data files that are larger than that, my computer runs out of RAM and hangs (I have 8 GB of RAM). In particular I've tried swarming on a 13 MB data file and it froze very quickly. Memory usage seems to climb monotonically during the swarming process, and released all at the end on completion. I am wondering if anyone has had experience swarming large (>10 MB) csv files and your experiences with the memory consumption. Ideally I'd like to be able to swarm over much larger datasets (on the order of a hundred megs). Thanks, Ritchie Lee Research Engineer Carnegie Mellon University-Silicon Valley NASA Ames Research Center Bldg 23, Rm 115 Moffett Field, CA 94035 (650) 335-2847 _______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
