Hi Ritchie, I did some experiments (not swarming) with large scale CLAs, it turns out python objects take a lot of resources. Your memory demands would depend on: encoders you use, the size and type of input data, and implementation of spatial pooler and temporal pooler (currently we have 2 - python and cpp). Could you share these details or best the "OPF settings file" where you describe the structure of data to Nupic (description.py or something)?
Cheers, Mark On Thu, Apr 10, 2014 at 12:31 PM, Scheele, Manuel < [email protected]> wrote: > Hi Ritchie, > > Unfortunately, I can't help you as to why your swarm uses up so much > memory, but I have swarmed over data files with sizes of 8MB without > problems (it only takes some time, as you would expect). I have 6GB of RAM. > According to the resources monitor the swarming process uses about 320MB > initially and grows to a total of 460MB. > > However, I don't think file size matters too much when swarming. What is > more relevant is the number of fields in your file and the two things are > not necessarily connected (a field can have any byte size, so a large file > may not indicate a large number of fields). But I am not too confident > about this. Let's see what the rest of the community has to say about it ;). > > A workaround would for now would be to limit the number of lines you swarm > over (in the search_def.json), but that is equivalent to swarming over a > smaller file. > > Manuel > > > > > ________________________________________ > From: nupic [[email protected]] on behalf of Ritchie Lee [ > [email protected]] > Sent: 09 April 2014 22:07 > To: [email protected] > Subject: [nupic-discuss] Swarming process using a large amount of memory > > Hi friends of NuPIC! > > I have been running swarms on csv data files that are around 3 megabytes > in size, and I have found that it is using about >6 gigabytes of RAM during > the process. If I run swarms on data files that are larger than that, my > computer runs out of RAM and hangs (I have 8 GB of RAM). In particular > I've tried swarming on a 13 MB data file and it froze very quickly. Memory > usage seems to climb monotonically during the swarming process, and > released all at the end on completion. > > I am wondering if anyone has had experience swarming large (>10 MB) csv > files and your experiences with the memory consumption. Ideally I'd like > to be able to swarm over much larger datasets (on the order of a hundred > megs). > > Thanks, > > Ritchie Lee > Research Engineer > Carnegie Mellon University-Silicon Valley > NASA Ames Research Center > Bldg 23, Rm 115 > Moffett Field, CA 94035 > (650) 335-2847 > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > -- Marek Otahal :o)
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
