Re: parsing and chunking large xyz files

2014-12-27 Thread cej38
I just noticed that there was an extra line of code (which doesn't work) in my original post. Sorry. I edited it out below. In molecular dynamics a popular format for writing out the positions of the atoms in a system is the xyz file format (see: http://en.wikipedia.org/wiki/XYZ_file_format

Re: parsing and chunking large xyz files

2014-12-26 Thread cej38
Francis, Thank you for the gist. I looked through it briefly just now, but won't be able to look at it in detail until later tonight or tomorrow morning. On Friday, December 26, 2014 2:40:53 PM UTC-5, Francis Avila wrote: > > If you need parallelism, you need to do an indexing pass first to

Re: parsing and chunking large xyz files

2014-12-26 Thread Francis Avila
If you need parallelism, you need to do an indexing pass first to determine the group boundaries. Then you can process them in parallel because you know the units-of-work. Iota is an ok fit for this, so I suggest trying it first. (You may have to dial down the parallelism of r/fold to avoid str

Re: parsing and chunking large xyz files

2014-12-26 Thread cej38
Line-by-line is the problem. I need groups of lines at a time. On Friday, December 26, 2014 10:33:27 AM UTC-5, Jony Hudson wrote: > > I think clojure.csv reads CSV files lazily, line-by-line, so might be > useful to take a look at: > > https://github.com/clojure/data.csv > > > Jony > > On Fri

Re: parsing and chunking large xyz files

2014-12-26 Thread Jony Hudson
I think clojure.csv reads CSV files lazily, line-by-line, so might be useful to take a look at: https://github.com/clojure/data.csv Jony On Friday, 26 December 2014 14:49:59 UTC, cej38 wrote: > > In molecular dynamics a popular format for writing out the positions of > the atoms in a system i

parsing and chunking large xyz files

2014-12-26 Thread cej38
In molecular dynamics a popular format for writing out the positions of the atoms in a system is the xyz file format (see: http://en.wikipedia.org/wiki/XYZ_file_format and/or http://www.ks.uiuc.edu/Research/vmd/plugins/molfile/xyzplugin.html). The format allows for storing the positions of the