Hi J'ers, I'm trying to perform calculations on massive data sets using mapped files and after searching the forum/essays am stuck.
I've load 50 million integer digits of pi and trying to do a moving average on them. I 've created two mapped files one for the data and one for the calculated results, but I get out of memory errors. I'm not seeing what I'm doing wrong here. My goal is to work with floating point numbers eventually, but i'd figure i'd start with integers first, since there are more examples for integers than floats in the 'mapped files' Labs (if anyone has the code for the example mention in the 67 of 68 slide of the Lab on mapped files using NYSE stock prices - I'd love to see it). load 'jmf files dir' NB. data from http://zenwerx.com/projects/pi-digits/pi/ NB. used perl to write each digits on 1 line of file data pidata =: fread '/home/joe/pi-study/data' NB. type is 2 (literal) 3!:0 pidata NB. convert to 4-byte integers piInt =: _2 (3!:4) pidata NB. shape is 50,000,000 $piInt ]sz=:7!:5 <'piInt' NB. define moving average ma =: mean \ NB. out of memory ma, expected since storing results in ram smoothPi =: 100 ma piInt NB. create jmf for results fn =: jpath '~temp/pi-results.jmf' ]sz=: 7!:5 <'piInt' createjmf_jmf_ fn;sz JFL map_jmf_ 'piResults'; fn piResults =. 100 ma piInt NB. still get out of memory! Much thanks ! Joe ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
