Being a fan of things pi, I'm wondering about several things in your post.
The URL you give points to a site purporting to have 4 (not 50) million
digits of pi. Maybe I missed a pointer to a larger dataset.
I have a file on my machine comprising 33,554,433 digits of pi - so
using that to do what you say you are trying to do -
50 {. pi =. fread 'Pi33554433.txt'
3.141592653589793238462643383279502884197169399375
NB. a bit silly that the long string of characters includes the
decimal place... So getting rid of it -
20 {. npi =. 3, '0123456789'i. 2}. pi NB. npi is integer values of
the digits of pi
3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3 2 3 8 4
ma =: (+/ % #)\
NB. as pointed out in another post, this definition of moving
average is quite efficient.
$ av100 =. 100 ma npi
33554334
>./av100
6.06
<./av100
3.07
time and space for the longer of the above operations (done in 64 bit J
so takes more space than 32 bit...) -
timex 'npi =. 3, ''0123456789''i. 2}. pi'
0.74283 1.14085e9
timex 'av100 =. 100 ma npi'
0.465046 2.68438e8
On another point, I don't understand your conversion to 4-byte integers
- I assumed above that you meant the integers 3 1 4 1 5 9 2 6 .... but
your expression converts the character codes (in groups of 4) into
machine words - i.e.
_2 (3!:4) '314159265353'
825504051 909261109 859124533
I don't think these are the integers you wanted to take the moving
average of... They do demonstrate the little endian internal storage
involved -
256 256 256 256 #: 825504051 909261109 859124533
49 52 49 51
54 50 57 53
51 53 51 53
a. {~ 256 256 256 256 #: 825504051 909261109 859124533
1413
6295
3535
Plus, you make a remark about having Perl store the string of literal
digits as lines in a file. If that is in fact what you read from your
"data" file, then there are a LOT of line feeds (one for each digit)
included that further scramble any results you may be getting...
If your goal is to create a file of the results for some purpose beyond
studying the patterns in the digits of pi, then you should be more
specific about what the format of the stored data should be for further use.
On 2012/05/08 03:56 , Joe Bohart wrote:
> Hi J'ers,
>
> I'm trying to perform calculations on massive data sets using mapped files
> and after searching the forum/essays am stuck.
>
> I've load 50 million integer digits of pi and trying to do a moving average
> on them. I 've created two mapped files one for the data and one for the
> calculated results, but I get out of memory errors. I'm not seeing what I'm
> doing wrong here.
>
> My goal is to work with floating point numbers eventually, but i'd figure
> i'd start with integers first, since there are more examples for integers
> than floats in the 'mapped files' Labs (if anyone has the code for the
> example mention in the 67 of 68 slide of the Lab on mapped files using NYSE
> stock prices - I'd love to see it).
>
> load 'jmf files dir'
> NB. data from http://zenwerx.com/projects/pi-digits/pi/
> NB. used perl to write each digits on 1 line of file data
> pidata =: fread '/home/joe/pi-study/data'
> NB. type is 2 (literal)
> 3!:0 pidata
> NB. convert to 4-byte integers
> piInt =: _2 (3!:4) pidata
> NB. shape is 50,000,000
> $piInt
> ]sz=:7!:5<'piInt'
> NB. define moving average
> ma =: mean \
> NB. out of memory ma, expected since storing results in ram
> smoothPi =: 100 ma piInt
>
> NB. create jmf for results
> fn =: jpath '~temp/pi-results.jmf'
> ]sz=: 7!:5<'piInt'
> createjmf_jmf_ fn;sz
> JFL map_jmf_ 'piResults'; fn
> piResults =. 100 ma piInt
> NB. still get out of memory!
>
> Much thanks !
> Joe
>
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
