On Sat, Aug 21, 2010 at 10:00 AM, <numpy-discussion-requ...@scipy.org>wrote:
> Date: Fri, 20 Aug 2010 14:30:58 -0500 > From: Robert Kern <robert.k...@gmail.com> > Subject: Re: [Numpy-discussion] Making MATLAB and Python "play nice" > To: Discussion of Numerical Python <numpy-discussion@scipy.org> > Message-ID: > > <aanlktikrwzd0vtjisk+6xh2djbca1v1sxx_ln6g4g...@mail.gmail.com<aanlktikrwzd0vtjisk%2b6xh2djbca1v1sxx_ln6g4g...@mail.gmail.com> > > > Content-Type: text/plain; charset=UTF-8 > > On Fri, Aug 20, 2010 at 14:25, David Goldsmith <d.l.goldsm...@gmail.com> > wrote: > > Hi! ?Please forgive the re-post: I forgot to change the subject line > > and I haven't seen a response to this yet, so I'm assuming the former > > might be the cause of the latter. > > Or perhaps because the licenses are plainly visible at the links? > Ah, I see: if I can't be bothered to click on the links, no one else can be bothered to tell me that that's all I need to do to get my question answered. Unfortunately, the solutions are useless to me if they're not freely redistributable, so I have no incentive to click on the links--which do not advertise that they answer the licensing question--'til that question is answered - catch-22. DG > > -- > Robert Kern > > "I have come to believe that the whole world is an enigma, a harmless > enigma that is made terrible by our own mad attempt to interpret it as > though it had an underlying truth." > ? -- Umberto Eco > > > ------------------------------ > > Message: 4 > Date: Sat, 21 Aug 2010 00:31:03 +0100 > From: Francesc Alted <fal...@pytables.org> > Subject: Re: [Numpy-discussion] [ANN] carray: an in-memory compressed > data container > To: Discussion of Numerical Python <numpy-discussion@scipy.org> > Message-ID: > > <aanlktimuaahqvgg4xwktwtv+cw+s_ypavxtpxnxfa...@mail.gmail.com<aanlktimuaahqvgg4xwktwtv%2bcw%2bs_ypavxtpxnxfa...@mail.gmail.com> > > > Content-Type: text/plain; charset=ISO-8859-1 > > 2010/8/20, Zbyszek Szmek <zbys...@in.waw.pl>: > > OK, I've got a case where carray really shines :| > > > > zbys...@escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python > > bench/concat.py numpy 800000 1000 4 1 > > problem size: (800000) x 1000 = 10^8.90309 > > time for concat: 4.806s > > size of the final container: 6103.516 MB > > zbys...@escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python > > bench/concat.py concat 800000 1000 4 1 > > problem size: (800000) x 1000 = 10^8.90309 > > time for concat: 3.475s > > size of the final container: 6103.516 MB > > zbys...@escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python > > bench/concat.py carray 800000 1000 4 1 > > problem size: (800000) x 1000 = 10^8.90309 > > time for concat: 1.434s > > size of the final container: 373.480 MB > > > > Size is set to NOT hit the swap. This is still the easily compressible > > arange... but still, the results are very nice. > > Wow, the results with your processor are much nicer than with my Atom > indeed. But yeah, I somewhat expected this because Blosc works much > faster with recent processors, as can be seen in: > > http://blosc.pytables.org/trac/wiki/SyntheticBenchmarks > > BTW, the difference between memcpy and memmove times for this > benchmark is almost 40% for your computer, which is really large :-/ > Hmm, something must go really wrong with memcpy in some glibc > distributions... > > At any rate, for real data that is less compressible the advantages of > carray will be less apparent, but at least the proof of concept seems > to work as intended, so I'm very happy with it. I'm also expecting > that the combination carray/numexpr would perform faster than plain > computations programmed in C, most specially with modern processors, > but will see how much faster exactly. > > > Of course when the swap is hit, the ratio between carray and a normal > array > > can grow to infinity :) > > > > zbys...@escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python > > bench/concat.py numpy 1000000 1000 3 1 > > problem size: (1000000) x 1000 = 10^9 > > time for concat: 35.700s > > size of the final container: 7629.395 MB > > zbys...@escher:~/python/numpy/carray-0.1.dev$ PYTHONPATH=. python > > bench/concat.py carray 1000000 1000 3 1 > > problem size: (1000000) x 1000 = 10^9 > > time for concat: 1.751s > > size of the final container: 409.633 MB > > Exactly. This is another scenario where the carray concept can be > really useful. > > -- > Francesc Alted > > > ------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > End of NumPy-Discussion Digest, Vol 47, Issue 61 > ************************************************ > -- Privacy is overrated; Identity isn't.
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
