On 3/28/12 10:15 AM, Alvaro Tejero Cantero wrote:
> That is a perfectly fine solution for me, as long as the arrays aren't
> copied in memory for the query.
No, the arrays are not copied in memory. They are just read from disk
block-by-block and then the output is directed to the iterator, or an
array (depending on the context).
>
> Thank you!
>
> Thinking that your proposed solution uses iterables to avoid it I tried
>
> boolcond = pt.Expr('(exp(a)<0.9)&(a*b>0.7)|(b*sin(a)<0.1)')
> indices = [i for i,v in boolcond if v]
> (...) TypeError: 'numpy.bool_' object is not iterable
>
> I can, however, do
> boolarr = boolcond.eval()
> indices = np.nonzero(boolarr)
>
> but then I get boolarr into memory.
>
> Did I miss something?
Yes, that was an error on my part. The correct way is:
indices = [i for i,v in boolcond if enumerate(v)]
> What is your advice on how to monitor the use of
> memory? (I need this until PyTables is second skin).
top?
>
> It is very rewarding to see that these numexpr's are 3-4 times faster
> than the same with arrays in memory. However, I didn't find a way to
> set the number of threads used
Well, you can use the `MAX_THREADS` variable in 'parameters.py', but
this do not offer separate controls for numexpr and blosc. Feel free to
open a ticket asking for imporving this functionality.
>
> When evaluating the blosc benchmarks I found that in my system with
> two 6-core processors , using 12 is best for writing and 6 for
> reading. Interesting...
Yes, it is :)
>
> Another question (maybe for a separate thread): is there any way to
> shrink memory usage of booleans to 1 bit? It might well be that this
> optimizes the use of the memory bus (at some processing cost). But I
> am not aware of a numpy container for this.
Maybe a compressed array? That would lead to using less that 1 bit per
element in many situations. If you are interested in this, look into:
https://github.com/FrancescAlted/carray
--
Francesc Alted
------------------------------------------------------------------------------
This SF email is sponsosred by:
Try Windows Azure free for 90 days Click Here
http://p.sf.net/sfu/sfd2d-msazure
_______________________________________________
Pytables-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/pytables-users
