Hi,
I'm new to pytables so I hope this isn't too obvious...
I have some existing h5 files that I read in with:
> >> import tables
> >> h5slow = tables.openFile(data.h5, 'r')
The data is then directly in the root directory, for example:
> >> h5slow.root.tag
/tag (Array(393816, 2)) ''
atom := Int32Atom(shape=(), dflt=0)
maindim := 0
flavor := 'numpy'
byteorder := 'big'
chunkshape := None
The problem is when I do a query on this it is very slow. For example, I
know that the last entry in my data has a value of 7848107, so searching
for this is my cheap test for speed. In Ipython:
%timeit result = [x for x in h5slow.root.tag if x[1]==7848107]
1 loops, best of 3: 5.66 s per loop
using the read() method gives a slight speed up:
%timeit result = [x for x in h5slow.root.tag.read() if x[1]==7848107]
1 loops, best of 3: 2.21 s per loop
However when I wrote a little script to convert my data and save it in a
new h5 file this is all MUCH faster (100x):
%timeit result = [x for x in h5fast.root.detector.readout if x[1]==7848107]
10 loops, best of 3: 53.8 ms per loop
In the faster case I created a new h5 file using the data from my
original h5 file 'Getting started' instructions in the user manual. The
faster case is also a table.Table object and the slow case a array.Array
object.
> >>h5fast.root.detector.readout
/detector/readout (Table(393816,)) 'Readout example'
description := {
"cpu": Int8Col(shape=(), dflt=0, pos=0),
"pnum": Int32Col(shape=(), dflt=0, pos=1)}
byteorder := 'little'
chunkshape := (1638,)
Question:
Why is openFile reading the original data file as an array.Array object
if querying tables is so much faster?
Thanks in advance for any help!
Raphael
------------------------------------------------------------------------------
Special Offer-- Download ArcSight Logger for FREE (a $49 USD value)!
Finally, a world-class log management solution at an even better price-free!
Download using promo code Free_Logger_4_Dev2Dev. Offer expires
February 28th, so secure your free ArcSight Logger TODAY!
http://p.sf.net/sfu/arcsight-sfd2d
_______________________________________________
Pytables-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/pytables-users
