2009/10/30 Stephen Simmons :
> I should clarify what I meant..
>
> Suppose I have a recarray with 50 fields and want to read just one of
> those fields. PyTables/HDF will read in the compressed data for chunks
> of complete rows, decompress the full 50 fields, and then give me back
> the data f
I should clarify what I meant..
Suppose I have a recarray with 50 fields and want to read just one of
those fields. PyTables/HDF will read in the compressed data for chunks
of complete rows, decompress the full 50 fields, and then give me back
the data for just one field.
I'm after a solut
On Fri, Oct 30, 2009 at 08:18, Stephen Simmons wrote:
> Thoughts about a new format
>
> It seems that numpy could benefit from a new storage format.
While you may indeed need a new format, I'm not sure that numpy does.
Lord knows I've gotten enough flak for inven
A Friday 30 October 2009 14:18:05 Stephen Simmons escrigué:
> - Pytables (HDF using chunked storage for recarrays with LZO
> compression and shuffle filter)
> - can't extract individual field from a recarray
Er... Have you tried the ``cols`` accessor?
http://www.pytables.org/docs/manual/ch04
Unless I read your request or the documentation wrong, h5py already
supports pulling specific fields out of "compound data types":
http://h5py.alfven.org/docs-1.1/guide/hl.html#id3
> For compound data, you can specify multiple field names alongside
> the numeric slices:
> >>> dset["FieldA"]
>
Stephen Simmons wrote:
> P.S. Maybe this will be too much work, and I'd be better off sticking
> with Pytables.
I can't judge that, but I want to share some thoughts (rant?):
- Are you ready to not only write the code, but maintain it over years to
come, and work through nasty bugs, and thin
Dag Sverre Seljebotn:
> Hi,
>
> Is anyone working on alternative storage options for numpy arrays, and
> specifically recarrays? My main application involves processing series
> of large recarrays (say 1000 recarrays, each with 5M rows having 50
> fields). Existing options meet some but not all of
Hi,
Is anyone working on alternative storage options for numpy arrays, and
specifically recarrays? My main application involves processing series
of large recarrays (say 1000 recarrays, each with 5M rows having 50
fields). Existing options meet some but not all of my requirements.
Requirement
