Hatem Nassrat (el 2008-02-08 a les 00:12:48 -0400) va dir:: > As requested I am replying with some feedback regarding some timing > tests of the different variations of sollutions to the problem mentioned > earlier. > [...] > Experiment 1: VLArray of Object Atoms > > Writing: 28.397 s > Reading: 2.744 s > [...] > Experiment 2: 2 EArrays > > One for Single Chars = Pickled data > One for data offsets > > I am not sure if there is a better way of doing this. Maybe someone can > inspire me. > > Writing: 32.343s > Reading: 3.441s
I don't quite get this arrangement. Do you mean one ``EArray`` of
characters with the pickled representation of each list one after the
other, and another ``EArray`` storing the indices where each pickled
list starts in the first one? Now this is an interesting approach, and
you may be approaching how ``VLArray`` objects work internally...
> [...]
> Experiment 3:
>
> This is essentially the same as Exp 2 but more intuitive.
>
> Writing: 27.394s
> Reading: 20.755s
>
> Type 3 was consistently faster in writing which is intuitive as it does
> not invlove pickling. Where as recreating the structure through the
> access of two arrays slows everything down.
Well, this may either be a problem of inherent slowness in ``VLArray``,
or a problem with the particular way you rebuild your lists. It'd be
interesting to measure both times separately. Also, this is the only
language-independent arrangement, which may iterest you portability-
wise.
> Type 2 is as consistent as Type 1 with variations between reading and
> writing however is a very convoluted method of doing things.
>
> Type 1 is the most intuitive to program and use and happens to be the
> fastest to read which will be done more often that writing. And is only
> slightly slower than type 3.
>
> I would be looking ofrward to the inclusion of variable arrays or
> variable length strings or pickled objects or variable length within the
> standard table as that would be the most suitable for my application.
> Until then I believe that I will be going with pickling into a variable
> length array with a link in the table linking to the index of its object
> into the vlarray.
The demand for variable length fields has been a recurring topic in the
list, but the implementation doesn't seem trivial at all and we would
most probably need some external funding.
Thanks Hatem for the very interesting feedback report. I'm sure that
some people will find it illustrative and interesting too.
Best wishes,
::
Ivan Vilata i Balaguer >qo< http://www.carabos.com/
Cárabos Coop. V. V V Enjoy Data
""
signature.asc
Description: Digital signature
------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________ Pytables-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/pytables-users
