Thank you for the information. I will run a few more tests over the next
couple of days, one day with no compression, and one day with a chunksize
similar to what will be appended each cycle, hopefully I will get a chance
to report back.
A ptrepack into a file with no compression is half the size
Hi Thadeus,
HDF5 does not guarantee that the data is contiguous on disk between blocks.
hat is, there may be empty space in your file. Furthermore, compression
really messes with HDF5's ability to predict how large blocks will end up
being. To avoid accidental data loss, HDF5 tends to over pred
I have a PyTables file that receives many appends to a Table throughout the
day, the file is opened, a small bit of data is appended, and the file is
closed. The open/append/close can happen many times in a minute. Anywhere
from 1-500 rows are appended at any given time. By the end of the day, this
Hey Tim,
Awesome dataset! And neat image!
As per your request, a couple of minor things I noticed were that you
probably don't need to do the sanity check each time (great for debugging,
but not needed always), you are using masked arrays which while
sometimes convenient are generally slower than
