[Pytables-users] adding column index hides data(?)
Hello,
I am attempting to use PyTables (v2.3.1) to store timestamped data and
things were going well until I added a column index. While the column
is indexed no data is returned from a table.where call!
This behavior is demonstrated with the following test code:
---begin test.py---
import tables
import random
class Descr(tables.IsDescription):
when = tables.Time64Col(pos = 1)
value = tables.Float32Col(pos = 2)
h5f = tables.openFile('/tmp/tmp.h5', 'w')
tbl = h5f.createTable('/', 'test', Descr)
tbl.cols.when.createIndex(_verbose = True)
t = 1321031471.0 # 11/11/11 11:11:11
tbl.append([(t + i, random.random()) for i in range(1000)])
tbl.flush()
def query(s):
print 'is_index =', tbl.cols.when.is_indexed
print [(row['when'], row['value']) for row in tbl.where(s)]
print tbl.readWhere(wherestr)
wherestr = '(when >= %d) & (when < %d)'%(t, t+5)
query(wherestr)
tbl.cols.when.removeIndex()
query(wherestr)
h5f.close()
---end test.py---
This creates the table for storing time/value pairs, inserts some
synthetic data, and then checks to see if there is data in the table.
When the table is created there is an index added to the 'where'
column. The first query returns no data (which is incorrect). Then
the column index is removed (via table.removeIndex) and the query is
repeated. This time 5 results are returned as expected. The data is
clearly there however the index is somehow breaking the where logic.
Here is the output I get:
---begin output---
is_index = True
[]
[]
is_index = False
[(1321031471.0, 0.6449417471885681), (1321031472.0,
0.7889317274093628), (1321031473.0, 0.609708845615387), (1321031474.0,
0.9120397567749023), (1321031475.0, 0.2386845201253891)]
[(1321031471.0, 0.6449417471885681) (1321031472.0, 0.7889317274093628)
(1321031473.0, 0.609708845615387) (1321031474.0, 0.9120397567749023)
(1321031475.0, 0.2386845201253891)]
---end output---
Creating the index after the data has been inserted produces the same
behavior (no data is returned while the index exists). Any
suggestions would be greatly appreciated.
Alan
--
All the data continuously generated in your IT infrastructure
contains a definitive record of customers, application performance,
security threats, fraudulent activity, and more. Splunk takes this
data and makes sense of it. IT sense. And common sense.
http://p.sf.net/sfu/splunk-novd2d
___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] adding column index hides data(?)
On Fri, Nov 18, 2011 at 9:21 AM, Josh Moore wrote: > > On Nov 17, 2011, at 10:35 PM, Alan Marchiori wrote: > >> I am attempting to use PyTables (v2.3.1) to store timestamped data and >> things were going well until I added a column index. While the column >> is indexed no data is returned from a table.where call! >> > > I've reproduced with a number of different index configurations. If I change > the column type to Float64, then the index works as expected. > > BEFORE: > Initial index: verbose has_index= True use_index= > frozenset(['Awhen']) where= 0 readWhere= 0 > remove index has_index= False use_index= > frozenset([]) where= 5 readWhere= 5 > re-add index (non-verbose) has_index= True use_index= > frozenset(['Awhen']) where= 0 readWhere= 0 > remove again has_index= False use_index= > frozenset([]) where= 5 readWhere= 5 > re-add index (with flush) has_index= True use_index= > frozenset(['Awhen']) where= 0 readWhere= 0 > re-add index (full) has_index= True use_index= > frozenset(['Awhen']) where= 0 readWhere= 0 > re-add index (ultralight) has_index= True use_index= > frozenset(['Awhen']) where= 0 readWhere= 0 > re-add index (o=0) has_index= True use_index= > frozenset(['Awhen']) where= 0 readWhere= 0 > re-add index (o=9) has_index= True use_index= > frozenset(['Awhen']) where= 0 readWhere= 0 > re-index has_index= True use_index= > frozenset(['Awhen']) where= 0 readWhere= 0 > also index value has_index= True use_index= > frozenset(['Awhen']) where= 0 readWhere= 0 > > > AFTER: > Initial index: verbose has_index= True use_index= > frozenset(['Awhen']) where= 5 readWhere= 5 > remove index has_index= False use_index= > frozenset([]) where= 5 readWhere= 5 > re-add index (non-verbose) has_index= True use_index= > frozenset(['Awhen']) where= 5 readWhere= 5 > remove again has_index= False use_index= > frozenset([]) where= 5 readWhere= 5 > re-add index (with flush) has_index= True use_index= > frozenset(['Awhen']) where= 5 readWhere= 5 > re-add index (full) has_index= True use_index= > frozenset(['Awhen']) where= 5 readWhere= 5 > re-add index (ultralight) has_index= True use_index= > frozenset(['Awhen']) where= 5 readWhere= 5 > re-add index (o=0) has_index= True use_index= > frozenset(['Awhen']) where= 5 readWhere= 5 > re-add index (o=9) has_index= True use_index= > frozenset(['Awhen']) where= 5 readWhere= 5 > re-index has_index= True use_index= > frozenset(['Awhen']) where= 5 readWhere= 5 > also index value has_index= True use_index= > frozenset(['Awhen']) where= 5 readWhere= 5 > Josh, Thanks for confirming this. It would seem indexing is broken on Time64's (also the pytables unit tests do not catch this and unit tests should probably be added for all indexable Col datatypes and/or an error raised if you try to index on a non-indexable column). Switching to Float64 works (floating point time since epoch) I just have to ensure that gives sufficient precision for my data. Thanks, problem solved and hopefully can be better handled in the next release. Alan -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-novd2d ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
[Pytables-users] Problems with flush(): RuntimeError: dictionary changed size during iteration
I am trying to allow multiple threads read/write access to pytables data
and found it is necessary to call flush() before any read. If not, the
latest data is not returned. However, this can cause a RuntimeError. I
have tried protecting pytables access with both locks and queues as done by
joshayers (
https://github.com/PyTables/PyTables/blob/develop/examples/multiprocess_access_queues.py).
In either case I still get RuntimeError: dictionary changed size during
iteration when doing the flush. (incidentally using the Locks appears to
be much faster than using queues in my unscientific tests...)
I have tried versions 2.4 and 2.3.1 with the same results. Interestingly
this only appears to happen if there are multiple tables/groups in the H5
file. To investigate this behavior further I create a test program to
illustrate (below). When run with num_groups = 5 num_tables = 5 (or
greater) I see the runtime error every time. When these values are smaller
than this it doesn't (at least in a short test period).
I might be doing something unexpected with pytables, but this seems pretty
straight forward to me. Any help is appreciated.
import tables
import threading
import random
import time
lock = threading.Lock()
def synchronized(lock):
def wrap(f):
def newFunction(*args, **kw):
lock.acquire()
try:
return f(*args, **kw)
finally:
lock.release()
return newFunction
return wrap
class TableValue(tables.IsDescription):
a = tables.Int64Col(pos=1)
b = tables.UInt32Col(pos=2)
class Test():
def __init__(self):
self.h5 = None
self.h5 = tables.openFile('/data/test.h5', mode='w')
self.num_groups = 5
self.num_tables = 5
self.groups = [self.h5.createGroup('/', "group%d"%i) for i in
range(self.num_groups)]
self.tables = []
for group in self.groups:
tbls = [self.h5.createTable(group, 'table%d'%i, TableValue) for
i in range(self.num_tables)]
self.tables.append (tbls)
for table in tbls:
table.cols.a.createIndex()
self.stats = {'read': 0,
'write': 0}
@synchronized(lock)
def __del__(self):
if self.h5 != None:
self.h5.close()
self.h5 = None
@synchronized(lock)
def write(self):
x = self.tables[random.randint(0,
self.num_groups-1)][random.randint(0, self.num_tables-1)].row
x['a'] = random.randint(0, 100)
x['b'] = random.randint(0, 100)
x.append()
self.stats['write'] += 1
@synchronized(lock)
def read(self):
# flush so we can query latest data
self.h5.flush()
table = self.tables[random.randint(0,
self.num_groups-1)][random.randint(0, self.num_tables-1)]
# do some query
results = table.readWhere('a > %d'%(random.randint(0, 100)))
#print 'Query got %d hits'%(len(results))
self.stats['read'] += 1
class Worker(threading.Thread):
def __init__(self, method):
threading.Thread.__init__(self)
self.method = method
self.stopEvt = threading.Event()
def run(self):
while not self.stopEvt.is_set():
self.method()
time.sleep(random.random()/100.0)
def stop(self):
self.stopEvt.set()
def main():
t = Test()
threads = [Worker(t.write) for _i in range(10)]
threads.extend([Worker(t.read) for _i in range(10)])
for thread in threads:
thread.start()
time.sleep(5)
for thread in threads:
thread.stop()
for thread in threads:
thread.join()
print t.stats
if __name__ == "__main__":
main()
--
LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
Remotely access PCs and mobile devices and provide instant support
Improve your efficiency, and focus on delivering more value-add services
Discover what IT Professionals Know. Rescue delivers
http://p.sf.net/sfu/logmein_12329d2d___
Pytables-users mailing list
Pytables-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Problems with flush(): RuntimeError: dictionary changed size during iteration
vt.set() def main(): t = Test() threads = [Worker(t.write) for _i in range(10)] threads.extend([Worker(t.read) for _i in range(10)]) for thread in threads: thread.start() time.sleep(5) for thread in threads: thread.stop() for thread in threads: thread.join() t.close() print t.stats if __name__ == "__main__": main() On Wed, Dec 5, 2012 at 10:52 PM, Josh Ayers wrote: > Alan, > > Unfortunately, the underlying HDF5 library isn't thread-safe by default. > It can be built in a thread-safe mode that serializes all API calls, but > still doesn't allow actual parallel access to the disk. See [1] for more > details. Here's [2] another interesting discussion concerning whether > multithreaded access is actually beneficial for an I/O limited library like > HDF5. Ultimately, if one thread can read at the disk's maximum transfer > rate, then multiple threads don't provide any benefit. > > Beyond the limitations of HDF5, PyTables also maintains global state in > various module-level variables. One example is the _open_file cache in the > file.py module. I made an attempt in the past to work around this to allow > read-only access from multiple threads, but didn't make much progress. > > In general, I think your best bet is to serialize all access through a > single process. There is another example in the PyTables/examples > directory that benchmarks different methods of transferring data from > PyTables to another process [3]. It compares Python's > multiprocessing.Queue, sockets, and memory-mapped files. In my testing, > the latter two are 5-10x faster than using a queue. > > Another option would be to use multiple threads, but handle all access to > the HDF5 file in one thread. PyTables will release the GIL when making > HDF5 library calls, so the other threads will be able to run. You could > use a Queue.Queue or some other mechanism to transfer data between > threads. No actual copying would be needed since their memory is shared, > which should make it faster than the multi-process techniques. > > Hope that helps. > > Josh Ayers > > > [1]: http://www.hdfgroup.org/hdf5-quest.html#mthread > > [2]: > https://visitbugs.ornl.gov/projects/8/wiki/Multi-threaded_cores_and_HPC-HDF5 > > [3]: > https://github.com/PyTables/PyTables/blob/develop/examples/multiprocess_access_benchmarks.py > > > On Wed, Dec 5, 2012 at 2:24 PM, Alan Marchiori wrote: > >> I am trying to allow multiple threads read/write access to pytables data >> and found it is necessary to call flush() before any read. If not, the >> latest data is not returned. However, this can cause a RuntimeError. I >> have tried protecting pytables access with both locks and queues as done by >> joshayers ( >> https://github.com/PyTables/PyTables/blob/develop/examples/multiprocess_access_queues.py). >> In either case I still get RuntimeError: dictionary changed size during >> iteration when doing the flush. (incidentally using the Locks appears to >> be much faster than using queues in my unscientific tests...) >> >> I have tried versions 2.4 and 2.3.1 with the same results. Interestingly >> this only appears to happen if there are multiple tables/groups in the H5 >> file. To investigate this behavior further I create a test program to >> illustrate (below). When run with num_groups = 5 num_tables = 5 (or >> greater) I see the runtime error every time. When these values are smaller >> than this it doesn't (at least in a short test period). >> >> I might be doing something unexpected with pytables, but this seems >> pretty straight forward to me. Any help is appreciated. >> >> >> -- LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial Remotely access PCs and mobile devices and provide instant support Improve your efficiency, and focus on delivering more value-add services Discover what IT Professionals Know. Rescue delivers http://p.sf.net/sfu/logmein_12329d2d___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Problems with flush(): RuntimeError: dictionary changed size during iteration
I'm continuing to fight this error. As a sanity check I rewrote my sample
app as a single thread only. With interleaved read/writes to multiple
tables I still get "RuntimeError: dictionary changed size during iteration"
in flush. I still think there is some underlying problem or something I
don't understand about pytables/hdf5. I'm far from an expert on either of
these so I appreciate any suggestions or even confirmation that I'm not
completely crazy? The following code should work, right?
import tables
import random
import datetime
# a simple table
class TableValue(tables.IsDescription):
a = tables.Int64Col(pos=1)
b = tables.UInt32Col(pos=2)
class Test():
def __init__(self):
self.stats = {'read': 0,
'write': 0,
'read_error': 0,
'write_error': 0}
self.h5 = None
self.h5 = tables.openFile('/data/test.h5', mode='w')
self.num_groups = 5
self.num_tables = 5
# create num_groups
self.groups = [self.h5.createGroup('/', "group%d"%i) for i in
range(self.num_groups)]
self.tables = []
# create num_tables in each group we just created
for group in self.groups:
tbls = [self.h5.createTable(group, 'table%d'%i, TableValue) for
i in range(self.num_tables)]
self.tables.append (tbls)
for table in tbls:
# add an index for good measure
table.cols.a.createIndex()
def write(self):
# select a random table and write to it
x = self.tables[random.randint(0,
self.num_groups-1)][random.randint(0, self.num_tables-1)].row
x['a'] = random.randint(0, 100)
x['b'] = random.randint(0, 100)
x.append()
self.stats['write'] += 1
def read(self):
# first flush any cached data
self.h5.flush()
# then select a random table
table = self.tables[random.randint(0,
self.num_groups-1)][random.randint(0, self.num_tables-1)]
# and do some random query
table.readWhere('a > %d'%(random.randint(0, 100)))
self.stats['read'] += 1
def close(self):
self.h5.close()
def main():
t = Test()
start = datetime.datetime.now()
# run for 10 seconds
while (datetime.datetime.now() - start <
datetime.timedelta(seconds=10)):
# randomly do a read or a write
if random.random() > 0.5:
t.write()
else:
t.read()
print t.stats
print "Done"
t.close()
if __name__ == "__main__":
main()
On Thu, Dec 6, 2012 at 9:55 AM, Alan Marchiori wrote:
> Josh,
>
> Thanks for the detailed response. I would like to avoid going through a
> separate process if at all possible due to the performance penalty. I have
> also tried your last suggestion to create a dedicated pytables thread and
> send everything through that but still see the same problem (Runtime error
> in flush). This leads me to believe something strange is going on behind
> the scenes. ??
>
> Updated test program with dedicated pytables thread reading an input
> Queue.Queue:
>
> import tables
> import threading
> import random
> import time
> import Queue
>
> # a simple table
> class TableValue(tables.IsDescription):
> a = tables.Int64Col(pos=1)
> b = tables.UInt32Col(pos=2)
>
> class TablesThread(threading.Thread):
> def __init__(self):
> threading.Thread.__init__(self)
> self.name = 'HDF5 io thread'
> # create the dummy HDF5 file
> self.h5 = None
> self.h5 = tables.openFile('/data/test.h5', mode='w')
> self.num_groups = 5
> self.num_tables = 5
> self.groups = [self.h5.createGroup('/', "group%d"%i) for i in
> range(self.num_groups)]
> self.tables = []
> for group in self.groups:
> tbls = [self.h5.createTable(group, 'table%d'%i, TableValue)
> for i in range(self.num_tables)]
> self.tables.append (tbls)
> for table in tbls:
> # add an index for good measure
> table.cols.a.createIndex()
> self.stopEvt = threading.Event()
> self.stoppedEvt = threading.Event()
> self.inputQ = Queue.Queue()
>
> def run(self):
> try:
> while not self.stopEvt.is_set():
> # get a command
> try:
> cmd, args, result = self.inputQ.get(timeout = 0.5)
> except Queue.Empty:
> # poll stopEvt so we c
Re: [Pytables-users] Problems with flush(): RuntimeError: dictionary changed size during iteration
I think I have found a viable work around.
Previously, I was flushing the whole HDF5 file:
self.h5.flush()
By replacing this with a flush on just the table of interest:
table = self.tables[random.randint(0, self.num_groups-1)][random.randint(0,
self.num_tables-1)]
table.flush()
The RuntimeError seems to be gone in all versions of my test program (both
single threaded and threaded with locks). Hope this helps someone else and
eventually maybe someone will figure out what is wrong with File.flush().
On Mon, Dec 10, 2012 at 10:52 AM, Alan Marchiori wrote:
> I'm continuing to fight this error. As a sanity check I rewrote my sample
> app as a single thread only. With interleaved read/writes to multiple
> tables I still get "RuntimeError: dictionary changed size during iteration"
> in flush. I still think there is some underlying problem or something I
> don't understand about pytables/hdf5. I'm far from an expert on either of
> these so I appreciate any suggestions or even confirmation that I'm not
> completely crazy? The following code should work, right?
>
> import tables
> import random
> import datetime
>
> # a simple table
> class TableValue(tables.IsDescription):
> a = tables.Int64Col(pos=1)
> b = tables.UInt32Col(pos=2)
>
> class Test():
> def __init__(self):
> self.stats = {'read': 0,
> 'write': 0,
> 'read_error': 0,
> 'write_error': 0}
> self.h5 = None
> self.h5 = tables.openFile('/data/test.h5', mode='w')
> self.num_groups = 5
> self.num_tables = 5
> # create num_groups
> self.groups = [self.h5.createGroup('/', "group%d"%i) for i in
> range(self.num_groups)]
> self.tables = []
> # create num_tables in each group we just created
> for group in self.groups:
> tbls = [self.h5.createTable(group, 'table%d'%i, TableValue)
> for i in range(self.num_tables)]
> self.tables.append (tbls)
> for table in tbls:
> # add an index for good measure
> table.cols.a.createIndex()
>
> def write(self):
> # select a random table and write to it
> x = self.tables[random.randint(0,
> self.num_groups-1)][random.randint(0, self.num_tables-1)].row
> x['a'] = random.randint(0, 100)
> x['b'] = random.randint(0, 100)
> x.append()
> self.stats['write'] += 1
>
> def read(self):
> # first flush any cached data
> self.h5.flush()
> # then select a random table
> table = self.tables[random.randint(0,
> self.num_groups-1)][random.randint(0, self.num_tables-1)]
> # and do some random query
> table.readWhere('a > %d'%(random.randint(0, 100)))
> self.stats['read'] += 1
>
> def close(self):
> self.h5.close()
>
> def main():
> t = Test()
>
> start = datetime.datetime.now()
>
> # run for 10 seconds
> while (datetime.datetime.now() - start <
> datetime.timedelta(seconds=10)):
> # randomly do a read or a write
> if random.random() > 0.5:
> t.write()
> else:
> t.read()
>
> print t.stats
> print "Done"
> t.close()
>
> if __name__ == "__main__":
> main()
>
>
> On Thu, Dec 6, 2012 at 9:55 AM, Alan Marchiori wrote:
>
>> Josh,
>>
>> Thanks for the detailed response. I would like to avoid going through a
>> separate process if at all possible due to the performance penalty. I have
>> also tried your last suggestion to create a dedicated pytables thread and
>> send everything through that but still see the same problem (Runtime error
>> in flush). This leads me to believe something strange is going on behind
>> the scenes. ??
>>
>> Updated test program with dedicated pytables thread reading an input
>> Queue.Queue:
>>
>> import tables
>> import threading
>> import random
>> import time
>> import Queue
>>
>> # a simple table
>> class TableValue(tables.IsDescription):
>> a = tables.Int64Col(pos=1)
>> b = tables.UInt32Col(pos=2)
>>
>> class TablesThread(threading.Thread):
>> def __init__(self):
>> threading.Thread.__init__(self)
>> self.name = 'HDF5 io thread'
>> # create the dummy HDF5 file
>> self.h5 = None
>> self
