[Pytables-users] pytables: could not find the HDF5 runtime
Hi, I'm trying to install pytables and its proving difficult (using MAC OS 10.6.4). I have installed in "/usr/local/hdf5" and set the environment variable $HDF5_DIR to /usr/local/hdf5. When I run setup, I get a warning about not being able to find the HDF5 runtime. ndmmac149:tables-2.4.0 jflegg$ sudo python setup.py install --hdf5="/usr/local/hdf5" * Found numpy 1.6.1 package installed. * Found numexpr 2.0.1 package installed. * Found Cython 0.17.2 package installed. * Found HDF5 headers at ``/usr/local/hdf5/include``, library at ``/usr/local/hdf5/lib``. .. WARNING:: Could not find the HDF5 runtime. The HDF5 shared library was *not* found in the default library paths. In case of runtime problems, please remember to install it. ld: library not found for -llzo2 collect2: ld returned 1 exit status ld: library not found for -llzo2 collect2: ld returned 1 exit status * Could not find LZO 2 headers and library; disabling support for it. ld: library not found for -llzo collect2: ld returned 1 exit status ld: library not found for -llzo collect2: ld returned 1 exit status * Could not find LZO 1 headers and library; disabling support for it. * Found bzip2 headers at ``/usr/include``, library at ``/usr/lib``. running install running build running build_py creating build creating build/lib.macosx-10.5-i386-2.7 creating build/lib.macosx-10.5-i386-2.7/tables copying tables/__init__.py -> build/lib.macosx-10.5-i386-2.7/tables copying tables/array.py -> build/lib.macosx-10.5-i386-2.7/tables When I import pytables in python, I get the following error message In [1]: import tables - ImportError Traceback (most recent call last) /Users/jflegg/ in () > 1 import tables /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site- packages/tables/__init__.py in () 28 29 # Necessary imports to get versions stored on the Pyrex extension ---> 30 from tables.utilsExtension import getPyTablesVersion, getHDF5Version 31 32 ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/7.3 /lib/python2.7/site-packages/tables/utilsExtension.so, 2): Symbol not found: _H5E_CALLBACK_g Referenced from: /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site- packages/tables/utilsExtension.so Expected in: flat namespace in /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site- packages/tables/utilsExtension.so Any help would be greatly appreciated. Jennifer -- LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial Remotely access PCs and mobile devices and provide instant support Improve your efficiency, and focus on delivering more value-add services Discover what IT Professionals Know. Rescue delivers http://p.sf.net/sfu/logmein_12329d2d ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] pytables: could not find the HDF5 runtime
Try leaving the pytables source dir and then running then running IPython. On Mon, Dec 10, 2012 at 9:20 AM, Jennifer Flegg wrote: > Hi, > I'm trying to install pytables and its proving difficult (using MAC OS > 10.6.4). > I have installed in "/usr/local/hdf5" and set the environment variable > $HDF5_DIR to /usr/local/hdf5. When I run setup, I get a warning about > not being able to find the HDF5 runtime. > > ndmmac149:tables-2.4.0 jflegg$ sudo python setup.py install > --hdf5="/usr/local/hdf5" > * Found numpy 1.6.1 package installed. > * Found numexpr 2.0.1 package installed. > * Found Cython 0.17.2 package installed. > * Found HDF5 headers at ``/usr/local/hdf5/include``, > library at ``/usr/local/hdf5/lib``. > .. WARNING:: Could not find the HDF5 runtime. >The HDF5 shared library was *not* found in the default library >paths. In case of runtime problems, please remember to install it. > ld: library not found for -llzo2 > collect2: ld returned 1 exit status > ld: library not found for -llzo2 > collect2: ld returned 1 exit status > * Could not find LZO 2 headers and library; disabling support for it. > ld: library not found for -llzo > collect2: ld returned 1 exit status > ld: library not found for -llzo > collect2: ld returned 1 exit status > * Could not find LZO 1 headers and library; disabling support for it. > * Found bzip2 headers at ``/usr/include``, library at ``/usr/lib``. > running install > running build > running build_py > creating build > creating build/lib.macosx-10.5-i386-2.7 > creating build/lib.macosx-10.5-i386-2.7/tables > copying tables/__init__.py -> build/lib.macosx-10.5-i386-2.7/tables > copying tables/array.py -> build/lib.macosx-10.5-i386-2.7/tables > > When I import pytables in python, I get the following error message > > In [1]: import tables > - > ImportError Traceback (most recent call last) > /Users/jflegg/ in () > > 1 import tables > > /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site- > packages/tables/__init__.py in () > 28 > 29 # Necessary imports to get versions stored on the Pyrex extension > > ---> 30 from tables.utilsExtension import getPyTablesVersion, > getHDF5Version > 31 > 32 > > ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/7.3 > /lib/python2.7/site-packages/tables/utilsExtension.so, 2): > Symbol not found: _H5E_CALLBACK_g Referenced from: > /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site- > packages/tables/utilsExtension.so > Expected in: flat namespace > in /Library/Frameworks/Python.framework/Versions/7.3/lib/python2.7/site- > packages/tables/utilsExtension.so > > > Any help would be greatly appreciated. > Jennifer > > > > > > -- > LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial > Remotely access PCs and mobile devices and provide instant support > Improve your efficiency, and focus on delivering more value-add services > Discover what IT Professionals Know. Rescue delivers > http://p.sf.net/sfu/logmein_12329d2d > ___ > Pytables-users mailing list > Pytables-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/pytables-users > -- LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial Remotely access PCs and mobile devices and provide instant support Improve your efficiency, and focus on delivering more value-add services Discover what IT Professionals Know. Rescue delivers http://p.sf.net/sfu/logmein_12329d2d___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Problems with flush(): RuntimeError: dictionary changed size during iteration
I'm continuing to fight this error. As a sanity check I rewrote my sample
app as a single thread only. With interleaved read/writes to multiple
tables I still get "RuntimeError: dictionary changed size during iteration"
in flush. I still think there is some underlying problem or something I
don't understand about pytables/hdf5. I'm far from an expert on either of
these so I appreciate any suggestions or even confirmation that I'm not
completely crazy? The following code should work, right?
import tables
import random
import datetime
# a simple table
class TableValue(tables.IsDescription):
a = tables.Int64Col(pos=1)
b = tables.UInt32Col(pos=2)
class Test():
def __init__(self):
self.stats = {'read': 0,
'write': 0,
'read_error': 0,
'write_error': 0}
self.h5 = None
self.h5 = tables.openFile('/data/test.h5', mode='w')
self.num_groups = 5
self.num_tables = 5
# create num_groups
self.groups = [self.h5.createGroup('/', "group%d"%i) for i in
range(self.num_groups)]
self.tables = []
# create num_tables in each group we just created
for group in self.groups:
tbls = [self.h5.createTable(group, 'table%d'%i, TableValue) for
i in range(self.num_tables)]
self.tables.append (tbls)
for table in tbls:
# add an index for good measure
table.cols.a.createIndex()
def write(self):
# select a random table and write to it
x = self.tables[random.randint(0,
self.num_groups-1)][random.randint(0, self.num_tables-1)].row
x['a'] = random.randint(0, 100)
x['b'] = random.randint(0, 100)
x.append()
self.stats['write'] += 1
def read(self):
# first flush any cached data
self.h5.flush()
# then select a random table
table = self.tables[random.randint(0,
self.num_groups-1)][random.randint(0, self.num_tables-1)]
# and do some random query
table.readWhere('a > %d'%(random.randint(0, 100)))
self.stats['read'] += 1
def close(self):
self.h5.close()
def main():
t = Test()
start = datetime.datetime.now()
# run for 10 seconds
while (datetime.datetime.now() - start <
datetime.timedelta(seconds=10)):
# randomly do a read or a write
if random.random() > 0.5:
t.write()
else:
t.read()
print t.stats
print "Done"
t.close()
if __name__ == "__main__":
main()
On Thu, Dec 6, 2012 at 9:55 AM, Alan Marchiori wrote:
> Josh,
>
> Thanks for the detailed response. I would like to avoid going through a
> separate process if at all possible due to the performance penalty. I have
> also tried your last suggestion to create a dedicated pytables thread and
> send everything through that but still see the same problem (Runtime error
> in flush). This leads me to believe something strange is going on behind
> the scenes. ??
>
> Updated test program with dedicated pytables thread reading an input
> Queue.Queue:
>
> import tables
> import threading
> import random
> import time
> import Queue
>
> # a simple table
> class TableValue(tables.IsDescription):
> a = tables.Int64Col(pos=1)
> b = tables.UInt32Col(pos=2)
>
> class TablesThread(threading.Thread):
> def __init__(self):
> threading.Thread.__init__(self)
> self.name = 'HDF5 io thread'
> # create the dummy HDF5 file
> self.h5 = None
> self.h5 = tables.openFile('/data/test.h5', mode='w')
> self.num_groups = 5
> self.num_tables = 5
> self.groups = [self.h5.createGroup('/', "group%d"%i) for i in
> range(self.num_groups)]
> self.tables = []
> for group in self.groups:
> tbls = [self.h5.createTable(group, 'table%d'%i, TableValue)
> for i in range(self.num_tables)]
> self.tables.append (tbls)
> for table in tbls:
> # add an index for good measure
> table.cols.a.createIndex()
> self.stopEvt = threading.Event()
> self.stoppedEvt = threading.Event()
> self.inputQ = Queue.Queue()
>
> def run(self):
> try:
> while not self.stopEvt.is_set():
> # get a command
> try:
> cmd, args, result = self.inputQ.get(timeout = 0.5)
> except Queue.Empty:
> # poll stopEvt so we can shutdown
> continue
>
> # do the command
> if cmd == 'write':
> x = self.tables[args[0]][args[1]].row
> x['a'] = args[2]
> x['b'] = args[3]
> x.append()
> elif cmd == 'read':
> self.h5.flush()
> table = self.tables[args[0]][args[1]]
> r
Re: [Pytables-users] Problems with flush(): RuntimeError: dictionary changed size during iteration
I think I have found a viable work around.
Previously, I was flushing the whole HDF5 file:
self.h5.flush()
By replacing this with a flush on just the table of interest:
table = self.tables[random.randint(0, self.num_groups-1)][random.randint(0,
self.num_tables-1)]
table.flush()
The RuntimeError seems to be gone in all versions of my test program (both
single threaded and threaded with locks). Hope this helps someone else and
eventually maybe someone will figure out what is wrong with File.flush().
On Mon, Dec 10, 2012 at 10:52 AM, Alan Marchiori wrote:
> I'm continuing to fight this error. As a sanity check I rewrote my sample
> app as a single thread only. With interleaved read/writes to multiple
> tables I still get "RuntimeError: dictionary changed size during iteration"
> in flush. I still think there is some underlying problem or something I
> don't understand about pytables/hdf5. I'm far from an expert on either of
> these so I appreciate any suggestions or even confirmation that I'm not
> completely crazy? The following code should work, right?
>
> import tables
> import random
> import datetime
>
> # a simple table
> class TableValue(tables.IsDescription):
> a = tables.Int64Col(pos=1)
> b = tables.UInt32Col(pos=2)
>
> class Test():
> def __init__(self):
> self.stats = {'read': 0,
> 'write': 0,
> 'read_error': 0,
> 'write_error': 0}
> self.h5 = None
> self.h5 = tables.openFile('/data/test.h5', mode='w')
> self.num_groups = 5
> self.num_tables = 5
> # create num_groups
> self.groups = [self.h5.createGroup('/', "group%d"%i) for i in
> range(self.num_groups)]
> self.tables = []
> # create num_tables in each group we just created
> for group in self.groups:
> tbls = [self.h5.createTable(group, 'table%d'%i, TableValue)
> for i in range(self.num_tables)]
> self.tables.append (tbls)
> for table in tbls:
> # add an index for good measure
> table.cols.a.createIndex()
>
> def write(self):
> # select a random table and write to it
> x = self.tables[random.randint(0,
> self.num_groups-1)][random.randint(0, self.num_tables-1)].row
> x['a'] = random.randint(0, 100)
> x['b'] = random.randint(0, 100)
> x.append()
> self.stats['write'] += 1
>
> def read(self):
> # first flush any cached data
> self.h5.flush()
> # then select a random table
> table = self.tables[random.randint(0,
> self.num_groups-1)][random.randint(0, self.num_tables-1)]
> # and do some random query
> table.readWhere('a > %d'%(random.randint(0, 100)))
> self.stats['read'] += 1
>
> def close(self):
> self.h5.close()
>
> def main():
> t = Test()
>
> start = datetime.datetime.now()
>
> # run for 10 seconds
> while (datetime.datetime.now() - start <
> datetime.timedelta(seconds=10)):
> # randomly do a read or a write
> if random.random() > 0.5:
> t.write()
> else:
> t.read()
>
> print t.stats
> print "Done"
> t.close()
>
> if __name__ == "__main__":
> main()
>
>
> On Thu, Dec 6, 2012 at 9:55 AM, Alan Marchiori wrote:
>
>> Josh,
>>
>> Thanks for the detailed response. I would like to avoid going through a
>> separate process if at all possible due to the performance penalty. I have
>> also tried your last suggestion to create a dedicated pytables thread and
>> send everything through that but still see the same problem (Runtime error
>> in flush). This leads me to believe something strange is going on behind
>> the scenes. ??
>>
>> Updated test program with dedicated pytables thread reading an input
>> Queue.Queue:
>>
>> import tables
>> import threading
>> import random
>> import time
>> import Queue
>>
>> # a simple table
>> class TableValue(tables.IsDescription):
>> a = tables.Int64Col(pos=1)
>> b = tables.UInt32Col(pos=2)
>>
>> class TablesThread(threading.Thread):
>> def __init__(self):
>> threading.Thread.__init__(self)
>> self.name = 'HDF5 io thread'
>> # create the dummy HDF5 file
>> self.h5 = None
>> self.h5 = tables.openFile('/data/test.h5', mode='w')
>> self.num_groups = 5
>> self.num_tables = 5
>> self.groups = [self.h5.createGroup('/', "group%d"%i) for i in
>> range(self.num_groups)]
>> self.tables = []
>> for group in self.groups:
>> tbls = [self.h5.createTable(group, 'table%d'%i, TableValue)
>> for i in range(self.num_tables)]
>> self.tables.append (tbls)
>> for table in tbls:
>> # add an index for good measure
>> table.cols.a.createIndex()
>> self.stopEvt = threading.Event()
>> self.stoppedEvt = threading.Event()
>> self.
Re: [Pytables-users] pytables: could not find the HDF5 runtime
HI Anthony, I'm not in the pytables source dir when I'm running IPython, so I don't think this is the problem. Thanks, Jennifer -- LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial Remotely access PCs and mobile devices and provide instant support Improve your efficiency, and focus on delivering more value-add services Discover what IT Professionals Know. Rescue delivers http://p.sf.net/sfu/logmein_12329d2d ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] pytables: could not find the HDF5 runtime
Hi Jennifer, Oh, right, I am sorry. Your end error message looks very similar to another, more common issue. How did you install HDF5? On Mac I typically use MacPorts or have to install it from source. IIRC the macports build fails to make the shared libraries and you typically have to configure & compile manually? Is there a reason you are building PyTables yourself? On Mac, I typically use EPD or Anaconda Even when I am making edits to the PyTables (or other projects source), I use these distributions as a base and link against the HDF5 provided in them. Be Well Anthony On Mon, Dec 10, 2012 at 1:23 PM, Jennifer Flegg wrote: > HI Anthony, > I'm not in the pytables source dir when I'm running IPython, > so I don't think this is the problem. > Thanks, > Jennifer > > > > -- > LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial > Remotely access PCs and mobile devices and provide instant support > Improve your efficiency, and focus on delivering more value-add services > Discover what IT Professionals Know. Rescue delivers > http://p.sf.net/sfu/logmein_12329d2d > ___ > Pytables-users mailing list > Pytables-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/pytables-users > -- LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial Remotely access PCs and mobile devices and provide instant support Improve your efficiency, and focus on delivering more value-add services Discover what IT Professionals Know. Rescue delivers http://p.sf.net/sfu/logmein_12329d2d___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] pytables: could not find the HDF5 runtime
Hi Anthony, Thanks for your reply. I installed HDF5 also from source. The reason I'm building hdf5 and pytables myself is that they don't seem to be available through EPD any more (at least in the free version: http://www.enthought.com/products/epdlibraries.php) They used to both come bundled in EPD, but not anymore, which is a pain. Many thanks, Jennifer -- LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial Remotely access PCs and mobile devices and provide instant support Improve your efficiency, and focus on delivering more value-add services Discover what IT Professionals Know. Rescue delivers http://p.sf.net/sfu/logmein_12329d2d ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] pytables: could not find the HDF5 runtime
Hi Jennifer, Yeah, that is right, they are not in EPD Free. However, they are in Anaconda CE (http://continuum.io/downloads.html). Note the CE rather than the full version. Be Well Anthony On Mon, Dec 10, 2012 at 4:07 PM, Jennifer Flegg wrote: > Hi Anthony, > Thanks for your reply. I installed HDF5 also from source. The > reason I'm building hdf5 and pytables myself is that they don't > seem to be available through EPD any more (at least in the free > version: http://www.enthought.com/products/epdlibraries.php) > They used to both come bundled in EPD, but not anymore, which is > a pain. > Many thanks, > Jennifer > > > > > -- > LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial > Remotely access PCs and mobile devices and provide instant support > Improve your efficiency, and focus on delivering more value-add services > Discover what IT Professionals Know. Rescue delivers > http://p.sf.net/sfu/logmein_12329d2d > ___ > Pytables-users mailing list > Pytables-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/pytables-users > -- LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial Remotely access PCs and mobile devices and provide instant support Improve your efficiency, and focus on delivering more value-add services Discover what IT Professionals Know. Rescue delivers http://p.sf.net/sfu/logmein_12329d2d___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
