Re: [Pytables-users] User block in HDF5 files
Created at https://github.com/PyTables/PyTables/issues/123. If I get a chance, I'll see if I can work out how to add this functionality but it won't be for a few weeks. Thanks! Brad On Thu, Jan 19, 2012 at 2:10 PM, Anthony Scopatz scop...@gmail.com wrote: On Thu, Jan 19, 2012 at 1:02 PM, Antonio Valentino antonio.valent...@tiscali.it wrote: Hi Brad, Il 19/01/2012 04:13, Brad Buran ha scritto: Hi Antonio: The user block I am referring to is a region at the beginning of the file that is reserved for user metadata. I pasted the description from the HDF5 documentation below (http://www.hdfgroup.org/HDF5/doc1.6/UG/08_TheFile.html). I believe H5Py added this feature (http://code.google.com/p/h5py/source/detail?r=8a3010d07e14); however, I don't see a similar way to do this via PyTables. I suppose I could use h5py to create a blank HDF5 file with the required user block then reopen it in PyTables, however, I'd like to minimize code dependencies and everything I've written is built around PyTables rather than h5py. Sorry, for the misunderstanding, I was not aware of this feature. Please file a feature request on https://github.com/PyTables/PyTables Yes, I agree that this should be an added feature. It seems like it should be a keyword argument to the File() constructor. As Antonio said, please open an issue for this. On the other hand if you want to try your hand implementing it, we'll review any pull request you send our way. Be Well Anthony best regards Thanks! Brad Documentation from HDF5: User-block size herr_t H5Pset_userblock (hid_t plist, hsize_t size) herr_t H5Pget_userblock (hid_t plist, hsize_t *size) The user-block is a fixed-length block of data located at the beginning of the file and which is ignored by the HDF5 library. This block is specifically set aside for any data or information that developers determine to be useful to their application but that will not be used by the HDF5 library. The size of the user-block is defined in bytes and may be set to any power of two, with a minimum size of 512 bytes (i.e. 512, 1024, 2048, etc). This property is set with H5Pset_userblock and queried via H5Pget_userblock. For example, if an application was thought to reqire a 4K user-block, that could be set with the following function call: status = H5Pset_userblock(fcpl_id, 4096) The property list could later be queried with status = H5Pget_userblock(fcpl_id, size) and the value 4096 would be returned in the parameter size. On Wed, Jan 18, 2012 at 4:52 PM, Antonio Valentino antonio.valent...@tiscali.it wrote: Hi Brad, Il 18/01/2012 21:49, Brad Buran ha scritto: Is there a way to set the size of the user block when creating a file in PyTables? This would be useful for using PyTables to generate Matlab 7.3 compatible files. Brad Try to use CArrays and set the chunkshape explicitly http://pytables.github.com/usersguide/libref.html#tables.File.createCArray Does it answers your question? best regards -- Antonio Valentino -- Antonio Valentino -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] User block in HDF5 files
Thanks for bringing this to our attention Brad! On Tue, Jan 24, 2012 at 9:50 AM, Brad Buran bbu...@cns.nyu.edu wrote: Created at https://github.com/PyTables/PyTables/issues/123. If I get a chance, I'll see if I can work out how to add this functionality but it won't be for a few weeks. Thanks! Brad On Thu, Jan 19, 2012 at 2:10 PM, Anthony Scopatz scop...@gmail.comwrote: On Thu, Jan 19, 2012 at 1:02 PM, Antonio Valentino antonio.valent...@tiscali.it wrote: Hi Brad, Il 19/01/2012 04:13, Brad Buran ha scritto: Hi Antonio: The user block I am referring to is a region at the beginning of the file that is reserved for user metadata. I pasted the description from the HDF5 documentation below (http://www.hdfgroup.org/HDF5/doc1.6/UG/08_TheFile.html). I believe H5Py added this feature (http://code.google.com/p/h5py/source/detail?r=8a3010d07e14); however, I don't see a similar way to do this via PyTables. I suppose I could use h5py to create a blank HDF5 file with the required user block then reopen it in PyTables, however, I'd like to minimize code dependencies and everything I've written is built around PyTables rather than h5py. Sorry, for the misunderstanding, I was not aware of this feature. Please file a feature request on https://github.com/PyTables/PyTables Yes, I agree that this should be an added feature. It seems like it should be a keyword argument to the File() constructor. As Antonio said, please open an issue for this. On the other hand if you want to try your hand implementing it, we'll review any pull request you send our way. Be Well Anthony best regards Thanks! Brad Documentation from HDF5: User-block size herr_t H5Pset_userblock (hid_t plist, hsize_t size) herr_t H5Pget_userblock (hid_t plist, hsize_t *size) The user-block is a fixed-length block of data located at the beginning of the file and which is ignored by the HDF5 library. This block is specifically set aside for any data or information that developers determine to be useful to their application but that will not be used by the HDF5 library. The size of the user-block is defined in bytes and may be set to any power of two, with a minimum size of 512 bytes (i.e. 512, 1024, 2048, etc). This property is set with H5Pset_userblock and queried via H5Pget_userblock. For example, if an application was thought to reqire a 4K user-block, that could be set with the following function call: status = H5Pset_userblock(fcpl_id, 4096) The property list could later be queried with status = H5Pget_userblock(fcpl_id, size) and the value 4096 would be returned in the parameter size. On Wed, Jan 18, 2012 at 4:52 PM, Antonio Valentino antonio.valent...@tiscali.it wrote: Hi Brad, Il 18/01/2012 21:49, Brad Buran ha scritto: Is there a way to set the size of the user block when creating a file in PyTables? This would be useful for using PyTables to generate Matlab 7.3 compatible files. Brad Try to use CArrays and set the chunkshape explicitly http://pytables.github.com/usersguide/libref.html#tables.File.createCArray Does it answers your question? best regards -- Antonio Valentino -- Antonio Valentino -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users -- Keep Your Developer Skills Current with LearnDevNow! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-d2d ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net
Re: [Pytables-users] Performance problems in indexed tables.
Hi Ümit, On inspecting the issue a bit more, it seems like the culprit is Index.get_chuckmap() in index.py. I can't tell what exactly in this function is causing the slowdown, but I do see a couple of possible problem spots. More profiling in is probably needed. Running kernprof on the index.py file would be helpful. (Or a demonstration script too.) Also it turns out that we have some profiling reporting built in to pytables. If you include the following in your script, you should see the results (not as detailed as line_profiler): import tables.index tables.index.profile = True Unfortunately, the non_indexed.lprof file that you sent earlier doesn't seem to work... I look forward to hearing from you soon... Be Well Anthony On Mon, Jan 23, 2012 at 6:46 AM, Ümit Seren uemit.se...@gmail.com wrote: Because the profile output is probably not formatted properly in the mail I attached the two line_profiler profile output files. In addition to this I also added the profile for _table__whereIndexed() function. On Mon, Jan 23, 2012 at 12:13 PM, Ümit Seren uemit.se...@gmail.com wrote: Hi Anthony I did some profiling with line_profiler. Here are the results for an table with index: Line # Hits Time Per Hit % Time Line Contents == 1469 def readWhere( self, condition, condvars=None, field=None, 1470 start=None, stop=None, step=None ): 1471 1480 1481 18 8.0 0.0 self._checkFieldIfNumeric(field) 1482 1483 12 2.0 0.0 coords = [ p.nrow for p in 1484 6951894 25942594 3.7 92.4 self._where(condition, condvars, start, stop, step) ] 1485 15 5.0 0.0 self._whereCondition = None # reset the conditions 1486 16 6.0 0.0 if len(coords) 1: 1487 1 13 13.0 0.0 cstart, cstop = coords[0], coords[-1]+1 1488 15 5.0 0.0 if cstop - cstart == len(coords): 1489 # Chances for monotonically increasing row values. Refine. 1490 14 4.0 0.0 inc_seq = numpy.alltrue( 1491 1 1124713 1124713.0 4.0 numpy.arange(cstart, cstop) == numpy.array(coords)) 1492 15 5.0 0.0 if inc_seq: 1493 1 1000172 1000172.0 3.6 return self.read(cstart, cstop, field=field) 1494 return self.readCoordinates(coords, field) File: profile.py Function: function_to_test at line 11 Total time: 39.1439 s the profile for the _where() function looks like this: Line # Hits Time Per Hit % Time Line Contents == 1423 def _where( self, condition, condvars, 1424 start=None, stop=None, step=None ): 1425 Low-level counterpart of `self.where()`. 1426 12 2.0 0.0 if profile: tref = time() 1427 11 1.0 0.0 if profile: show_stats(Entering table._where, tref) 1428 # Adjust the slice to be used. 1429 1 20 20.0 0.0 (start, stop, step) = self._processRangeRead(start, stop, step) 1430 12 2.0 0.0 if start = stop: # empty range, reset conditions 1431 self._useIndex = False 1432 self._whereCondition = None 1433 return iter([]) 1434 1435 # Compile the condition and extract usable index conditions. 1436 1 153153.0 0.0 condvars = self._requiredExprVars(condition, condvars, depth=3) 1437 1 1299 1299.0 0.3 compiled = self._compileCondition(condition, condvars) 1438 1439 # Can we use indexes? 1440 12 2.0 0.0 if compiled.index_expressions: 1441 11 1.0 0.0 chunkmap = _table__whereIndexed( 1442 1 439602 439602.0 99.6 self, compiled, condition, condvars, start, stop, step) 1443 15 5.0 0.0 if type(chunkmap) != numpy.ndarray: 1444
Re: [Pytables-users] Installation problems windows
I seem to several MSVCR*.DLL on my system. I've checked every MSVCR90.DLL that I have on my system and the properties indicate for all of them Product Name: Microsoft Visual Studio 2008 My expertise in this domain is limited so I'll check with a colleague. Furthermore it is bizar that if I run the same script (just import tables) as a 'normal' script (in spyder, but at the command prompt 'python script.py' works also fine) So, to me it seems that my settings should be ok. PS. my installation of python, pytables,... is done with python(x,y) at once. Message: 3 Date: Mon, 23 Jan 2012 10:16:05 -0600 From: Anthony Scopatz scop...@gmail.com Subject: Re: [Pytables-users] Installation problems windows To: Discussion list for PyTables pytables-users@lists.sourceforge.net Message-ID: CAPk-6T67dXddkD8djBgosNtZhH5Who=adk0ceozwmvgcc2c...@mail.gmail.com Content-Type: text/plain; charset=iso-8859-1 Hello Luc, This is because you do not have the correct version of the Windows C runtime library installed. Python is distributed with the Virtual C 2008 (and not anything newer!). You need to install MSVCR90.DLL correctly to get this to work. This is a well known issue. Google for ImportError and windows and MSVCR90.DLL you should be able to find a solution online. Be Well Anthony On Mon, Jan 23, 2012 at 7:04 AM, Luc Kesters luc.kest...@hotmail.comwrote: ** I've noticed that this issue occured already in the past but I can't figure out my problem. My settings: - python 2.7 - windows vista - pytables is installed using python(x,y) If I import pandas in a regular script it works fine. I also use python scripting in an scripting engine that's embedded in another system (windows exe) which can use vbscript, javascript and also python. Other python scripting seems to work fine but if I want to use pytables I get this error message: [Error] Message=Syntax error in script Number=44804 [Error Context] Error_Description_1=Traceback (most recent call last): Error_Description_2= File Script Block , line 1, in module Error_Description_3=import tables Error_Description_4= File C:\Python27\lib\site-packages\tables\__init__.py, line 63, in module Error_Description_5=from tables.utilsExtension import getPyTablesVersion, getHDF5Version Error_Description_6=ImportError: DLL load failed: The specified module could not be found. Error_Source=Python ActiveX Scripting Engine HResult=$80020009 PreProcessor_Source_LineNr=3 Script_Error_Column=-1 Script_Error_LineNr=1 Script_Error_Text=import tables Win32errorMessage=Exception occurred I'd tried to figure it out with a tool dependecy walker (see www.dependencywalker.com) I get the following error. 00:00:15.179: LoadLibraryExA(C:\Python27\lib\site-packages\tables\utilsExtension.pyd, 0x, LOAD_WITH_ALTERED_SEARCH_PATH) called from c:\windows\system32\PYTHON27.DLL at address 0x03CB8BD1 by thread 1. 00:00:15.194: *Loaded c:\python27\lib\site-packages\tables\UTILSEXTENSION.PYD at address 0x02CE by thread 1. Successfully hooked module. * 00:00:15.194: *Loaded c:\python27\lib\site-packages\tables\HDF5DLL.DLL at address 0x04D6 by thread 1. Successfully hooked module. * 00:00:15.210: *Loaded c:\python27\lib\site-packages\tables\ZLIB1.DLL at address 0x02D4 by thread 1. Successfully hooked module. * 00:00:15.225: *Loaded c:\python27\lib\site-packages\tables\SZIP.DLL at address 0x6C54 by thread 1. Successfully hooked module. * 00:00:15.225: Unloaded c:\python27\lib\site-packages\tables\UTILSEXTENSION.PYD at address 0x02CE by thread 1. 00:00:15.225: Unloaded c:\python27\lib\site-packages\tables\HDF5DLL.DLL at address 0x04D6 by thread 1. 00:00:15.241: Unloaded c:\python27\lib\site-packages\tables\SZIP.DLL at address 0x6C54 by thread 1. 00:00:15.241: Unloaded c:\python27\lib\site-packages\tables\ZLIB1.DLL at address 0x02D4 by thread 1. 00:00:15.257: *LoadLibraryExA(C:\Python27\lib\site-packages\tables\utilsExtension.pyd, 0x, LOAD_WITH_ALTERED_SEARCH_PATH) returned NULL by thread 1. Error: The specified module could not be found (126). * 00:00:15.491: First chance exception 0x0EEDFADE (Unknown) occurred in c:\windows\system32\KERNEL32.DLL at address 0x76A3FC56 by thread 1. 00:00:15.506: LoadLibraryExW(C:\Windows\system32\btmmhook.dll, 0x, LOAD_WITH_ALTERED_SEARCH_PATH) called from c:\windows\system32\USER32.DLL at address 0x763BE710 by thread 6. If I take UTILSEXTENSION.pyd, I get an error: *Error: At least one required implicit or forwarded dependency was not found. Warning: At least one delay-load dependency module was not found. Warning: At least one module has an unresolved import due to a missing export function in a delay-load dependent module.* That seems to be related to MSVCR90.DLL and it gives: C:\Python27\lib\site-packages\tables\MSVCR90.DLL was not found UTILSEXTENSION.pyd is also related to
