On Wed, Aug 15, 2007 at 08:40:28AM +1000, David Worrall wrote: > Thanks Francesc, > > >>> tables.print_versions() > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > =-=-=-= > PyTables version: 2.0 > HDF5 version: 1.6.5 > NumPy version: 1.0.3 > Zlib version: 1.2.3 > BZIP2 version: 1.0.2 (30-Dec-2001) > Python version: 2.4.3 (#1, Mar 30 2006, 11:02:16) > [GCC 4.0.1 (Apple Computer, Inc. build 5250)] > Platform: darwin-i386 > Byte-ordering: little > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- > =-=-=-=
Mmm, seems good. > My RAM is 256Mb on OSX 10.4.10 (the latest). Mmmm, not so good ;) I think what is happening is that your computer is getting out of memory resources. A Table node should use around 1 MB of memory, and the number of slots in the PyTables' node cache (NODE_MAX_SLOTS) is 256 by default. This means that with 256 MB you are completely filling it (i.e. the system will start using swap), and that could be the source of the problem you are experiencing. My advice is that you should try with a machine with at least 512 MB, or better yet, try lowering NODE_MAX_SLOTS to, say, 64. > It may be something to do with the way the OS manages memory. I've > heard it's pretty aggressive, so if the number of groups is large, > parts may be cached out before being accessed again. I don't know > what I'm talking about really :-) but if you think that it is likely, > perhaps I could raise the issue on the pythonmac-SIG. ? I don't know either ;), but IMO it is most probably a memory issue. > Also, is there any reason why I can't/shouldn't use 2 or more .h5 > files in parallel (split the 3500+ top-level groups into two DBs)? As Ivan has already said, this perfectly fine with PyTables. However, this will only worse the memory problem, as each opened file will have its own node cache, raising still more your memory needs. So, if you are dealing with a very large amount of nodes and have not much memory available, it is better to stay with only one opened file at a time. Cheers, Francesc ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Pytables-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/pytables-users
