On 15/08/2007, at 5:36 PM, [EMAIL PROTECTED] wrote: > On Wed, Aug 15, 2007 at 08:40:28AM +1000, David Worrall wrote: > >> My RAM is 256Mb on OSX 10.4.10 (the latest). > > Mmmm, not so good ;) I think what is happening is that your > computer is getting out of memory resources. A Table node should use > around 1 MB of memory, and the number of slots in the PyTables' node > cache (NODE_MAX_SLOTS) is 256 by default. This means that with 256 MB > you are completely filling it (i.e. the system will start using swap), > and that could be the source of the problem you are experiencing. > My advice is that you should try with a machine with at least 512 MB, > or better yet, try lowering NODE_MAX_SLOTS to, say, 64. > Arrrrrh! I made a mistake - it is actually 512Mb - sorry. ....
>> Also, is there any reason why I can't/shouldn't use 2 or more .h5 >> files in parallel (split the 3500+ top-level groups into two DBs)? > > As Ivan has already said, this perfectly fine with PyTables. However, > this will only worse the memory problem, as each opened file will have > its own node cache, raising still more your memory needs. So, if > you are dealing with a very large amount of nodes and have not much > memory available, it is better to stay with only one opened file at a > time. I understand that, however it means I might be able to find a way to meaningfully segment the DB so each segment-DB is smaller. As I think I mentioned earlier, I did success in generating a complete DB by loading 1/2 of it, closing the file, quitting python and then starting again with the same file, adding the 2nd 1/2. All that aside, 1. it is a worry that there is no exception generated, by python, at least, don't you agree? 2. Is there a way of purging memory after a flush or are we up against python's memory management? > Cheers, > > Francesc > thanks a lot for your help, David _________________________________________________ experimental polymedia: www.avatar.com.au Sonic Communications Research Group, University of Canberra: creative.canberra.edu.au/scrg ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ Pytables-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/pytables-users
