El dl 11 de 09 del 2006 a les 20:02 +0200, en/na Francesc Altet va escriure: > > -------- Missatge reenviat -------- > > De: [EMAIL PROTECTED] > > Per a: [email protected] > > Assumpte: More than 4096 tables in a group > > Data: Mon, 11 Sep 2006 16:34:12 +0200 > > > > Hello, > > > > I'm currently building a storage backend for so-called interest regions > > in images. I have 90,000 images. > > I started out by having one big group with one table per image (since > > the number of regions per image is variable, this seemed like a nice > > solution to me). All these tables have the exact same structure. > > Now, I received a PerformanceWarning that having more than 4096 'image > > tables' in a group will increase memory usage. > > > > My question: should should I keep this big group? Or should I split it > > into groups (which is possible in a natural way) of 200-500 images? > > Which is more memory efficient? Does this make a big difference in > > lookup performance? Or in 'creation/write' performance?
To say the truth, this has been a warning that existed in pre-1.2 versions when all the *metainformation* about nodes was loaded completely in-memory. After the introduction of a cache for nodes in 1.2, we decided to keep this warning because we felt that keeping too much leaves on a single tree was not a good way of structuring datasets. Having said that, I'm not very sure whether this recommendation is still valid or not (from the performance point of view). You may want to experiment with spliting and without spliting (you can avoid the warning by increasing the MAX_GROUP_WIDTH in tables/constants.py) and decide which schema is better for you. Also, if you do a lot of browsing on the object tree, you may want to play as well with changing the value of NODE_CACHE_SIZE (also in tables/constants.py) before to decide the value you should stick in. I'm insterested in your findings, so please, report back your experiences. -- >0,0< Francesc Altet http://www.carabos.com/ V V Cárabos Coop. V. Enjoy Data "-" ------------------------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 _______________________________________________ Pytables-users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/pytables-users
