Sorry Andreas.. My product is rewritten in python placed in the product folder. So yes, it is a instance of a class.
In the future, I will be storing more data into one instance from other dbase file (The total columns may vary). I'm still in the developing stage and this is just a test run to know that is the length of processing time I'm look at. I want each entry to be an instance because I'm planning to create other interactive functions (edit and query(ZCatalog maybe, I'm not sure yet) and more ) for particular info. I've never thought of using a BTree because I don't know enough about it. I'll look into it but will BTree still be a better choice than making instance if I'm going to make interactive functions? Any other suggestions? ----- Original Message ---- From: Andreas Jung <[EMAIL PROTECTED]> To: Allen Huang <[EMAIL PROTECTED]>; Zope <zope@zope.org> Sent: Tuesday, January 9, 2007 1:55:34 PM Subject: Re: [Zope] need advice on mass data processing --On 8. Januar 2007 19:28:32 -0800 Allen Huang <[EMAIL PROTECTED]> wrote: > I have a data file that has over 110000 entry of 3 column data (string, > float, float) > currently I have written my program so it will do an entry by entry > processing with zope. This operation is like this > 1. read data (the data file) > 2. create product (a python product that store three field data: one > string and two float data) 3. update product (update the three field > entries) Please name things the right way. A "Product" is basically a Zope/Python package that contains definitions of classes, scripts, templates etc. You mean instances of a particular class? > > when I first tried it out with the first 1000 entries it took about 30 > seconds. That means its going to take 50 ~ 60 minutes for 110000 entries. You're creating 110k instances for storing a string and two floats? If yes, that's stupid idea. You can persistent large amounts of data within a single instances by using Zope BTrees. > It not every day that you have to process over 110000 data entries but > processing over 60 minutes is still kind of long. > What kind of processing? -aj __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )