[Pytables-users] Performance issue?

[Louis Wicker]

Hi: I have a 300,000+ line Pytable having 25 columns of various data, mostly floats.   I need to do a set of simultaneous searches on this data, based on observation type, time, location... Here is the code I am using to do the search (here I am just searching for 5 columns of data, I would like to do all 25 and have in other ways, not much diff):       for o in table.iterrows():             ob_date = datetime.datetime(o['year'], o['month'], o['day'], o['hour'], o['min'], o['sec'])             if o['status'] == status_in:                   if o['type'] == obtype_in                        and  \                      tgrid[0]  <= ob_date  < tgrid[1]          and  \                      xgrid[1]  <=  o['x']  < xgrid[2]             and  \                      ygrid[1]  <=  o['y']  < ygrid[2]             and  \                      zgrid[1]  <=  o['z']  < zgrid[2]:                         value.append(o['value'])                         Hxa.append(o['Hxa'])                         Hxf.append(o['Hxf'])                         sdHxa.append(o['sdHxa'])                         sdHxf.append(o['sdHxf']) the only "funny" deal here is that I am using the python datetime module to compare the times, but from my tests and trials with and without it I don't think that is the problem (?). It takes about 7 sec on the Mac G5 (2.5 ghz) and 8.5 sec on an SGI Altix.  I think that is not very good, right? I have also tried "inlining" each variable with a search (really slow), or try using the table.where functions, again, slow. Another problem I am having with this is that I need to search over 20-30 individual time bins (like above), and I cannot figure out a way to bin up the data in one pass, e.g., I am calling the above loop 20-30 times, so to data mine this table I have a longer wait than I thought I would.... Ideas or comments?   Lou Wicker ---------------------------------------------------------------------------- |  Dr. Louis J. Wicker |  Research Scientist, National Severe Storms Lab | 1313 Halley Cir, Norman, OK 73069 | E-mail:   [EMAIL PROTECTED] | HTTP:  www.nssl.noaa.gov/~lwicker | Phone:    (405) 366-0416 | Fax:       (405) 366-0472 | | | Two wrongs don't make a right, but three left's do...  ---------------------------------------------------------------------------- | | "The contents  of this message are mine personally and  | do not reflect any position of  the Government or NOAA." | ----------------------------------------------------------------------------