Re: [Pytables-users] Searching for duplicate keys...

2012-07-02 Thread Anthony Scopatz
Why not read in just the date and ID columns to start with, then do a numpy.unique() or python set() on theses, then query based on the unique values? Seems like it might be faster Be Well Anthony On Mon, Jul 2, 2012 at 5:16 PM, Aquil H. Abdullah wrote: > Hello All, > > I have a table that

Re: [Pytables-users] How would you handle file openings for reading by an API?

2012-07-02 Thread Anthony Scopatz
No worries ;) On Mon, Jul 2, 2012 at 5:41 PM, Jacob Bennett wrote: > Cool, this seems pretty straightforward. Thanks again Anthony! > > -Jacob > > > On Mon, Jul 2, 2012 at 1:10 PM, Anthony Scopatz wrote: > >> Hello Jacob, >> >> It seems like you have answered your own question ;). The thing is

Re: [Pytables-users] How would you handle file openings for reading by an API?

2012-07-02 Thread Jacob Bennett
Cool, this seems pretty straightforward. Thanks again Anthony! -Jacob On Mon, Jul 2, 2012 at 1:10 PM, Anthony Scopatz wrote: > Hello Jacob, > > It seems like you have answered your own question ;). The thing is that > the locking doesn't have to be all that hard. You can simply check if the >

[Pytables-users] Searching for duplicate keys...

2012-07-02 Thread Aquil H. Abdullah
Hello All, I have a table that is indexed by two keys, and I would like to search for duplicate keys. So here is my naive slow implementation: (code I posted on stackoverflow) import tables h5f = tables.openFile('filename.h5') tbl = h5f.getNode('/data','data_table') # assumes group data and tabl

Re: [Pytables-users] How would you handle file openings for reading by an API?

2012-07-02 Thread Anthony Scopatz
Hello Jacob, It seems like you have answered your own question ;). The thing is that the locking doesn't have to be all that hard. You can simply check if the file is already in the open files cache (from previous thread). If it isn't, the thread will open the file. Or you don't even have to d

[Pytables-users] How would you handle file openings for reading by an API?

2012-07-02 Thread Jacob Bennett
Hello PyTables Users, I am developing an API to access the current data stored in my pytables instance. Note at this point that this is only reading, no writing to the files. The big question on my mind at this point is how am I supposed to handle the opening and closing of files on read requests