Re: [Pytables-users] Some method like a table.readWhereSorted
I am also interested in the this capability, if it exists in some way... Lou On Apr 10, 2013, at 12:35 PM, Julio Trevisan juliotrevi...@gmail.com wrote: Hi, Is there a way that I could have the ability of readWhere (i.e., specify condition, and fast result) but also using a CSIndex so that the rows come sorted in a particular order? I checked readSorted() but it is iterative and does not allow to specify a condition. Julio -- Precog is a next-generation analytics platform capable of advanced analytics on semi-structured data. The platform includes APIs for building apps and a phenomenal toolset for data science. Developers can use our toolset for easy data analysis visualization. Get a free account! http://www2.precog.com/precogplatform/slashdotnewsletter___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users | Dr. Louis J. Wicker | NSSL/WRDD Rm 4366 | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072 | | E-mail: louis.wic...@noaa.gov | HTTP:http://www.nssl.noaa.gov/~lwicker | Phone:(405) 325-6340 | Fax:(405) 325-6780 | | I For every complex problem, there is a solution that is simple, | neat, and wrong. | | -- H. L. Mencken | | The contents of this message are mine personally and | do not reflect any position of the Government or NOAA. -- Precog is a next-generation analytics platform capable of advanced analytics on semi-structured data. The platform includes APIs for building apps and a phenomenal toolset for data science. Developers can use our toolset for easy data analysis visualization. Get a free account! http://www2.precog.com/precogplatform/slashdotnewsletter ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Odd behavior on Mac os x 10.6
Tony - thanks - here is the dropbox link: http://dl.dropbox.com/u/4017006/testconvert.tar 2.63 GB. Just untar it, and type python convertNC.py that will do it, and check the file sizes after its run. On my Mac Pro - about 1.5 minutes for the whole thing. Thanks again so much. Lou On Jun 25, 2011, at 1:25 AM, Tony Theodore wrote: On 25 June 2011 08:43, Louis Wicker louis.wic...@noaa.gov wrote: Anthony: No thanks, think there is a bug, and so I was offering up code and files for testing. The files are an order of magnitude larger than I thought. The base netCDF4 files, compressed, are 1.4 GB. I dont have time to pare down the files and see if that is the problem - my guess is, since I have played with this conversion before on smaller files is that it does not. I am uploading 2 files and the code to dropbox (I have a big space up there) if anyone wants to try this on their machine. Let me know and I will send a link to the files, but keep in mind the tar file is ~ 3 GB. Hi Louis, I'm running OSX 10.6 and am happy to test if the issue occurs here. Cheers, Tony -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense.. http://p.sf.net/sfu/splunk-d2d-c1 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users | Dr. Louis J. Wicker | NSSL/FRDD Rm 4368 | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072 | | E-mail: louis.wic...@noaa.gov | HTTP:www.nssl.noaa.gov/~lwicker | Phone:(405) 325-6340 | Fax:(405) 325-6780 | | If you start a day without a clear plan about how you’re | going to spend your attention, you’ll end up wasting most | of it.—Martha Beck (2002) | | Assemby of Japanese bicycle requires great peace of mind. |Robert Pirsig, ZAMM | | | The contents of this message are mine personally and | do not reflect any position of the Government or NOAA. | -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense.. http://p.sf.net/sfu/splunk-d2d-c1 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
[Pytables-users] sorry for message
did not mean to spam the entire pyTables list with that reply to Tony. Still not awake yet, apparently. L | Dr. Louis J. Wicker | NSSL/FRDD Rm 4368 | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072 | | E-mail: louis.wic...@noaa.gov | HTTP:www.nssl.noaa.gov/~lwicker | Phone:(405) 325-6340 | Fax:(405) 325-6780 | | If you start a day without a clear plan about how you’re | going to spend your attention, you’ll end up wasting most | of it.—Martha Beck (2002) | | Assemby of Japanese bicycle requires great peace of mind. |Robert Pirsig, ZAMM | | | The contents of this message are mine personally and | do not reflect any position of the Government or NOAA. | -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense.. http://p.sf.net/sfu/splunk-d2d-c1 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Odd behavior on Mac os x 10.6
sorry - that was test code to see that the uncompressed files were the same size as what I was seeng. I have tried complevel 1-5, and get the same result. L On Jun 24, 2011, at 7:45 AM, Francesc Alted wrote: 2011/6/24 Dr. Louis Wicker louis.wic...@noaa.gov All Perhaps I am just missing the obvious, but I have some code that is converting netCDF4 files to pyTables Carray with blosc compression (playing around). I loop over the files in a directory, and I have found that the first file is written compressed, but the second file onward is NOT. I have checked the filter settings after each file write, and they are the same. Any ideas here? If I turn off compression, the file sizes match. Code below Lou array_atom = tables.Float32Atom() filters = tables.Filters(complevel=0, complib='blosc') clip Any reason why you are specifying 'complevel=0'? 0 means 'not compression'. Try with levels from 1 to 9. -- Francesc Alted -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense.. http://p.sf.net/sfu/splunk-d2d-c1___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users | Dr. Louis J. Wicker | NSSL/FRDD Rm 4368 | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072 | | E-mail: louis.wic...@noaa.gov | HTTP:www.nssl.noaa.gov/~lwicker | Phone:(405) 325-6340 | Fax:(405) 325-6780 | | If you start a day without a clear plan about how you’re | going to spend your attention, you’ll end up wasting most | of it.—Martha Beck (2002) | | Assemby of Japanese bicycle requires great peace of mind. |Robert Pirsig, ZAMM | | | The contents of this message are mine personally and | do not reflect any position of the Government or NOAA. | -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense.. http://p.sf.net/sfu/splunk-d2d-c1 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] Odd behavior on Mac os x 10.6
Thanks. Except that when I only do one data set, it works. E.g, If I start on a different file - it then compresses. Let me check, and if someone (not you, I guess) wants, I can tar 2 files up and the code and put it on dropbox. These are big files, the uncompressed files are 0.5Gb L On Jun 24, 2011, at 12:22 PM, Francesc Alted wrote: A Friday 24 June 2011 18:31:28 Dr. Louis Wicker escrigué: sorry - that was test code to see that the uncompressed files were the same size as what I was seeng. I have tried complevel 1-5, and get the same result. It is possible that Blosc cannot compress your datasets, and as consequence, it leaves them uncompressed. You may want to try with level 9 just to see if Blosc can actually compress them. -- Francesc Alted -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense.. http://p.sf.net/sfu/splunk-d2d-c1 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users | Dr. Louis J. Wicker | NSSL/FRDD Rm 4368 | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072 | | E-mail: louis.wic...@noaa.gov | HTTP:www.nssl.noaa.gov/~lwicker | Phone:(405) 325-6340 | Fax:(405) 325-6780 | | If you start a day without a clear plan about how you’re | going to spend your attention, you’ll end up wasting most | of it.—Martha Beck (2002) | | Assemby of Japanese bicycle requires great peace of mind. |Robert Pirsig, ZAMM | | | The contents of this message are mine personally and | do not reflect any position of the Government or NOAA. | -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense.. http://p.sf.net/sfu/splunk-d2d-c1 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
[Pytables-users] Odd behavior on Mac os x 10.6
All Perhaps I am just missing the obvious, but I have some code that is converting netCDF4 files to pyTables Carray with blosc compression (playing around). I loop over the files in a directory, and I have found that the first file is written compressed, but the second file onward is NOT. I have checked the filter settings after each file write, and they are the same. Any ideas here? If I turn off compression, the file sizes match. Code below Lou array_atom = tables.Float32Atom() filters = tables.Filters(complevel=0, complib='blosc') for file in files[0:2]: print \n Processing file: %s \n % file f = netCDF4.Dataset(file) fid = tables.openFile(file[:-2]+h5,w) t0 = time() # Copy all arrays for array_name in f.variables: data = f.variables[array_name][:] print array_name, data.shape field = fid.createCArray(fid.root, array_name, array_atom, data.shape, filters=filters) field[:] = data[:] f.close() fid.close() | Dr. Louis J. Wicker | NSSL/FRDD Rm 4368 | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072 | | E-mail: louis.wic...@noaa.gov | HTTP:www.nssl.noaa.gov/~lwicker | Phone:(405) 325-6340 | Fax:(405) 325-6780 | | If you start a day without a clear plan about how you’re | going to spend your attention, you’ll end up wasting most | of it.—Martha Beck (2002) | | Assemby of Japanese bicycle requires great peace of mind. |Robert Pirsig, ZAMM | | | The contents of this message are mine personally and | do not reflect any position of the Government or NOAA. | -- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense.. http://p.sf.net/sfu/splunk-d2d-c1 ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
Re: [Pytables-users] PyTables Installation
Todd: I have found that sometimes I needed to go into the setup.py scripts, turn off looking for the HDF install, and set the directory right inside. I dont know what that makes a diff. L On Feb 12, 2011, at 10:21 AM, Todd Matthews wrote: Hello, I am struggling with PyTables installation. It has been successfully installed for Python 2.6; but, I want to use Pytables for a second Python 2.7 installation. When installing with the following - easy_install-2.7 tables The following message (initial output not included) is produced - ... Running tables-2.2.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-gMhvzt/tables-2.2.1/egg-dist-tmp-mS3ZYC * Found numpy 1.5.1 package installed. .. ERROR:: Could not find a local HDF5 installation. You may need to explicitly state where your local HDF5 headers and library can be found by setting the ``HDF5_DIR`` environment variable or by using the ``--hdf5`` command-line option. error: Setup script exited with 1 I have attempted to be explicit with header file locations as suggested. Also, I have changed the hdf scripts according to HDF docs. But, without any luck. Not sure what to try next and I hope that some one can provide guidance. Thanks, Todd -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users | Dr. Louis J. Wicker | NSSL/FRDD Rm 4368 | National Weather Center | 120 David L. Boren Boulevard, Norman, OK 73072 | | E-mail: louis.wic...@noaa.gov | HTTP:www.nssl.noaa.gov/~lwicker | Phone:(405) 325-6340 | Fax:(405) 325-6780 | | If you start a day without a clear plan about how you’re | going to spend your attention, you’ll end up wasting most | of it.—Martha Beck (2002) | | Assemby of Japanese bicycle requires great peace of mind. |Robert Pirsig, ZAMM | | | The contents of this message are mine personally and | do not reflect any position of the Government or NOAA. | -- The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE: Pinpoint memory and threading errors before they happen. Find and fix more than 250 security defects in the development cycle. Locate bottlenecks in serial and parallel code that limit performance. http://p.sf.net/sfu/intel-dev2devfeb ___ Pytables-users mailing list Pytables-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/pytables-users
