Hi Francesc, I tried your example as it is, could not get time to modify and try some thing new.
ran the $ python csv_demo.py it did create a CSV file with 10 columns, populating the columns with random no. The demo.h5 was created, and I used HDFView 2.9 to see the contents of the demo.h5 file. created were a directory table, and data table - table. In the data table - table, there are 2 columns index | value_block_0 empty | no value no data | but 10 commas So that I can relate to your guidance with respect to the issue, please find attached 2 sample files. Also, note the first row in CSVs attached, this was created to initialise the start point of data sequence. Will it be a good practice to have them in h5 tables also ? Last column has string values, need them. ALIGN data goes into file1 and GRADE data into File2, so I am looking for a write function to write into respective tables and then read function to read from them. After the data is in H5 file, can I insert/add/append a new row in between other rows or at end of file ? Which editor to use or method to do it in ? Thank you, Nitin On 30 January 2017 at 23:01, nitin chandra <[email protected]> wrote: > Thank you Francesc, > > Please give me 2-3 days try your example ... do some reading and > testes based as per the link mentioned. > > I shall repost soon. > > Thank you > > Nitin > > On 30 January 2017 at 17:14, Francesc Altet <[email protected]> wrote: >> Hi Nitin, >> >> >> I think before getting into details, you need to look into how to >> efficiently read and write data from CSV files into HDF5 in Python. For >> this, pandas is a great library to use. My advice is to have a look at the >> excellent documentation in pandas website: >> >> >> http://pandas.pydata.org/pandas-docs/stable/io.html >> >> >> In particular, you want to use the `pandas.read_csv()` which one of the >> fastest ways to read CSV files that I am aware of. Also, for storing the >> data in HDF5, `pandas.HDFStore()` comes handy because it can generate HDF5 >> files out of pandas Dataframes. In addition, in order to avoid loading all >> the data in a Dataframe in memory, you want to use the `chunksize` keyword >> that will allow to read the CSV files in chunks before storing. >> >> >> I have prepared an example for you (attached) so that you can have a look at >> how to use all of this (it is simpler than it may seem). Here it is the >> output on my machine: >> >> >> $ python csv_demo.py >> CSV creation time: 1.491 (67.092 Krow/s) >> CSV reading time: 0.134 (748.360 Krow/s) >> HDF5 store time: 0.322 (310.228 Krow/s) >> HDF5 read time: 0.006 (15622.990 Krow/s) >> >> >> so, once the data is stored in HDF5, the read times will be much faster than >> using CSV (as expected). >> >> >> HTH, >> >> >> Francesc >> >> >> _______________________________________________ >> Hdf-forum is for HDF software users discussion. >> [email protected] >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org >> Twitter: https://twitter.com/hdf5
,,27,,, 27,0,182,,, 27.1,0,181,,, 27.2,0,180.5,,, 27.3,0,181.5,,, 27.4,0,183,,, 27.5,0,185,,, 27.6,0,183,,, 27.7,0,182.5,,, 27.8,0,180.7,,, 27.9,0,180.3,,, 28,0,179,,, 28.1,0,178.4,,, 28.2,0,178.6,,, 28.3,0,178.8,,, 28.4,0,178.9,,, 28.5,0,178,,, 28.6,0,178,,, 28.7,0,178,,, 28.78,1,177,,, 28.8,0,178,,, 28.9,0,178,,, 29,0,177.5,,, 29.1,0,178,,, 29.2,0,178,,, 29.3,0,178,,, 29.4,0,178.5,,, 29.5,0,179,,, 29.6,0,179,,, 29.7,0,179,,, 29.8,0,179,,, 29.82,1,176,,, 29.9,0,179,,, 29.92,1,178.5,,, 30,0,180,,, 30.1,0,179,,, 30.2,0,178.5,,, 30.3,0,178,,, 30.4,0,178.5,,, 30.5,0,179,,, 30.6,0,177.5,,, 30.7,0,177,,, 30.8,0,177,,, 30.9,0,176,,, 31,0,178,,, 31.1,0,176.5,,, 31.2,0,176,,, 31.3,0,176,,, 31.4,0,175,,, 31.5,0,173,,, 31.6,0,172,,, 31.7,0,172.5,,, 31.8,0,171,,, 31.9,0,170,,, 32,0,169,,, 32.1,0,172,,, 32.2,0,171,,, 32.3,0,172,,, 32.4,0,172.5,,, 32.5,0,173,,, 32.6,0,174,,, 32.7,0,175,,, 32.8,0,176,,, 32.9,0,177,,, 33,0,176.5,,, 33.1,0,177,,, 33.2,0,177.5,,, 33.3,0,178,,, 33.4,0,179,,, 33.5,0,180,,, 33.6,0,178,,, 33.7,0,178,,, 33.8,0,178,,, 33.9,0,178.5,,, 34,0,182,,, 34.1,0,177,,, 34.2,0,178,,, 34.3,0,178,,, 34.4,0,178.5,,, 34.5,0,180,,, 34.6,0,179,,, 34.7,0,178.5,,, 34.8,0,178,,, 34.9,0,178,,, 35,0,177,,, 35.1,0,178,,, 35.2,0,178,,, 35.3,0,178,,, 35.4,0,178,,, 35.5,0,178,,, 35.6,0,178,,, 35.7,0,178,,, 35.8,0,178,,, 35.9,0,178,,, 36,0,177.5,,, 36.1,0,178,,, 36.2,0,178,,, 36.3,0,176.55,,, 36.4,0,177,,, 36.5,0,178,,, 36.6,0,178,,, 36.7,0,178,,, 36.8,0,178,,, 36.9,0,178,,, 37,0,177,,, 37.1,0,178,,, 37.2,0,177,,, 37.3,0,177.5,,, 37.4,0,177,,, 37.5,0,177,,, 37.6,0,177,,, 37.7,0,177,,, 37.8,0,177,,, 37.9,0,177,,, 38,0,176,,, 38.1,0,176,,, 38.2,0,176,,, 38.3,0,176.5,,, 38.4,0,177,,, 38.5,0,177,,, 38.6,0,177,,, 38.7,0,178,,, 38.8,0,179,,, 38.9,0,180.5,,, 39,0,179,,, 39.1,0,179.5,,, 39.2,0,179,,, 39.3,0,179,,, 39.4,0,179,,, 39.5,0,180,,, 39.6,0,179,,, 39.7,0,180,,, 39.8,0,181,,, 39.9,0,181.5,,, 40,0,182,,, 40.1,0,181,,, 40.2,0,180,,, 40.3,0,177.9,,, 40.4,0,178,,, 40.5,0,179,,, 40.6,0,179,,, 40.7,0,179,,, 40.8,0,179,,, 40.9,0,179.5,,, 41,0,180,,, 41.1,0,179,,, 41.2,0,179,,, 41.3,0,179,,, 41.4,0,179,,, 41.5,0,179,,, 41.6,0,179,,, 41.7,0,179,,, 41.8,0,179,,, 41.9,0,179,,, 42,0,179,,, 42.1,0,179,,, 42.2,0,179,,, 42.3,0,179,,, 42.4,0,179,,, 42.5,0,179,,, 42.6,0,178,,, 42.7,0,179,,, 42.8,0,179,,, 42.9,0,179,,, 43,0,179,,, 43.1,0,182,,, 43.2,0,188,,, 43.3,0,190,,,Bridge 43.4,0,195,,,Bridge 43.5,0,194,,,Bridge 43.6,0,195.5,,, 43.7,0,197,,, 43.8,0,198.7,,, 43.9,0,199.8,,, 44,0,210,,, 44.1,0,215,,, 44.2,0,216.5,,, 44.3,0,218,,, 44.4,0,220,,, 44.5,0,230,,, 44.6,0,232,,, 44.7,0,233.5,,, 44.8,0,234.6,,, 44.9,0,240.8,,, 45,0,250,,, 45.1,0,249,,, 45.2,0,248,,, 45.3,0,247,,, 45.4,0,246,,, 45.5,0,245,,, 45.6,0,243,,, 45.7,0,243.6,,, 45.8,0,242,,, 45.9,0,241,,, 46,0,240,,, 46.1,0,244,,, 46.2,0,246.5,,, 46.3,0,248.9,,, 46.4,0,252,,, 46.5,0,255,,, 46.6,0,257,,, 46.7,0,260,,,ecut 46.8,0,265,,,ecut 46.9,0,267,,,ecut 47,0,270,,,ecut 47.1,0,263,,,ecut 47.2,0,260,,, 47.3,0,255,,, 47.4,0,250,,, 47.5,0,245,,, 47.6,0,240,,, 47.7,0,235,,, 47.8,0,230,,, 47.9,0,225,,, 48,0,220,,, 48.1,0,225,,, 48.2,0,227,,, 48.3,0,230,,, 48.4,0,233,,, 48.5,0,235,,, 48.6,0,240,,, 48.7,0,242,,, 48.8,0,243,,, 48.9,0,244,,, 49,0,250,,, 49.1,0,238,,, 49.2,0,240,,, 49.3,0,245,,, 49.4,0,248,,, 49.5,0,250,,, 49.6,0,250,,, 49.7,0,250,,, 49.8,0,250,,, 49.9,0,250,,, 50,0,250,,,
27,,199.206, 29.5,,-0.5,FALL 1 IN 200 32,,-0.084,FALL 1 IN 1200 33,,0.2,RISE 1 IN 130 40,,0.083,RISE 1 IN 1200 50,,0.769,RISE 1 IN 130
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org Twitter: https://twitter.com/hdf5
