Kalpa, If I read your example correctly, your ‘SLOW’ example is 15 times bigger than your ‘quite quick’ example. So, assuming everything scales linearly:
2.5seconds * 15 = 37.5 seconds… which is pretty close to the 38.4 seconds you measured and called ‘SLOW’. Also, 600K * 15 = 8.8 MB… so 8 MB is doing pretty well. What were you expecting? Scott From: Hdf-forum [mailto:[email protected]] On Behalf Of Karpaga Rajadurai Sent: Wednesday, April 23, 2014 7:42 AM To: [email protected] Subject: [Hdf-forum] Writing a large Compound dataset - slow [Java API] I tried writing a 2D compound dataset with dimension [10000 x 150]. int DIM_X = 10000; int DIM_Y = 150; int MAX_DIMX = -1; int MAX_DIMY = -1; int CHUNK_X =1000; int CHUNK_Y = 10; long[] dims = {DIM_X, DIM_Y}; long[] maxdims = {MAX_DIMX, MAX_DIMY}; // UNLIMITED FILE SIZE long[] chunks = {CHUNK_X, CHUNK_Y}; // NOT USED AS OF NOW int gzip = 9; String[] Column1 = new String[DIM_X*DIM_Y]; String[] Column2 = new String[DIM_X*DIM_Y]; String[] Column3 = new String[DIM_X*DIM_Y] /* Column1, Column2, Column3 are String arrays of size DIM_X * DIM_Y */ Vector data = new Vector(); data.add(0, Column1); data.add(1, Column2); data.add(2, Column3); long b = System.currentTimeMillis(); System.out.println("<<< Creating HDF5 File >>>"); Dataset d = file.createCompoundDS(FILENAME, null, dims, maxdims, null, gzip, memberNames, memberDatatypes, memberSizes, data); System.out.println("Time taken for Writing all cells to H5 file >> "+(System.currentTimeMillis()-b)); It is found that the time taken for writing the above H5 file (10000x150) takes around 38393ms(SLOW). The file size is around 8MB(too big). If, my DIM_X is 10000 and DIM_Y is 10 then, the time taken for writing the file is just 2543ms(quite quick). The file size is around 600KB. Is there any better way to reduce the time taken for writing huge compound dataset file? Also, how to reduce the file size? Will chunking in any way do the need? Please, throw some light on this. Thanks in advance, kalpa ________________________________ This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Exelis Inc. The recipient should check this e-mail and any attachments for the presence of viruses. Exelis Inc. accepts no liability for any damage caused by any virus transmitted by this e-mail.
_______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
