There is no bug in the laszip library that will affect users, but you have found a problem.
First, I was using %d for an unsigned int, when I should have been using %u. I've fixed that. However, using 100,000,000 for the -n option seems to make more than a file of more than 4GB of bytes, and the zip routines only carry around uint32's for byte counts. I think it should be straight-forward to move those counters to u64's - Martin, what do you think? -mpg From: [email protected] [mailto:[email protected]] On Behalf Of David Burken Sent: Monday, February 07, 2011 10:01 AM To: [email protected] Subject: Re: [Liblas-devel] I/O performance? -- your help requested! OK I feel better now. I see Michael's and mine are about the same. I did 100x and something went out of range I think: $ time laszippertest -n 100000000 Settings: num_points=100000000, use_iostream=false, run_forever=false, use_random=false, user_seed=1297100531 laszipper wrote -1589934592 bytes in 58.12 seconds Note the negative, also had to control-c. 6 10x below and 2 million points. Hope this helps... [dburken@tazz build]$ time laszippertest -n 10000000 Settings: num_points=10000000, use_iostream=false, run_forever=false, use_random=false, user_seed=1297100699 laszipper wrote 700000000 bytes in 5.85 seconds SUCCESS: lasunzipper read 700000000 bytes in 3.85 seconds laszipper wrote 381625423 bytes in 17.62 seconds SUCCESS: lasunzipper read 381625422 bytes in 28.49 seconds Finished 1 runs real 0m59.820s user 0m51.617s sys 0m5.002s [dburken@tazz build]$ [dburken@tazz build]$ time laszippertest -n 10000000 Settings: num_points=10000000, use_iostream=false, run_forever=false, use_random=false, user_seed=1297100974 laszipper wrote 700000000 bytes in 4.91 seconds SUCCESS: lasunzipper read 700000000 bytes in 2.63 seconds laszipper wrote 381625423 bytes in 18.51 seconds SUCCESS: lasunzipper read 381625422 bytes in 30.89 seconds Finished 1 runs real 0m58.074s user 0m53.394s sys 0m3.840s [dburken@tazz build]$ time laszippertest -n 10000000 Settings: num_points=10000000, use_iostream=false, run_forever=false, use_random=false, user_seed=1297101035 laszipper wrote 700000000 bytes in 4.79 seconds SUCCESS: lasunzipper read 700000000 bytes in 2.67 seconds laszipper wrote 381625423 bytes in 20.48 seconds SUCCESS: lasunzipper read 381625422 bytes in 23.44 seconds Finished 1 runs real 0m52.201s user 0m47.626s sys 0m4.031s [dburken@tazz build]$ time laszippertest -n 10000000 -s Settings: num_points=10000000, use_iostream=true, run_forever=false, use_random=false, user_seed=1297101129 laszipper wrote 700000000 bytes in 6.81 seconds SUCCESS: lasunzipper read 700000000 bytes in 4.76 seconds laszipper wrote 381625423 bytes in 21.66 seconds SUCCESS: lasunzipper read 381625422 bytes in 28.8 seconds Finished 1 runs real 1m2.768s user 0m55.831s sys 0m6.590s [dburken@tazz build]$ time laszippertest -n 10000000 -s Settings: num_points=10000000, use_iostream=true, run_forever=false, use_random=false, user_seed=1297101209 laszipper wrote 700000000 bytes in 5.09 seconds SUCCESS: lasunzipper read 700000000 bytes in 3.53 seconds laszipper wrote 381625423 bytes in 17.39 seconds SUCCESS: lasunzipper read 381625422 bytes in 27.96 seconds Finished 1 runs real 0m54.278s user 0m49.023s sys 0m5.235s [dburken@tazz build]$ time laszippertest -n 10000000 -s Settings: num_points=10000000, use_iostream=true, run_forever=false, use_random=false, user_seed=1297101272 laszipper wrote 700000000 bytes in 5.2 seconds SUCCESS: lasunzipper read 700000000 bytes in 3.58 seconds laszipper wrote 381625423 bytes in 17.02 seconds SUCCESS: lasunzipper read 381625422 bytes in 24.59 seconds Finished 1 runs real 0m50.899s user 0m45.558s sys 0m5.113s [dburken@tazz build]$ time laszippertest -n 1000000 Settings: num_points=1000000, use_iostream=false, run_forever=false, use_random=false, user_seed=1297101353 laszipper wrote 70000000 bytes in 0.47 seconds SUCCESS: lasunzipper read 70000000 bytes in 0.27 seconds laszipper wrote 38271321 bytes in 1.55 seconds SUCCESS: lasunzipper read 38271321 bytes in 2.45 seconds Finished 1 runs real 0m5.060s user 0m4.435s sys 0m0.594s [dburken@tazz build]$ time laszippertest -n 1000000 -s Settings: num_points=1000000, use_iostream=true, run_forever=false, use_random=false, user_seed=1297101361 laszipper wrote 70000000 bytes in 0.5 seconds SUCCESS: lasunzipper read 70000000 bytes in 0.36 seconds laszipper wrote 38271321 bytes in 1.69 seconds SUCCESS: lasunzipper read 38271321 bytes in 2.48 seconds Finished 1 runs On 02/07/2011 12:02 PM, Michael P. Gerlek wrote: Your box is too fast. :-( Could you maybe do the runs again, but with "-n 100000000" (100x larger)? -mpg From: Smith, Michael ERDC-CRREL-NH [mailto:[email protected]] Sent: Saturday, February 05, 2011 4:48 AM To: [email protected]; [email protected] Subject: Re: [Liblas-devel] I/O performance? -- your help requested! lidar@lidarora1 tmp]$ uname -a Linux lidarora1 2.6.32-100.0.19.el5 #1 SMP Fri Sep 17 17:51:41 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux 2 CPU, 4cores/cpu + hyperthreading = 16 virtual cores processor : 15 vendor_id : GenuineIntel cpu family : 6 model : 26 model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz stepping : 5 cpu MHz : 1600.000 cache size : 8192 KB [lidar@lidarora1 tmp]$ time laszippertest -n 1000000 (skipping range coder test) laszipper1 wrote 7000000 bytes in 0.03 seconds laszipper2 wrote 3898142 bytes in 0.13 seconds SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds SUCCESS: lasunzipper2 read 3898142 bytes in 0.18 seconds real 0m0.379s user 0m0.349s sys 0m0.029s [lidar@lidarora1 tmp]$ time laszippertest -n 1000000 (skipping range coder test) laszipper1 wrote 7000000 bytes in 0.03 seconds laszipper2 wrote 3898142 bytes in 0.13 seconds SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds SUCCESS: lasunzipper2 read 3898142 bytes in 0.17 seconds real 0m0.366s user 0m0.336s sys 0m0.029s [lidar@lidarora1 tmp]$ time laszippertest -n 1000000 (skipping range coder test) laszipper1 wrote 7000000 bytes in 0.03 seconds laszipper2 wrote 3898142 bytes in 0.13 seconds SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds SUCCESS: lasunzipper2 read 3898142 bytes in 0.16 seconds real 0m0.366s user 0m0.338s sys 0m0.026s [lidar@lidarora1 tmp]$ time laszippertest -n 1000000 -s (skipping range coder test) laszipper1 wrote 7000000 bytes in 0.03 seconds laszipper2 wrote 3898142 bytes in 0.13 seconds SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds SUCCESS: lasunzipper2 read 3898142 bytes in 0.16 seconds real 0m0.364s user 0m0.338s sys 0m0.024s [lidar@lidarora1 tmp]$ time laszippertest -n 1000000 -s (skipping range coder test) laszipper1 wrote 7000000 bytes in 0.03 seconds laszipper2 wrote 3898142 bytes in 0.13 seconds SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds SUCCESS: lasunzipper2 read 3898142 bytes in 0.17 seconds real 0m0.369s user 0m0.339s sys 0m0.028s [lidar@lidarora1 tmp]$ time laszippertest -n 1000000 -s (skipping range coder test) laszipper1 wrote 7000000 bytes in 0.03 seconds laszipper2 wrote 3898142 bytes in 0.13 seconds SUCCESS: lasunzipper1 read 7000000 bytes in 0.02 seconds SUCCESS: lasunzipper2 read 3898142 bytes in 0.17 seconds real 0m0.364s user 0m0.335s sys 0m0.027s -- Michael Smith Remote Sensing/GIS Center US Army Corps of Engineers On 2/4/11 11:39 AM, "Michael P. Gerlek" <[email protected]> wrote: Do you have 5-10 minutes to spare today? Your libLAS team (well, me anyway) is wondering about I/O performance of the liblas kit -- specifically, when doing binary reading and writing, is there any fundamental performance difference between using C-style FILE* I/O and C++-style stream I/O? And if streams are better, would boost's stream be better still? If you google around a bit, you'll find lots of contradictory (and sometimes overly passionate) statements about this topic. At the end of the day, though, the consensus seems to be that: (1) you need to be "smart" if you're using C++ I/O -- it is easy to shoot yourself in the foot (2) modern C++ streams are implemented on top of the native OS APIs (3) under Visual Studio, FILE* operations and streams are both implemented using the win32 APIs, but streams have an additional lock (that is claimed by some to be not needed) and, most importantly, (4) performance varies greatly with different I/O patterns, e.g. large sequential block reads vs small random reads Very fortunately, we happen to already have a rough, 1st-order I/O performance test built into the laszip tree. If you have that tree built (http://hg.liblas.org/zip), in Release mode, could you please send me the results of running the "laszippertest" test app, as follows? time ./laszippertest -n 1000000 time ./laszippertest -n 1000000 time ./laszippertest -n 1000000 time ./laszippertest -n 1000000 -s time ./laszippertest -n 1000000 -s time ./laszippertest -n 1000000 -s The first three runs will encode and decode 1 million random points using FILEs, and the second three will do it with streams. This is not a perfect test, but it represents something approximating the real I/O footprint or traces that liblas uses. Oh, and be sure to include the kind of platform (processor speed, compiler, OS) you're running it on. Thanks much! -mpg _______________________________________________ Liblas-devel mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/liblas-devel _______________________________________________ Liblas-devel mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/liblas-devel
_______________________________________________ Liblas-devel mailing list [email protected] http://lists.osgeo.org/mailman/listinfo/liblas-devel
