Thank you very much for the answer. I have a lot to learn! Yes, my question was under the assumption that small chunks would make the program run slower. :). I tried setting the OTB_MAX_RAM_HINT to 4000, and could see an improvement (~50 %)!
leo@leo-ThinkPad-E480:~/devel/otbapps/image_segmentation/build$ export OTB_MAX_RAM_HINT=128 leo@leo-ThinkPad-E480:~/devel/otbapps/image_segmentation/build$ time ./ConnectedThresholdImageFilter_NC ../test_img/fire_test.tif /tmp/example1.tif 2073 2372 200 1500 2072 2371 2018-05-16 20:28:59 (INFO): No kwl metadata found in file ../test_img/fire_test.tif 2018-05-16 20:28:59 (INFO): Default RAM limit for OTB is 128 MB 2018-05-16 20:28:59 (INFO): GDAL maximum cache size is 393 MB 2018-05-16 20:28:59 (INFO): OTB will use at most 8 threads 2018-05-16 20:28:59 (INFO): Estimated memory for full processing: 3.94151e+06MB (avail.: 128 MB), optimal image partitioning: 30794 blocks 2018-05-16 20:28:59 (INFO): File /tmp/example1.tif will be written in 32396 blocks of 43x43 pixels real 0m20.951s user 1m3.094s sys 0m9.747s leo@leo-ThinkPad-E480:~/devel/otbapps/image_segmentation/build$ export OTB_MAX_RAM_HINT=4000 leo@leo-ThinkPad-E480:~/devel/otbapps/image_segmentation/build$ time ./ConnectedThresholdImageFilter_NC ../test_img/fire_test.tif /tmp/example1.tif 2073 2372 200 1500 2072 2371 2018-05-16 20:29:35 (INFO): No kwl metadata found in file ../test_img/fire_test.tif 2018-05-16 20:29:35 (INFO): Default RAM limit for OTB is 4000 MB 2018-05-16 20:29:35 (INFO): GDAL maximum cache size is 393 MB 2018-05-16 20:29:35 (INFO): OTB will use at most 8 threads 2018-05-16 20:29:35 (INFO): Estimated memory for full processing: 3.94151e+06MB (avail.: 4000 MB), optimal image partitioning: 986 blocks 2018-05-16 20:29:35 (INFO): File /tmp/example1.tif will be written in 1830 blocks of 256x128 pixels real 0m13.211s user 0m59.665s sys 0m1.873s I was checking htop and the process did not take 4000 mb of ram but only 1200 mb. Also, what does GDAL cache size do? The logs are very useful! Great addition! I am actually using the master version (self compiled) on ubuntu 18.04, because I tried the superbuild and it failed. I can try to reproduce the failure and let you know what it was. As I mentioned, the code I am using is https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/blob/develop/Examples/Segmentation/ConnectedThresholdImageFilter.cxx Do you think it could run faster by tweaking other parameters? The plan is to replace an old IDL script that is used in my workplace to aid in the mapping of bushfires. Ideally I would like to implement it as a QGIS plugin, but the to convince my supervisor it needs to run as fast as the IDL code, which takes about 4 second to run. Thanks again! On Wednesday, May 16, 2018 at 4:18:53 PM UTC+10, Julien Michel wrote: > > Hi, > > > > You can add a RAM parameter to your application that will allow you to > control this (AddRamParameter()). You can also set the OTB_MAX_RAM_HINT > environment variable (this will be documented in cookbook for next release > [1]. > > > > Looking at your logs (btw those logs are pretty new in OTB, congrats for > using a bleeding edge version ;)), OTB is splitting the data into very > small chunks (one line at a time). This is likely to be slower than larger > chunks. To get larger chunks, you can increase the RAM parameter, but you > can even set the chunk size yourself using extended filenames [1]. > > > > How long is the inner pipeline of your application ? If OTB decides that > it can only process one line at a time with 128 Mb, I expect the inner > pipeline to be quite long (or include non-streamed filters). > > > > Hope that helps, > > > > Julien > > > > [1] > https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/blob/develop/Documentation/Cookbook/rst/AdvancedUse.rst > > > > *De :* otb-...@googlegroups.com <javascript:> [mailto: > otb-...@googlegroups.com <javascript:>] *De la part de* Leonardo Hardtke > *Envoyé :* mercredi 16 mai 2018 06:47 > *À :* otb-users <otb-...@googlegroups.com <javascript:>> > *Objet :* [otb-users] How to change default RAM limit? > > > > Dear all, > > I am starting a new project and was investigating the likelihood of doing > it in OTB. I used otb apps from python many times before but never tried to > write my own apps. Part of my application will be based on the code > /Examples/Segmentation/ConnectedThresholdImageFilter.cxx > > > > It compiles and runs OK, but it is a little bit slow (8 secs) for what I > need (interactive application with Landsat images [7591, 7731]). The info > shows it uses only 128 Mb of RAM while I have at least 4 Gb of RAM free: > > > > 2018-05-16 14:33:10 (INFO): No kwl metadata found in file fire_test.tif > 2018-05-16 14:33:10 (INFO): Default RAM limit for OTB is 128 MB > 2018-05-16 14:33:10 (INFO): GDAL maximum cache size is 393 MB > 2018-05-16 14:33:10 (INFO): OTB will use at most 8 threads > 2018-05-16 14:33:10 (INFO): Estimated memory for full processing: > 3.94151e+06MB (avail.: 128 MB), optimal image partitioning: 30794 blocks > 2018-05-16 14:33:10 (INFO): File fire_segment.tif will be written in 30924 > blocks of 1898x1 pixels > > > > How do I configure the amount of RAM that the application uses? > > > > Thanks in advance! > > > > -- > -- > Check the OTB FAQ at > http://www.orfeo-toolbox.org/FAQ.html > > You received this message because you are subscribed to the Google > Groups "otb-users" group. > To post to this group, send email to otb-...@googlegroups.com > <javascript:> > To unsubscribe from this group, send email to > otb-users+...@googlegroups.com <javascript:> > For more options, visit this group at > http://groups.google.com/group/otb-users?hl=en > --- > You received this message because you are subscribed to the Google Groups > "otb-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to otb-users+...@googlegroups.com <javascript:>. > For more options, visit https://groups.google.com/d/optout. > -- -- Check the OTB FAQ at http://www.orfeo-toolbox.org/FAQ.html You received this message because you are subscribed to the Google Groups "otb-users" group. To post to this group, send email to otb-users@googlegroups.com To unsubscribe from this group, send email to otb-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/otb-users?hl=en --- You received this message because you are subscribed to the Google Groups "otb-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to otb-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
