Sure it helps! Thank you very much. Cheers.
On Friday, 18 May 2018 02:32:27 UTC+10, Julien Michel wrote: > > GDAL_CACHEMAX is the size of gdal cache to store blocks it already reads > for further use. Depending on you processing chain, it may or may not be > relevant. > > > > You can try setting the chunk size yourself : > > > > “/tmp/example1.tif?&streaming:sizemode=nbsplits&streaming:sizevalue=1” > > > > Regards, > > > > Julien > > > > > > > > > > > > *De :* otb-...@googlegroups.com <javascript:> [mailto: > otb-...@googlegroups.com <javascript:>] *De la part de* Leonardo Hardtke > *Envoyé :* mercredi 16 mai 2018 12:42 > *À :* otb-users <otb-...@googlegroups.com <javascript:>> > *Objet :* Re: [otb-users] How to change default RAM limit? > > > > Thank you very much for the answer. I have a lot to learn! > > > > Yes, my question was under the assumption that small chunks would make the > program run slower. :). I tried setting the OTB_MAX_RAM_HINT to 4000, and > could see an improvement (~50 %)! > > > > leo@leo-ThinkPad-E480:~/devel/otbapps/image_segmentation/build$ export > OTB_MAX_RAM_HINT=128 > leo@leo-ThinkPad-E480:~/devel/otbapps/image_segmentation/build$ time > ./ConnectedThresholdImageFilter_NC ../test_img/fire_test.tif > /tmp/example1.tif 2073 2372 200 1500 2072 2371 > 2018-05-16 20:28:59 (INFO): No kwl metadata found in file > ../test_img/fire_test.tif > 2018-05-16 20:28:59 (INFO): Default RAM limit for OTB is 128 MB > 2018-05-16 20:28:59 (INFO): GDAL maximum cache size is 393 MB > 2018-05-16 20:28:59 (INFO): OTB will use at most 8 threads > 2018-05-16 20:28:59 (INFO): Estimated memory for full processing: > 3.94151e+06MB (avail.: 128 MB), optimal image partitioning: 30794 blocks > 2018-05-16 20:28:59 (INFO): File /tmp/example1.tif will be written in > 32396 blocks of 43x43 pixels > > real 0m20.951s > user 1m3.094s > sys 0m9.747s > leo@leo-ThinkPad-E480:~/devel/otbapps/image_segmentation/build$ export > OTB_MAX_RAM_HINT=4000 > leo@leo-ThinkPad-E480:~/devel/otbapps/image_segmentation/build$ time > ./ConnectedThresholdImageFilter_NC ../test_img/fire_test.tif > /tmp/example1.tif 2073 2372 200 1500 2072 2371 > 2018-05-16 20:29:35 (INFO): No kwl metadata found in file > ../test_img/fire_test.tif > 2018-05-16 20:29:35 (INFO): Default RAM limit for OTB is 4000 MB > 2018-05-16 20:29:35 (INFO): GDAL maximum cache size is 393 MB > 2018-05-16 20:29:35 (INFO): OTB will use at most 8 threads > 2018-05-16 20:29:35 (INFO): Estimated memory for full processing: > 3.94151e+06MB (avail.: 4000 MB), optimal image partitioning: 986 blocks > 2018-05-16 20:29:35 (INFO): File /tmp/example1.tif will be written in 1830 > blocks of 256x128 pixels > > real 0m13.211s > user 0m59.665s > sys 0m1.873s > > > > I was checking htop and the process did not take 4000 mb of ram but only > 1200 mb. Also, what does GDAL cache size do? > > > > The logs are very useful! Great addition! I am actually using the master > version (self compiled) on ubuntu 18.04, because I tried the superbuild and > it failed. I can try to reproduce the failure and let you know what it was. > > > > As I mentioned, the code I am using is > https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/blob/develop/Examples/Segmentation/ConnectedThresholdImageFilter.cxx > > > > Do you think it could run faster by tweaking other parameters? > > > > The plan is to replace an old IDL script that is used in my workplace to > aid in the mapping of bushfires. Ideally I would like to implement it as a > QGIS plugin, but the to convince my supervisor it needs to run as fast as > the IDL code, which takes about 4 second to run. > > > > Thanks again! > > > > > > > > > > > > > On Wednesday, May 16, 2018 at 4:18:53 PM UTC+10, Julien Michel wrote: > > Hi, > > > > You can add a RAM parameter to your application that will allow you to > control this (AddRamParameter()). You can also set the OTB_MAX_RAM_HINT > environment variable (this will be documented in cookbook for next release > [1]. > > > > Looking at your logs (btw those logs are pretty new in OTB, congrats for > using a bleeding edge version ;)), OTB is splitting the data into very > small chunks (one line at a time). This is likely to be slower than larger > chunks. To get larger chunks, you can increase the RAM parameter, but you > can even set the chunk size yourself using extended filenames [1]. > > > > How long is the inner pipeline of your application ? If OTB decides that > it can only process one line at a time with 128 Mb, I expect the inner > pipeline to be quite long (or include non-streamed filters). > > > > Hope that helps, > > > > Julien > > > > [1] > https://gitlab.orfeo-toolbox.org/orfeotoolbox/otb/blob/develop/Documentation/Cookbook/rst/AdvancedUse.rst > > > > *De :* otb-...@googlegroups.com [mailto:otb-...@googlegroups.com] *De la > part de* Leonardo Hardtke > *Envoyé :* mercredi 16 mai 2018 06:47 > *À :* otb-users <otb-...@googlegroups.com> > *Objet :* [otb-users] How to change default RAM limit? > > > > Dear all, > > I am starting a new project and was investigating the likelihood of doing > it in OTB. I used otb apps from python many times before but never tried to > write my own apps. Part of my application will be based on the code > /Examples/Segmentation/ConnectedThresholdImageFilter.cxx > > > > It compiles and runs OK, but it is a little bit slow (8 secs) for what I > need (interactive application with Landsat images [7591, 7731]). The info > shows it uses only 128 Mb of RAM while I have at least 4 Gb of RAM free: > > > > 2018-05-16 14:33:10 (INFO): No kwl metadata found in file fire_test.tif > 2018-05-16 14:33:10 (INFO): Default RAM limit for OTB is 128 MB > 2018-05-16 14:33:10 (INFO): GDAL maximum cache size is 393 MB > 2018-05-16 14:33:10 (INFO): OTB will use at most 8 threads > 2018-05-16 14:33:10 (INFO): Estimated memory for full processing: > 3.94151e+06MB (avail.: 128 MB), optimal image partitioning: 30794 blocks > 2018-05-16 14:33:10 (INFO): File fire_segment.tif will be written in 30924 > blocks of 1898x1 pixels > > > > How do I configure the amount of RAM that the application uses? > > > > Thanks in advance! > > > > -- > -- > Check the OTB FAQ at > http://www.orfeo-toolbox.org/FAQ.html > > You received this message because you are subscribed to the Google > Groups "otb-users" group. > To post to this group, send email to otb-...@googlegroups.com > To unsubscribe from this group, send email to > otb-users+...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/otb-users?hl=en > --- > You received this message because you are subscribed to the Google Groups > "otb-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to otb-users+...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > -- > -- > Check the OTB FAQ at > http://www.orfeo-toolbox.org/FAQ.html > > You received this message because you are subscribed to the Google > Groups "otb-users" group. > To post to this group, send email to otb-...@googlegroups.com > <javascript:> > To unsubscribe from this group, send email to > otb-users+...@googlegroups.com <javascript:> > For more options, visit this group at > http://groups.google.com/group/otb-users?hl=en > --- > You received this message because you are subscribed to the Google Groups > "otb-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to otb-users+...@googlegroups.com <javascript:>. > For more options, visit https://groups.google.com/d/optout. > -- -- Check the OTB FAQ at http://www.orfeo-toolbox.org/FAQ.html You received this message because you are subscribed to the Google Groups "otb-users" group. To post to this group, send email to otb-users@googlegroups.com To unsubscribe from this group, send email to otb-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/otb-users?hl=en --- You received this message because you are subscribed to the Google Groups "otb-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to otb-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
