Thanks, Remi. That's what I imagined. In fact, I processed them using two subsets and using two threads but the results were not satisfactory. Your splitting strategy makes much more sense of course.
On Saturday, September 29, 2018 at 12:37:45 PM UTC+2, Rémi wrote: > > Hello Gus, > > The logs help, thanks. > The application crashes when the statistics are summarized, because there > is not enough memory to store the mutual statistics for your 3582 images. > > In short, the mutual statistics between couple of images (i,j) are stored > in matrices of size NxN (N = 3582 images in your case). > Each matrix holds something useful (e.g. pixel count, pixel sum, ...) for > each band. > In the end, all these matrices occupies a lot of memory! that's why the > Mosaic application consumes so much RAM when you enable stats computations. > > You have different ways for solving this problem (alternatively to run on > a computer with more RAM!): > 1. Give a try with a single thread. This will reduce the number of > internal accumulators for statistics (matrices) because each thread has its > own. To do that, set the ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS to 1 (type > "export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=1" in terminal before > running the command). But... this will last 4 times longer that your > precedent try, something like 25 hours instead of 6 just for the stats > computation. But you should save about 6Gb (approx for 3k images). However, > let's hope that the later quadratic optimization step, which also > requires to allocate space for matrix, will not fail. > > [image: path4668-3.png] > 2. Split your dataset in two (or more, if two isn't enough) the right way. > You can theoretically obtain the exact same result as if you process the > whole dataset. For this, you need to split the dataset the right way to > make the quadratic optimization equivalent. This can be done in finding a > number of overlaping images that are connected only with one single image > (i.e. this image would have 2 overlap with the adjacent images, and those 2 > adjacent images does not intersect). Then you process independently these > two split independently (using the same harmo/comp options) and later you > can process the mosicing of those two resulting mosaics (using the same > harmo/options as you did for each one). The harmonization should be > equivalent. See attached figure > > Hope that helps ! > > Le samedi 29 septembre 2018 00:07:15 UTC+2, Gus a écrit : >> >> Ok. Here there are the logs and the output of gdalinfo on some random >> file (all them were processed with gdal_translate). >> The harmonization pipeline failed. During statistics evaluation it showed >> a progressively increasing memory footprint. I add a capture when it failed. >> >> [image: Screenshot at 2018-09-28 23-57-06.png] >> Thanks >> >> -- -- Check the OTB FAQ at http://www.orfeo-toolbox.org/FAQ.html You received this message because you are subscribed to the Google Groups "otb-users" group. To post to this group, send email to otb-users@googlegroups.com To unsubscribe from this group, send email to otb-users+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/otb-users?hl=en --- You received this message because you are subscribed to the Google Groups "otb-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to otb-users+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
