Hi, did you re-run the tests after making the change? The test cases for re-harvesting an existing file, at least those in the netcdf module, should fail now.
The reason why we remove granules is that when handling mosaics of multidimensional files (NetCDF/Grib/...) that are updated regularly, there is a common case of having daily files, and "today"'s file keeps on getting new slices and it's re-harvested multiple times a day, when that happens the slices collected from the previous version of the same file need to be removed. Another common case is having to add back a file that was ruined, incomplete, or erroneous, with a fixed version, that might result in some attributes fetched by property collectors to change. So, the code cannot be removed, but it could conceivably be limited to the harvest case, leaving the first pass during indexing free of these checks You might want to open a ticket requesting for such improvement. Cheers Andrea On Wed, Aug 3, 2016 at 11:16 AM, GeoUser <[email protected]> wrote: > Hey! > > I was building a mosaic index on a fairly large tiff collection and the > progress was not "linear". It seemed as if the more files get added to the > index, the slower the process became. So i did some profiling and the > profiler pointed out that the majority of time was being spent in the > CatalogManager.updateCatalog, specifically this part of the code: > > // drop all the granules associated to the same > Filter filter = Utils.FF.equal(Utils.FF.property(locationAttribute), > Utils.FF.literal(fileLocation), > !isCaseSensitiveFileSystem(fileBeingProcessed)); > store.removeGranules(filter); > > I did not have time to investigate further, so if anyone here knows why do > we have to remove the granules for every file added, please explain. > > I also compiled a version with these lines commented out and the generated > mosaic was correct (and done much much faster). > > I was testing this on geotools 14.3. > > Cheers! > > > > -- > View this message in context: > http://osgeo-org.1560.x6.nabble.com/slow-mosaic-index-builder-tp5279228.html > Sent from the geotools-gt2-users mailing list archive at Nabble.com. > > > ------------------------------------------------------------------------------ > _______________________________________________ > GeoTools-GT2-Users mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users > -- == GeoServer Professional Services from the experts! Visit http://goo.gl/it488V for more information. == Ing. Andrea Aime @geowolf Technical Lead GeoSolutions S.A.S. Via di Montramito 3/A 55054 Massarosa (LU) phone: +39 0584 962313 fax: +39 0584 1660272 mob: +39 339 8844549 http://www.geo-solutions.it http://twitter.com/geosolutions_it *AVVERTENZE AI SENSI DEL D.Lgs. 196/2003* Le informazioni contenute in questo messaggio di posta elettronica e/o nel/i file/s allegato/i sono da considerarsi strettamente riservate. Il loro utilizzo è consentito esclusivamente al destinatario del messaggio, per le finalità indicate nel messaggio stesso. Qualora riceviate questo messaggio senza esserne il destinatario, Vi preghiamo cortesemente di darcene notizia via e-mail e di procedere alla distruzione del messaggio stesso, cancellandolo dal Vostro sistema. Conservare il messaggio stesso, divulgarlo anche in parte, distribuirlo ad altri soggetti, copiarlo, od utilizzarlo per finalità diverse, costituisce comportamento contrario ai principi dettati dal D.Lgs. 196/2003. The information in this message and/or attachments, is intended solely for the attention and use of the named addressee(s) and may be confidential or proprietary in nature or covered by the provisions of privacy act (Legislative Decree June, 30 2003, no.196 - Italy's New Data Protection Code).Any use not in accord with its purpose, any disclosure, reproduction, copying, distribution, or either dissemination, either whole or partial, is strictly forbidden except previous formal approval of the named addressee(s). If you are not the intended recipient, please contact immediately the sender by telephone, fax or e-mail and delete the information in this message that has been received in error. The sender does not give any warranty or accept liability as the content, accuracy or completeness of sent messages and accepts no responsibility for changes made after they were sent or for other risks which arise as a result of e-mail transmission, viruses, etc. -------------------------------------------------------
------------------------------------------------------------------------------
_______________________________________________ GeoTools-GT2-Users mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/geotools-gt2-users
