Thinking further ... listening to "gwc-layers/*" is a poor idea and unnecessary.
Since we completely know the names of these files (from their matching layer objects). There should be no need to scan the directory for changes ... we have enough information to do this file by file which should be more efficient when using opening system watcher. Is it possible to detect when we are only of these mapped file systems in order to warn the user that a different approach will be required to keep nodes in the cluster advised of changes? -- Jody Garnett On Fri, 18 Oct 2019 at 07:12, Gabriel Roldan <[email protected]> wrote: > > > On Fri, 18 Oct 2019 at 10:17, Jody Garnett <[email protected]> wrote: > >> That is ... unexpected. >> >> So there are other solutions to notify tile cache state across the >> cluster right. Are they more appropriate then depending on file system >> polling here? >> > > I would suspect so. The catalog does so it should be possible to do the > same with the tile layer catalog. Meanwhile I wouldn't mind this > improvement. > > >> >> On Thu, Oct 17, 2019 at 11:27 PM Gabriel Roldan <[email protected]> >> wrote: >> >>> >>> >>> On Thu, 17 Oct 2019 at 13:46, Jody Garnett <[email protected]> >>> wrote: >>> >>>> Bad news is FileSystemWatcher is killing the machine. For each instance >>>>> it gets one thread eating 100% of one CPU all the time. Reported this >>>>> before, just sharing my pain. >>>>> >>>>> Been looking at replacing the internals of FileSystemWatcher to use >>>>> java.nio.file.WatchService, and although at a first glance it looks pretty >>>>> straightforward, it's not gonna be that simple. >>>>> >>>> >>>> Darn, that was the point of making the interface (since we could not >>>> use WatchService on Java 7 yet). >>>> >>>> >>>>> There are a number of situations where it just won't work, basically >>>>> when the data dir is on any sort of mapped filesystem (NFS, SAMBA, Docker >>>>> bind mounts), for which the Linux implementation doesn't work as it relies >>>>> on inotify. >>>>> >>>>> The MacOS implementation is a polling one, but it's gonna be better >>>>> than the current custom one anyways. And for Windows, I don't know yet. >>>>> >>>> >>>> Still this was only supposed to be used for a few files like the >>>> security settings which require active notification? If a watcher is being >>>> setup beyond that ... is it a mistake? >>>> >>>> In any case it's clear that anyone that has a large catalog will most >>>>> probably have it on a shared filesystem to be used by several instances. >>>>> So this might require a GSIP, which I'm willing to start, in order to >>>>> tackle on the different scenarios, as it can get even more involved as >>>>> above described. >>>>> >>>> >>>> Before you go that far can we talk about how FileSystemWatcher is being >>>> used... >>>> >>> >>> Right. It's DefaultTileLayerCatalog listening to changes to gwc-layers/* >>> in order to update tile layer configs when they're updated externally on a >>> cluster (added at commit 966a8af90fc6b68725a78b09829c3cbc7360242e). >>> >>> For the time being, I managed to reduce the polling runtime from over 2 >>> minutes to ~350 milliseconds, the CPU usage from constant 100% to peaks >>> bellow 25% (or so says htop). Memory usage: old gen up to 477MB from 450MB, >>> eden space peaks and down quickly of 1GB instead of fixed at around 850MB. >>> >>> Got some clean up to do before issueing a PR, but here it is: >>> >>> https://github.com/groldan/geoserver/commit/0ba1614a7f9b1b2bea5647160441f3bd534c9979 >>> >>> Gabriel >>> >>>> >>>> -- >>>> Jody >>>> >>> >>> >>> -- >>> Gabriel Roldán >>> >> -- >> -- >> Jody Garnett >> > > > -- > Gabriel Roldán >
_______________________________________________ Geoserver-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/geoserver-devel
