I can verify that ML 8.07 resolves the memory leak in the profiler. I can now profile 100s of 1000s of tasks no problem.
Cheers, E. -- Eliot Kimber http://contrext.com On 8/28/17, 1:41 PM, "[email protected] on behalf of Eliot Kimber" <[email protected] on behalf of [email protected]> wrote: Thanks—I should be able to test with latest ML 8 in a couple of days. Cheers, E. -- Eliot Kimber http://contrext.com On 8/28/17, 12:37 PM, "[email protected] on behalf of Christopher Hamlin" <[email protected] on behalf of [email protected]> wrote: There was a bug where, under certain circumstances, the profiler will result in a query deadlock &/or a resource leak (#45569). It could be that this is what you are seeing. It was noticed in 8.0-2 and is fixed in the latest release (8.0-7). On Mon, Aug 28, 2017 at 1:11 PM, Eliot Kimber <[email protected]> wrote: > I reported earlier that my profiling application was causing MarkLogic to restart after handling about 20,000 tasks. Turns out it was an out-of-memory issue on the server itself (currently configured with 256GB of RAM). We could see a distinct spike in memory usage, at which point the server restarted MarkLogic. I tried different input data sets so it doesn’t appear to be an issue with a particular input document (my data set has a few outliers that are much larger than typical but only a few). > > Subsequent testing determined that it was the use of the MarkLogic profiler that was causing the memory spike: if I turned off the profiler then memory usage was flat and all the tasks completed as expected. > > This is ML 8.03. I’m still working on getting my server upgraded to a newer version of MarkLogic so I can see if this is an issue that has already been fixed. > > So it looks like there’s some kind of memory leak related to the profiler and I’d like to understand what that issue and either understand how to avoid it or report it formally. > > If it’s a general potential problem with large-scale processing would like to understand how to avoid it or plan for it. If it’s a problem specific to the profiler then need to report it formally and provide appropriate diagnostics. > > So my questions: > > 1. Is this a known issue with profiling? I’m guessing not in that I’m probably doing something out-of-the-ordinary vis-à-vis profiling and is something that nobody would see in typical single-instance ad-hoc profiling. > 2. What types of MarkLogic processing would cause this kind of memory spike that lasts across the execution of multiple tasks? I would expect the memory required for a given task to be released as soon as the task is complete so I’m guessing it must be an issue with caches or something? > > Thanks, > > Eliot > -- > Eliot Kimber > http://contrext.com > > > > > _______________________________________________ > General mailing list > [email protected] > Manage your subscription at: > http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general _______________________________________________ General mailing list [email protected] Manage your subscription at: http://developer.marklogic.com/mailman/listinfo/general
