Re: On pool / memory usage debugging
On Mon, Dec 08, 2014 at 09:23:10PM +, Philip Martin wrote: Philip Martin philip.mar...@wandisco.com writes: I think this extra walk is pointless most of the time and is just making our nominally O(1) copy operation slower. Locks on the copy source do not prevent a copy. Oops! I meant to write Locks on the copy source held by somebody else do not prevent a copy. -- Philip Thanks for digging up the history. I was wondering too whether this walk is really needed but ended up assuming it had always been there. Should we raise an issue with mod_dav developers? (gstein?)
Re: On pool / memory usage debugging
On Mon, Dec 8, 2014 at 8:42 PM, Stefan Sperling s...@elego.de wrote: On Mon, Dec 08, 2014 at 08:30:31PM +0100, Stefan Fuhrmann Finally, to minimize cache usage, make sure to disable fulltext caching as well (enabled by default in 1.9) and set the cache size to *1*, not 0. The latter would fall back to 1.6-style caches, Which option are you referring? The SVNInMemoryCacheSize option? The doc for that option says 0 deactivates the cache. Is this an error? /* per server */ AP_INIT_TAKE1(SVNInMemoryCacheSize, SVNInMemoryCacheSize_cmd, NULL, RSRC_CONF, specifies the maximum size in kB per process of Subversion's in-memory object cache (default value is 16384; 0 deactivates the cache).), It's technically correct, because the static per-process cache does get disabled and all that's left is memory dynamically allocated for an open connection (svn_fs_t, actually). But the doc string is misleading. I would suggest to change it to 0 switches to dynamically sized caches. Yes please, that would help a lot! I can never remember what the various caching knobs do. The clearer the documentation, the better :) Done in r1644035. -- Stefan^2.
Re: On pool / memory usage debugging
Stefan Sperling s...@elego.de writes: Should we raise an issue with mod_dav developers? (gstein?) I suspect any mod_dav fixes are probably going to be written by us. I think the fix might be to have dav_validate_request() identify the conditions that mean dav_validate_walker() will always return NULL, i.e. a copy with no if header, and skip the walk. -- Philip Martin | Subversion Committer WANdisco // *Non-Stop Data*
Re: On pool / memory usage debugging
On Mon, Dec 08, 2014 at 06:42:38PM +0100, Stefan Fuhrmann wrote: IOW, pool debugging is nice for tracing allocations but if you want to measure memory consumption on the OS side, turn pool debugging off. All measurements I mentioned in the issue were done with pool debugging disabled. Measuring memory usage of the issue #4531 copy operation with pool debugging enabled was impossible because the copy operation never completed in a reasonable amount of time due to pool-debugging-induced logging overhead hogging the CPU. Finally, to minimize cache usage, make sure to disable fulltext caching as well (enabled by default in 1.9) and set the cache size to *1*, not 0. The latter would fall back to 1.6-style caches, Which option are you referring? The SVNInMemoryCacheSize option? The doc for that option says 0 deactivates the cache. Is this an error? /* per server */ AP_INIT_TAKE1(SVNInMemoryCacheSize, SVNInMemoryCacheSize_cmd, NULL, RSRC_CONF, specifies the maximum size in kB per process of Subversion's in-memory object cache (default value is 16384; 0 deactivates the cache).),
Re: On pool / memory usage debugging
Stefan Fuhrmann stefan.fuhrm...@wandisco.com writes: This post has been prompted by issue 4531 and r1643834 (http://subversion.tigris.org/issues/show_bug.cgi?id=4531 http://svn.apache.org/viewvc?view=revisionrevision=r1643834). IOW, pool debugging is nice for tracing allocations but if you want to measure memory consumption on the OS side, turn pool debugging off. I have been looking at the proposed 1.8/1.7 backport for issue 4531. The problem is easy to reproduce with pool debugging enabled and the patch does reduce memory use, but with a normal build I don't see the excessive memory in the first place. The runtime of the copy is similarly affected: it is negligible with a normal build but takes several seconds when pool debugging is enabled. Was this issue raised in response to a problem observed in a normal build? Can the problem be reproduced in a normal build? Perhaps the large tree produced by the script in the issue is not large enough to cause the problem in a normal build? -- Philip Martin | Subversion Committer WANdisco // *Non-Stop Data*
Re: On pool / memory usage debugging
Stefan Sperling s...@elego.de writes: Which option are you referring? The SVNInMemoryCacheSize option? The doc for that option says 0 deactivates the cache. Is this an error? http://subversion.apache.org/docs/release-notes/1.7.html#data-caches Please note that a cache size of 0 will deactivate the new caching facilities and cause the server to fall back to 1.6 caching mechanisms. -- Philip Martin | Subversion Committer WANdisco // *Non-Stop Data*
Re: On pool / memory usage debugging
On Mon, Dec 8, 2014 at 7:46 PM, Stefan Sperling s...@elego.de wrote: On Mon, Dec 08, 2014 at 06:42:38PM +0100, Stefan Fuhrmann wrote: IOW, pool debugging is nice for tracing allocations but if you want to measure memory consumption on the OS side, turn pool debugging off. All measurements I mentioned in the issue were done with pool debugging disabled. Measuring memory usage of the issue #4531 copy operation with pool debugging enabled was impossible because the copy operation never completed in a reasonable amount of time due to pool-debugging-induced logging overhead hogging the CPU. Hm. 381 MB are massive, then. Maybe I can find reproduce it and help tracking it down with a modified APR. Finally, to minimize cache usage, make sure to disable fulltext caching as well (enabled by default in 1.9) and set the cache size to *1*, not 0. The latter would fall back to 1.6-style caches, Which option are you referring? The SVNInMemoryCacheSize option? The doc for that option says 0 deactivates the cache. Is this an error? /* per server */ AP_INIT_TAKE1(SVNInMemoryCacheSize, SVNInMemoryCacheSize_cmd, NULL, RSRC_CONF, specifies the maximum size in kB per process of Subversion's in-memory object cache (default value is 16384; 0 deactivates the cache).), It's technically correct, because the static per-process cache does get disabled and all that's left is memory dynamically allocated for an open connection (svn_fs_t, actually). But the doc string is misleading. I would suggest to change it to 0 switches to dynamically sized caches. -- Stefan^2.
Re: On pool / memory usage debugging
On Mon, Dec 08, 2014 at 07:19:57PM +, Philip Martin wrote: I have been looking at the proposed 1.8/1.7 backport for issue 4531. The problem is easy to reproduce with pool debugging enabled and the patch does reduce memory use, but with a normal build I don't see the excessive memory in the first place. That's surprising. Was this issue raised in response to a problem observed in a normal build? Can the problem be reproduced in a normal build? Perhaps the large tree produced by the script in the issue is not large enough to cause the problem in a normal build? It was found and reproduced first with a set of CollabNet SVN 1.8 binaries, both stand-alone and Edge ones, where it was crashing the server which evidently maxed out memory as displayed by top(1) on Linux. The original repository was much larger than the sample created by the gentree script, though, in the order of 40GB in total size. I'm not sure which repository format exactly. As mentioned in the issue I can reproduce the problem with a trunk build. The copy doesn't abort but uses roughly 2GB of memory before it completes.
Re: On pool / memory usage debugging
On Mon, Dec 08, 2014 at 08:30:31PM +0100, Stefan Fuhrmann wrote: Hm. 381 MB are massive, then. Maybe I can find reproduce it and help tracking it down with a modified APR. Yes, we should be able to manage with much less. Though perhaps what's left is in in mod_dav rather than mod_dav_svn. Finally, to minimize cache usage, make sure to disable fulltext caching as well (enabled by default in 1.9) and set the cache size to *1*, not 0. The latter would fall back to 1.6-style caches, Which option are you referring? The SVNInMemoryCacheSize option? The doc for that option says 0 deactivates the cache. Is this an error? /* per server */ AP_INIT_TAKE1(SVNInMemoryCacheSize, SVNInMemoryCacheSize_cmd, NULL, RSRC_CONF, specifies the maximum size in kB per process of Subversion's in-memory object cache (default value is 16384; 0 deactivates the cache).), It's technically correct, because the static per-process cache does get disabled and all that's left is memory dynamically allocated for an open connection (svn_fs_t, actually). But the doc string is misleading. I would suggest to change it to 0 switches to dynamically sized caches. Yes please, that would help a lot! I can never remember what the various caching knobs do. The clearer the documentation, the better :)
Re: On pool / memory usage debugging
Philip Martin philip.mar...@wandisco.com writes: I'm still trying to determine why this extra walk is present, perhaps the new version is doing an unnecessary walk or perhaps the old version has an unfixed bug. Debian's httpd doesn't have r1497441 which introduced the extra walk for copies, apparently to fix PR 54610. This was followed by r1515569 and r1540728 which changed the checks on the copy source but did not remove the walk. -- Philip Martin | Subversion Committer WANdisco // *Non-Stop Data*
Re: On pool / memory usage debugging
Philip Martin philip.mar...@wandisco.com writes: I think this extra walk is pointless most of the time and is just making our nominally O(1) copy operation slower. Locks on the copy source do not prevent a copy. Oops! I meant to write Locks on the copy source held by somebody else do not prevent a copy. -- Philip
Re: On pool / memory usage debugging
Philip Martin philip.mar...@wandisco.com writes: My pool-debug build uses my own httpd build from 2.2.x@1562432 while my normal build used the system httpd, Debian's 2.2.22-13+deb7u3. It looks like some change to mod_dav has added an extra walk over the copy source in the pool-debug build and the absence of this walk means there is no problem in my normal build. I'm still trying to determine why this extra walk is present, perhaps the new version is doing an unnecessary walk or perhaps the old version has an unfixed bug. I think this extra walk is pointless most of the time and is just making our nominally O(1) copy operation slower. Locks on the copy source do not prevent a copy. This walk appears to be checking that locks (ETags?) supplied by the COPY requests match the copy source. On an URL-URL copy a Subversion client will not supply any tokens, so the walk scans the whole tree for no point. The walk repeatedly calls back into mod_dav's dav_validate_resource_state() and it always returns NULL as DAV_VALIDATE_NO_MODIFY is set and if_header is NULL. -- Philip Martin | Subversion Committer WANdisco // *Non-Stop Data*
