Re: AuthzSVNAccessFile size issues
On Thu, Mar 22, 2012 at 01:49:44PM -0700, Joel Eidsath wrote: For a longer term solution, I am looking at contributing some code to mod_authz_svn to 1. allow rule caching, or 2. provide some sort of db support. I would be interested in reviewing patches that add some sort of multi-backend support to our authz system. This problem comes up quite often especially among enterprise users of Subversion. Some would prefer to store authz rules directly in databases like LDAP, for example. A multi-backend mechanism for authz, of which the plaintext file is one possible backend, would certainly make sense.
Re: AuthzSVNAccessFile size issues
On Thu, Mar 22, 2012 at 9:11 AM, Joel Eidsath jeids...@gmail.com wrote: I was just handed a large SVN install with thousands of users and hundreds of individual repositories. It is experiencing serious performance issues. I believe that it mostly boils down to a 14MB AuthzSVNAccessFile. What can I do to speed this up? Is there a database solution to use instead of the flatfile? Can I implement caching somehow? I am willing to code something up if I have to. Thanks for any help! Please CC me on any replies. Joel Eidsath Joel, Copying Dave's reply from an earlier thread this week. Sounds like it would be really helpful for you: -- Forwarded message -- From: David Weintraub qazw...@gmail.com Date: Tue, Mar 20, 2012 at 6:44 PM Subject: Re: preventing commits (this is *not* a classic hook question) To: mich...@huettermann.net Cc: bob.arc...@amsi.com, lesmikes...@gmail.com, nka...@gmail.com, users@subversion.apache.org I have a pre-commit hook that stores its configuration inside your repository. You'll need access to the Subversion server to set it up, but once it's setup, you can control access by checking out the control file from the repository, making your changes, and then checking it back in. This is a modification of a hook that I've been using for years. Originally, the control file was kept on the server -- usually inside the hooks directory. However, that meant logging onto the server, and that was getting too difficult to do all the time. Besides, this way, I can track who changed the control file and why. What prevents anyone from changing the control file? The control file is configured, so only the administrators can modify it. You can take a look at it at https://github.com/qazwart/SVN-Precommit-Kitchen-Sink-Hook. (Yes, a Subversion hook is stored in GitHub). -- This email, including any attachments, is for the sole use of the intended recipient and may contain confidential information. If you are not the intended recipient, please immediately notify us by reply email or by telephone, delete this email and destroy any copies. Thank you.
Re: AuthzSVNAccessFile size issues
The relevant code is in libsvn_repos/authz.c. Look at the call to svn_repos_authz_read() in mod_authz_svn.c: the file is read and parsed once per connection. I guess you could cache the parsed svn_authz_t struct in a longer-lived pool, or perhaps increase the lifetime of connections (compare the configuration suggestions in http://subversion.apache.org/docs/release-notes/1.7#serf). The impact of this will be, of course, that changes to the authz config file might not take effect as soon as they do now. Joel Eidsath wrote on Thu, Mar 22, 2012 at 09:11:48 -0700: I was just handed a large SVN install with thousands of users and hundreds of individual repositories. It is experiencing serious performance issues. I believe that it mostly boils down to a 14MB AuthzSVNAccessFile. What can I do to speed this up? Is there a database solution to use instead of the flatfile? Can I implement caching somehow? I am willing to code something up if I have to. Thanks for any help! Please CC me on any replies. Joel Eidsath
RE: AuthzSVNAccessFile size issues
Joel Eidsath wrote on Thu, Mar 22, 2012 at 09:11:48 -0700: I was just handed a large SVN install with thousands of users and hundreds of individual repositories. It is experiencing serious performance issues. I believe that it mostly boils down to a 14MB AuthzSVNAccessFile. What can I do to speed this up? Is there a database solution to use instead of the flatfile? Can I implement caching somehow? I am willing to code something up if I have to. Thanks for any help! Please CC me on any replies. The relevant code is in libsvn_repos/authz.c. Look at the call to svn_repos_authz_read() in mod_authz_svn.c: the file is read and parsed once per connection. I guess you could cache the parsed svn_authz_t struct in a longer-lived pool, or perhaps increase the lifetime of connections (compare the configuration suggestions in http://subversion.apache.org/docs/release-notes/1.7#serf). The impact of this will be, of course, that changes to the authz config file might not take effect as soon as they do now. Cruise control .Net caches it's config info... however if uses a file watcher to see if the file changes... and if so reparses it. Could svn take this approach. BOb
Re: AuthzSVNAccessFile size issues
Joel Eidsath jeids...@gmail.com writes: I was just handed a large SVN install with thousands of users and hundreds of individual repositories. It is experiencing serious performance issues. I believe that it mostly boils down to a 14MB AuthzSVNAccessFile. What can I do to speed this up? Use per-repository authz files? -- Philip
Re: AuthzSVNAccessFile size issues
On Thu, Mar 22, 2012 at 15:24, Bob Archer bob.arc...@amsi.com wrote: Joel Eidsath wrote on Thu, Mar 22, 2012 at 09:11:48 -0700: I was just handed a large SVN install with thousands of users and hundreds of individual repositories. It is experiencing serious performance issues. I believe that it mostly boils down to a 14MB AuthzSVNAccessFile. What can I do to speed this up? Is there a database solution to use instead of the flatfile? Can I implement caching somehow? I am willing to code something up if I have to. Thanks for any help! Please CC me on any replies. The relevant code is in libsvn_repos/authz.c. Look at the call to svn_repos_authz_read() in mod_authz_svn.c: the file is read and parsed once per connection. I guess you could cache the parsed svn_authz_t struct in a longer-lived pool, or perhaps increase the lifetime of connections (compare the configuration suggestions in http://subversion.apache.org/docs/release-notes/1.7#serf). The impact of this will be, of course, that changes to the authz config file might not take effect as soon as they do now. Cruise control .Net caches it's config info... however if uses a file watcher to see if the file changes... and if so reparses it. Could svn take this approach. Wouldn't that depend on where the bottleneck is? If the issues is file I/O, this could help. If the issue is the time it takes to check every rule, the improvement may not be significant.
Re: AuthzSVNAccessFile size issues
Thanks. strace seems to indicate I/O issues as the bottleneck. The file is loaded from NFS without caching. The per-repository access file seems like my best bet for an immediate solution. It is example 4 in this config: http://svn.apache.org/repos/asf/subversion/trunk/subversion/mod_authz_svn/INSTALL For a longer term solution, I am looking at contributing some code to mod_authz_svn to 1. allow rule caching, or 2. provide some sort of db support. Joel Eidsath On Thu, Mar 22, 2012 at 1:11 PM, Andy Levy andy.l...@gmail.com wrote: On Thu, Mar 22, 2012 at 15:24, Bob Archer bob.arc...@amsi.com wrote: Joel Eidsath wrote on Thu, Mar 22, 2012 at 09:11:48 -0700: I was just handed a large SVN install with thousands of users and hundreds of individual repositories. It is experiencing serious performance issues. I believe that it mostly boils down to a 14MB AuthzSVNAccessFile. What can I do to speed this up? Is there a database solution to use instead of the flatfile? Can I implement caching somehow? I am willing to code something up if I have to. Thanks for any help! Please CC me on any replies. The relevant code is in libsvn_repos/authz.c. Look at the call to svn_repos_authz_read() in mod_authz_svn.c: the file is read and parsed once per connection. I guess you could cache the parsed svn_authz_t struct in a longer-lived pool, or perhaps increase the lifetime of connections (compare the configuration suggestions in http://subversion.apache.org/docs/release-notes/1.7#serf). The impact of this will be, of course, that changes to the authz config file might not take effect as soon as they do now. Cruise control .Net caches it's config info... however if uses a file watcher to see if the file changes... and if so reparses it. Could svn take this approach. Wouldn't that depend on where the bottleneck is? If the issues is file I/O, this could help. If the issue is the time it takes to check every rule, the improvement may not be significant.
Re: AuthzSVNAccessFile size issues
strace seems to indicate I/O issues as the bottleneck. The file is loaded from NFS without caching. The per-repository access file seems like my best bet for an immediate solution. It is example 4 in this config: http://svn.apache.org/repos/asf/subversion/trunk/subversion/ mod_authz_svn/INSTALL For a longer term solution, I am looking at contributing some code to mod_authz_svn to 1. allow rule caching, or 2. provide some sort of db support. You may also want to try: SVNPathAuthz short_circuit I'll let you google the potential problems with using that value. It can result in significant performance gains if you work within the restrictions. Kevin R.
Re: AuthzSVNAccessFile size issues
That option definitely looks interesting. From my reading, we could use it to get apache to ignore all access restrictions except for the SVN access file that I specify. Unfortunately my current performance woes mostly come from the issues involved with processing the SVN access file itself. Joel Eidsath On Thu, Mar 22, 2012 at 2:39 PM, kmra...@rockwellcollins.com wrote: strace seems to indicate I/O issues as the bottleneck. The file is loaded from NFS without caching. The per-repository access file seems like my best bet for an immediate solution. It is example 4 in this config: http://svn.apache.org/repos/asf/subversion/trunk/subversion/ mod_authz_svn/INSTALL For a longer term solution, I am looking at contributing some code to mod_authz_svn to 1. allow rule caching, or 2. provide some sort of db support. You may also want to try: SVNPathAuthz short_circuit I'll let you google the potential problems with using that value. It can result in significant performance gains if you work within the restrictions. Kevin R.
