[fossil-users] Fossil server load control
A new feature was recently added to Fossil that allows it to deny expensive requests (such as blame or tarball on a large repository) if the server load average is too high. See http://www.fossil-scm.org/fossil/doc/tip/www/server.wiki#loadmgmt for further information. This new feature was recently added to the server that self-hosts Fossil and which also hosts a number of other projects including SQLite and System.Data.SQLite. I am pleased to announce that this new feature has passed its first test. About three hours ago, a single user in Beijing began downloading multiple copies of the same System.Data.SQLite tarball. As of this writing, he has so far attempted to download that one tarball 11,784 times (at last count - the download attempts are continuing...) Download requests are arriving at a rate of about one per second, and each request takes about 3.1 seconds of CPU time in order to compute the 80MB tarball. Since requests are arriving faster than they can be serviced, this would formerly have resulted in unlimited growth of the run-queue, essentially shutting down the server. The effect is the same as having been slashdotted. But thanks to the recent enhancements, most of these massive download requests are rejected with a 503 error and the server load average is staying below 4.0 at all times and thus continues to provide quick responses to other 10 requests/second that the server normally receives. If you are running your own publicly accessible Fossil server, you might want to consider updating to the latest Fossil trunk and activating the load-average limiter too. (Note to Tclers: I have already done this on core.tcl.tk.) And if you have alternative suggestions about how to keep a light-weight host running smoothly under a massive Fossil request load, please post follow-up comments. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Fossil server load control
On Wed, Mar 12, 2014 at 6:40 AM, Richard Hipp d...@sqlite.org wrote: A new feature was recently added to Fossil that allows it to deny expensive requests (such as blame or tarball on a large repository) if the server load average is too high. See http://www.fossil-scm.org/fossil/doc/tip/www/server.wiki#loadmgmt for further information. Interesting. I am pleased to announce that this new feature has passed its first test. About three hours ago, a single user in Beijing began downloading multiple copies of the same System.Data.SQLite tarball. As of this writing, he has so far attempted to download that one tarball 11,784 times (at last count - a rate of about one per second, and each request takes about 3.1 seconds of CPU time in order to compute the 80MB tarball. And if you have alternative suggestions about how to keep a light-weight host running smoothly under a massive Fossil request load, please post follow-up comments. How sensible do you think would it be to have a (limited-size) (in-memory|disk) cache to hold the most recently requested tarballs ? That way a high-demand tarball, etc. would be computed only once and then served statically from the cache. Note that I actually see this as a possible complement to the load mgmt feature. The cache would help if demand is high for a small number of revisions, whereas load mgmt would kick in and restrict load if the access pattern of revisions is sufficiently random/spread out to negate the cache (i.e. cause it to thrash). Side note: While the same benefits could be had by putting a regular web cache in front of the fossil server, i.e. a squid or the like this would require more work to set up and admin. And might be a problem for the truly dynamic parts of the fossil web ui. An integrated cache just for the assets which are expensive to compute and yet (essentially) static does not have these issues. I mentioned in-memory and disk ... I can see that a two-level scheme here ... A smaller in-memory cache for the really high-demand pieces with LRU, and a larger disk cache for the things not so much in-demand at the moment, but possibly in the future. The disk cache could actually be much larger (disks are large and cheap these days), this would help with random access attacks (as they would become asymptotically more difficult as the disk cache over time extends its net of quickly served assets). -- Andreas Kupries Senior Tcl Developer Code to Cloud: Smarter, Safer, Faster(tm) F: 778.786.1133 andre...@activestate.com http://www.activestate.com Learn about Stackato for Private PaaS: http://www.activestate.com/stackato EuroTcl'2014, July 12-13, Munich, GER ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] copying a branch and history into a new repository
I host my fossil repository on a private server. As a little background, I do mostly scripting work and keep all my projects in separate branches. This has worked great for me and I will continue to use this method. However, I am almost finished with a project I would like to host on chiselapp so the greater Internet can reach it. My Google-fu seems to have failed me and this was the best search I could come up, fossil branch history new repository. I only found one try that did not talk about just branching an existing repository and this mailing list entry from January 2012http://www.mail-archive.com/fossil-users%40lists.fossil-scm.org/msg07227.htmldoes not have any responses to it. I have run fossil export, but the resulting file for my repository is too large for me to realistically manually edit down to just the branch I want. If anybody has some scripting-fu for that, that would be awesome. I am on Windows, but any language will work; I can port the functionality to a Windows scripting language for my purposes (and would happily share that back with the list). Otherwise, does anybody have any other ideas? Thanks. Also, Stephan Beal, I cannot wait until libfossil can be used natively on Windows (without Cygwin). I have some ideas for using it in my workflow... JR ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] html file - cannot compute difference between binary files
On 10/03/2014 22:24, Richard Hipp wrote: The diff-generator in Fossil is unable to cope with lines longer than 8192 bytes. That could be changed. (It's a #define, though there are adverse consequences for making it too large.) But is a diff on a file with huge lines like that really useful? Would a human be able to understand such a diff? That would depend on how the diff is presented. As two long lines? No, I doubt a human could. But if you present the diff as two wrapped lines side by side diff A | diff B and wrapped each at say 35 chars with a clear border and space down the middle and highlighted using colour each difference I can see that working even on a 80 char wide console. But equally, the diff command is used to present a difference between files. A human might not care about understanding it, as long as patch(1) does. Roy ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Fossil server load control
On Wed, Mar 12, 2014 at 1:13 PM, Andreas Kupries andre...@activestate.comwrote: On Wed, Mar 12, 2014 at 6:40 AM, Richard Hipp d...@sqlite.org wrote: And if you have alternative suggestions about how to keep a light-weight host running smoothly under a massive Fossil request load, please post follow-up comments. How sensible do you think would it be to have a (limited-size) (in-memory|disk) cache to hold the most recently requested tarballs ? That way a high-demand tarball, etc. would be computed only once and then served statically from the cache. It's on my to-do list, actually. The idea is to have a separate database that holds the cache. And yes it is complementary to the load management feature. Side note: While the same benefits could be had by putting a regular web cache in front of the fossil server, No they can't actually, at least not by any technology I'm aware of. The problem is that these request must be authenticated. Downloads might be only authorized for certain users. If an authorized user does a download, and squid caches it, some other unauthorized user might be able to obtain the download from cache. Even if downloads are currently authorized for anybody (which is the common case, at least on public repos), I don't think you want them being cached, since to do so would mean that turning off public downloads would be ineffective until the caches all expired. I mentioned in-memory and disk ... I can see that a two-level scheme here ... A smaller in-memory cache for the really high-demand pieces with LRU, and a larger disk cache for the things not so much in-demand at the moment, but possibly in the future. The disk cache could actually be much larger (disks are large and cheap these days), this would help with random access attacks (as they would become asymptotically more difficult as the disk cache over time extends its net of quickly served assets). The current Fossil implementation runs a separate process for each HTTP request. So an in-memory cache wouldn't be helpful. It has to be disk-based. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Fossil server load control
On Wed, Mar 12, 2014 at 6:13 PM, Andreas Kupries andre...@activestate.comwrote: How sensible do you think would it be to have a (limited-size) (in-memory|disk) cache to hold the most recently requested tarballs ? That way a high-demand tarball, etc. would be computed only once and then served statically from the cache. FWIW: i was scratching down ideas for this very idea today for the libfossil CGI demos because i don't like the memory cost of generate ZIP files from script code. Caching the (say) 10 most recent ZIPs could alleviate some of my load concerns. It need not be a synchable table, nor in one which survives a rebuild. Note that I actually see this as a possible complement to the load mgmt feature. The cache would help if demand is high for a small number of revisions, whereas load mgmt would kick in and restrict load if the access pattern of revisions is sufficiently random/spread out to negate the cache (i.e. cause it to thrash). +1 would require more work to set up and admin. And might be a problem for the truly dynamic parts of the fossil web ui. An integrated cache just for the assets which are expensive to compute and yet (essentially) static does not have these issues. In my experience, most proxies won't cache for requests which have URL parameters. Whether or not that's generally true, i can't say. For static content (lots of what fossil serves is static), the URLs can/should be written as /path/arg1/arg2, rather than /path?arg1=...arg2=..., to make them potentially more cacheable. -- - stephan beal http://wanderinghorse.net/home/stephan/ http://gplus.to/sgbeal Freedom is sloppy. But since tyranny's the only guaranteed byproduct of those who insist on a perfect world, freedom will have to do. -- Bigby Wolf ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Fossil server load control
The current Fossil implementation runs a separate process for each HTTP request. So an in-memory cache wouldn't be helpful. It has to be disk- based. Does not FastCGI do exactly the opposite? RR 2014-03-12 18:25 GMT+01:00 Richard Hipp d...@sqlite.org: On Wed, Mar 12, 2014 at 1:13 PM, Andreas Kupries andre...@activestate.com wrote: On Wed, Mar 12, 2014 at 6:40 AM, Richard Hipp d...@sqlite.org wrote: And if you have alternative suggestions about how to keep a light-weight host running smoothly under a massive Fossil request load, please post follow-up comments. How sensible do you think would it be to have a (limited-size) (in-memory|disk) cache to hold the most recently requested tarballs ? That way a high-demand tarball, etc. would be computed only once and then served statically from the cache. It's on my to-do list, actually. The idea is to have a separate database that holds the cache. And yes it is complementary to the load management feature. Side note: While the same benefits could be had by putting a regular web cache in front of the fossil server, No they can't actually, at least not by any technology I'm aware of. The problem is that these request must be authenticated. Downloads might be only authorized for certain users. If an authorized user does a download, and squid caches it, some other unauthorized user might be able to obtain the download from cache. Even if downloads are currently authorized for anybody (which is the common case, at least on public repos), I don't think you want them being cached, since to do so would mean that turning off public downloads would be ineffective until the caches all expired. I mentioned in-memory and disk ... I can see that a two-level scheme here ... A smaller in-memory cache for the really high-demand pieces with LRU, and a larger disk cache for the things not so much in-demand at the moment, but possibly in the future. The disk cache could actually be much larger (disks are large and cheap these days), this would help with random access attacks (as they would become asymptotically more difficult as the disk cache over time extends its net of quickly served assets). The current Fossil implementation runs a separate process for each HTTP request. So an in-memory cache wouldn't be helpful. It has to be disk-based. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Fossil server load control
On Wed, Mar 12, 2014 at 1:26 PM, Stephan Beal sgb...@googlemail.com wrote: In my experience, most proxies won't cache for requests which have URL parameters. Whether or not that's generally true, i can't say. For static content (lots of what fossil serves is static), the URLs can/should be written as /path/arg1/arg2, rather than /path?arg1=...arg2=..., to make them potentially more cacheable. With a few carefully chosen exceptions, Fossil always sets Cache-control: no-cache in the header of its replies, due in large part to those pesky authentication cookies. -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Fossil server load control
On Wed, Mar 12, 2014 at 6:31 PM, Ramon Ribó ram...@compassis.com wrote: The current Fossil implementation runs a separate process for each HTTP request. So an in-memory cache wouldn't be helpful. It has to be disk- based. Does not FastCGI do exactly the opposite? FastCGI requires that there be some sort of state object which is can re-set between calls, and feed that state into each child. Fossil doesn't have such a state object (it has one, but not one which can simply be re-set/re-used), so FastCGI can't really do its magic with fossil. libfossil (currently under construction and moving along nicely) provides such a construct. -- - stephan beal http://wanderinghorse.net/home/stephan/ http://gplus.to/sgbeal Freedom is sloppy. But since tyranny's the only guaranteed byproduct of those who insist on a perfect world, freedom will have to do. -- Bigby Wolf ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] importing/forking from Git
On Mon, Mar 10, 2014 at 06:31:25PM -0400, Ron Wilson wrote: On Sat, Mar 8, 2014 at 5:17 PM, Chad Perrin c...@apotheon.net wrote: Should I take it at this point that --incremental is deprecated and should not be used or expected to be present in future Fossil versions? So far, I have seen no responses that specifically addressed whether the behavior I expected from it was the intended behavior, whether anyone else might have the same problem indicating a bug in the software or whether this was confined to my situation and thus might be a local configuration issue, whether there was anything I could do to provide more information to help, and so on. The closest thing to meaningfully addressing the issue that I have seen was akin to the old joke . . . patient: Doctor, doctor, it hurts when I do this! doctor: Don't do that. I very much doubt that --incremental would be deprecated. I suspect that no one else has tried your use case. I'm stuck in kind of a no-win situation with this, then. It's not really appropriate to log a bug in an issue tracker without knowing it's a bug, but nobody will tell me if it's a bug, how I should judge whether it's a bug, or whatever else applies here. I'm not even sure yet that what I believe to be expected behavior is expected behavior, or whether I'm just using it wrong. . . . so while I wait to see if someone will give me some guidance, it will fall off everyone's radar, and nothing will be done, so that my only option going forward for some of these projects will be to just use Git. -- Chad Perrin [ original content licensed OWL: http://owl.apotheon.org ] ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] importing/forking from Git
On Wed, Mar 12, 2014 at 5:04 PM, Chad Perrin c...@apotheon.net wrote: I'm stuck in kind of a no-win situation with this, then. It's not really appropriate to log a bug in an issue tracker without knowing it's a bug, but nobody will tell me if it's a bug, how I should judge whether it's a bug, or whatever else applies here. I'm not even sure yet that what I believe to be expected behavior is expected behavior, or whether I'm just using it wrong. You could put in an enhancement request. What you want to do, import in to a repository with only the initial, empty commit, seems reasonable. . . . so while I wait to see if someone will give me some guidance, it will fall off everyone's radar, and nothing will be done, so that my only option going forward for some of these projects will be to just use Git. If the only possible way to start a new repository on the server is fossil new repoName.fossil, then a possible work around would be to do a normal import to Fossil, then create a marks file (see fossil help export) with the RID of the initial, empty commit, then use the --import-marks option to the fossil export command to (hopefully) create something usable with fossil import --incremental. Not sure when I will be able to experiemnt with this more. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
