Hi, I was introduced to Fossil about three years ago. It is the first VCS I have actually used extensively for my own work and as such I haven't really had much to compare it with. However, now I'm in the realm of contributing code I am having to learn how to make use of other systems as well. For the purposes of my experiments, I have been comparing Fossil alongside Git, as this seems to be the popular alternative and there is a comparison sheet describing many areas of difference between Fossil and Git. For the most part, Fossil has many advantages over Git. Though I couldn't help noticing there seemed to be a silence on speed comparisons. There is a separate article on performance statistics that again doesn't even mention speed.
After cloning and working with several publicised Fossil repositories, I can't help but notice that the majority of them are rather small. The system.data.sqlite repository is probably the biggest I've seen. As of 11th August 2017: Flint: 468KB, 65 commits Hydra: 648KB, 37 commits Fossilbook: 37.2MB, 82 commits Fossil: 56.1MB, 10696 commits SQLite: 92MB, 19275 commits SLT: 142.5MB, 165 commits TCL: 212.3MB, 21227 commits system.data.sqlite: 395.4MB, 3072 commits Most of the projects that I am involved with are games designed to run on Windows systems. These are anywhere from a small arcade game (10-100MB), or a fully fledged action packed or role playing adventure game (the biggest so far being 5GB). Of course these will contain binary files (generally an executable, dependency libraries and, most of all, media). It is the big ones where Fossil seems to be tripping over its own feet... My machine: Windows 7 64-bit quad-core hyperthreaded, 32GB ram, 31.42TB total storage. Fossil version: 2.3 [f7914bfdfa] 2017-07-21 03:19:30 UTC Note: None of the below repositories have any ticket or custom wiki entries, and all only have the main trunk branch. Also note that all commits were tests only and so weren't synced to remotes. Naturally this means that commits are even slower when syncing. I have included export times because I have also done some experimentation with import/export. While the below tests don't seem to be too taxing in regards to exporting, try git-cloning https://github.com/torvalds/linux (roughly a 15 minute operation) and import that to Fossil. On my machine it hadn't finished importing after three days, and the bigger it got the more it slowed. Took an hour to go from 2 to 3GB, and roughly 10 hours to go from 4 to 5GB. Of course you wouldn't find me so much as touching the Linux code - it is very different to Windows and even using it has turned out to be a pain in the rear and a bullet in the leg. I did this merely to test Fossil's handling of large repositories. Similar slowdowns also seem to be occurring when building zip files from repositories. In fact there have been various scenarios when I have wanted to visit a certain checkin, and have had to open that checkin as a separate checkout and zip it with another utility, because Fossil gives me out of memory errors. With Fossil, we get the following results: Repo1: 76.5MB, 3 total commits Clone: 14S (approx. 5.1MBPS) Open: 3S (Latest commit, 322 files, 89.1MB, approx. 107 files or 29.7MB per second) Commit: 12S (Deleted 5 small files) Export: 7S (approx. 10.93MBPS) Repo2: 554.4MB, 315 total commits Clone: 2M27S (Download=1M10S, finalise=1M17S, approx. 7.92MBPS) Open: 10S (Latest commit, 446 files, 385MB, approx. 44 files or 38.5MB per second) Commit: 8S (Deleted one small file) Export: 1M1S (Approx. 9.09MBPS) Repo3: 792MB, 340 total commits Clone: 2M16S (Download=1M40S, finalise=36S, approx. 7.92MBPS) Open: 20S (Latest commit, 1087 files, 906MB, Approx. 54 files or 45.3MB per second) Commit: 25S (Changed a few audio files) Export: 1M24S (Approx. 9.43MBPS) Repo4: 2.4GB, 7 commits Clone: 6M30S (Download=5M, finalise=1M30S, approx. 8MBPS) Open: 43S (Latest commit, 574 files, 2.36GB, approx. 13 files or 181.5MB per second) Commit: 1M18S (Edited perhaps 10 lines of code) Export: 3M55S (Approx. 10MBPS) Repo5: 5.6GB, 24 total commits Clone: 9M1S (Download=6M38S, Finalise=2M23S, approx. 14.07MBPS) Open: 1M35S (Latest commit, 1380 files, 4.18GB, approx. 14 files or 45MB per second) Commit: 1M30S (Added 2 source code files totalling 62.2KB) Export: 11M49S (Approx. 9.17MBPS) Importing the same repositories into Git and doing similar commits yields the following results. Unfortunately I don't have a private Git server from which to test clones, however Fossil's cloning is pretty fast anyway so I'm not too concerned about this. Repo1: 76.5MB, 3 total commits Git size: 75.1MB Open: 3S (Latest commit, 322 files, 89.1MB, approx. 107 files or 29.7MB per second) Commit: Instantaneous (Edited code) Repo2: 554.4MB, 315 total commits Git size: 655.5MB Open: 5S (Latest commit, 446 files, 385MB, approx. 88 files or 77MB per second) Commit: instantaneous (Source tree directory restructuring, some deletions) Repo3: 792MB, 340 total commits Git size: 801.1MB Open: 13S (Latest commit, 1087 files, 906MB, Approx. 84 files or 70MB per second) Commit: 16s (Changed a few sounds) Repo4: 2.4GB, 7 commits Git size: 2.4GB Open: 17S (Latest commit, 574 files, 2.36GB, approx. 33 files or 181.5MB per second) Commit: instantaneous (Edited perhaps 10 lines of code) Repo5: 5.6GB, 24 total commits Git size: 5.6GB Open: 1M41S (Latest commit, 1380 files, 4.18GB, approximately 14 files or 42MB per second) Commit: Instantaneous (Added 2 source code files totalling 62.2KB) Observations: 1. Git seems to do better at compressing and opening smaller repositories, while Fossil triumphs over larger ones. 2. In some cases (notably repo1), Fossil is only 14% faster at performing a commit than a full clone. 3. The speed of a commit in Git seems to be dependent on the size of the change. The bigger the changes, naturally the slower the commits. Commits in Fossil seem, with a few discrepancies (notably commit times in repos 1 and 2), to be dependent on the size of the repository. 4. Fossil clone - finalising procedure took 36 seconds for repo 3, 77 seconds (just over twice the length) for repo2, and 90 seconds for repo4. Yet repo3 has 25 more checkins than repo2, and plenty more files. Repo 4 only has 7 checkins in total. Does this have to do with the state of the remote repository? 5. Export seems to be slowing down significantly the bigger the repository, but when you look at the average MBPS from one to the other it actually doesn't seem too bad. Again though, attempt the Linux import above and watch it take three days to create an incomplete, 12GB fossil repository...Of course this could easily also be Git's export system - I haven't done a similar test on Git with other VCS systems in order to create a detailed performance comparison. I was very interested to find an article hidden in the depths of the Fossil website (http://fossil-scm.org/index.html/event?name=be8f2f3447ef2ea3344f8058b6733aa08c08336f) which, though containing details a little too technical for my knowledge, discusses large scale repositories, however this seemed to point more towards the issue of huge numbers of small files. It made frequent reference to the checkout size of SLT (over 1GB) as an example of a small number of larger files, however as you see above, some of these size boundaries are being exceeded in both repository and checkout size. Also the articles discuss cloning and rebuilding and don't seem to cover committing. If a local commit can take 90 seconds on a 5GB repository, how long would it take to send all that data back to the remote server... I think I remember some rather small commits sometimes taking 10 minutes, including remote sync. Of course these issues may well already be known. I'm sure I'm not the only person to have raised this issue since Fossil was conceived back in 2006. However, other than the above article, I can find no mention of them being addressed. Are there any configurations I may have missed that may help to optimise Fossil in these areas? If not, are there any plans to optimise Fossil in the future? Cheers. Damien.
_______________________________________________ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
