Re: [fossil-users] Old problem not entirely gone?
On Mon, Feb 2, 2015 at 5:35 PM, Warren Young wrote: > After sending that prior message, I did think of a way to allow retries > without inconsistency, but it would surely slow Fossil down: there could be > a mode that turns cloning into a replay of the master repo’s timeline. > > That is, every change made to the master gets applied to the slave, in the > order it was made. This means local files get updated multiple times, even > if later changes wipe out prior ones. The advantage is that after each > changeset is applied, the tree is again in a consistent state. I don't think it would be much of a slow down. All the artifacts will get sent, just the ordering will be more controlled. At the start of the first attempt to clone, send the oldest manifest and a cluster "listing" the remaining manifests in chronological order, plus as many of the files "listed" in the first manifest as will fit in the initial transfer. Then the client will process the manifests in the order they appear in in the "manifest clusters". (If there are too many manifests to list in a single cluster, the final entry in the cluster would be the ID of the next manifest cluster.) When attempting to resume, the client would include the pending "gimme" cards, also in the order the IDs appeared in the manifest clusters Other than insuring the client receives the manifests in chronological order, the clone protocol would operate as it currently does and would be compatible with clients that don't support resuming. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Old problem not entirely gone?
On Tue, Feb 03, 2015 at 01:48:52PM +0100, Jan Danielsson wrote: >In terms of the type of data, our data and fossil's data is very > different, but in terms of the time it takes to synchronize large data > stores/repositories, we're in the exact same situation. We don't expect > synchronizations to fail; they rarely will, but it will happen sooner or > later, so we were forced to find a way to skip work that has already > been done. Correct. One possible approach is to mark new artefacts in a separate "to parse" table, add them to that list after every round trip and maybe even do a commit in case of power down. Afterwards, process that table. It doesn't even require many changes to the code as the approach taken is essentially the same, just using in-memory storage. Joerg ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Old problem not entirely gone?
On 02/02/15 23:23, Warren Young wrote: >> The annoying thing is that when it fails, it wipes away whatever >> progress it has made. > Yes, well, that’s the nature of transactional DB updates: all or nothing. Implied: There's only one way to use transactions when performing initial synchronizations. >> How difficult would it be to allow fossil to pick up where it left >> off in such a case? > Are you seriously asking for Fossil to allow a local clone to be in an > inconsistent state after an error? I'm working on a distributed storage system which borrows a few of the basic ideas from fossil (immutable artifacts, etc). We perform a DB commit after an entire *application layer* transactional unit has been transferred. When an initial synchronization breaks in our system, we use a simple method to skip artifacts when it resumes; we divide the timeline into week-long chunks, and hash all the artifact hashes in order of time in each chunk; once a week is encountered where the hashes don't match, we perform a fine-grained sync of the artifacts for that week. We deal with up to GB-size artifacts (indexed in DB, but not stored there), and the entire data storage can be in the region of terabytes. The system runs over the regular Internet. It must be tolerant to faults like "pull network cable", or ISP hiccups, because restarting from scratch after 90% has been synchronized simply isn't an option. In terms of the type of data, our data and fossil's data is very different, but in terms of the time it takes to synchronize large data stores/repositories, we're in the exact same situation. We don't expect synchronizations to fail; they rarely will, but it will happen sooner or later, so we were forced to find a way to skip work that has already been done. We don't remove all the work which has been done after a failed initial synchronization. Nor do we leave it in an "inconsistent state" -- we leave it in an "incomplete/resumable" state, and it's clearly tagged as such. I'm not even asking if fossil could be made fully resumable after an exit either, a "An error was detected; retry [y/n]?" would be another option. If it's impossible for fossil to do anything like that without major rewiring of the internals, then *that* would be relevant to my question. So to answer your question: No. Not that I ever implied anything of the sort. -- Kind Regards, Jan ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Old problem not entirely gone?
Thus said Jan Danielsson on Sun, 01 Feb 2015 15:08:07 +0100: > In that thread the commit > http://www.fossil-scm.org/index.html/info/b4dffdac5e706980d911a0e672526ad461ec0640 > was brought up as a potential fix. I updated to get the fix and then > tried running a clone, and I could indeed get the entire repository. This fix is server side, and the NetBSD repository is running version: http://www.fossil-scm.org/index.html/info/3d49f04587 And as far as I can tell, did not have the fix in for this until http://www.fossil-scm.org/index.html/info/ebac09bcf7 Which means perhaps the NetBSD repositories need a newer Fossil? Andy -- TAI64 timestamp: 400054d01abc ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Old problem not entirely gone?
On Mon, Feb 02, 2015 at 03:35:13PM -0700, Warren Young wrote: > > On Feb 2, 2015, at 3:28 PM, Joerg Sonnenberger > > wrote: > > > > On Mon, Feb 02, 2015 at 03:23:46PM -0700, Warren Young wrote: > >> > >> Are you seriously asking for Fossil to allow a local clone to be in an > >> inconsistent state after an error? > > > > Why does it have to be an inconsistent state? At the very least, it > > could ask for isatty(stdout) whether it should just retry. > > SQLite already has retry behavior in its locking. I presume Fossil hasn’t > disabled that. Retry on the *client* side, which is completely unrelated to sqlite. Joerg ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Old problem not entirely gone?
> On Feb 2, 2015, at 3:28 PM, Joerg Sonnenberger > wrote: > > On Mon, Feb 02, 2015 at 03:23:46PM -0700, Warren Young wrote: >> >> Are you seriously asking for Fossil to allow a local clone to be in an >> inconsistent state after an error? > > Why does it have to be an inconsistent state? At the very least, it > could ask for isatty(stdout) whether it should just retry. SQLite already has retry behavior in its locking. I presume Fossil hasn’t disabled that. After sending that prior message, I did think of a way to allow retries without inconsistency, but it would surely slow Fossil down: there could be a mode that turns cloning into a replay of the master repo’s timeline. That is, every change made to the master gets applied to the slave, in the order it was made. This means local files get updated multiple times, even if later changes wipe out prior ones. The advantage is that after each changeset is applied, the tree is again in a consistent state. That might not help the NetBSD repo, though, depending on how it was constructed. If it’s a history-preserving conversion from CVS or similar, this would possibly work. If on the other hand there was a point where a huge existing tree was imported into an SCM (Fossil or a predecessor) as-is, you have a good chance that the timeout will happen during processing of that initial huge commit. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Old problem not entirely gone?
On Mon, Feb 02, 2015 at 03:23:46PM -0700, Warren Young wrote: > > On Feb 1, 2015, at 7:08 AM, Jan Danielsson > > wrote: > > > > The annoying thing is that when it fails, it wipes away whatever > > progress it has made. > > Yes, well, that’s the nature of transactional DB updates: all or nothing. > > > How difficult would it be to allow fossil to pick up where it left > > off in such a case? > > Are you seriously asking for Fossil to allow a local clone to be in an > inconsistent state after an error? Why does it have to be an inconsistent state? At the very least, it could ask for isatty(stdout) whether it should just retry. Joerg ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Old problem not entirely gone?
> On Feb 1, 2015, at 7:08 AM, Jan Danielsson wrote: > > The annoying thing is that when it fails, it wipes away whatever > progress it has made. Yes, well, that’s the nature of transactional DB updates: all or nothing. > How difficult would it be to allow fossil to pick up where it left > off in such a case? Are you seriously asking for Fossil to allow a local clone to be in an inconsistent state after an error? ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
