Re: FYI: SVN to GIT converter currently broken, github is falling behind
On 12/4/2015 10:49 AM, Ulrich Spörlein wrote: > 2015-11-08 12:06 GMT+01:00 Ulrich Spörlein : >> 2015-11-08 11:32 GMT+01:00 Ulrich Spörlein : >>> 2015-11-08 2:51 GMT+01:00 Alfred Perlstein : > Uli, One of the biggest concerns I've heard from folks using FreeBSD's git mirror is that the hashes can change. I have a question about this. Is it possible to keep track of what the "official" git mirror (on github) is doing and keep that as a log. Then that log can be used to replay commits when there is a divergence problem. What I'm basically saying is that let's take this small example: importer is working fine @rev 1 imports 1 imports 10001 imports 10002 something happens to importer to give indeterminate shas. imports 10003 - sha is "unstable" sha3 imports 10004 - sha is "unstable" sha4 imports 10005 - sha is "unstable" sha5 imports 10006 - sha is "unstable" sha6 importer is fixed At this point normally we'd rewind the importer to 10002 and then force update the affected branches. My question is... can the imports of 10003, 10004, 10005 and 10006 be put into the importer such that any "mirror site" that re-does the import using the most up to date importer will get the same shas. That would allow to proceed with 10007, etc without force pushing. This should be possible based on querying "git" for the meta data associated with sha3..sha6 and then forcing those commits to have the same meta data. This would eliminate the concern about shas in the mirror changing that I've heard. >>> >>> The goal of the conversion is that everyone can re-do the conversion >>> in their basement and come up with the same history and checksums. >>> This was not the case when I first started, as there was some >>> non-deterministic hash structure being used in svn2git. This was fixed >>> in the code and then all converter runs produced the very same >>> results. >>> >>> The scenario that we have right now, is that one of the merge commits >>> done about two weeks ago is being handled different by svn2git w/ svn >>> v1.8 vs. svn v1.9 and I haven't investigated yet how the API's >>> behavior changed to cause this. I'm afraid I also swapped out all my >>> knowledge about svn2git internals and will have to redo this all from >>> scratch :/ >>> >>> Your suggestion could only work, if we hard-code this svn revision >>> special handling into svn2git, either in the code or by providing more >>> mappings and rules to the process. svn2git should run hermetic and not >>> poke at github's commits to see how things were handled in the past. >>> It has to be self-sufficient and must not depend on github. >>> >>> This would also only work, if the "breakage" window was very small, >>> but it is already about two weeks long and will surely increase till I >>> find the proper fix. >>> >>> So, to take a stand here: this sort of kludge is unlikely to ever >>> happen. Git commit hashes *might* change in the future. I really don't >>> see how this is a big deal anyway. It happened once and I'm trying to >>> have it never happen again. But why are people afraid of this >>> happening? Every "official" git commit is tagged with a SVN revision >>> and the contents of those revisions are obviously correct (just not >>> the ancestry and the commit objects, possibly). So it would be easy to >>> write a script that replays VendorA's git history and swaps out the >>> new official commits for the old official commits. There would be no >>> merge conflicts. >>> >>> I can see how this would be annoying if you have 100 developers and >>> dozens of branches that are far from mainline FreeBSD. But I'm sure >>> these companies that depend on git will come forward and donate some >>> of their developer manpower to help me with keeping the converter >>> stable/deterministic. Right? Right? :) :) >>> >>> Cheers, >>> Uli >> >> Quick update: doc is so far unaffected by svn 1.9, but for ports, the >> drift happened as of Jul 18, so you'd need to special case a lot of >> commits. >> >> Here's the same commit, and the difference between 1.8 and 1.9: >> >> % git cat-file commit 803795d >> tree 7fc83aba022834da5c218114b09ad4640735bcc0 >> parent c96fb0418e545a569b5975b4d878a30a948c29d5 >> author olgeni 1437203525 + >> committer olgeni 1437203525 + >> >> Upgrade to version 0.4.1. >> % git cat-file commit 61ca43b >> tree 7fc83aba022834da5c218114b09ad4640735bcc0 >> parent c96fb0418e545a569b5975b4d878a30a948c29d5 >> author olgeni 1437203529 + >> committer olgeni 1437203529 + >> >> Upgrade to version 0.4.1. >> >> >> In case you don't see it, there's a 4s difference in the timestamps >> for authoring and committing. Here's the original: >> >> % svn log -vc392405 svn://svn.freebsd.org/ports >> --
Re: FYI: SVN to GIT converter currently broken, github is falling behind
2015-11-08 12:06 GMT+01:00 Ulrich Spörlein : > 2015-11-08 11:32 GMT+01:00 Ulrich Spörlein : >> 2015-11-08 2:51 GMT+01:00 Alfred Perlstein : >>> Uli, >>> >>> One of the biggest concerns I've heard from folks using FreeBSD's git mirror >>> is that the hashes can change. >>> >>> I have a question about this. Is it possible to keep track of what the >>> "official" git mirror (on github) is doing and keep that as a log. Then >>> that log can be used to replay commits when there is a divergence problem. >>> >>> What I'm basically saying is that let's take this small example: >>> >>> importer is working fine @rev 1 >>> imports 1 >>> imports 10001 >>> imports 10002 >>> something happens to importer to give indeterminate shas. >>> imports 10003 - sha is "unstable" sha3 >>> imports 10004 - sha is "unstable" sha4 >>> imports 10005 - sha is "unstable" sha5 >>> imports 10006 - sha is "unstable" sha6 >>> importer is fixed >>> >>> >>> At this point normally we'd rewind the importer to 10002 and then force >>> update the affected branches. >>> >>> My question is... can the imports of 10003, 10004, 10005 and 10006 be put >>> into the importer such that any "mirror site" that re-does the import using >>> the most up to date importer will get the same shas. >>> >>> That would allow to proceed with 10007, etc without force pushing. >>> >>> This should be possible based on querying "git" for the meta data associated >>> with sha3..sha6 and then forcing those commits to have the same meta data. >>> >>> This would eliminate the concern about shas in the mirror changing that I've >>> heard. >> >> The goal of the conversion is that everyone can re-do the conversion >> in their basement and come up with the same history and checksums. >> This was not the case when I first started, as there was some >> non-deterministic hash structure being used in svn2git. This was fixed >> in the code and then all converter runs produced the very same >> results. >> >> The scenario that we have right now, is that one of the merge commits >> done about two weeks ago is being handled different by svn2git w/ svn >> v1.8 vs. svn v1.9 and I haven't investigated yet how the API's >> behavior changed to cause this. I'm afraid I also swapped out all my >> knowledge about svn2git internals and will have to redo this all from >> scratch :/ >> >> Your suggestion could only work, if we hard-code this svn revision >> special handling into svn2git, either in the code or by providing more >> mappings and rules to the process. svn2git should run hermetic and not >> poke at github's commits to see how things were handled in the past. >> It has to be self-sufficient and must not depend on github. >> >> This would also only work, if the "breakage" window was very small, >> but it is already about two weeks long and will surely increase till I >> find the proper fix. >> >> So, to take a stand here: this sort of kludge is unlikely to ever >> happen. Git commit hashes *might* change in the future. I really don't >> see how this is a big deal anyway. It happened once and I'm trying to >> have it never happen again. But why are people afraid of this >> happening? Every "official" git commit is tagged with a SVN revision >> and the contents of those revisions are obviously correct (just not >> the ancestry and the commit objects, possibly). So it would be easy to >> write a script that replays VendorA's git history and swaps out the >> new official commits for the old official commits. There would be no >> merge conflicts. >> >> I can see how this would be annoying if you have 100 developers and >> dozens of branches that are far from mainline FreeBSD. But I'm sure >> these companies that depend on git will come forward and donate some >> of their developer manpower to help me with keeping the converter >> stable/deterministic. Right? Right? :) :) >> >> Cheers, >> Uli > > Quick update: doc is so far unaffected by svn 1.9, but for ports, the > drift happened as of Jul 18, so you'd need to special case a lot of > commits. > > Here's the same commit, and the difference between 1.8 and 1.9: > > % git cat-file commit 803795d > tree 7fc83aba022834da5c218114b09ad4640735bcc0 > parent c96fb0418e545a569b5975b4d878a30a948c29d5 > author olgeni 1437203525 + > committer olgeni 1437203525 + > > Upgrade to version 0.4.1. > % git cat-file commit 61ca43b > tree 7fc83aba022834da5c218114b09ad4640735bcc0 > parent c96fb0418e545a569b5975b4d878a30a948c29d5 > author olgeni 1437203529 + > committer olgeni 1437203529 + > > Upgrade to version 0.4.1. > > > In case you don't see it, there's a 4s difference in the timestamps > for authoring and committing. Here's the original: > > % svn log -vc392405 svn://svn.freebsd.org/ports > > r392405 | olgeni | 2015-07-18 09:12:05 +0200 (Sat, 18 Jul 2015) | 2 lines > Changed paths: >M /head/www/elixir-maru/Makefile >M /head/www/elixir-maru/distinf
Re: FYI: SVN to GIT converter currently broken, github is falling behind
Hi Ulrich, This might be one of the reasons why the git converter was broken recently: https://reviews.reviewboard.org/r/7617/diff (it seems that svn is now reporting "nonexistent" for new files instead of "Revision 0"). It broke rbt with svn 1.9 :(... Thanks! -NGie ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FYI: SVN to GIT converter currently broken, github is falling behind
Lars, Try to remove .git/gc.log then re-run fetch. If that doesn't work then move ".git/refs/remotes/origin/HEAD" to backup location outside of your .git directory and try again. -Alfred On 11/11/15 4:03 AM, Eggert, Lars wrote: Hi, I just got this error when fetching from remote; related? [elars@laurel: ~/src] git fetch --all Fetching origin Auto packing the repository in background for optimum performance. See "git help gc" for manual housekeeping. Fetching upstream remote: Counting objects: 557, done. remote: Compressing objects: 100% (543/543), done. remote: Total 557 (delta 213), reused 2 (delta 2), pack-reused 0 Receiving objects: 100% (557/557), 1.15 MiB | 433.00 KiB/s, done. Resolving deltas: 100% (213/213), completed with 2 local objects. From github.com:/freebsd/freebsd b4eb11a..3eb0ea4 master -> upstream/master f147893..9c319c0 stable/10 -> upstream/stable/10 e901edd..b3c9fd2 stable/8 -> upstream/stable/8 81ab2b1..2fc7a9a stable/9 -> upstream/stable/9 c2c933c..cc76737 svn_head -> upstream/svn_head Auto packing the repository in background for optimum performance. See "git help gc" for manual housekeeping. error: The last gc run reported the following. Please correct the root cause and remove .git/gc.log. Automatic cleanup will not be performed until the file is removed. fatal: bad object refs/remotes/origin/HEAD error: failed to run repack Auto packing the repository in background for optimum performance. See "git help gc" for manual housekeeping. error: The last gc run reported the following. Please correct the root cause and remove .git/gc.log. Automatic cleanup will not be performed until the file is removed. fatal: bad object refs/remotes/origin/HEAD error: failed to run repack Lars ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FYI: SVN to GIT converter currently broken, github is falling behind
Hi, I just got this error when fetching from remote; related? [elars@laurel: ~/src] git fetch --all Fetching origin Auto packing the repository in background for optimum performance. See "git help gc" for manual housekeeping. Fetching upstream remote: Counting objects: 557, done. remote: Compressing objects: 100% (543/543), done. remote: Total 557 (delta 213), reused 2 (delta 2), pack-reused 0 Receiving objects: 100% (557/557), 1.15 MiB | 433.00 KiB/s, done. Resolving deltas: 100% (213/213), completed with 2 local objects. From github.com:/freebsd/freebsd b4eb11a..3eb0ea4 master -> upstream/master f147893..9c319c0 stable/10 -> upstream/stable/10 e901edd..b3c9fd2 stable/8 -> upstream/stable/8 81ab2b1..2fc7a9a stable/9 -> upstream/stable/9 c2c933c..cc76737 svn_head -> upstream/svn_head Auto packing the repository in background for optimum performance. See "git help gc" for manual housekeeping. error: The last gc run reported the following. Please correct the root cause and remove .git/gc.log. Automatic cleanup will not be performed until the file is removed. fatal: bad object refs/remotes/origin/HEAD error: failed to run repack Auto packing the repository in background for optimum performance. See "git help gc" for manual housekeeping. error: The last gc run reported the following. Please correct the root cause and remove .git/gc.log. Automatic cleanup will not be performed until the file is removed. fatal: bad object refs/remotes/origin/HEAD error: failed to run repack Lars signature.asc Description: Message signed with OpenPGP using GPGMail
Re: FYI: SVN to GIT converter currently broken, github is falling behind
2015-11-08 21:05 GMT+01:00 Oliver Pinter : > Hi Uli! > > I can not find your original svn2git repo in gitorius > (https://gitorious.org/ is down) , could you please the source code > somewhere to git-repo? For example github.com/freebsd/svn2git? You mean http://svn.freebsd.org/base/user/uqs/git_conv/ :) This doesn't have the v1.9 fixes though. I only notices this week that gitorious is down, but we also have https://github.com/freebsd/svn2git/commits/master for a while now. I'll push updated code there. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FYI: SVN to GIT converter currently broken, github is falling behind
> On Nov 8, 2015, at 20:51, Shane Ambler wrote: ... > 4 seconds?? > There have been 4 leap seconds added this century. > Did 1.9 add timestamp corrections relating to leap seconds? > > Did the developer not use leapsecs when the svn server does? Alternatively, was this impacted by a change in recent change to timekeeping (ntpd, etc)? Thanks, -NGie ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FYI: SVN to GIT converter currently broken, github is falling behind
On 08/11/2015 21:36, Ulrich Spörlein wrote: Here's the same commit, and the difference between 1.8 and 1.9: % git cat-file commit 803795d tree 7fc83aba022834da5c218114b09ad4640735bcc0 parent c96fb0418e545a569b5975b4d878a30a948c29d5 author olgeni 1437203525 + committer olgeni 1437203525 + Upgrade to version 0.4.1. % git cat-file commit 61ca43b tree 7fc83aba022834da5c218114b09ad4640735bcc0 parent c96fb0418e545a569b5975b4d878a30a948c29d5 author olgeni 1437203529 + committer olgeni 1437203529 + Upgrade to version 0.4.1. In case you don't see it, there's a 4s difference in the timestamps for authoring and committing. Here's the original: % svn log -vc392405 svn://svn.freebsd.org/ports r392405 | olgeni | 2015-07-18 09:12:05 +0200 (Sat, 18 Jul 2015) | 2 lines Changed paths: M /head/www/elixir-maru/Makefile M /head/www/elixir-maru/distinfo Upgrade to version 0.4.1. So yeah, svn 1.9 returned a timestamp that was off by 4s. WTF? 4 seconds?? There have been 4 leap seconds added this century. Did 1.9 add timestamp corrections relating to leap seconds? Did the developer not use leapsecs when the svn server does? -- FreeBSD - the place to B...Software Developing Shane Ambler ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FYI: SVN to GIT converter currently broken, github is falling behind
On Sun, Nov 8, 2015 at 12:06 PM, Ulrich Spörlein wrote: > 2015-11-08 11:32 GMT+01:00 Ulrich Spörlein : >> 2015-11-08 2:51 GMT+01:00 Alfred Perlstein : >>> Uli, >>> >>> One of the biggest concerns I've heard from folks using FreeBSD's git mirror >>> is that the hashes can change. >>> >>> I have a question about this. Is it possible to keep track of what the >>> "official" git mirror (on github) is doing and keep that as a log. Then >>> that log can be used to replay commits when there is a divergence problem. >>> >>> What I'm basically saying is that let's take this small example: >>> >>> importer is working fine @rev 1 >>> imports 1 >>> imports 10001 >>> imports 10002 >>> something happens to importer to give indeterminate shas. >>> imports 10003 - sha is "unstable" sha3 >>> imports 10004 - sha is "unstable" sha4 >>> imports 10005 - sha is "unstable" sha5 >>> imports 10006 - sha is "unstable" sha6 >>> importer is fixed >>> >>> >>> At this point normally we'd rewind the importer to 10002 and then force >>> update the affected branches. >>> >>> My question is... can the imports of 10003, 10004, 10005 and 10006 be put >>> into the importer such that any "mirror site" that re-does the import using >>> the most up to date importer will get the same shas. >>> >>> That would allow to proceed with 10007, etc without force pushing. >>> >>> This should be possible based on querying "git" for the meta data associated >>> with sha3..sha6 and then forcing those commits to have the same meta data. >>> >>> This would eliminate the concern about shas in the mirror changing that I've >>> heard. >> >> The goal of the conversion is that everyone can re-do the conversion >> in their basement and come up with the same history and checksums. >> This was not the case when I first started, as there was some >> non-deterministic hash structure being used in svn2git. This was fixed >> in the code and then all converter runs produced the very same >> results. >> >> The scenario that we have right now, is that one of the merge commits >> done about two weeks ago is being handled different by svn2git w/ svn >> v1.8 vs. svn v1.9 and I haven't investigated yet how the API's >> behavior changed to cause this. I'm afraid I also swapped out all my >> knowledge about svn2git internals and will have to redo this all from >> scratch :/ >> >> Your suggestion could only work, if we hard-code this svn revision >> special handling into svn2git, either in the code or by providing more >> mappings and rules to the process. svn2git should run hermetic and not >> poke at github's commits to see how things were handled in the past. >> It has to be self-sufficient and must not depend on github. >> >> This would also only work, if the "breakage" window was very small, >> but it is already about two weeks long and will surely increase till I >> find the proper fix. >> >> So, to take a stand here: this sort of kludge is unlikely to ever >> happen. Git commit hashes *might* change in the future. I really don't >> see how this is a big deal anyway. It happened once and I'm trying to >> have it never happen again. But why are people afraid of this >> happening? Every "official" git commit is tagged with a SVN revision >> and the contents of those revisions are obviously correct (just not >> the ancestry and the commit objects, possibly). So it would be easy to >> write a script that replays VendorA's git history and swaps out the >> new official commits for the old official commits. There would be no >> merge conflicts. >> >> I can see how this would be annoying if you have 100 developers and >> dozens of branches that are far from mainline FreeBSD. But I'm sure >> these companies that depend on git will come forward and donate some >> of their developer manpower to help me with keeping the converter >> stable/deterministic. Right? Right? :) :) >> >> Cheers, >> Uli > Hi Uli! I can not find your original svn2git repo in gitorius (https://gitorious.org/ is down) , could you please the source code somewhere to git-repo? For example github.com/freebsd/svn2git? > Quick update: doc is so far unaffected by svn 1.9, but for ports, the > drift happened as of Jul 18, so you'd need to special case a lot of > commits. > > Here's the same commit, and the difference between 1.8 and 1.9: > > % git cat-file commit 803795d > tree 7fc83aba022834da5c218114b09ad4640735bcc0 > parent c96fb0418e545a569b5975b4d878a30a948c29d5 > author olgeni 1437203525 + > committer olgeni 1437203525 + > > Upgrade to version 0.4.1. > % git cat-file commit 61ca43b > tree 7fc83aba022834da5c218114b09ad4640735bcc0 > parent c96fb0418e545a569b5975b4d878a30a948c29d5 > author olgeni 1437203529 + > committer olgeni 1437203529 + > > Upgrade to version 0.4.1. > > > In case you don't see it, there's a 4s difference in the timestamps > for authoring and committing. Here's the original: > > % svn log -vc392405 svn://svn.freebsd.org/ports > ---
Re: FYI: SVN to GIT converter currently broken, github is falling behind
On 8 Nov 2015, at 02:32, Ulrich Spörlein wrote: > > > Git commit hashes *might* change in the future. I really don't > see how this is a big deal anyway. It happened once and I'm trying to > have it never happen again. But why are people afraid of this > happening? Every "official" git commit is tagged with a SVN revision > and the contents of those revisions are obviously correct (just not > the ancestry and the commit objects, possibly). So it would be easy to > write a script that replays VendorA's git history and swaps out the > new official commits for the old official commits. There would be no > merge conflicts. Git commit hashes must not change, or we completely destroy the utility of the git mirror for all downstream users. You can no longer do a merge from upstream git if the hashes in your local clone do not match the hashes downstream. Your answer of expecting every downstream user of FreeBSD’s git repo (GitHub tracks over 600 of them, there are likely more that are using private clones) to write a script to fix the mess for themselves is completely unacceptable. If there has been a window in which incorrect hashes were generated, this can be fixed by moving that to a branch, doing the correct thing in master, and then using git-imerge in rebase-with-history mode. After the point of the fix, there will be a single set of commits, but before that there will be two options as parents, one for each version of the export. Please remember that a guarantee of not changing the hashes of the history was one of the conditions that Core had for promoting this to an official FreeBSD service. David ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FYI: SVN to GIT converter currently broken, github is falling behind
2015-11-08 11:32 GMT+01:00 Ulrich Spörlein : > 2015-11-08 2:51 GMT+01:00 Alfred Perlstein : >>> >> Uli, >> >> One of the biggest concerns I've heard from folks using FreeBSD's git mirror >> is that the hashes can change. >> >> I have a question about this. Is it possible to keep track of what the >> "official" git mirror (on github) is doing and keep that as a log. Then >> that log can be used to replay commits when there is a divergence problem. >> >> What I'm basically saying is that let's take this small example: >> >> importer is working fine @rev 1 >> imports 1 >> imports 10001 >> imports 10002 >> something happens to importer to give indeterminate shas. >> imports 10003 - sha is "unstable" sha3 >> imports 10004 - sha is "unstable" sha4 >> imports 10005 - sha is "unstable" sha5 >> imports 10006 - sha is "unstable" sha6 >> importer is fixed >> >> >> At this point normally we'd rewind the importer to 10002 and then force >> update the affected branches. >> >> My question is... can the imports of 10003, 10004, 10005 and 10006 be put >> into the importer such that any "mirror site" that re-does the import using >> the most up to date importer will get the same shas. >> >> That would allow to proceed with 10007, etc without force pushing. >> >> This should be possible based on querying "git" for the meta data associated >> with sha3..sha6 and then forcing those commits to have the same meta data. >> >> This would eliminate the concern about shas in the mirror changing that I've >> heard. > > The goal of the conversion is that everyone can re-do the conversion > in their basement and come up with the same history and checksums. > This was not the case when I first started, as there was some > non-deterministic hash structure being used in svn2git. This was fixed > in the code and then all converter runs produced the very same > results. > > The scenario that we have right now, is that one of the merge commits > done about two weeks ago is being handled different by svn2git w/ svn > v1.8 vs. svn v1.9 and I haven't investigated yet how the API's > behavior changed to cause this. I'm afraid I also swapped out all my > knowledge about svn2git internals and will have to redo this all from > scratch :/ > > Your suggestion could only work, if we hard-code this svn revision > special handling into svn2git, either in the code or by providing more > mappings and rules to the process. svn2git should run hermetic and not > poke at github's commits to see how things were handled in the past. > It has to be self-sufficient and must not depend on github. > > This would also only work, if the "breakage" window was very small, > but it is already about two weeks long and will surely increase till I > find the proper fix. > > So, to take a stand here: this sort of kludge is unlikely to ever > happen. Git commit hashes *might* change in the future. I really don't > see how this is a big deal anyway. It happened once and I'm trying to > have it never happen again. But why are people afraid of this > happening? Every "official" git commit is tagged with a SVN revision > and the contents of those revisions are obviously correct (just not > the ancestry and the commit objects, possibly). So it would be easy to > write a script that replays VendorA's git history and swaps out the > new official commits for the old official commits. There would be no > merge conflicts. > > I can see how this would be annoying if you have 100 developers and > dozens of branches that are far from mainline FreeBSD. But I'm sure > these companies that depend on git will come forward and donate some > of their developer manpower to help me with keeping the converter > stable/deterministic. Right? Right? :) :) > > Cheers, > Uli Quick update: doc is so far unaffected by svn 1.9, but for ports, the drift happened as of Jul 18, so you'd need to special case a lot of commits. Here's the same commit, and the difference between 1.8 and 1.9: % git cat-file commit 803795d tree 7fc83aba022834da5c218114b09ad4640735bcc0 parent c96fb0418e545a569b5975b4d878a30a948c29d5 author olgeni 1437203525 + committer olgeni 1437203525 + Upgrade to version 0.4.1. % git cat-file commit 61ca43b tree 7fc83aba022834da5c218114b09ad4640735bcc0 parent c96fb0418e545a569b5975b4d878a30a948c29d5 author olgeni 1437203529 + committer olgeni 1437203529 + Upgrade to version 0.4.1. In case you don't see it, there's a 4s difference in the timestamps for authoring and committing. Here's the original: % svn log -vc392405 svn://svn.freebsd.org/ports r392405 | olgeni | 2015-07-18 09:12:05 +0200 (Sat, 18 Jul 2015) | 2 lines Changed paths: M /head/www/elixir-maru/Makefile M /head/www/elixir-maru/distinfo Upgrade to version 0.4.1. So yeah, svn 1.9 returned a timestamp that was off by 4s. WTF? For base it's actu
Re: FYI: SVN to GIT converter currently broken, github is falling behind
2015-11-08 2:51 GMT+01:00 Alfred Perlstein : >> > Uli, > > One of the biggest concerns I've heard from folks using FreeBSD's git mirror > is that the hashes can change. > > I have a question about this. Is it possible to keep track of what the > "official" git mirror (on github) is doing and keep that as a log. Then > that log can be used to replay commits when there is a divergence problem. > > What I'm basically saying is that let's take this small example: > > importer is working fine @rev 1 > imports 1 > imports 10001 > imports 10002 > something happens to importer to give indeterminate shas. > imports 10003 - sha is "unstable" sha3 > imports 10004 - sha is "unstable" sha4 > imports 10005 - sha is "unstable" sha5 > imports 10006 - sha is "unstable" sha6 > importer is fixed > > > At this point normally we'd rewind the importer to 10002 and then force > update the affected branches. > > My question is... can the imports of 10003, 10004, 10005 and 10006 be put > into the importer such that any "mirror site" that re-does the import using > the most up to date importer will get the same shas. > > That would allow to proceed with 10007, etc without force pushing. > > This should be possible based on querying "git" for the meta data associated > with sha3..sha6 and then forcing those commits to have the same meta data. > > This would eliminate the concern about shas in the mirror changing that I've > heard. The goal of the conversion is that everyone can re-do the conversion in their basement and come up with the same history and checksums. This was not the case when I first started, as there was some non-deterministic hash structure being used in svn2git. This was fixed in the code and then all converter runs produced the very same results. The scenario that we have right now, is that one of the merge commits done about two weeks ago is being handled different by svn2git w/ svn v1.8 vs. svn v1.9 and I haven't investigated yet how the API's behavior changed to cause this. I'm afraid I also swapped out all my knowledge about svn2git internals and will have to redo this all from scratch :/ Your suggestion could only work, if we hard-code this svn revision special handling into svn2git, either in the code or by providing more mappings and rules to the process. svn2git should run hermetic and not poke at github's commits to see how things were handled in the past. It has to be self-sufficient and must not depend on github. This would also only work, if the "breakage" window was very small, but it is already about two weeks long and will surely increase till I find the proper fix. So, to take a stand here: this sort of kludge is unlikely to ever happen. Git commit hashes *might* change in the future. I really don't see how this is a big deal anyway. It happened once and I'm trying to have it never happen again. But why are people afraid of this happening? Every "official" git commit is tagged with a SVN revision and the contents of those revisions are obviously correct (just not the ancestry and the commit objects, possibly). So it would be easy to write a script that replays VendorA's git history and swaps out the new official commits for the old official commits. There would be no merge conflicts. I can see how this would be annoying if you have 100 developers and dozens of branches that are far from mainline FreeBSD. But I'm sure these companies that depend on git will come forward and donate some of their developer manpower to help me with keeping the converter stable/deterministic. Right? Right? :) :) Cheers, Uli ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FYI: SVN to GIT converter currently broken, github is falling behind
On 11/5/15 6:46 AM, Ulrich Spörlein wrote: 2015-11-04 18:57 GMT+01:00 Ulrich Spörlein : The recent SVN update on the cluster broke svn2git in certain circumstances. To fix this and make sure no content was dropped, the converter has been stopped and we're working on bringing a fixed version online, as well as vetting the correctness of the published git repositories. ETA is currently unknown, expect an update to this thread within 24h. Sorry for the inconvenience! Uli, on behalf of git-admin An independent run of the converter produces a different git repository starting at the commit following this one: https://github.com/freebsd/freebsd/commit/bf66c97c4a64e64410bf0223d221a54ca9555f52 This is from 9d ago and will likely require a force push to github that will necessitate people to rebase or merge there work (a fast-forward merge will fail). This is the preliminary inspection and a third verification run is currently underway. Expect another update within 24h. Uli, One of the biggest concerns I've heard from folks using FreeBSD's git mirror is that the hashes can change. I have a question about this. Is it possible to keep track of what the "official" git mirror (on github) is doing and keep that as a log. Then that log can be used to replay commits when there is a divergence problem. What I'm basically saying is that let's take this small example: importer is working fine @rev 1 imports 1 imports 10001 imports 10002 something happens to importer to give indeterminate shas. imports 10003 - sha is "unstable" sha3 imports 10004 - sha is "unstable" sha4 imports 10005 - sha is "unstable" sha5 imports 10006 - sha is "unstable" sha6 importer is fixed At this point normally we'd rewind the importer to 10002 and then force update the affected branches. My question is... can the imports of 10003, 10004, 10005 and 10006 be put into the importer such that any "mirror site" that re-does the import using the most up to date importer will get the same shas. That would allow to proceed with 10007, etc without force pushing. This should be possible based on querying "git" for the meta data associated with sha3..sha6 and then forcing those commits to have the same meta data. This would eliminate the concern about shas in the mirror changing that I've heard. -Alfred ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FYI: SVN to GIT converter currently broken, github is falling behind
2015-11-05 15:46 GMT+01:00 Ulrich Spörlein : > 2015-11-04 18:57 GMT+01:00 Ulrich Spörlein : >> The recent SVN update on the cluster broke svn2git in certain circumstances. >> >> To fix this and make sure no content was dropped, the converter has >> been stopped and we're working on bringing a fixed version online, as >> well as vetting the correctness of the published git repositories. >> >> ETA is currently unknown, expect an update to this thread within 24h. >> Sorry for the inconvenience! >> >> Uli, on behalf of git-admin > > An independent run of the converter produces a different git > repository starting at the commit following this one: > https://github.com/freebsd/freebsd/commit/bf66c97c4a64e64410bf0223d221a54ca9555f52 > > This is from 9d ago and will likely require a force push to github > that will necessitate people to rebase or merge there work (a > fast-forward merge will fail). > > This is the preliminary inspection and a third verification run is > currently underway. Expect another update within 24h. > > Uli We've rolled back to SVN 1.8 and have resumed normal operations. We're still working on getting svn2git to work nice with SVN 1.9 and produce the same repository checksums. Sorry for the inconvenience! Uli ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: FYI: SVN to GIT converter currently broken, github is falling behind
2015-11-04 18:57 GMT+01:00 Ulrich Spörlein : > The recent SVN update on the cluster broke svn2git in certain circumstances. > > To fix this and make sure no content was dropped, the converter has > been stopped and we're working on bringing a fixed version online, as > well as vetting the correctness of the published git repositories. > > ETA is currently unknown, expect an update to this thread within 24h. > Sorry for the inconvenience! > > Uli, on behalf of git-admin An independent run of the converter produces a different git repository starting at the commit following this one: https://github.com/freebsd/freebsd/commit/bf66c97c4a64e64410bf0223d221a54ca9555f52 This is from 9d ago and will likely require a force push to github that will necessitate people to rebase or merge there work (a fast-forward merge will fail). This is the preliminary inspection and a third verification run is currently underway. Expect another update within 24h. Uli ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"