Re: [Wikitech-l] Gerrit server issues
On Wed, Sep 12, 2012 at 3:51 PM, Platonides wrote: > Doesn't formey have core.logAllRefUpdates set to true? > Wouldn't that have prevented git gc from removing commits referenced in > the reflog? (at least until two weeks without the references passed, it > should probably have been run with --no-prune) > I did indeed manage to recover some commits that were lost (apparently gc'ed) on manganese but still present on formey, so that sounds plausible. Roan ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Gerrit server issues
http://blog.wikimedia.org/2012/09/07/recovery-of-broken-gerrit-repositories/ writes: > While Git does make it very difficult to get rid of data forever, it is > possible. For example, if we had let the cron run again, git gc would’ve > merrily deleted all of the unreferenced objects since they were unused. > So, now it just became a matter of sorting out which refs these commits > could go to. Playing around with git show makes it pretty easy to figure > out where the commits belong, and so this just turned into a process of > editing the appropriate refs/* files to add the commit hash (and what >seems so obvious now took several hours of hand-wringing, I assure you). > So for the next several hours, I took the time to sort out these dangling > commits, and the repos ended up working again. Doesn't formey have core.logAllRefUpdates set to true? Wouldn't that have prevented git gc from removing commits referenced in the reflog? (at least until two weeks without the references passed, it should probably have been run with --no-prune) ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Gerrit server issues
On Wed, Sep 12, 2012 at 4:53 PM, Raimond Spekking wrote: > Am 11.09.2012 06:52, schrieb Roan Kattouw: >> On Fri, Sep 7, 2012 at 2:12 PM, Chad wrote: >>> All of these have been fixed other than Nonlinear (more heavily >>> broken). TranslationNotifications' master is intact, but some of >>> the changes are still in a bad state and I need to finish cleaning >>> it up. >>> >> TranslationNotifications has now been fixed. >> > Could you please check the permissions? L10n-bot is not longer allowed > to merge its commits: > > https://gerrit.wikimedia.org/r/#/c/23455/ > https://gerrit.wikimedia.org/r/#/c/23565/ > > It fails with > > Merging 23565... > error: Verified=1 not permitted > one or more approvals failed; review output above > The permissions were not inheriting from mediawiki/extensions, they were instead inheriting from All-Projects. This is now fixed. -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Gerrit server issues
Am 11.09.2012 06:52, schrieb Roan Kattouw: > On Fri, Sep 7, 2012 at 2:12 PM, Chad wrote: >> All of these have been fixed other than Nonlinear (more heavily >> broken). TranslationNotifications' master is intact, but some of >> the changes are still in a bad state and I need to finish cleaning >> it up. >> > TranslationNotifications has now been fixed. > Could you please check the permissions? L10n-bot is not longer allowed to merge its commits: https://gerrit.wikimedia.org/r/#/c/23455/ https://gerrit.wikimedia.org/r/#/c/23565/ It fails with Merging 23565... error: Verified=1 not permitted one or more approvals failed; review output above Thanks. Raimond. signature.asc Description: OpenPGP digital signature ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Gerrit server issues
On Fri, Sep 7, 2012 at 2:12 PM, Chad wrote: > All of these have been fixed other than Nonlinear (more heavily > broken). TranslationNotifications' master is intact, but some of > the changes are still in a bad state and I need to finish cleaning > it up. > TranslationNotifications has now been fixed. > software, wikimedia-lvs-realserver, wikimedia-search-qa and > wikistats are all back and fine. mysqlatfacebook is very broken > like Nonlinear. > I fixed mysqlatfacebook, which had been completely garbage-collected on manganese (the primary server). I restored the commits from formey (the replication slave) where they'd been orphaned but not deleted. Nonlinear is also fixed. Roan ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Gerrit server issues
On Fri, 07 Sep 2012 08:30:47 -0700, Santhosh Thottingal wrote: 2012/9/7 Chad : And mediawiki/extensions/TranslationNotifications is back up too, master was intact. The remaining ones in extensions/* and debs/* are under repair. It seems TranslationNotifications extension has some problem. Git review is always failing with this error: error: unpack failed: error Missing unknown 4ee9dc02a655cb376a8f20e6d5c1ee95a81b1a37 And remote rejected with unpack error. ps: Some information about this corruption available at http://asheepapart.blogspot.com/2011/10/gerrit-code-review-unpack-error-missing.html Thanks Santhosh Thottingal I'd like to know what kind of blobs it is that Gerrit depends on so much but git is happy to prune. -- ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name] ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Gerrit server issues
On Thu, Sep 6, 2012 at 5:11 PM, Chad wrote: > In mediawiki/extensions/*: > Comments, FacebookOpenGraph, GoogleDocs4MW, Nonlinear, > OnlineStatusBar, Phalanx, RandomImageByCategory, > SemanticImageInput, ShoutWikiAds, SphinxSearch, > TranslationNotifcations > All of these have been fixed other than Nonlinear (more heavily broken). TranslationNotifications' master is intact, but some of the changes are still in a bad state and I need to finish cleaning it up. > In operations/*: > debs/mysqlatfacebook, debs/wikimedia-lvs-realserver > debs/wikimedia-search-qa, debs/wikistats, software > software, wikimedia-lvs-realserver, wikimedia-search-qa and wikistats are all back and fine. mysqlatfacebook is very broken like Nonlinear. -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Gerrit server issues
On Fri, Sep 7, 2012 at 11:30 AM, Santhosh Thottingal wrote: > 2012/9/7 Chad : >> And mediawiki/extensions/TranslationNotifications is back up too, master >> was intact. >> >> The remaining ones in extensions/* and debs/* are under repair. >> > > It seems TranslationNotifications extension has some problem. Git > review is always failing with this error: > > error: unpack failed: error Missing unknown > 4ee9dc02a655cb376a8f20e6d5c1ee95a81b1a37 > > And remote rejected with unpack error. > > ps: Some information about this corruption available at > http://asheepapart.blogspot.com/2011/10/gerrit-code-review-unpack-error-missing.html > Ah indeed, that repo does need some work. This is the only thing I'm working on today, and I figured out how to resolve it late last night--things will be back to normal by the end of today. -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Gerrit server issues
2012/9/7 Chad : > And mediawiki/extensions/TranslationNotifications is back up too, master > was intact. > > The remaining ones in extensions/* and debs/* are under repair. > It seems TranslationNotifications extension has some problem. Git review is always failing with this error: error: unpack failed: error Missing unknown 4ee9dc02a655cb376a8f20e6d5c1ee95a81b1a37 And remote rejected with unpack error. ps: Some information about this corruption available at http://asheepapart.blogspot.com/2011/10/gerrit-code-review-unpack-error-missing.html Thanks Santhosh Thottingal ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Gerrit server issues
On Thu, Sep 6, 2012 at 9:43 PM, Chad wrote: > On Thu, Sep 6, 2012 at 5:11 PM, Chad wrote: >> Hi everyone, >> >> Just letting everyone know that we're having some issues with the Gerrit >> server at the moment. Sometime since last night, a little over a dozen >> repositories (listed below) have gone a wonky and are missing their >> branches & tags (objects are intact). >> >> We're working on recovering everything right now, and I'll be sure to let >> everyone know the status when we're done. >> >> In the meantime, if you use any of the following repositories, please >> refrain from trying to use them until I give the all-clear (I don't want to >> exacerbate any problems): >> >> In mediawiki/extensions/*: >> Comments, FacebookOpenGraph, GoogleDocs4MW, Nonlinear, >> OnlineStatusBar, Phalanx, RandomImageByCategory, >> SemanticImageInput, ShoutWikiAds, SphinxSearch, >> TranslationNotifcations >> >> In operations/*: >> debs/mysqlatfacebook, debs/wikimedia-lvs-realserver >> debs/wikimedia-search-qa, debs/wikistats, software >> >> Also note: operations/mediawiki-config was busted, but we went ahead >> and just rebuilt the repo from the live copy so as to not interfere with >> site operations. >> >> I thank you for your patience, and please find me on IRC if you have >> any questions or find other repositories that are misbehaving. >> > > Just to let everyone know, I'm still working on restoring these projects, > but they're not ready yet. > > operations/software.git is back up, though, master was intact after all. > And mediawiki/extensions/TranslationNotifications is back up too, master was intact. The remaining ones in extensions/* and debs/* are under repair. -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Re: [Wikitech-l] Gerrit server issues
On Thu, Sep 6, 2012 at 5:11 PM, Chad wrote: > Hi everyone, > > Just letting everyone know that we're having some issues with the Gerrit > server at the moment. Sometime since last night, a little over a dozen > repositories (listed below) have gone a wonky and are missing their > branches & tags (objects are intact). > > We're working on recovering everything right now, and I'll be sure to let > everyone know the status when we're done. > > In the meantime, if you use any of the following repositories, please > refrain from trying to use them until I give the all-clear (I don't want to > exacerbate any problems): > > In mediawiki/extensions/*: > Comments, FacebookOpenGraph, GoogleDocs4MW, Nonlinear, > OnlineStatusBar, Phalanx, RandomImageByCategory, > SemanticImageInput, ShoutWikiAds, SphinxSearch, > TranslationNotifcations > > In operations/*: > debs/mysqlatfacebook, debs/wikimedia-lvs-realserver > debs/wikimedia-search-qa, debs/wikistats, software > > Also note: operations/mediawiki-config was busted, but we went ahead > and just rebuilt the repo from the live copy so as to not interfere with > site operations. > > I thank you for your patience, and please find me on IRC if you have > any questions or find other repositories that are misbehaving. > Just to let everyone know, I'm still working on restoring these projects, but they're not ready yet. operations/software.git is back up, though, master was intact after all. -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
[Wikitech-l] Gerrit server issues
Hi everyone, Just letting everyone know that we're having some issues with the Gerrit server at the moment. Sometime since last night, a little over a dozen repositories (listed below) have gone a wonky and are missing their branches & tags (objects are intact). We're working on recovering everything right now, and I'll be sure to let everyone know the status when we're done. In the meantime, if you use any of the following repositories, please refrain from trying to use them until I give the all-clear (I don't want to exacerbate any problems): In mediawiki/extensions/*: Comments, FacebookOpenGraph, GoogleDocs4MW, Nonlinear, OnlineStatusBar, Phalanx, RandomImageByCategory, SemanticImageInput, ShoutWikiAds, SphinxSearch, TranslationNotifcations In operations/*: debs/mysqlatfacebook, debs/wikimedia-lvs-realserver debs/wikimedia-search-qa, debs/wikistats, software Also note: operations/mediawiki-config was busted, but we went ahead and just rebuilt the repo from the live copy so as to not interfere with site operations. I thank you for your patience, and please find me on IRC if you have any questions or find other repositories that are misbehaving. -Chad ___ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
