[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Mark A. Hershberger changed: What|Removed |Added Target Milestone|1.21.x release |--- --- Comment #29 from Mark A. Hershberger --- Removing target milestone that was in the past. If you want this in a specific release, have a good reason AND you are willing to find resources to fix this bug, feel free to change it to something appropriate. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #28 from Chris Koerner --- After changing $wgCliPhp to be blank my job queue now runs and clears out jobs without issue. $wgPhpCli = ""; As discovered in this thread: https://www.mediawiki.org/w/index.php?title=Project:Support_desk&offset=20140227154556&lqt_mustshow=40130#Mediawiki_1.22.2B_Causes_two_of_my_servers_to_hang_indefinitely._37734 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Andre Klapper changed: What|Removed |Added See Also||https://bugzilla.wikimedia. ||org/show_bug.cgi?id=60719 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #27 from Aaron Schulz --- If the only jobs that linger have job_attempts set to something other than zero, then this is just a problem of failed jobs. You'd want to set wgDebugLogGroups as above to possibly get more insights on why jobs are failing sometimes. Jobs can fail for any number of reasons, mostly specific to the code the job classes run. That in itself doesn't indicate any problem with the job queue itself. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #26 from Chris Koerner --- Aaron, can you explain a little more about the claim TTL? I don't recall setting that anywhere. I'm trying to pinpoint the cause as much as I can without touching my prod environment. We didn't have this issue in test (berating me for not having more sophisticated QA is justified!) and in order to make changes to prod I have many hoops I must jump through now. One interesting note is that trying to specify the --memory-limit when running runJobs.php throws an error. I'm beginning to think this might be related to available ram. (Server has 2gb, php.ini has 128mb) php runJobs.php --memory-limit 1024 PHP Fatal error: Allowed memory size of 262144 bytes exhausted (tried to allocate 122880 bytes) in /var/www/html/w/includes/AutoLoader.php on line 290 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #25 from Aaron Schulz --- What is $wgJobTypeConf set to? All the jobs you posted had at least 1 attempt. They won't run again until the claim TTL is reached. I don't know what you set that too. By default, jobs the fail are never retried and get deleted after a week. You can try using: $wgJobTypeConf['default']['claimTTL'] = 3600; // 1 hour ...this will let the jobs be retried (after 1 hour of failure). You can also set: $wgDebugLogGroups['runJobs'] = "" ...this will log all jobs run, and may show some failures (fatal errors will not show here though). -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #24 from Chris Koerner --- Another note (and someone tell me to shut up if this isn't proper etiquette) it appears that running php refreshLinks.php will queue up a number of jobs. Running runJobs.php afterward will kick off many jobs (in a queue of 2000, 1500 or so) but does not complete all jobs as a result of refreshLinks.php or any of the other jobs queued. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #23 from Chris Koerner --- I should note that certain actions on the site, such as modifying a template or running refreshLinks.php appears to not only add new jobs to the queue (as it should) but I can also run "runJobs.php" and a number of jobs will process. I don't see a pattern or commonality in what is run however. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #22 from Chris Koerner --- Nothing appears to run. http://i.imgur.com/eypighU.jpg -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #21 from Aaron Schulz --- What does <> show? What about <>? Does <> actually run anything? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #19 from Chris Koerner --- Here's the job table from one of our wikis (total separate installations, but configured identically) http://pastebin.com/i4Qatgpa -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #20 from Chris Koerner --- Note, this is after I removed some of the older jobs in an attempt to 'kick start' the queue. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #18 from Aaron Schulz --- (In reply to comment #17) > I had a similar thing happen. We were running 1.16.3 (yeah!) and upgraded to > 1.23. Made a copy of the 1.16.3 database, pointed new 1.23wmf11 installation > to > new database. Ran update.php, everything is hunky-dory. Notice the next day > that job queue is backed up. A few old jobs existed from the day of the > upgrade > (in 1.16.3). Tried clearing job_token and a few would run. Tried clearing out > old pre 1.23 jobs, still not running. showJobs.php spits out 0, while api > query > and db shows jobs in the queue. What types of jobs? What to some of the rows look like? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Chris Koerner changed: What|Removed |Added CC||nob...@gmail.com --- Comment #17 from Chris Koerner --- I had a similar thing happen. We were running 1.16.3 (yeah!) and upgraded to 1.23. Made a copy of the 1.16.3 database, pointed new 1.23wmf11 installation to new database. Ran update.php, everything is hunky-dory. Notice the next day that job queue is backed up. A few old jobs existed from the day of the upgrade (in 1.16.3). Tried clearing job_token and a few would run. Tried clearing out old pre 1.23 jobs, still not running. showJobs.php spits out 0, while api query and db shows jobs in the queue. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #16 from Nemo --- WikiApiary still had hundreds of lingering jobs since October, till they dropped them from the DB yesterday because it was impossible to run them. We're still waiting for a general solution to the migration problems. http://lists.thingelstad.com/pipermail/wikiapiary-l/2014-February/000104.html http://lists.thingelstad.com/pipermail/wikiapiary-l/attachments/20140202/fd838fb7/attachment-0002.png http://lists.thingelstad.com/pipermail/wikiapiary-l/attachments/20140202/fd838fb7/attachment-0003.png -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Mark A. Hershberger changed: What|Removed |Added Status|RESOLVED|REOPENED Resolution|WONTFIX |--- Target Milestone|--- |1.21.x release --- Comment #15 from Mark A. Hershberger --- (In reply to comment #14) > hexmode said this wasn't deemed worth fixing for 1.21 release. I don't know > the reasons, but better a partial update than nothing. What I meant is that this isn't going to stop 1.21.0 from being released. It is still a valid bug that should be fixed at some point. If we can get it fixed in a 1.21 point release that would be great. I don't know enough about the problem to fix it, though. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Nemo changed: What|Removed |Added Status|NEW |RESOLVED Resolution|--- |WONTFIX Target Milestone|1.21.x release |--- --- Comment #14 from Nemo --- hexmode said this wasn't deemed worth fixing for 1.21 release. I don't know the reasons, but better a partial update than nothing. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Nemo changed: What|Removed |Added Target Milestone|1.21.0 release |1.21.x release -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #13 from Aaron Schulz --- (In reply to comment #12) > (In reply to comment #11) > > I mean job_random of course, not the token. > > Ah. :) Well, isn't setting job_random exactly what the user in comment 0 > didn't > do and should have done? The complaint was that 0 valued ones didn't work, so I set all mine to that value and they still worked. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #12 from Nemo --- (In reply to comment #11) > I mean job_random of course, not the token. Ah. :) Well, isn't setting job_random exactly what the user in comment 0 didn't do and should have done? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #11 from Aaron Schulz --- (In reply to comment #10) > In master and REL1_21 I've tried > adding > a bunch of jobs and setting the token to 0 for all of them, and runJobs.php > works just fine. They weren't lost for me. I mean job_random of course, not the token. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #10 from Aaron Schulz --- (In reply to comment #0) > See http://lists.wikimedia.org/pipermail/mediawiki-l/2013-April/040970.html > > I remember that WMF had a similar problem, is there a solution apart from > dropping the old jobs? Did we? The only problem is that they have a harder time getting picked if there always a bunch of other new jobs. In master and REL1_21 I've tried adding a bunch of jobs and setting the token to 0 for all of them, and runJobs.php works just fine. They weren't lost for me. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #9 from Tyler Romeo --- (In reply to comment #8) > Could maintenance/runJobs.php be run with the wiki in read-only mode? Nope, I meant making it read-only after clearing the job queue. Still not a complete solution, but I can't think of anything else. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #8 from Mark A. Hershberger --- Could maintenance/runJobs.php be run with the wiki in read-only mode? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Tyler Romeo changed: What|Removed |Added CC||tylerro...@gmail.com --- Comment #7 from Tyler Romeo --- (In reply to comment #6) > (In reply to comment #3) > > Do we seriously have no way to fix this? Should we just tell people not to > > upgrade if they have something in the job queue? > > I'd like to include something in the installation information telling users > to > clear their job queue before upgrading, but I don't know a lot about this. > What would be appropriate? > > I would like to get this fixed ASAP for a point release. Running maintenance/runJobs.php should clear the job queue. But depending on how long upgrading takes, one or two jobs might still be lost. Maybe put the wiki into read-only mode once the job queue is cleared? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Mark A. Hershberger changed: What|Removed |Added CC||m...@everybody.org --- Comment #6 from Mark A. Hershberger --- (In reply to comment #3) > Do we seriously have no way to fix this? Should we just tell people not to > upgrade if they have something in the job queue? I'd like to include something in the installation information telling users to clear their job queue before upgrading, but I don't know a lot about this. What would be appropriate? I would like to get this fixed ASAP for a point release. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #5 from Andre Klapper --- Aaron: Could you comment on this please, as the 1.21 tarball release is imminent? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #4 from Andre Klapper --- Aaron: Could you comment on this, please? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 --- Comment #3 from Nemo --- Do we seriously have no way to fix this? Should we just tell people not to upgrade if they have something in the job queue? -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Andre Klapper changed: What|Removed |Added Priority|Unprioritized |High CC||g...@wikimedia.org -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Bartosz Dziewoński changed: What|Removed |Added CC||matma@gmail.com --- Comment #2 from Bartosz Dziewoński --- Quoting from the duped bug: (Quoting bug 46971 comment #0) > After I upgraded from 1.20.3, in my database (MySQL) some > pre-upgrade jobs have job_random set to 0 and do not seem to be > picked up -not even when I try to run them by providing their type > as an option: php runJobs.php --type=replaceText. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 46934] Job queue items from 1.20 get lost – need exit strategy for upgrade
https://bugzilla.wikimedia.org/show_bug.cgi?id=46934 Bartosz Dziewoński changed: What|Removed |Added CC||isabel.ganc...@gmail.com --- Comment #1 from Bartosz Dziewoński --- *** Bug 46971 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
