[Wikidata-bugs] [Maniphest] [Commented On] T165074: CI tests failing with segfault
gerritbot added a comment. Change 358341 merged by Muehlenhoff: [operations/puppet@production] contint: install HHVM from main https://gerrit.wikimedia.org/r/358341TASK DETAILhttps://phabricator.wikimedia.org/T165074EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hashar, gerritbotCc: gerritbot, ArielGlenn, Stashbot, TerraCodes, Jay8g, Anomie, Ladsgroup, Mattflaschen-WMF, Paladox, greg, MoritzMuehlenhoff, Lucas_Werkmeister_WMDE, Jdforrester-WMF, hashar, daniel, Aklapper, Reedy, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Marostegui, Litanwei, Lewizho99, Zppix, Maathavan, Liudvikas, Izno, Luke081515, thcipriani, Wikidata-bugs, aude, faidon, zeljkofilipin, Mbch331, fgiunchedi, Joe, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T165074: CI tests failing with segfault
MoritzMuehlenhoff added a comment. @Ladsgroup I'm not sure how that Travis setup is configured, but if you make it install HHVM 3.18.2+dfsg-1+wmf3, that should be fixed.TASK DETAILhttps://phabricator.wikimedia.org/T165074EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hashar, MoritzMuehlenhoffCc: gerritbot, ArielGlenn, Stashbot, TerraCodes, Jay8g, Anomie, Ladsgroup, Mattflaschen-WMF, Paladox, greg, MoritzMuehlenhoff, Lucas_Werkmeister_WMDE, Jdforrester-WMF, hashar, daniel, Aklapper, Reedy, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Litanwei, Lewizho99, Zppix, Maathavan, Izno, thcipriani, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi, Joe, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T165074: CI tests failing with segfault
Ladsgroup added a comment. Just saying this also happens in travis instances causing Wikibase travis tests to fail. https://travis-ci.org/wikimedia/mediawiki-extensions-Wikibase/jobs/232431659 if there is anything specific needs to be done (like using 3.17 instead of 3.18) please tell me to apply to travis config.TASK DETAILhttps://phabricator.wikimedia.org/T165074EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hashar, LadsgroupCc: gerritbot, ArielGlenn, Stashbot, TerraCodes, Jay8g, Anomie, Ladsgroup, Mattflaschen-WMF, Paladox, greg, MoritzMuehlenhoff, Lucas_Werkmeister_WMDE, Jdforrester-WMF, hashar, daniel, Aklapper, Reedy, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Litanwei, Lewizho99, Zppix, Maathavan, Izno, thcipriani, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi, Joe, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T165074: CI tests failing with segfault
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2017-05-15T13:27:45Z] uploaded HHVM 3.18.2+dfsg-1+wmf3 to apt.wikimedia.org (addresses segfault in XML reader (T162586, T165074)TASK DETAILhttps://phabricator.wikimedia.org/T165074EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hashar, StashbotCc: gerritbot, ArielGlenn, Stashbot, TerraCodes, Jay8g, Anomie, Ladsgroup, Mattflaschen-WMF, Paladox, greg, MoritzMuehlenhoff, Lucas_Werkmeister_WMDE, Jdforrester-WMF, hashar, daniel, Aklapper, Reedy, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Litanwei, Lewizho99, Zppix, Maathavan, Izno, thcipriani, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi, Joe, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T165074: CI tests failing with segfault
MoritzMuehlenhoff added a comment. (Tested in mediawiki-vagrant)TASK DETAILhttps://phabricator.wikimedia.org/T165074EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hashar, MoritzMuehlenhoffCc: gerritbot, ArielGlenn, Stashbot, TerraCodes, Jay8g, Anomie, Ladsgroup, Mattflaschen-WMF, Paladox, greg, MoritzMuehlenhoff, Lucas_Werkmeister_WMDE, Jdforrester-WMF, hashar, daniel, Aklapper, Reedy, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Litanwei, Lewizho99, Zppix, Maathavan, Izno, thcipriani, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi, Joe, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T165074: CI tests failing with segfault
MoritzMuehlenhoff added a comment. I've built new HHVM packages with a patch as proposed by upstream in https://github.com/facebook/hhvm/issues/7779, that avoids the segfault and makes the phpunit tests pass with 3.18 again. I'll upload the new packages later on, after that we can revert the HHVM 3.12 hack again.TASK DETAILhttps://phabricator.wikimedia.org/T165074EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hashar, MoritzMuehlenhoffCc: gerritbot, ArielGlenn, Stashbot, TerraCodes, Jay8g, Anomie, Ladsgroup, Mattflaschen-WMF, Paladox, greg, MoritzMuehlenhoff, Lucas_Werkmeister_WMDE, Jdforrester-WMF, hashar, daniel, Aklapper, Reedy, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Litanwei, Lewizho99, Zppix, Maathavan, Izno, thcipriani, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi, Joe, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T165074: CI tests failing with segfault
gerritbot added a comment. Change 353533 merged by Muehlenhoff: [operations/puppet@production] contint: pin HHVM packages to use experimental component https://gerrit.wikimedia.org/r/353533TASK DETAILhttps://phabricator.wikimedia.org/T165074EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hashar, gerritbotCc: gerritbot, ArielGlenn, Stashbot, TerraCodes, Jay8g, Anomie, Ladsgroup, Mattflaschen-WMF, Paladox, greg, MoritzMuehlenhoff, Lucas_Werkmeister_WMDE, Jdforrester-WMF, hashar, daniel, Aklapper, Reedy, GoranSMilovanovic, Adik2382, Th3d3v1ls, Hfbn0, Ramalepe, Liugev6, QZanden, Litanwei, Lewizho99, Zppix, Maathavan, Izno, thcipriani, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi, Joe, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T165074: CI tests failing with segfault
gerritbot added a comment. Change 353533 had a related patch set uploaded (by Hashar; owner: Hashar): [operations/puppet@production] contint: pin HHVM packages to use experimental component https://gerrit.wikimedia.org/r/353533TASK DETAILhttps://phabricator.wikimedia.org/T165074EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hashar, gerritbotCc: gerritbot, ArielGlenn, Stashbot, TerraCodes, Jay8g, Anomie, Ladsgroup, Mattflaschen-WMF, Paladox, greg, MoritzMuehlenhoff, Lucas_Werkmeister_WMDE, Jdforrester-WMF, hashar, daniel, Aklapper, Reedy, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Litanwei, Zppix, Izno, thcipriani, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi, Joe, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T165074: CI tests failing with segfault
MoritzMuehlenhoff added a comment. The new HHVM version has been extensively tested on five canary servers in production for 5-6 week now. As per Reedy's backtrace from above it has in fact identified the crash we already knew https://phabricator.wikimedia.org/T162586 (reported upstream at https://github.com/facebook/hhvm/issues/7779 but so far there wasn't a better reproducer than "expose 3 days of Wikimedia production traffic" for it :-) So the test suite has proven to be really useful. Going forward from here, we should extend CI to also cover the "experimental" component from our repo (at least for partial CI runs). That would have identified that much quicker.TASK DETAILhttps://phabricator.wikimedia.org/T165074EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hashar, MoritzMuehlenhoffCc: Stashbot, TerraCodes, Jay8g, Anomie, Ladsgroup, Mattflaschen-WMF, Paladox, greg, MoritzMuehlenhoff, Lucas_Werkmeister_WMDE, Jdforrester-WMF, hashar, daniel, Aklapper, Reedy, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Litanwei, Zppix, Izno, thcipriani, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi, Joe, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T165074: CI tests failing with segfault
Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2017-05-11T20:57:55Z] CI Phpunit jobs were segfaulting due to an upgrade of HHVM to 3.18. Got rolled back to 3.12 - T165074TASK DETAILhttps://phabricator.wikimedia.org/T165074EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hashar, StashbotCc: Stashbot, TerraCodes, Jay8g, Anomie, Ladsgroup, Mattflaschen-WMF, Paladox, greg, MoritzMuehlenhoff, Lucas_Werkmeister_WMDE, Jdforrester-WMF, hashar, daniel, Aklapper, Reedy, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Litanwei, Zppix, Izno, thcipriani, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi, Joe, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T165074: CI tests failing with segfault
greg added a comment. We should really use the new HHVM in testing first before going to production. If the tests are broken it means fix the tests/the underlying issue, not work around it by disabling the test and continuing to use the (now uncertain if that test actually found a bug) in production. CI is fine now, we've downgraded there. But we should be in sync on this going forward.TASK DETAILhttps://phabricator.wikimedia.org/T165074EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gregCc: Stashbot, TerraCodes, Jay8g, Anomie, Ladsgroup, Mattflaschen-WMF, Paladox, greg, MoritzMuehlenhoff, Lucas_Werkmeister_WMDE, Jdforrester-WMF, hashar, daniel, Aklapper, Reedy, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Litanwei, Zppix, Izno, thcipriani, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi, Joe, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T165074: CI tests failing with segfault
Reedy added a comment. Ok, so a clean vagrant vm (with 4GB ram!), will segfault by running phpunit with no extensions From gdb attached... Continuing. [New Thread 0x7f1b053ff700 (LWP 2337)] [Thread 0x7f1b053ff700 (LWP 2337) exited] [New Thread 0x7f1b053ff700 (LWP 2338)] [Thread 0x7f1b053ff700 (LWP 2338) exited] Program received signal SIGSEGV, Segmentation fault. HPHP::UserFile::close (this=0x7f1afd461090) at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/base/user-file.cpp:212 212 /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/base/user-file.cpp: No such file or directory. (gdb) bt #0 HPHP::UserFile::close (this=0x7f1afd461090) at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/base/user-file.cpp:212 #1 0x01a74ec2 in HPHP::XMLReader::close (this=0x7f1afd411ec0) at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/ext/xmlreader/ext_xmlreader.cpp:95 #2 0x02133130 in HPHP::MemoryManager::sweep (this=0x7f1b25a8c840, this@entry=) at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/base/memory-manager.cpp:471 #3 0x021324dd in HPHP::hphp_memory_cleanup () at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/base/program-functions.cpp:2322 #4 0x00fd6605 in HPHP::hphp_session_exit () at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/base/program-functions.cpp:2370 #5 0x00fda5df in HPHP::execute_command_line_end (xhprof=0, coverage=coverage@entry=true, program=0x7f1b0ee48d58 "tests/phpunit/phpunit.php") at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/base/program-functions.cpp:721 #6 0x00fe2d95 in HPHP::execute_program_impl (argc=argc@entry=4, argv=argv@entry=0x7fff36a84068) at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/base/program-functions.cpp:1830 #7 0x00fe50be in HPHP::execute_program (argc=argc@entry=4, argv=argv@entry=0x7fff36a84068) at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/runtime/base/program-functions.cpp:1148 #8 0x00b56d6a in main (argc=, argv=0x7fff36a84068) at /tmp/buildd/hhvm-3.18.2+dfsg/hphp/hhvm/main.cpp:83TASK DETAILhttps://phabricator.wikimedia.org/T165074EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: ReedyCc: Stashbot, TerraCodes, Jay8g, Anomie, Ladsgroup, Mattflaschen-WMF, Paladox, greg, MoritzMuehlenhoff, Lucas_Werkmeister_WMDE, Jdforrester-WMF, hashar, daniel, Aklapper, Reedy, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Litanwei, Zppix, Izno, thcipriani, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi, Joe, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T165074: CI tests failing with segfault
greg added a comment. @MoritzMuehlenhoff we should probably downgrade the HHVM version from Beta and CI and work on repro'ing elsewhere. This is preventing merges and is UBN.TASK DETAILhttps://phabricator.wikimedia.org/T165074EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gregCc: Stashbot, TerraCodes, Jay8g, Anomie, Ladsgroup, Mattflaschen-WMF, Paladox, greg, MoritzMuehlenhoff, Lucas_Werkmeister_WMDE, Jdforrester-WMF, hashar, daniel, Aklapper, Reedy, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Litanwei, Zppix, Izno, thcipriani, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi, Joe, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T165074: CI tests failing with segfault
Stashbot added a comment. Mentioned in SAL (#wikimedia-releng) [2017-05-11T20:43:40Z] nodepool: delete today jessie image snapshot. It comes with HHVM 3.18 which segfault with MediaWiki/PHPUnit. Rolled back to snapshot-ci-jessie-1494425642 from 30 hours ago. T165074TASK DETAILhttps://phabricator.wikimedia.org/T165074EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: StashbotCc: Stashbot, TerraCodes, Jay8g, Anomie, Ladsgroup, Mattflaschen-WMF, Paladox, greg, MoritzMuehlenhoff, Lucas_Werkmeister_WMDE, Jdforrester-WMF, hashar, daniel, Aklapper, Reedy, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Litanwei, Zppix, Izno, thcipriani, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi, Joe, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T165074: CI tests failing with segfault
hashar added a comment. The snapshots we have: IDProviderImageHostnameVersionImage IDServer IDStateAge (hours) 1537wmflabs-eqiadsnapshot-ci-jessiesnapshot-ci-jessie-14945120421494512042f8c28074-e1aa-4703-b09f-b23fa4d4fc08806124d0-8b6c-4064-b3e9-9f561105acabready6.27 1535wmflabs-eqiadsnapshot-ci-jessiesnapshot-ci-jessie-1494425642149442564221671a4d-1d8a-4f2d-aa8f-a929bde2f3d81cbf9a1a-e634-42d2-ae39-8254d7c0e991ready30.23 TASK DETAILhttps://phabricator.wikimedia.org/T165074EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: hasharCc: TerraCodes, Jay8g, Anomie, Ladsgroup, Mattflaschen-WMF, Paladox, greg, MoritzMuehlenhoff, Lucas_Werkmeister_WMDE, Jdforrester-WMF, hashar, daniel, Aklapper, Reedy, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Litanwei, Zppix, Izno, thcipriani, Wikidata-bugs, aude, faidon, Mbch331, fgiunchedi, Joe, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T165074: CI tests failing with segfault
greg added a comment. Can someone create a simple repo case? Or at least a backtrace?TASK DETAILhttps://phabricator.wikimedia.org/T165074EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: gregCc: Paladox, greg, MoritzMuehlenhoff, Lucas_Werkmeister_WMDE, Jdforrester-WMF, hashar, daniel, Aklapper, Reedy, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Litanwei, Zppix, Izno, thcipriani, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, fgiunchedi, Joe, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
[Wikidata-bugs] [Maniphest] [Commented On] T165074: CI tests failing with segfault
Paladox added a comment. May be related T165043TASK DETAILhttps://phabricator.wikimedia.org/T165074EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: PaladoxCc: Paladox, greg, MoritzMuehlenhoff, Lucas_Werkmeister_WMDE, Jdforrester-WMF, hashar, daniel, Aklapper, Reedy, GoranSMilovanovic, Th3d3v1ls, Hfbn0, QZanden, Litanwei, Zppix, Izno, thcipriani, Wikidata-bugs, aude, faidon, Mbch331, Jay8g, fgiunchedi, Joe, jeremyb___ Wikidata-bugs mailing list Wikidata-bugs@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs
