https://bugzilla.wikimedia.org/show_bug.cgi?id=27113
Summary: be able to restart history dump after breakage, from where it was interrupted Product: XML Snapshots Version: unspecified Platform: All OS/Version: All Status: NEW Severity: enhancement Priority: Normal Component: General AssignedTo: ar...@wikimedia.org ReportedBy: ar...@wikimedia.org CC: tf...@wikimedia.org Blocks: 27110 Dumping the page-meta-history file is the phase that takes the longest; when some external factor causes the dumps to fail (a code push that breaks them, network/db/power/space/other issues), they currently must be restarted from the beginning. Even when they complete in 2 weeks instead of 6 weeks, the odds of something going wrong in that time is quite high. Being able to restart from the point of interruption would mean being able to produce them on a reasonable schedule. Code available: find last page id in file form interrupted run (works only for bz2 files), by seeking to the end and walking through compressed blocks. Code needed: stream this file to a filter which writes out the MediaWiki header, writes everything up to but excluding the last pageID, writes the MediaWiki footer; this output can be piped to bzip2 to produce an intact bzip2 file. We can then run from that pageID to the end, take the two bzip2 files, recombine them and be done. Why can't we just find the truncated bzip2 block, toss it, and start from there? Because at the end of a file bzip2 requires a cumulative crc algorithm, which means rereading all the text the minute we want to add blocks at the end. -- Configure bugmail: https://bugzilla.wikimedia.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug. _______________________________________________ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l