Bug#724610: Acknowledgement (incrementally updated databases eventually become corrupt)

[Olly Betts]

On Tue, Dec 03, 2013 at 01:51:23PM -0400, Joey Hess wrote:
> Olly Betts wrote:
> > I think a good test would be to take the git repo for an ikiwiki
> > instance and replay the commits to reproduce an actual wiki being
> > updated as closely as possible.
> > 
> > Can you suggest some suitable ikiwiki instances with a public git repos?
> 
> Yes, it's happened repeatedly to my personal website, joeyh.name, and
> although I don't remember the exact times, I don't make too many commits
> there so it would not be hard to replay eg, a year's commits. It looks
> like I last deleted and rebuilt the xapian database there on Nov 3,
> possibly after it was broken for a few days.
> 
> git clone git://joeyh.branchable.com/

I've written a script (attached) to replay a git repo one commit at a
time, updating ikiwiki's xapian index each time.  So far it's not
reproduced the problem in a few test runs, but I'll leave it going in a
loop and see what happens.

I'm using this (from perl) to invoke ikiwiki to regenerate the wiki:

my $url = 'http://joeyh.branchable.com/';
my $cgiurl = 'http://joeyh.branchable.com/cgi-bin/';
my $outdir = '../output.tmp';
system('ikiwiki', '--cgiurl', $cgiurl, '--url', $url, '--plugin', 'search', 
'.', $outdir)

Are there are other plugins or other options which might be relevant?

Also, would the ikiwiki update normally happen after every commit, or
do multiple commits sometimes get processed together?

Cheers,
    Olly

#!/usr/bin/perl
use strict;
use warnings;
use autodie;

my $giturl = 'git://joeyh.branchable.com/';
my $gitdir = 'joeyh.branchable.com';
my $url = 'http://joeyh.branchable.com/';
my $cgiurl = 'http://joeyh.branchable.com/cgi-bin/';
my $outdir = '../output.tmp';

my $xapian_check = $ENV{XAPIAN_CHECK} || 'xapian-check';

if (! -d $gitdir) {
    # Initial clone.
    system("git", "clone", $giturl) == 0
        or die "git clone failed: $!";
}

chdir $gitdir;

# Clean up any generated files from previous run.
system 'rm', '-rf', 'html', '.ikiwiki', $outdir;

open REVLIST, 'git log --reverse --format=format:%H master|';
while (<REVLIST>) {
    chomp;
    my $rev = $_;
    system('git', 'checkout', $rev) == 0
        or die "git checkout $rev failed: $!";
    system('ikiwiki', '--cgiurl', $cgiurl, '--url', $url, '--plugin', 'search', 
'.', $outdir) == 0
        or die "ikiwiki failed: $!";
    system($xapian_check, '.ikiwiki/xapian/default') == 0
        or die "xapian-check failed: $!";
}
close REVLIST;