Bug#724610: Acknowledgement (incrementally updated databases eventually become corrupt)

[Joey Hess]

Here is a cleaner reproduction of this.

I ran attached foo.pl once. I hit ctrl-c after a few seconds.
xapian-check reported the db was ok, and all on the same base.

I ran:

while xapian-check foo; do perl foo.pl & sleep 10s; kill -9 %1; done

This failed in under 2 minutes. Result:

record:
baseA blocksize=8K items=50000 lastblock=589 revision=30 levels=1 root=6
B-tree checked okay
record table structure checked OK

termlist:
baseA blocksize=8K items=50000 lastblock=359 revision=30 levels=1 root=5
B-tree checked okay
termlist table structure checked OK

postlist:
baseB blocksize=8K items=299962 lastblock=1936 revision=31 levels=2 root=24
B-tree checked okay
document id 50001 in doclen stream is larger than get_last_docid() 50000
document id 50001: length 7 doesn't match 0 in the termlist table
document id 50002 in doclen stream is larger than get_last_docid() 50000
document id 50002: length 7 doesn't match 0 in the termlist table
document id 50003 in doclen stream is larger than get_last_docid() 50000
...
document id 59998 in doclen stream is larger than get_last_docid() 50000
document id 59998: length 7 doesn't match 0 in the termlist table
document id 59999 in doclen stream is larger than get_last_docid() 50000
document id 59999: length 7 doesn't match 0 in the termlist table
document id 60000 in doclen stream is larger than get_last_docid() 50000
document id 60000: length 7 doesn't match 0 in the termlist table
postlist table errors found: 20000

position:
baseB blocksize=8K items=180000 lastblock=483 revision=31 levels=1 root=319
B-tree checked okay
position table structure checked OK

spelling:
Lazily created, and not yet used.

synonym:
Lazily created, and not yet used.

Total errors found: 20000


Archive of complete database: http://tmp.kitenet.net/xapian-1.tar.bz2

-- 
see shy jo

use Search::Xapian::WritableDatabase;
use Search::Xapian;

my $stemmer=Search::Xapian::Stem->new("english");

my $db=Search::Xapian::WritableDatabase->new("foo", Search::Xapian::DB_CREATE_OR_OPEN);
for (1..1000000) {
	my $pageterm="U:$_\n";
	my $doc=Search::Xapian::Document->new();
	$doc->set_data(
		"url=foo\n".
		"sample=foo($_)oobar($_)baz\n".
		"modtime=".localtime(time)."\n"
	);
	my $tg = Search::Xapian::TermGenerator->new();
	$tg->set_stemmer($stemmer);
	$tg->set_document($doc);
	$tg->index_text("$_ $_$_", 2);
	$tg->index_text("foo$_", 1, "XLINK");
	$doc->add_term($pageterm);
	$db->replace_document_by_term($pageterm, $doc);
}

signature.asc
Description: Digital signature