Gupta, Rajiv wrote on 12/6/16 3:16 PM:
I thought since I'm doing read and write together I may be getting file error
so I tried to use FastUpdate method described here -
http://lucy.apache.org/docs/perl/Lucy/Docs/Cookbook/FastUpdates.html
But now I'm more frequently getting below error. Error input 57 too high
20161206 150630 [] S_fibonacci at core/Lucy/Index/IndexManager.c line 129
My use case is. While my application generating multiple logs, I'm indexing
them parallelly. To achieve this I'm storing docs at multiple locations at
each directory level. In a directory there could be multiple log files so for
that directory I'm having one indexer directory. When file get closed I
insert an end marker doc to indicate that indexing on that file is done.
However, sometimes same file get open multiple times with additional data. In
such case I search in existing indexing directory if there is any end marker
is set, if there is end marker then I delete that end marker, and index
additional data and again insert end marker. In this process I search as well
as write at the same time.
I started seeing these type of error only after I inserted the logic of
search and deleting. Should I catch this and retry? How many docs I should
limit to commit together?
It wasn't clear to me from your description, but you should be running only one
indexer at time per invindex.
You should also destroy any open searchers once the invindex changes (the
indexer commits).
You can see here how Dezi approaches this:
https://metacpan.org/source/KARMAN/Dezi-App-0.014/lib/Dezi/Lucy/Searcher.pm#L406
TL;DR Dezi keeps track of a invindex header file with a UUID in it, which
changes whenever the indexer finishes. Both the UUID and a md5 checksum of the
header file are checked on every search, and the searcher is destroyed and a new
one created if the old searcher is stale.
--
Peter Karman . http://peknet.com/ . [email protected]
