Re: Will doc ids ever change if nothing is deleted?

2010-05-18 Thread Michael McCandless
If you never delete docs, then w/ the default merge policy, the docIDs should never change. But... this should be considered an impl detail of Lucene. In theory someday this could change. EG there's an issue open (LUCENE-1076) to allow a merge policy to select out-of-order merges, which they can

Re: Will doc ids ever change if nothing is deleted?

2010-05-14 Thread Nigel
Right, but my question was whether merging segments will renumber docs *if no documents are deleted*. Empirically, the answer is no. I've written test code that indexes documents with a field equal to each document's current id, and verified that the ids still match the field values even after me

Re: Will doc ids ever change if nothing is deleted?

2010-05-14 Thread Chris Lu
The doc id will get changed if the segments are merged. The doc id is more depending on the order of documents being added. Just think about it. The doc ids are starting from 0 to N. And when some documents are deleted, they are marked deleted on .del file. So no change there. When some docume

Re: Will doc ids ever change if nothing is deleted?

2010-05-14 Thread Nigel
We do assign GUIDs to everything in the index for cases where longer-term identity is necessary. For this case, using GUIDs would be prohibitively expensive as we'd need to load the GUIDs for all search results. We might have tens of thousands of results and only want to load a random 100, so hav

Re: Will doc ids ever change if nothing is deleted?

2010-05-14 Thread Chris Harris
Could you address your needs by assigning each document a unique identifier (maybe you have a natural key, or maybe you could generate a new GUID or something for each doc), and using those identifiers, rather than internal Lucene docids, to track documents between the search stage and the loading

Re: Will doc ids ever change if nothing is deleted?

2010-05-13 Thread Nigel
Yes, I realize that storing document IDs persistently (for example) is a Bad Idea. Partly I'm asking just to make sure I understand what's going on. There is a use case, though. In some cases between when we do a search and return some doc ids, and when we use those doc ids to load some documents

Re: Will doc ids ever change if nothing is deleted?

2010-05-13 Thread Erick Erickson
Why do you care? That is, what do you want to accomplish that makes document ID renumbering relevant? In general, it is unwise to rely on Lucene-assigned document IDs. If you need an invariant document ID, assign it yourself. If this is off base, could you supply your use-case? Best Erick On Th

Will doc ids ever change if nothing is deleted?

2010-05-13 Thread Nigel
The FAQ clearly states that document IDs will not be re-assigned unless something was deleted. http://wiki.apache.org/lucene-java/LuceneFAQ#When_is_it_possible_for_document_IDs_to_change.3F However, a number of other emails and posts I've read mention that renumbering occurs when segments are merg