[
https://issues.apache.org/jira/browse/LUCENE-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12662464#action_12662464
]
Michael McCandless commented on LUCENE-1479:
--------------------------------------------
{quote}
> I'll feed it with some mock documents (String), which simulate the bug.
{quote}
Right, either that or make a mocked set of docs in a .gz file and just use
TrecDocMaker directly.
{quote}
> Also, I'm not near the code now, so I can't tell if DocData allows for a null
> Date.
{quote}
I think DocData ought to be able to handle a null Date, especially since the
real data (GOV2 in your OP) has real documents lacking a Date.
To pretend the document did have a Date that happens to be today's date doesn't
seem good (I don't know of anything in contrib/benchmark that depends on
accurate dates now, but we may have something in the future where shifting
dates could confuse things).
> TrecDocMaker skips over documents when "Date" is missing from documents
> -----------------------------------------------------------------------
>
> Key: LUCENE-1479
> URL: https://issues.apache.org/jira/browse/LUCENE-1479
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/benchmark
> Reporter: Shai Erera
> Assignee: Michael McCandless
> Fix For: 2.4.1, 2.9
>
> Attachments: LUCENE-1479.patch
>
>
> TrecDocMaker skips over Trec documents if they do not have a "Date" line.
> When such a document is encountered, the code may skip over several documents
> until the next tag that is searched for is found.
> The result is, instead of reading ~25M documents from the GOV2 collection,
> the code reads only ~23M (don't remember the actual numbers).
> The fix adds a terminatingTag to read() such that the code looks for prefix,
> but only until terminatingTag is found. Appropriate changes were made in
> getNextDocData().
> Patch to follow
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
