[
https://issues.apache.org/jira/browse/LUCENE-532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Muir resolved LUCENE-532.
--------------------------------
Resolution: Fixed
Fix Version/s: 4.0
This is fixed by LUCENE-2373, just set your codec to AppendingCodec.
> [PATCH] Indexing on Hadoop distributed file system
> --------------------------------------------------
>
> Key: LUCENE-532
> URL: https://issues.apache.org/jira/browse/LUCENE-532
> Project: Lucene - Java
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 1.9
> Reporter: Igor Bolotin
> Priority: Minor
> Fix For: 4.0
>
> Attachments: SegmentTermEnum.patch, TermInfosWriter.patch,
> cfs-patch.txt, indexOnDFS.patch
>
>
> In my current project we needed a way to create very large Lucene indexes on
> Hadoop distributed file system. When we tried to do it directly on DFS using
> Nutch FsDirectory class - we immediately found that indexing fails because
> DfsIndexOutput.seek() method throws UnsupportedOperationException. The reason
> for this behavior is clear - DFS does not support random updates and so
> seek() method can't be supported (at least not easily).
>
> Well, if we can't support random updates - the question is: do we really need
> them? Search in the Lucene code revealed 2 places which call
> IndexOutput.seek() method: one is in TermInfosWriter and another one in
> CompoundFileWriter. As we weren't planning to use CompoundFileWriter - the
> only place that concerned us was in TermInfosWriter.
>
> TermInfosWriter uses IndexOutput.seek() in its close() method to write total
> number of terms in the file back into the beginning of the file. It was very
> simple to change file format a little bit and write number of terms into last
> 8 bytes of the file instead of writing them into beginning of file. The only
> other place that should be fixed in order for this to work is in
> SegmentTermEnum constructor - to read this piece of information at position =
> file length - 8.
>
> With this format hack - we were able to use FsDirectory to write index
> directly to DFS without any problems. Well - we still don't index directly to
> DFS for performance reasons, but at least we can build small local indexes
> and merge them into the main index on DFS without copying big main index back
> and forth.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
