[ https://issues.apache.org/jira/browse/LUCENE-532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986934#action_12986934 ]
Shai Erera commented on LUCENE-532: ----------------------------------- I see some progress in that direction was made under LUCENE-2373 but am not sure if this Codec is a generic one (i.e. can support any file we write today) or tailored for StandardTermDict. It'd be great if Lucene can support append-only FS ! > [PATCH] Indexing on Hadoop distributed file system > -------------------------------------------------- > > Key: LUCENE-532 > URL: https://issues.apache.org/jira/browse/LUCENE-532 > Project: Lucene - Java > Issue Type: Improvement > Components: Index > Affects Versions: 1.9 > Reporter: Igor Bolotin > Priority: Minor > Attachments: cfs-patch.txt, indexOnDFS.patch, SegmentTermEnum.patch, > TermInfosWriter.patch > > > In my current project we needed a way to create very large Lucene indexes on > Hadoop distributed file system. When we tried to do it directly on DFS using > Nutch FsDirectory class - we immediately found that indexing fails because > DfsIndexOutput.seek() method throws UnsupportedOperationException. The reason > for this behavior is clear - DFS does not support random updates and so > seek() method can't be supported (at least not easily). > > Well, if we can't support random updates - the question is: do we really need > them? Search in the Lucene code revealed 2 places which call > IndexOutput.seek() method: one is in TermInfosWriter and another one in > CompoundFileWriter. As we weren't planning to use CompoundFileWriter - the > only place that concerned us was in TermInfosWriter. > > TermInfosWriter uses IndexOutput.seek() in its close() method to write total > number of terms in the file back into the beginning of the file. It was very > simple to change file format a little bit and write number of terms into last > 8 bytes of the file instead of writing them into beginning of file. The only > other place that should be fixed in order for this to work is in > SegmentTermEnum constructor - to read this piece of information at position = > file length - 8. > > With this format hack - we were able to use FsDirectory to write index > directly to DFS without any problems. Well - we still don't index directly to > DFS for performance reasons, but at least we can build small local indexes > and merge them into the main index on DFS without copying big main index back > and forth. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org