This can also be a big performance win. Jason Venner reports significant index and cluster start time improvements by indexing to local disk, zipping and then uploading the resulting zip file. Hadoop has significant file open overhead so moving one zip file wins big over many index component files. There is a secondary bandwidth win as well.
On Fri, Jan 15, 2010 at 8:34 AM, Andrzej Bialecki (JIRA) <j...@apache.org>wrote: > > HDFS doesn't support enough POSIX to support writing Lucene indexes > directly to HDFS - for this reason indexes are always created on local > storage of each node, and then after closing they are copied to HDFS. > > >
