Hello, I ran into a problem while trying to make a utility which loads all documents in the index one by one. Loading was super slow. Orders of magnitude slower then it is supposed to be. After some debugging and looking at the code I figured that the culprit was the index compression which I set to BEST_COMPRESSION since index size is critical for me. Every time lucene loads a document it decompresses the whole 64K block which this document is in my documents are very small and I have ~ 10^9 documents per index which makes scanning the index practically impossible. There was an addition recently made to the CompressingStoredFieldsReader to eagerly decompress all docs in the block and cache results but the is only available during merging. I was able to get access to the merging reader by doing this:
DirectoryReader reader = DirectoryReader.open(directory); for (LeafReaderContext leafReaderContext : reader.leaves()) { StoredFieldsReader fieldsReader = SlowCodecReaderWrapper.wrap(leafReaderContext.reader()).getFieldsReader().getMergeInstance(); for (int i = 0; i < leafReader.maxDoc(); i++) { DocumentStoredFieldVisitor visitor = new DocumentStoredFieldVisitor(); fieldsReader.visitDocument(i, visitor); visitor.getDocument(); } } I was wondering if there is better way of doing this and if there are plans to make access to the faster document loading through some API. Should I try to come up with a patch for this? Thanks! Anton