I am quite interested by this story, including sample code. Back in Lucene 1.4 and 2.0 times, the reader vs string loading abilities was inconsistently handled and I switched to have one directory with thousands of files for our ActiveMath content storage. It works but fairly badly on smaller machines (laptops among others).
If I'd be able to get lazy loading to work faithfully I think it'd be quite a win! The overhead can be handled by just separating the index for "mass-storage" and only store keys in the "small storage". That's what I do currently (with mass-storage in files). It'd be good this is better than Hibernate or SQL storage which has always been crappy wrt streaming field contents to my experience. paul >> More specifically - if I have a bunch of large stored fields, >> would it significantly impact search performance in the >> cases when those fields aren't fetched. > > Make sure you have lazy field loading on. > >> >> Searches are very common in this system, and it's very rare >> that someone actually opens up one of these attachments >> so I'm not really worried about the time it takes to fetch >> them when someone does actually want one. > > > You would be adding some overhead to the system in that Solr now has to > manage these files as stored fields. I guess I would do some benchmarking to > see.
