Thanks to you both. I've pinged the people who can reproduce this problem, I suspect they'll try the patch out. I'm out next week, far far away from internet connections so I'll be silent for a while. But don't be surprised if new messages appear.
Thanks again Erick On Fri, May 10, 2013 at 3:46 AM, Adrien Grand <[email protected]> wrote: > Erick, > > On Thu, May 9, 2013 at 10:28 PM, Erick Erickson <[email protected]> > wrote: >> Yeah, this is getting warmer. Some of the >> CompressingStoredFieldsReader objects are 240M. The documents aren't >> nearly that large I don't think (but I'll verify). > > Wow, 240M is a lot! Would you be able to know which member of these > instances is making them so large? Since you were mentioning > threading, I first thought that the culprit was the buffer used for > decompression but it could be the fields index as well (which is > stored in memory but should be shared across instances for the same > segment, so threading shouldn't make the situation worse). > >> But still, over 700 of these objects live at once? I _think_ I'm >> seeing the number go up significantly when the number of indexing >> threads increases, but that's a bit of indirect evidence. My other >> question would be whether you'd expect the number of these objects to >> go up as the number of segments goes up, i.e. I assume they're >> per-segment.... > > Indeed, these objects are managed per-segment, so increasing the > number of segments increases the number of stored fields reader > instances. > >> So the pattern here is atomic updates on documents where some of the >> fields get quite large. So the underlying reader has to decompress >> these a lot. Do you have any suggestions how to mitigate this? Other >> than "don't do that<G>".... > > As Savia suggests, lowering the number of indexing threads should > help. If the merge factor is highish, lowering it could help as well. > On my end, I'll fix the stored fields reader to not reuse the buffer > for decompression (https://issues.apache.org/jira/browse/LUCENE-4995, > even if we find out that it is not the problem here, it shouldn't > hurt). > > -- > Adrien > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
