[ https://issues.apache.org/jira/browse/SOLR-14249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17033334#comment-17033334 ]
Jason Gerlowski commented on SOLR-14249: ---------------------------------------- Awesome, I suspected SPNEGO as the reason for the buffering, but wasn't certain. The alternatives you suggested above seem reasonable - especially fixing the Expect: 100-continue behavior. (SOLR-13270 looks like it might be relevant there too?) > Krb5HttpClientBuilder should not buffer requests > ------------------------------------------------- > > Key: SOLR-14249 > URL: https://issues.apache.org/jira/browse/SOLR-14249 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Authentication, SolrJ > Affects Versions: 7.4, master (9.0), 8.4.1 > Reporter: Jason Gerlowski > Priority: Major > Attachments: SOLR-14249-reproduction.patch > > > When SolrJ clients enable Kerberos authentication, a request interceptor is > set up which wraps the actual HttpEntity in a BufferedHttpEntity. This > BufferedHttpEntity, well, buffers the request body in a {{byte[]}} so it can > be repeated if needed. This works fine for small requests, but when requests > get large storing the entire request in memory causes contention or > OutOfMemoryErrors. > The easiest way for this to manifest is to use ConcurrentUpdateSolrClient, > which opens a connection to Solr and streams documents out in an ever > increasing request entity until the doc queue held by the client is emptied. > I ran into this while troubleshooting a DIH run that would reproducibly load > a few hundred thousand documents before progress stalled out. Solr never > crashed and the DIH thread was still alive, but the > ConcurrentUpdateSolrClient used by DIH had its "Runner" thread disappear > around the time of the stall and an OOM like the one below could be seen in > solr-8983-console.log: > {code} > WARNING: Uncaught exception in thread: > Thread[concurrentUpdateScheduler-28-thread-1,5,TGRP-TestKerberosClientBuffering] > java.lang.OutOfMemoryError: Java heap space > at __randomizedtesting.SeedInfo.seed([371A00FBA76D31DF]:0) > at java.base/java.util.Arrays.copyOf(Arrays.java:3745) > at > java.base/java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:120) > at > java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:95) > at > java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:156) > at > org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:213) > at > org.apache.solr.common.util.FastOutputStream.write(FastOutputStream.java:94) > at > org.apache.solr.common.util.ByteUtils.writeUTF16toUTF8(ByteUtils.java:145) > at org.apache.solr.common.util.JavaBinCodec.writeStr(JavaBinCodec.java:848) > at > org.apache.solr.common.util.JavaBinCodec.writePrimitive(JavaBinCodec.java:932) > at > org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:328) > at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228) > at > org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:616) > at > org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355) > at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228) > at > org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:764) > at > org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383) > at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228) > at > org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:705) > at > org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:367) > at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228) > at > org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:223) > at > org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:330) > at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:228) > at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:155) > at > org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:91) > at > org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:83) > at > org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner$1.writeTo(ConcurrentUpdateSolrClient.java:264) > at org.apache.http.entity.EntityTemplate.writeTo(EntityTemplate.java:73) > at > org.apache.http.entity.BufferedHttpEntity.<init>(BufferedHttpEntity.java:62) > at > org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder.lambda$new$3(Krb5HttpClientBuilder.java:155) > at > org.apache.solr.client.solrj.impl.Krb5HttpClientBuilder$$Lambda$459/0x0000000800623840.process(Unknown > Source) > at > org.apache.solr.client.solrj.impl.HttpClientUtil$DynamicInterceptor$1.accept(HttpClientUtil.java:177) > {code} > We took heap dumps and were able to confirm that the entire 8gb heap was > taken up with a single massive CUSC request body that was being buffered! > (As an aside, I had no idea that OutOfMemoryError's could happen without > killing the entire JVM. But apparently they can. CUSC.Runner propagates the > OOM as it should and the OOM kills the Runner thread. Since that thread is > the gc-root for the massive BufferedHttpEntity though, a garbage collection > frees up most of the heap space and the JVM survives its memory trouble. > Solr's oom script never triggers.) > I've attached a JUnit test which reproduces the OOM issue by using a "fake" > Kerberos config. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org