Re: ConcurrentModificationException in SolrInputDocument writeMap

2019-11-18 Thread Tim Swetland 
 
Finally had a chance to get back to this and figured out what our problem
was. Thanks to Mikhail for pointing me in the right direction. For some
reason one of our custom UpdateRequestProcessors was calling
super.processAdd as the first line in our overridden processAdd method. It
looked something like this:

@Override
public void processAdd(AddUpdateCommand cmd) throws IOException {
  super.processAdd(cmd);

  SolrInputDocument doc = cmd.getSolrInputDocument();
  // Do stuff and set field in doc

  super.next.processAdd(cmd);
}

Once I got rid of the first line, everything worked fine. I suspect the
fact that we were going through a decent portion of the update request
processor chain twice was causing some unknown issues. The big issue
causing the ConcurrentModificationException, however, appeared to happen
when trying to distribute the update to a replica. It looks like it would
try and write out to the replica as we'd take the second trip through our
processor chain and thus two things would try and modify the
SolrInputDocument at the same time causing the exception. This would
additionally cause a ClassCastException (java.lang.String cannot be cast to
java.util.Map) on the replica as in issue SOLR-13471
.

Anyway, thanks for the insight everyone,
Tim

On Fri, Nov 8, 2019 at 12:26 AM Shawn Heisey  wrote:

> On 11/6/2019 8:17 AM, Tim Swetland wrote:
> > I'm currently running into a ConcurrentModificationException ingesting
> data
> > as we attempt to upgrade from Solr 8.1 to 8.2. It's not every document,
> but
> > it definitely appears regularly in our logs. We didn't run into this
> > problem in 8.1, so I'm not sure what might have changed. I feel like this
> > is probably a bug, but if there's a workaround or if there's an idea of
> > something I might be doing wrong, please let me know.
> >
> > Stack trace:
> > o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
> > SolrCmdDistributor$Req: cmd=add{_version=,id=};
> node=StdNode:
> > https:///solr/coll_shard1_replica_n2/ to
> > https:///solr/coll_shard1_replica_n2/
> > => java.util.ConcurrentModificationException
> >  at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
> > java.util.ConcurrentModificationException: null
> >at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
> >at
> >
> org.apache.solr.common.SolrInputDocument.writeMap(SolrInputDocument.java:51)
>
> This error, as mentioned in the SOLR-8028 issue linked by Edward
> Ribeiro, sounds like you are re-using objects when you are building
> SolrInputDocument instances for your indexing.
>
> Looking at the actual code involved in the stacktrace, I think what's
> happening is that at the same time SolrJ is converting a
> SolrInputDocument object to the javabin format so it can be sent to
> Solr, something else has modified that SolrInputDocument object.
>
> You should never re-use SolrJ objects that you construct for indexing.
> A brand new SolrInputDocument instance should be created every time you
> need one, and any objects that go into its creation should also be new
> objects.  This is especially important when the code is multi-threaded,
> but there are certain code constructs that can cause this to happen even
> in code that does not create multiple threads.  Also, code that uses
> CloudSolrClient can very easily become multi-threaded even if the user's
> code isn't.
>
> In Solr 8.2, clients and the server were updated to use HTTP/2.  I did
> not closely follow the work for this, but it would have required some
> pretty major changes to SolrJ, changes that very well could have altered
> the precise timing of operations.  Your additional note says you are
> also seeing this problem with 8.1, which does not surprise me.  The
> precise timing of the 8.1 code might have been such that the problem was
> far less noticeable.
>
> If you can share your code that uses SolrJ, we *might* be able to help
> you narrow down what's happening and get it fixed.
>
> Thanks,
> Shawn
>

Re: ConcurrentModificationException in SolrInputDocument writeMap

2019-11-07 Thread Shawn Heisey 

On 11/6/2019 8:17 AM, Tim Swetland wrote:

I'm currently running into a ConcurrentModificationException ingesting data
as we attempt to upgrade from Solr 8.1 to 8.2. It's not every document, but
it definitely appears regularly in our logs. We didn't run into this
problem in 8.1, so I'm not sure what might have changed. I feel like this
is probably a bug, but if there's a workaround or if there's an idea of
something I might be doing wrong, please let me know.

Stack trace:
o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
SolrCmdDistributor$Req: cmd=add{_version=,id=}; node=StdNode:
https:///solr/coll_shard1_replica_n2/ to
https:///solr/coll_shard1_replica_n2/
=> java.util.ConcurrentModificationException
 at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
java.util.ConcurrentModificationException: null
   at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
   at
org.apache.solr.common.SolrInputDocument.writeMap(SolrInputDocument.java:51)


This error, as mentioned in the SOLR-8028 issue linked by Edward 
Ribeiro, sounds like you are re-using objects when you are building 
SolrInputDocument instances for your indexing.


Looking at the actual code involved in the stacktrace, I think what's 
happening is that at the same time SolrJ is converting a 
SolrInputDocument object to the javabin format so it can be sent to 
Solr, something else has modified that SolrInputDocument object.


You should never re-use SolrJ objects that you construct for indexing. 
A brand new SolrInputDocument instance should be created every time you 
need one, and any objects that go into its creation should also be new 
objects.  This is especially important when the code is multi-threaded, 
but there are certain code constructs that can cause this to happen even 
in code that does not create multiple threads.  Also, code that uses 
CloudSolrClient can very easily become multi-threaded even if the user's 
code isn't.


In Solr 8.2, clients and the server were updated to use HTTP/2.  I did 
not closely follow the work for this, but it would have required some 
pretty major changes to SolrJ, changes that very well could have altered 
the precise timing of operations.  Your additional note says you are 
also seeing this problem with 8.1, which does not surprise me.  The 
precise timing of the 8.1 code might have been such that the problem was 
far less noticeable.


If you can share your code that uses SolrJ, we *might* be able to help 
you narrow down what's happening and get it fixed.


Thanks,
Shawn

Re: ConcurrentModificationException in SolrInputDocument writeMap

2019-11-07 Thread Edward Ribeiro 
You probably hit
https://issues.apache.org/jira/projects/SOLR/issues/SOLR-8028


Regards,
Edward


Em qua, 6 de nov de 2019 13:23, Mikhail Khludnev  escreveu:

> Hello, Tim.
> Please confirm my understanding. Does exception happens in standalone Java
> ingesting app?
> If, it's so, Does it reuse either SolrInputDocument instances of
> fields/values collections between update calls?
>
> On Wed, Nov 6, 2019 at 8:00 AM Tim Swetland  wrote:
>
> > Nevermind my comment on not having this problem in 8.1. We do have it
> there
> > as well, I just didn't look far enough back in our logs on my initial
> > search. Would still appreciate whatever thoughts anyone might have on the
> > exception.
> >
> > On Wed, Nov 6, 2019 at 10:17 AM Tim Swetland 
> wrote:
> >
> > > I'm currently running into a ConcurrentModificationException ingesting
> > > data as we attempt to upgrade from Solr 8.1 to 8.2. It's not every
> > > document, but it definitely appears regularly in our logs. We didn't
> run
> > > into this problem in 8.1, so I'm not sure what might have changed. I
> feel
> > > like this is probably a bug, but if there's a workaround or if there's
> an
> > > idea of something I might be doing wrong, please let me know.
> > >
> > > Stack trace:
> > > o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
> > > SolrCmdDistributor$Req: cmd=add{_version=,id=};
> > node=StdNode:
> > > https:///solr/coll_shard1_replica_n2/ to https://
> > /solr/coll_shard1_replica_n2/
> > > => java.util.ConcurrentModificationException
> > > at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
> > > java.util.ConcurrentModificationException: null
> > >   at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
> > >   at
> > >
> >
> org.apache.solr.common.SolrInputDocument.writeMap(SolrInputDocument.java:51)
> > >   at
> > >
> >
> org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:658)
> > >   at
> > >
> >
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
> > >   at
> > >
> org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
> > >   at
> > >
> >
> org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:813)
> > >
> > >   at
> > >
> >
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:411)
> > >
> > >   at
> > >
> org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
> > >   at
> > >
> >
> org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:750)
> > >
> > >   at
> > >
> >
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:395)
> > >
> > >   at
> > >
> org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
> > >   at
> > >
> >
> org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:248)
> > >
> > >   at
> > >
> >
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
> > >
> > >   at
> > >
> org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
> > >   at
> > > org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:167)
> > >   at
> > >
> >
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:102)
> > >   at
> > >
> >
> org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:83)
> > >   at
> > >
> >
> org.apache.solr.client.solrj.impl.Http2SolrClient.send(Http2SolrClient.java:338)
> > >
> > >   at
> > >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:231)
> > >
> > >   at
> > >
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:176)
> > >
> > >   at
> > >
> >
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
> > >   at
> > >
> >
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil
> > > .java:209)
> > >   at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > >   at
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >
> > >   at java.lang.Thread.run(Thread.java:748)
> > >
> > >
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>

Re: ConcurrentModificationException in SolrInputDocument writeMap

2019-11-06 Thread Mikhail Khludnev 
Hello, Tim.
Please confirm my understanding. Does exception happens in standalone Java
ingesting app?
If, it's so, Does it reuse either SolrInputDocument instances of
fields/values collections between update calls?

On Wed, Nov 6, 2019 at 8:00 AM Tim Swetland  wrote:

> Nevermind my comment on not having this problem in 8.1. We do have it there
> as well, I just didn't look far enough back in our logs on my initial
> search. Would still appreciate whatever thoughts anyone might have on the
> exception.
>
> On Wed, Nov 6, 2019 at 10:17 AM Tim Swetland  wrote:
>
> > I'm currently running into a ConcurrentModificationException ingesting
> > data as we attempt to upgrade from Solr 8.1 to 8.2. It's not every
> > document, but it definitely appears regularly in our logs. We didn't run
> > into this problem in 8.1, so I'm not sure what might have changed. I feel
> > like this is probably a bug, but if there's a workaround or if there's an
> > idea of something I might be doing wrong, please let me know.
> >
> > Stack trace:
> > o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
> > SolrCmdDistributor$Req: cmd=add{_version=,id=};
> node=StdNode:
> > https:///solr/coll_shard1_replica_n2/ to https://
> /solr/coll_shard1_replica_n2/
> > => java.util.ConcurrentModificationException
> > at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
> > java.util.ConcurrentModificationException: null
> >   at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
> >   at
> >
> org.apache.solr.common.SolrInputDocument.writeMap(SolrInputDocument.java:51)
> >   at
> >
> org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:658)
> >   at
> >
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
> >   at
> > org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
> >   at
> >
> org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:813)
> >
> >   at
> >
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:411)
> >
> >   at
> > org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
> >   at
> >
> org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:750)
> >
> >   at
> >
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:395)
> >
> >   at
> > org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
> >   at
> >
> org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:248)
> >
> >   at
> >
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
> >
> >   at
> > org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
> >   at
> > org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:167)
> >   at
> >
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:102)
> >   at
> >
> org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:83)
> >   at
> >
> org.apache.solr.client.solrj.impl.Http2SolrClient.send(Http2SolrClient.java:338)
> >
> >   at
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:231)
> >
> >   at
> >
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:176)
> >
> >   at
> >
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
> >   at
> >
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil
> > .java:209)
> >   at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >   at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >
> >   at java.lang.Thread.run(Thread.java:748)
> >
> >
>


-- 
Sincerely yours
Mikhail Khludnev

Re: ConcurrentModificationException in SolrInputDocument writeMap

2019-11-06 Thread Tim Swetland 
Nevermind my comment on not having this problem in 8.1. We do have it there
as well, I just didn't look far enough back in our logs on my initial
search. Would still appreciate whatever thoughts anyone might have on the
exception.

On Wed, Nov 6, 2019 at 10:17 AM Tim Swetland  wrote:

> I'm currently running into a ConcurrentModificationException ingesting
> data as we attempt to upgrade from Solr 8.1 to 8.2. It's not every
> document, but it definitely appears regularly in our logs. We didn't run
> into this problem in 8.1, so I'm not sure what might have changed. I feel
> like this is probably a bug, but if there's a workaround or if there's an
> idea of something I might be doing wrong, please let me know.
>
> Stack trace:
> o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
> SolrCmdDistributor$Req: cmd=add{_version=,id=}; node=StdNode:
> https:///solr/coll_shard1_replica_n2/ to 
> https:///solr/coll_shard1_replica_n2/
> => java.util.ConcurrentModificationException
> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
> java.util.ConcurrentModificationException: null
>   at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686)
>   at
> org.apache.solr.common.SolrInputDocument.writeMap(SolrInputDocument.java:51)
>   at
> org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:658)
>   at
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:383)
>   at
> org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
>   at
> org.apache.solr.common.util.JavaBinCodec.writeMapEntry(JavaBinCodec.java:813)
>
>   at
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:411)
>
>   at
> org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
>   at
> org.apache.solr.common.util.JavaBinCodec.writeIterator(JavaBinCodec.java:750)
>
>   at
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:395)
>
>   at
> org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
>   at
> org.apache.solr.common.util.JavaBinCodec.writeNamedList(JavaBinCodec.java:248)
>
>   at
> org.apache.solr.common.util.JavaBinCodec.writeKnownType(JavaBinCodec.java:355)
>
>   at
> org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:253)
>   at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:167)
>   at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:102)
>   at
> org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:83)
>   at
> org.apache.solr.client.solrj.impl.Http2SolrClient.send(Http2SolrClient.java:338)
>
>   at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:231)
>
>   at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:176)
>
>   at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
>   at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil
> .java:209)
>   at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>   at java.lang.Thread.run(Thread.java:748)
>
>