On 11/6/2019 8:17 AM, Tim Swetland wrote:
I'm currently running into a ConcurrentModificationException ingesting data as we attempt to upgrade from Solr 8.1 to 8.2. It's not every document, but it definitely appears regularly in our logs. We didn't run into this problem in 8.1, so I'm not sure what might have changed. I feel like this is probably a bug, but if there's a workaround or if there's an idea of something I might be doing wrong, please let me know.Stack trace: o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling SolrCmdDistributor$Req: cmd=add{_version=<version>,id=<id>}; node=StdNode: https://<server>/solr/coll_shard1_replica_n2/ to https://<server>/solr/coll_shard1_replica_n2/ => java.util.ConcurrentModificationException at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686) java.util.ConcurrentModificationException: null at java.util.LinkedHashMap.forEach(LinkedHashMap.java:686) at org.apache.solr.common.SolrInputDocument.writeMap(SolrInputDocument.java:51)
This error, as mentioned in the SOLR-8028 issue linked by Edward Ribeiro, sounds like you are re-using objects when you are building SolrInputDocument instances for your indexing.
Looking at the actual code involved in the stacktrace, I think what's happening is that at the same time SolrJ is converting a SolrInputDocument object to the javabin format so it can be sent to Solr, something else has modified that SolrInputDocument object.
You should never re-use SolrJ objects that you construct for indexing. A brand new SolrInputDocument instance should be created every time you need one, and any objects that go into its creation should also be new objects. This is especially important when the code is multi-threaded, but there are certain code constructs that can cause this to happen even in code that does not create multiple threads. Also, code that uses CloudSolrClient can very easily become multi-threaded even if the user's code isn't.
In Solr 8.2, clients and the server were updated to use HTTP/2. I did not closely follow the work for this, but it would have required some pretty major changes to SolrJ, changes that very well could have altered the precise timing of operations. Your additional note says you are also seeing this problem with 8.1, which does not surprise me. The precise timing of the 8.1 code might have been such that the problem was far less noticeable.
If you can share your code that uses SolrJ, we *might* be able to help you narrow down what's happening and get it fixed.
Thanks, Shawn
