[jira] [Commented] (SOLR-17413) UpdateLog Replay can throw ConcurrentModificationException from sharing the request
[
https://issues.apache.org/jira/browse/SOLR-17413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17945729#comment-17945729
]
ASF subversion and git services commented on SOLR-17413:
Commit 3dd3a96f09c89f5d6a9f2915256aface46a4a8a0 in solr's branch
refs/heads/add-pki-caching from Jason Gerlowski
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=3dd3a96f09c ]
SOLR-17413: ulog replay should copy SolrQueryRequest (#110)
SolrQueryRequest is a non-threadsafe type, but was being shared across
executor threads during UpdateLog replay. This introduces a number of
issues, not the least being a ConcurrentModificationException if
multiple threads happen to tweak the request 'context' simultaneously.
This commit fixes this issue by giving each executor thread a unique
LocalSolrQueryRequest instance to use.
NOTE: This fix is on its way to 9.8 upstream, but may take a slightly
different final form.
Co-authored-by: Jason Gerlowski
> UpdateLog Replay can throw ConcurrentModificationException from sharing the
> request
> ---
>
> Key: SOLR-17413
> URL: https://issues.apache.org/jira/browse/SOLR-17413
> Project: Solr
> Issue Type: Bug
>Reporter: David Smiley
>Assignee: Jason Gerlowski
>Priority: Major
> Labels: newdev, pull-request-available
> Fix For: 9.8
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> I saw org.apache.solr.cloud.BasicDistributedZkTest fail with a stack trace
> revealing we have a real issue with UpdateLog replay. Essentially, a
> SolrQueryRequest is not threadsafe but we replay logs in parallel sharing the
> same request instance. Creating DistributedUpdateProcessor ends up adding to
> a shared HashMap in req.getContext() that should not be shared.
> {noformat}
> 2> WARNING: Uncaught exception in thread:
> Thread[replayUpdatesExecutor-590-thread-2,5,TGRP-BasicDistributedZkTest]
> 2> java.util.ConcurrentModificationException
> 2> at __randomizedtesting.SeedInfo.seed([F2227B12A8FC234]:0)
> 2> at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1135)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessorFactory.addParamToDistributedRequestWhitelist(DistributedUpdateProcessorFactory.java:46)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.(DistributedUpdateProcessor.java:190)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.(DistributedUpdateProcessor.java:160)
> 2> at
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.(DistributedZkUpdateProcessor.java:114)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessorFactory.getInstance(DistributedUpdateProcessorFactory.java:59)
> 2> at
> org.apache.solr.update.processor.UpdateRequestProcessorChain.createProcessor(UpdateRequestProcessorChain.java:242)
> 2> at
> org.apache.solr.update.processor.UpdateRequestProcessorChain.createProcessor(UpdateRequestProcessorChain.java:214)
> 2> at
> org.apache.solr.update.UpdateLog$LogReplayer.lambda$doReplay$0(UpdateLog.java:2103)
> 2> at
> java.base/java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305)
> 2> at java.base/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
> 2> at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:172)
> 2> at
> org.apache.solr.update.UpdateLog$LogReplayer.lambda$execute$2(UpdateLog.java:2342)
> 2> at
> org.apache.solr.util.OrderedExecutor.lambda$execute$0(OrderedExecutor.java:68)
> 2> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$1(ExecutorUtil.java:449)
> 2> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> 2> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> 2> at java.base/java.lang.Thread.run(Thread.java:829)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17413) UpdateLog Replay can throw ConcurrentModificationException from sharing the request
[
https://issues.apache.org/jira/browse/SOLR-17413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17893571#comment-17893571
]
ASF subversion and git services commented on SOLR-17413:
Commit a15e6f4278523f2a23ad35246fe7e34bf531e12f in solr's branch
refs/heads/branch_9x from Jason Gerlowski
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=a15e6f42785 ]
SOLR-17413: ulog replay should copy SolrQueryRequest (#2765)
SolrQueryRequest is a non-threadsafe type, but was being shared across
executor threads during UpdateLog replay. This introduces a number of
issues, not the least being a ConcurrentModificationException if
multiple threads happen to tweak the request 'context' simultaneously.
This commit fixes this issue by giving each executor thread a unique
LocalSolrQueryRequest instance to use.
> UpdateLog Replay can throw ConcurrentModificationException from sharing the
> request
> ---
>
> Key: SOLR-17413
> URL: https://issues.apache.org/jira/browse/SOLR-17413
> Project: Solr
> Issue Type: Bug
>Reporter: David Smiley
>Priority: Major
> Labels: newdev, pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> I saw org.apache.solr.cloud.BasicDistributedZkTest fail with a stack trace
> revealing we have a real issue with UpdateLog replay. Essentially, a
> SolrQueryRequest is not threadsafe but we replay logs in parallel sharing the
> same request instance. Creating DistributedUpdateProcessor ends up adding to
> a shared HashMap in req.getContext() that should not be shared.
> {noformat}
> 2> WARNING: Uncaught exception in thread:
> Thread[replayUpdatesExecutor-590-thread-2,5,TGRP-BasicDistributedZkTest]
> 2> java.util.ConcurrentModificationException
> 2> at __randomizedtesting.SeedInfo.seed([F2227B12A8FC234]:0)
> 2> at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1135)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessorFactory.addParamToDistributedRequestWhitelist(DistributedUpdateProcessorFactory.java:46)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.(DistributedUpdateProcessor.java:190)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.(DistributedUpdateProcessor.java:160)
> 2> at
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.(DistributedZkUpdateProcessor.java:114)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessorFactory.getInstance(DistributedUpdateProcessorFactory.java:59)
> 2> at
> org.apache.solr.update.processor.UpdateRequestProcessorChain.createProcessor(UpdateRequestProcessorChain.java:242)
> 2> at
> org.apache.solr.update.processor.UpdateRequestProcessorChain.createProcessor(UpdateRequestProcessorChain.java:214)
> 2> at
> org.apache.solr.update.UpdateLog$LogReplayer.lambda$doReplay$0(UpdateLog.java:2103)
> 2> at
> java.base/java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305)
> 2> at java.base/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
> 2> at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:172)
> 2> at
> org.apache.solr.update.UpdateLog$LogReplayer.lambda$execute$2(UpdateLog.java:2342)
> 2> at
> org.apache.solr.util.OrderedExecutor.lambda$execute$0(OrderedExecutor.java:68)
> 2> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$1(ExecutorUtil.java:449)
> 2> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> 2> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> 2> at java.base/java.lang.Thread.run(Thread.java:829)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17413) UpdateLog Replay can throw ConcurrentModificationException from sharing the request
[
https://issues.apache.org/jira/browse/SOLR-17413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17893567#comment-17893567
]
ASF subversion and git services commented on SOLR-17413:
Commit 0b4157a8d83225d46e21087c4c0c610defc17e35 in solr's branch
refs/heads/main from Jason Gerlowski
[ https://gitbox.apache.org/repos/asf?p=solr.git;h=0b4157a8d83 ]
SOLR-17413: ulog replay should copy SolrQueryRequest (#2765)
SolrQueryRequest is a non-threadsafe type, but was being shared across
executor threads during UpdateLog replay. This introduces a number of
issues, not the least being a ConcurrentModificationException if
multiple threads happen to tweak the request 'context' simultaneously.
This commit fixes this issue by giving each executor thread a unique
LocalSolrQueryRequest instance to use.
> UpdateLog Replay can throw ConcurrentModificationException from sharing the
> request
> ---
>
> Key: SOLR-17413
> URL: https://issues.apache.org/jira/browse/SOLR-17413
> Project: Solr
> Issue Type: Bug
>Reporter: David Smiley
>Priority: Major
> Labels: newdev, pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> I saw org.apache.solr.cloud.BasicDistributedZkTest fail with a stack trace
> revealing we have a real issue with UpdateLog replay. Essentially, a
> SolrQueryRequest is not threadsafe but we replay logs in parallel sharing the
> same request instance. Creating DistributedUpdateProcessor ends up adding to
> a shared HashMap in req.getContext() that should not be shared.
> {noformat}
> 2> WARNING: Uncaught exception in thread:
> Thread[replayUpdatesExecutor-590-thread-2,5,TGRP-BasicDistributedZkTest]
> 2> java.util.ConcurrentModificationException
> 2> at __randomizedtesting.SeedInfo.seed([F2227B12A8FC234]:0)
> 2> at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1135)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessorFactory.addParamToDistributedRequestWhitelist(DistributedUpdateProcessorFactory.java:46)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.(DistributedUpdateProcessor.java:190)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.(DistributedUpdateProcessor.java:160)
> 2> at
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.(DistributedZkUpdateProcessor.java:114)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessorFactory.getInstance(DistributedUpdateProcessorFactory.java:59)
> 2> at
> org.apache.solr.update.processor.UpdateRequestProcessorChain.createProcessor(UpdateRequestProcessorChain.java:242)
> 2> at
> org.apache.solr.update.processor.UpdateRequestProcessorChain.createProcessor(UpdateRequestProcessorChain.java:214)
> 2> at
> org.apache.solr.update.UpdateLog$LogReplayer.lambda$doReplay$0(UpdateLog.java:2103)
> 2> at
> java.base/java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305)
> 2> at java.base/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
> 2> at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:172)
> 2> at
> org.apache.solr.update.UpdateLog$LogReplayer.lambda$execute$2(UpdateLog.java:2342)
> 2> at
> org.apache.solr.util.OrderedExecutor.lambda$execute$0(OrderedExecutor.java:68)
> 2> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$1(ExecutorUtil.java:449)
> 2> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> 2> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> 2> at java.base/java.lang.Thread.run(Thread.java:829)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17413) UpdateLog Replay can throw ConcurrentModificationException from sharing the request
[
https://issues.apache.org/jira/browse/SOLR-17413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17889664#comment-17889664
]
Jason Gerlowski commented on SOLR-17413:
The failure shows very rarely, but I am able to reproduce it on my local
machine when using a very high number of iterations:
{code}
./gradlew -p solr/core test --tests BasicDistributedZkTest -Ptests.iters=300
-Ptests.dups=4
{code}
I've created a PR for the fix [here|https://github.com/apache/solr/pull/2765],
which allows BasicDistributedZkTest to reliably succeed. Will kick off a few
more high-iteration runs to make sure I haven't just gotten "lucky", but seems
like this works (y)
> UpdateLog Replay can throw ConcurrentModificationException from sharing the
> request
> ---
>
> Key: SOLR-17413
> URL: https://issues.apache.org/jira/browse/SOLR-17413
> Project: Solr
> Issue Type: Bug
>Reporter: David Smiley
>Priority: Major
> Labels: newdev, pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I saw org.apache.solr.cloud.BasicDistributedZkTest fail with a stack trace
> revealing we have a real issue with UpdateLog replay. Essentially, a
> SolrQueryRequest is not threadsafe but we replay logs in parallel sharing the
> same request instance. Creating DistributedUpdateProcessor ends up adding to
> a shared HashMap in req.getContext() that should not be shared.
> {noformat}
> 2> WARNING: Uncaught exception in thread:
> Thread[replayUpdatesExecutor-590-thread-2,5,TGRP-BasicDistributedZkTest]
> 2> java.util.ConcurrentModificationException
> 2> at __randomizedtesting.SeedInfo.seed([F2227B12A8FC234]:0)
> 2> at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1135)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessorFactory.addParamToDistributedRequestWhitelist(DistributedUpdateProcessorFactory.java:46)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.(DistributedUpdateProcessor.java:190)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.(DistributedUpdateProcessor.java:160)
> 2> at
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.(DistributedZkUpdateProcessor.java:114)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessorFactory.getInstance(DistributedUpdateProcessorFactory.java:59)
> 2> at
> org.apache.solr.update.processor.UpdateRequestProcessorChain.createProcessor(UpdateRequestProcessorChain.java:242)
> 2> at
> org.apache.solr.update.processor.UpdateRequestProcessorChain.createProcessor(UpdateRequestProcessorChain.java:214)
> 2> at
> org.apache.solr.update.UpdateLog$LogReplayer.lambda$doReplay$0(UpdateLog.java:2103)
> 2> at
> java.base/java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305)
> 2> at java.base/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
> 2> at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:172)
> 2> at
> org.apache.solr.update.UpdateLog$LogReplayer.lambda$execute$2(UpdateLog.java:2342)
> 2> at
> org.apache.solr.util.OrderedExecutor.lambda$execute$0(OrderedExecutor.java:68)
> 2> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$1(ExecutorUtil.java:449)
> 2> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> 2> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> 2> at java.base/java.lang.Thread.run(Thread.java:829)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17413) UpdateLog Replay can throw ConcurrentModificationException from sharing the request
[
https://issues.apache.org/jira/browse/SOLR-17413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17889281#comment-17889281
]
Jason Gerlowski commented on SOLR-17413:
bq. This should be somewhat straightforward to fix by creating a copy of the
request in the ThreadLocal.withInitial lambda.
I like this approach for its simplicity. IMO it also tracks in terms of the
"Principle of Least Surprise" - creating copies is a pretty standard way to
work around threadsafety limitations. Gonna try to reproduce this in tests
today, and will create a PR if I can validate the fix.
> UpdateLog Replay can throw ConcurrentModificationException from sharing the
> request
> ---
>
> Key: SOLR-17413
> URL: https://issues.apache.org/jira/browse/SOLR-17413
> Project: Solr
> Issue Type: Bug
>Reporter: David Smiley
>Priority: Major
> Labels: newdev
>
> I saw org.apache.solr.cloud.BasicDistributedZkTest fail with a stack trace
> revealing we have a real issue with UpdateLog replay. Essentially, a
> SolrQueryRequest is not threadsafe but we replay logs in parallel sharing the
> same request instance. Creating DistributedUpdateProcessor ends up adding to
> a shared HashMap in req.getContext() that should not be shared.
> {noformat}
> 2> WARNING: Uncaught exception in thread:
> Thread[replayUpdatesExecutor-590-thread-2,5,TGRP-BasicDistributedZkTest]
> 2> java.util.ConcurrentModificationException
> 2> at __randomizedtesting.SeedInfo.seed([F2227B12A8FC234]:0)
> 2> at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1135)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessorFactory.addParamToDistributedRequestWhitelist(DistributedUpdateProcessorFactory.java:46)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.(DistributedUpdateProcessor.java:190)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.(DistributedUpdateProcessor.java:160)
> 2> at
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.(DistributedZkUpdateProcessor.java:114)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessorFactory.getInstance(DistributedUpdateProcessorFactory.java:59)
> 2> at
> org.apache.solr.update.processor.UpdateRequestProcessorChain.createProcessor(UpdateRequestProcessorChain.java:242)
> 2> at
> org.apache.solr.update.processor.UpdateRequestProcessorChain.createProcessor(UpdateRequestProcessorChain.java:214)
> 2> at
> org.apache.solr.update.UpdateLog$LogReplayer.lambda$doReplay$0(UpdateLog.java:2103)
> 2> at
> java.base/java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305)
> 2> at java.base/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
> 2> at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:172)
> 2> at
> org.apache.solr.update.UpdateLog$LogReplayer.lambda$execute$2(UpdateLog.java:2342)
> 2> at
> org.apache.solr.util.OrderedExecutor.lambda$execute$0(OrderedExecutor.java:68)
> 2> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$1(ExecutorUtil.java:449)
> 2> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> 2> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> 2> at java.base/java.lang.Thread.run(Thread.java:829)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17413) UpdateLog Replay can throw ConcurrentModificationException from sharing the request
[
https://issues.apache.org/jira/browse/SOLR-17413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17875506#comment-17875506
]
David Smiley commented on SOLR-17413:
-
I'd prefer to avoid catching this exception to take some other action -- would
feel too much like a hack. I'm optimistic a more elegant solution without
doing this can be found.
Perhaps SynchronousQueue? Maybe I can rule this out based on a little test I
did, as it throws a RejectedExecutionException (instead of blocking), which I'd
rather not catch.
Perhaps a LinkedBlockingQueue (thus infinite queue size) but have an
ExecutorService subclass that observes the queue size to see if a threshold is
reached and if so then runs the task directly? This way we never reject and
the caller threads receive back pressure by doing the work that they would have
done anyway with multiThreaded=false ! There is plenty of precedent for an
ExecutorService that runs the task in the caller thread -- I'm thinking of
Lucene's SameThreadExecutorService and Solr's SimpleFacets.directExecutor.
> UpdateLog Replay can throw ConcurrentModificationException from sharing the
> request
> ---
>
> Key: SOLR-17413
> URL: https://issues.apache.org/jira/browse/SOLR-17413
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: David Smiley
>Priority: Major
> Labels: newdev
>
> I saw org.apache.solr.cloud.BasicDistributedZkTest fail with a stack trace
> revealing we have a real issue with UpdateLog replay. Essentially, a
> SolrQueryRequest is not threadsafe but we replay logs in parallel sharing the
> same request instance. Creating DistributedUpdateProcessor ends up adding to
> a shared HashMap in req.getContext() that should not be shared.
> {noformat}
> 2> WARNING: Uncaught exception in thread:
> Thread[replayUpdatesExecutor-590-thread-2,5,TGRP-BasicDistributedZkTest]
> 2> java.util.ConcurrentModificationException
> 2> at __randomizedtesting.SeedInfo.seed([F2227B12A8FC234]:0)
> 2> at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1135)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessorFactory.addParamToDistributedRequestWhitelist(DistributedUpdateProcessorFactory.java:46)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.(DistributedUpdateProcessor.java:190)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.(DistributedUpdateProcessor.java:160)
> 2> at
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.(DistributedZkUpdateProcessor.java:114)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessorFactory.getInstance(DistributedUpdateProcessorFactory.java:59)
> 2> at
> org.apache.solr.update.processor.UpdateRequestProcessorChain.createProcessor(UpdateRequestProcessorChain.java:242)
> 2> at
> org.apache.solr.update.processor.UpdateRequestProcessorChain.createProcessor(UpdateRequestProcessorChain.java:214)
> 2> at
> org.apache.solr.update.UpdateLog$LogReplayer.lambda$doReplay$0(UpdateLog.java:2103)
> 2> at
> java.base/java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305)
> 2> at java.base/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
> 2> at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:172)
> 2> at
> org.apache.solr.update.UpdateLog$LogReplayer.lambda$execute$2(UpdateLog.java:2342)
> 2> at
> org.apache.solr.util.OrderedExecutor.lambda$execute$0(OrderedExecutor.java:68)
> 2> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$1(ExecutorUtil.java:449)
> 2> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> 2> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> 2> at java.base/java.lang.Thread.run(Thread.java:829)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-17413) UpdateLog Replay can throw ConcurrentModificationException from sharing the request
[
https://issues.apache.org/jira/browse/SOLR-17413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17875329#comment-17875329
]
David Smiley commented on SOLR-17413:
-
This should be somewhat straightforward to fix by creating a copy of the
request in the ThreadLocal.withInitial lambda.
> UpdateLog Replay can throw ConcurrentModificationException from sharing the
> request
> ---
>
> Key: SOLR-17413
> URL: https://issues.apache.org/jira/browse/SOLR-17413
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: David Smiley
>Priority: Major
>
> I saw org.apache.solr.cloud.BasicDistributedZkTest fail with a stack trace
> revealing we have a real issue with UpdateLog replay. Essentially, a
> SolrQueryRequest is not threadsafe but we replay logs in parallel sharing the
> same request instance. Creating DistributedUpdateProcessor ends up adding to
> a shared HashMap in req.getContext() that should not be shared.
> {noformat}
> 2> WARNING: Uncaught exception in thread:
> Thread[replayUpdatesExecutor-590-thread-2,5,TGRP-BasicDistributedZkTest]
> 2> java.util.ConcurrentModificationException
> 2> at __randomizedtesting.SeedInfo.seed([F2227B12A8FC234]:0)
> 2> at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1135)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessorFactory.addParamToDistributedRequestWhitelist(DistributedUpdateProcessorFactory.java:46)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.(DistributedUpdateProcessor.java:190)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.(DistributedUpdateProcessor.java:160)
> 2> at
> org.apache.solr.update.processor.DistributedZkUpdateProcessor.(DistributedZkUpdateProcessor.java:114)
> 2> at
> org.apache.solr.update.processor.DistributedUpdateProcessorFactory.getInstance(DistributedUpdateProcessorFactory.java:59)
> 2> at
> org.apache.solr.update.processor.UpdateRequestProcessorChain.createProcessor(UpdateRequestProcessorChain.java:242)
> 2> at
> org.apache.solr.update.processor.UpdateRequestProcessorChain.createProcessor(UpdateRequestProcessorChain.java:214)
> 2> at
> org.apache.solr.update.UpdateLog$LogReplayer.lambda$doReplay$0(UpdateLog.java:2103)
> 2> at
> java.base/java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305)
> 2> at java.base/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
> 2> at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:172)
> 2> at
> org.apache.solr.update.UpdateLog$LogReplayer.lambda$execute$2(UpdateLog.java:2342)
> 2> at
> org.apache.solr.util.OrderedExecutor.lambda$execute$0(OrderedExecutor.java:68)
> 2> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$1(ExecutorUtil.java:449)
> 2> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> 2> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> 2> at java.base/java.lang.Thread.run(Thread.java:829)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
