Jorren Hendriks created TOMEE-4699:
--------------------------------------
Summary: ThreadContext constructor is not thread-safe, causing
loss of submitted tasks.
Key: TOMEE-4699
URL: https://issues.apache.org/jira/browse/TOMEE-4699
Project: TomEE
Issue Type: Bug
Components: TomEE Core Server
Affects Versions: 10.2.0
Reporter: Jorren Hendriks
Attachments: stacktrace.txt
Some of our tomee applications have experienced issues losing tasks submitted
to a managed executor.
The root cause has been identified as a `{_}ConcurrentModificationException{_}`
in the ThreadContext constructor (stacktrace attached).
{code:java}
public ThreadContext(final ThreadContext that) {
this.beanContext = that.beanContext;
this.primaryKey = that.primaryKey;
this.data.putAll(that.data); // throws ConcurrentModificationException when
that.data is modified on another thread
this.oldClassLoader = that.oldClassLoader;
}{code}
`{_}HashMap.putAll{_}` will iterate the entrySet, which is not thread-safe on
the `{_}Collections.synchronizedMap{_}` wrapped HashMap currently in use.
A possible solution would be to use a thread-safe Map implementation.
This constructor is called when entering the snapshot in the
[{_}ApplicationThreadContextProvider{_}|[https://github.com/apache/tomee/blob/main/container/openejb-core/src/main/java/org/apache/openejb/threads/impl/ApplicationThreadContextProvider.java#L76|https://github.com/apache/tomee/blob/main/container/openejb-core/src/main/java/org/apache/openejb/threads/impl/ApplicationThreadContextProvider.java#L76].]]
This issue also revealed possible other issues for us:
1. `{_}CUTask{_}` does not handle exceptions before starting the task. Any
exceptions in `{_}contextService.enter(snapshot){_}` cancel invocation, but do
not abort the task.
2. Exceptions in a `{_}CUTask{_}`/`{_}CURunnable{_}` are propagated to the
`{_}ThreadPoolExecutor{_}`. The current implementation does not expose these
exceptions which makes this a silent failure.
If you'd prefer that I can file these as separate issues, but they are related
to this problem.
h2. Reproduction
This is a race condition which very rarely occurs. We were able to reproduce it
(under high load) within an hour on one of our applications with a debugger
attached. So far we were not able to create a minimal reproduction outside our
application. I will update the issue when we do.
The pattern we use in the application is as follows, the comments indicate my
interpretation of what happens:
{code:java}
// MyTaskExecutor.java
@Stateless
public class MyTaskExecutor {
@Resource
private ManagedExecutorService managedExecutorService; // default TomEE
managed executor service
@Override
public void execute(@NotNull Runnable runnable) {
// 1. runnable is wrapped in CUTask. ThreadContext is copied on the
executor thread.
managedExecutorService.submit(runnable);
} // 2. method returns immediately, causing an update to the current
ThreadContext.
} {code}
When the exception occurs, 1 & 2 both execute at the same time, accessing the
current ThreadContext concurrently.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
