[jira] [Updated] (CASSANDRA-19579) threads lingering after driver shutdown: session close starts thread and doesn't await its stop
[ https://issues.apache.org/jira/browse/CASSANDRA-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-19579: - Bug Category: Parent values: Degradation(12984) Complexity: Normal Discovered By: User Report Severity: Normal Status: Open (was: Triage Needed) > threads lingering after driver shutdown: session close starts thread and > doesn't await its stop > --- > > Key: CASSANDRA-19579 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19579 > Project: Cassandra > Issue Type: Bug > Components: Client/java-driver >Reporter: Thomas Klambauer >Priority: Normal > > We are checking remaining/lingering threads during shutdown. > we noticed some with naming pattern/thread factory: > ""globalEventExecutor-1-2" Id=146 TIMED_WAITING" > this one seems to be created during shutdown / session close and not > awaited/shut down: > {noformat} > addTask:156, GlobalEventExecutor (io.netty.util.concurrent) > execute0:225, GlobalEventExecutor (io.netty.util.concurrent) > execute:221, GlobalEventExecutor (io.netty.util.concurrent) > onClose:188, DefaultNettyOptions > (com.datastax.oss.driver.internal.core.context) > onChildrenClosed:589, DefaultSession$SingleThreaded > (com.datastax.oss.driver.internal.core.session) > lambda$close$9:552, DefaultSession$SingleThreaded > (com.datastax.oss.driver.internal.core.session) > run:-1, 860270832 > (com.datastax.oss.driver.internal.core.session.DefaultSession$SingleThreaded$$Lambda$9508) > tryFire$$$capture:783, CompletableFuture$UniRun (java.util.concurrent) > tryFire:-1, CompletableFuture$UniRun (java.util.concurrent) > - Async stack trace > addTask:-1, SingleThreadEventExecutor (io.netty.util.concurrent) > execute:836, SingleThreadEventExecutor (io.netty.util.concurrent) > execute0:827, SingleThreadEventExecutor (io.netty.util.concurrent) > execute:817, SingleThreadEventExecutor (io.netty.util.concurrent) > claim:568, CompletableFuture$UniCompletion (java.util.concurrent) > tryFire$$$capture:780, CompletableFuture$UniRun (java.util.concurrent) > tryFire:-1, CompletableFuture$UniRun (java.util.concurrent) > - Async stack trace > :767, CompletableFuture$UniRun (java.util.concurrent) > uniRunStage:801, CompletableFuture (java.util.concurrent) > thenRunAsync:2136, CompletableFuture (java.util.concurrent) > thenRunAsync:143, CompletableFuture (java.util.concurrent) > whenAllDone:75, CompletableFutures > (com.datastax.oss.driver.internal.core.util.concurrent) > close:551, DefaultSession$SingleThreaded > (com.datastax.oss.driver.internal.core.session) > access$1000:300, DefaultSession$SingleThreaded > (com.datastax.oss.driver.internal.core.session) > lambda$closeAsync$1:272, DefaultSession > (com.datastax.oss.driver.internal.core.session) > runTask:98, PromiseTask (io.netty.util.concurrent) > run:106, PromiseTask (io.netty.util.concurrent) > runTask$$$capture:174, AbstractEventExecutor (io.netty.util.concurrent) > runTask:-1, AbstractEventExecutor (io.netty.util.concurrent) > - Async stack trace > addTask:-1, SingleThreadEventExecutor (io.netty.util.concurrent) > execute:836, SingleThreadEventExecutor (io.netty.util.concurrent) > execute0:827, SingleThreadEventExecutor (io.netty.util.concurrent) > execute:817, SingleThreadEventExecutor (io.netty.util.concurrent) > submit:118, AbstractExecutorService (java.util.concurrent) > submit:118, AbstractEventExecutor (io.netty.util.concurrent) > on:57, RunOrSchedule (com.datastax.oss.driver.internal.core.util.concurrent) > closeSafely:286, DefaultSession > (com.datastax.oss.driver.internal.core.session) > closeAsync:272, DefaultSession (com.datastax.oss.driver.internal.core.session) > close:76, AsyncAutoCloseable (com.datastax.oss.driver.api.core) > -- custom shutdown code > run:829, Thread (java.lang) > {noformat} > the initial close here is called on > com.datastax.oss.driver.api.core.CqlSession. > netty framework suggests to call > io.netty.util.concurrent.GlobalEventExecutor#awaitInactivity > during shutdown to await event thread stopping > (slightly related issue in netty: > [https://github.com/netty/netty/issues/2084] ) > suggestion to add maybe GlobalEventExecutor.INSTANCE.awaitInactivity with > some timeout during close around here: > [https://github.com/apache/cassandra-java-driver/blob/4.x/core/src/main/java/com/datastax/oss/driver/internal/core/context/DefaultNettyOptions.java#L199] > noting that this might slow down closing for up to 2 seconds if the netty > issue comment is correct. > this is on latest datastax java driver version: 4.17, -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits
[jira] [Updated] (CASSANDRA-19579) threads lingering after driver shutdown: session close starts thread and doesn't await its stop
[ https://issues.apache.org/jira/browse/CASSANDRA-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Klambauer updated CASSANDRA-19579: - Description: We are checking remaining/lingering threads during shutdown. we noticed some with naming pattern/thread factory: ""globalEventExecutor-1-2" Id=146 TIMED_WAITING" this one seems to be created during shutdown / session close and not awaited/shut down: {noformat} addTask:156, GlobalEventExecutor (io.netty.util.concurrent) execute0:225, GlobalEventExecutor (io.netty.util.concurrent) execute:221, GlobalEventExecutor (io.netty.util.concurrent) onClose:188, DefaultNettyOptions (com.datastax.oss.driver.internal.core.context) onChildrenClosed:589, DefaultSession$SingleThreaded (com.datastax.oss.driver.internal.core.session) lambda$close$9:552, DefaultSession$SingleThreaded (com.datastax.oss.driver.internal.core.session) run:-1, 860270832 (com.datastax.oss.driver.internal.core.session.DefaultSession$SingleThreaded$$Lambda$9508) tryFire$$$capture:783, CompletableFuture$UniRun (java.util.concurrent) tryFire:-1, CompletableFuture$UniRun (java.util.concurrent) - Async stack trace addTask:-1, SingleThreadEventExecutor (io.netty.util.concurrent) execute:836, SingleThreadEventExecutor (io.netty.util.concurrent) execute0:827, SingleThreadEventExecutor (io.netty.util.concurrent) execute:817, SingleThreadEventExecutor (io.netty.util.concurrent) claim:568, CompletableFuture$UniCompletion (java.util.concurrent) tryFire$$$capture:780, CompletableFuture$UniRun (java.util.concurrent) tryFire:-1, CompletableFuture$UniRun (java.util.concurrent) - Async stack trace :767, CompletableFuture$UniRun (java.util.concurrent) uniRunStage:801, CompletableFuture (java.util.concurrent) thenRunAsync:2136, CompletableFuture (java.util.concurrent) thenRunAsync:143, CompletableFuture (java.util.concurrent) whenAllDone:75, CompletableFutures (com.datastax.oss.driver.internal.core.util.concurrent) close:551, DefaultSession$SingleThreaded (com.datastax.oss.driver.internal.core.session) access$1000:300, DefaultSession$SingleThreaded (com.datastax.oss.driver.internal.core.session) lambda$closeAsync$1:272, DefaultSession (com.datastax.oss.driver.internal.core.session) runTask:98, PromiseTask (io.netty.util.concurrent) run:106, PromiseTask (io.netty.util.concurrent) runTask$$$capture:174, AbstractEventExecutor (io.netty.util.concurrent) runTask:-1, AbstractEventExecutor (io.netty.util.concurrent) - Async stack trace addTask:-1, SingleThreadEventExecutor (io.netty.util.concurrent) execute:836, SingleThreadEventExecutor (io.netty.util.concurrent) execute0:827, SingleThreadEventExecutor (io.netty.util.concurrent) execute:817, SingleThreadEventExecutor (io.netty.util.concurrent) submit:118, AbstractExecutorService (java.util.concurrent) submit:118, AbstractEventExecutor (io.netty.util.concurrent) on:57, RunOrSchedule (com.datastax.oss.driver.internal.core.util.concurrent) closeSafely:286, DefaultSession (com.datastax.oss.driver.internal.core.session) closeAsync:272, DefaultSession (com.datastax.oss.driver.internal.core.session) close:76, AsyncAutoCloseable (com.datastax.oss.driver.api.core) -- custom shutdown code run:829, Thread (java.lang) {noformat} the initial close here is called on com.datastax.oss.driver.api.core.CqlSession. netty framework suggests to call io.netty.util.concurrent.GlobalEventExecutor#awaitInactivity during shutdown to await event thread stopping (slightly related issue in netty: [https://github.com/netty/netty/issues/2084] ) suggestion to add maybe GlobalEventExecutor.INSTANCE.awaitInactivity with some timeout during close around here: [https://github.com/apache/cassandra-java-driver/blob/4.x/core/src/main/java/com/datastax/oss/driver/internal/core/context/DefaultNettyOptions.java#L199] noting that this might slow down closing for up to 2 seconds if the netty issue comment is correct. this is on latest datastax java driver version: 4.17, was: We are checking remaining/lingering threads during shutdown. we noticed some with naming pattern/thread factory: ""globalEventExecutor-1-2" Id=146 TIMED_WAITING" this one seems to be created during shutdown / session close and not awaited/shut down: {noformat} addTask:156, GlobalEventExecutor (io.netty.util.concurrent) execute0:225, GlobalEventExecutor (io.netty.util.concurrent) execute:221, GlobalEventExecutor (io.netty.util.concurrent) onClose:188, DefaultNettyOptions (com.datastax.oss.driver.internal.core.context) onChildrenClosed:589, DefaultSession$SingleThreaded (com.datastax.oss.driver.internal.core.session) lambda$close$9:552, DefaultSession$SingleThreaded (com.datastax.oss.driver.internal.core.session) run:-1, 860270832 (com.datastax.oss.driver.internal.core.session.DefaultSession$SingleThreaded$$Lambda$9508) tryFire$$$capture:783, CompletableFuture$UniRun (java.util.concurrent) tryFire:-1, CompletableFuture$UniRun (java.util
[jira] [Updated] (CASSANDRA-19579) threads lingering after driver shutdown: session close starts thread and doesn't await its stop
[ https://issues.apache.org/jira/browse/CASSANDRA-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Klambauer updated CASSANDRA-19579: - Description: We are checking remaining/lingering threads during shutdown. we noticed some with naming pattern/thread factory: ""globalEventExecutor-1-2" Id=146 TIMED_WAITING" this one seems to be created during shutdown / session close and not awaited/shut down: {noformat} addTask:156, GlobalEventExecutor (io.netty.util.concurrent) execute0:225, GlobalEventExecutor (io.netty.util.concurrent) execute:221, GlobalEventExecutor (io.netty.util.concurrent) onClose:188, DefaultNettyOptions (com.datastax.oss.driver.internal.core.context) onChildrenClosed:589, DefaultSession$SingleThreaded (com.datastax.oss.driver.internal.core.session) lambda$close$9:552, DefaultSession$SingleThreaded (com.datastax.oss.driver.internal.core.session) run:-1, 860270832 (com.datastax.oss.driver.internal.core.session.DefaultSession$SingleThreaded$$Lambda$9508) tryFire$$$capture:783, CompletableFuture$UniRun (java.util.concurrent) tryFire:-1, CompletableFuture$UniRun (java.util.concurrent) - Async stack trace addTask:-1, SingleThreadEventExecutor (io.netty.util.concurrent) execute:836, SingleThreadEventExecutor (io.netty.util.concurrent) execute0:827, SingleThreadEventExecutor (io.netty.util.concurrent) execute:817, SingleThreadEventExecutor (io.netty.util.concurrent) claim:568, CompletableFuture$UniCompletion (java.util.concurrent) tryFire$$$capture:780, CompletableFuture$UniRun (java.util.concurrent) tryFire:-1, CompletableFuture$UniRun (java.util.concurrent) - Async stack trace :767, CompletableFuture$UniRun (java.util.concurrent) uniRunStage:801, CompletableFuture (java.util.concurrent) thenRunAsync:2136, CompletableFuture (java.util.concurrent) thenRunAsync:143, CompletableFuture (java.util.concurrent) whenAllDone:75, CompletableFutures (com.datastax.oss.driver.internal.core.util.concurrent) close:551, DefaultSession$SingleThreaded (com.datastax.oss.driver.internal.core.session) access$1000:300, DefaultSession$SingleThreaded (com.datastax.oss.driver.internal.core.session) lambda$closeAsync$1:272, DefaultSession (com.datastax.oss.driver.internal.core.session) runTask:98, PromiseTask (io.netty.util.concurrent) run:106, PromiseTask (io.netty.util.concurrent) runTask$$$capture:174, AbstractEventExecutor (io.netty.util.concurrent) runTask:-1, AbstractEventExecutor (io.netty.util.concurrent) - Async stack trace addTask:-1, SingleThreadEventExecutor (io.netty.util.concurrent) execute:836, SingleThreadEventExecutor (io.netty.util.concurrent) execute0:827, SingleThreadEventExecutor (io.netty.util.concurrent) execute:817, SingleThreadEventExecutor (io.netty.util.concurrent) submit:118, AbstractExecutorService (java.util.concurrent) submit:118, AbstractEventExecutor (io.netty.util.concurrent) on:57, RunOrSchedule (com.datastax.oss.driver.internal.core.util.concurrent) closeSafely:286, DefaultSession (com.datastax.oss.driver.internal.core.session) closeAsync:272, DefaultSession (com.datastax.oss.driver.internal.core.session) close:76, AsyncAutoCloseable (com.datastax.oss.driver.api.core) shutdown:172, DataStaxBackend (com.dynatrace.apm.server.core.persistence.dcc.impl.datastax.backend) shutdown:389, DccServiceImpl (com.dynatrace.apm.server.core.persistence.dcc.impl.common) shutdown:78, CassandraManagement (com.compuware.apm.server.core.lifecycle) shutdownWithoutExit:3811, ServerLifecycle (com.compuware.apm.server.core.api) shutdown:3648, ServerLifecycle (com.compuware.apm.server.core.api) run:121, ServerShutdownRunner (com.compuware.apm.server.core) run:829, Thread (java.lang) {noformat} the initial close here is called on com.datastax.oss.driver.api.core.CqlSession. netty framework suggests to call io.netty.util.concurrent.GlobalEventExecutor#awaitInactivity during shutdown to await event thread stopping (slightly related issue in netty: https://github.com/netty/netty/issues/2084 ) suggestion to add maybe GlobalEventExecutor.INSTANCE.awaitInactivity with some timeout during close around here: https://github.com/apache/cassandra-java-driver/blob/4.x/core/src/main/java/com/datastax/oss/driver/internal/core/context/DefaultNettyOptions.java#L199 noting that this might slow down closing for up to 2 seconds if the netty issue comment is correct. was: We are checking remaining/lingering threads during shutdown. we noticed some with naming pattern/thread factory: ""globalEventExecutor-1-2" Id=146 TIMED_WAITING" this one seems to be created during shutdown / session close and not awaited/shut down: {noformat} addTask:156, GlobalEventExecutor (io.netty.util.concurrent) execute0:225, GlobalEventExecutor (io.netty.util.concurrent) execute:221, GlobalEventExecutor (io.netty.util.concurrent) onClose:188, DefaultNettyOptions (com.datastax.oss.driver.internal.core.context) onChildrenClosed:589, DefaultS
[jira] [Updated] (CASSANDRA-19579) threads lingering after driver shutdown: session close starts thread and doesn't await its stop
[ https://issues.apache.org/jira/browse/CASSANDRA-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Klambauer updated CASSANDRA-19579: - Description: We are checking remaining/lingering threads during shutdown. we noticed some with naming pattern/thread factory: ""globalEventExecutor-1-2" Id=146 TIMED_WAITING" this one seems to be created during shutdown / session close and not awaited/shut down: {noformat} addTask:156, GlobalEventExecutor (io.netty.util.concurrent) execute0:225, GlobalEventExecutor (io.netty.util.concurrent) execute:221, GlobalEventExecutor (io.netty.util.concurrent) onClose:188, DefaultNettyOptions (com.datastax.oss.driver.internal.core.context) onChildrenClosed:589, DefaultSession$SingleThreaded (com.datastax.oss.driver.internal.core.session) lambda$close$9:552, DefaultSession$SingleThreaded (com.datastax.oss.driver.internal.core.session) run:-1, 860270832 (com.datastax.oss.driver.internal.core.session.DefaultSession$SingleThreaded$$Lambda$9508) tryFire$$$capture:783, CompletableFuture$UniRun (java.util.concurrent) tryFire:-1, CompletableFuture$UniRun (java.util.concurrent) - Async stack trace addTask:-1, SingleThreadEventExecutor (io.netty.util.concurrent) execute:836, SingleThreadEventExecutor (io.netty.util.concurrent) execute0:827, SingleThreadEventExecutor (io.netty.util.concurrent) execute:817, SingleThreadEventExecutor (io.netty.util.concurrent) claim:568, CompletableFuture$UniCompletion (java.util.concurrent) tryFire$$$capture:780, CompletableFuture$UniRun (java.util.concurrent) tryFire:-1, CompletableFuture$UniRun (java.util.concurrent) - Async stack trace :767, CompletableFuture$UniRun (java.util.concurrent) uniRunStage:801, CompletableFuture (java.util.concurrent) thenRunAsync:2136, CompletableFuture (java.util.concurrent) thenRunAsync:143, CompletableFuture (java.util.concurrent) whenAllDone:75, CompletableFutures (com.datastax.oss.driver.internal.core.util.concurrent) close:551, DefaultSession$SingleThreaded (com.datastax.oss.driver.internal.core.session) access$1000:300, DefaultSession$SingleThreaded (com.datastax.oss.driver.internal.core.session) lambda$closeAsync$1:272, DefaultSession (com.datastax.oss.driver.internal.core.session) runTask:98, PromiseTask (io.netty.util.concurrent) run:106, PromiseTask (io.netty.util.concurrent) runTask$$$capture:174, AbstractEventExecutor (io.netty.util.concurrent) runTask:-1, AbstractEventExecutor (io.netty.util.concurrent) - Async stack trace addTask:-1, SingleThreadEventExecutor (io.netty.util.concurrent) execute:836, SingleThreadEventExecutor (io.netty.util.concurrent) execute0:827, SingleThreadEventExecutor (io.netty.util.concurrent) execute:817, SingleThreadEventExecutor (io.netty.util.concurrent) submit:118, AbstractExecutorService (java.util.concurrent) submit:118, AbstractEventExecutor (io.netty.util.concurrent) on:57, RunOrSchedule (com.datastax.oss.driver.internal.core.util.concurrent) closeSafely:286, DefaultSession (com.datastax.oss.driver.internal.core.session) closeAsync:272, DefaultSession (com.datastax.oss.driver.internal.core.session) close:76, AsyncAutoCloseable (com.datastax.oss.driver.api.core) shutdown:172, DataStaxBackend (com.dynatrace.apm.server.core.persistence.dcc.impl.datastax.backend) shutdown:389, DccServiceImpl (com.dynatrace.apm.server.core.persistence.dcc.impl.common) shutdown:78, CassandraManagement (com.compuware.apm.server.core.lifecycle) shutdownWithoutExit:3811, ServerLifecycle (com.compuware.apm.server.core.api) shutdown:3648, ServerLifecycle (com.compuware.apm.server.core.api) run:121, ServerShutdownRunner (com.compuware.apm.server.core) run:829, Thread (java.lang) {noformat} the initial close here is called on com.datastax.oss.driver.api.core.CqlSession. netty framework suggests to call io.netty.util.concurrent.GlobalEventExecutor#awaitInactivity during shutdown to await event thread stopping (slightly related issue in netty: [https://github.com/netty/netty/issues/2084] ) suggestion to add maybe GlobalEventExecutor.INSTANCE.awaitInactivity with some timeout during close around here: [https://github.com/apache/cassandra-java-driver/blob/4.x/core/src/main/java/com/datastax/oss/driver/internal/core/context/DefaultNettyOptions.java#L199] noting that this might slow down closing for up to 2 seconds if the netty issue comment is correct. this is on latest datastax java driver version: 4.17, was: We are checking remaining/lingering threads during shutdown. we noticed some with naming pattern/thread factory: ""globalEventExecutor-1-2" Id=146 TIMED_WAITING" this one seems to be created during shutdown / session close and not awaited/shut down: {noformat} addTask:156, GlobalEventExecutor (io.netty.util.concurrent) execute0:225, GlobalEventExecutor (io.netty.util.concurrent) execute:221, GlobalEventExecutor (io.netty.util.concurrent) onClose:188, DefaultNettyOptions (com.datastax.oss.driver.i