[
https://issues.apache.org/jira/browse/CASSANDRA-19000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17784230#comment-17784230
]
Ekaterina Dimitrova edited comment on CASSANDRA-19000 at 11/10/23 2:32 PM:
---------------------------------------------------------------------------
I cannot reproduce either with
[5.0|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2557/workflows/be404529-e81a-4504-8ced-aaaedc647902],
or on
[trunk|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2558/workflows/44a50540-f04a-4459-9353-7d8ee46b8834].
However, I found the logs in nightlies from the failed test run in Jenkins -
[https://nightlies.apache.org/cassandra/trunk/Cassandra-trunk-test-cdc/1826/Cassandra-trunk-test-cdc/jdk=jdk_17_latest,label=cassandra,split=7/build/test/logs/cdc.jdk17.arch=x86_64.python2.7/]
Cassandra drivers docs point to threads with names: <cluster_name>- nio-worker
-<n>
And the logs - both of passing and the not passing test run contain:
{code:java}
DEBUG [main] 2023-11-02 18:57:49,940 [Control connection] error on
/127.9.9.1:9042 connection, no more host to try
com.datastax.driver.core.exceptions.TransportException: [/127.9.9.1:9042]
Cannot connect
at
com.datastax.driver.core.Connection$1.operationComplete(Connection.java:226)
at
com.datastax.driver.core.Connection$1.operationComplete(Connection.java:192)
at
com.datastax.shaded.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
at
com.datastax.shaded.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
at
com.datastax.shaded.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
at
com.datastax.shaded.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
at
com.datastax.shaded.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
at
com.datastax.shaded.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
at
com.datastax.shaded.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
at
com.datastax.shaded.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)
at
com.datastax.shaded.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at
com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at
com.datastax.shaded.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
com.datastax.shaded.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by:
com.datastax.shaded.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: /127.9.9.1:9042
Caused by: java.net.ConnectException: Connection refused
at java.base/sun.nio.ch.Net.pollConnect(Native Method)
at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
at
java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
at
com.datastax.shaded.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337)
at
com.datastax.shaded.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at
com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at
com.datastax.shaded.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
com.datastax.shaded.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833){code}
So I guess seeing those pop up was a matter of time.
My understanding of CASSANDRA-9054 is that tools do not have many of the
services initialized during regular startup and the test verifies that we do
not do that when using the BulkLoader and nothing new unneeded has popped up.
I suggest we add an exclusion as [this
|https://github.com/ekaterinadimitrova2/cassandra/commit/f14c92889eaeb9432e2ec716495b5bd313e969fe].
(I also fixed some indentation that was off)
Tested locally, same patch for 5.0 and trunk.
was (Author: e.dimitrova):
I cannot reproduce either with
[5.0|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2557/workflows/be404529-e81a-4504-8ced-aaaedc647902],
or on
[trunk|https://app.circleci.com/pipelines/github/ekaterinadimitrova2/cassandra/2558/workflows/44a50540-f04a-4459-9353-7d8ee46b8834].
However, I found the logs in nightlies from the failed test run in Jenkins -
[https://nightlies.apache.org/cassandra/trunk/Cassandra-trunk-test-cdc/1826/Cassandra-trunk-test-cdc/jdk=jdk_17_latest,label=cassandra,split=7/build/test/logs/cdc.jdk17.arch=x86_64.python2.7/]
Cassandra drivers docs point to threads with names: <cluster_name>- nio-worker
-<n>
And the logs - both of passing and the not passing test run contain:
{code:java}
DEBUG [main] 2023-11-02 18:57:45,809 -Dio.netty.maxThreadLocalCharBufferSize:
16384
DEBUG [cluster1-nio-worker-0] 2023-11-02 18:57:45,842
Connection[/127.9.9.1:9042-1, inFlight=0, closed=false] Error connecting to
/127.9.9.1:9042 (Connection refused: /127.9.9.1:9042)
DEBUG [cluster1-nio-worker-0] 2023-11-02 18:57:45,845 Defuncting
Connection[/127.9.9.1:9042-1, inFlight=0, closed=false] because:
[/127.9.9.1:9042] Cannot connect
DEBUG [cluster1-nio-worker-0] 2023-11-02 18:57:45,846 [/127.9.9.1:9042]
preventing new connections for the next 1000 ms
DEBUG [cluster1-nio-worker-0] 2023-11-02 18:57:45,846 [/127.9.9.1:9042]
Connection[/127.9.9.1:9042-1, inFlight=0, closed=false] failed, remaining = 0
DEBUG [cluster1-nio-worker-0] 2023-11-02 18:57:45,846
Connection[/127.9.9.1:9042-1, inFlight=0, closed=true] closing connection
DEBUG [main] 2023-11-02 18:57:45,849 [Control connection] error on
/127.9.9.1:9042 connection, no more host to try
com.datastax.driver.core.exceptions.TransportException: [/127.9.9.1:9042]
Cannot connect
at
com.datastax.driver.core.Connection$1.operationComplete(Connection.java:226)
at
com.datastax.driver.core.Connection$1.operationComplete(Connection.java:192)
at
com.datastax.shaded.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
at
com.datastax.shaded.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
at
com.datastax.shaded.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
at
com.datastax.shaded.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
at
com.datastax.shaded.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
at
com.datastax.shaded.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)
at
com.datastax.shaded.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)
at
com.datastax.shaded.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)
at
com.datastax.shaded.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at
com.datastax.shaded.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at
com.datastax.shaded.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at
com.datastax.shaded.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
com.datastax.shaded.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by:
com.datastax.shaded.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection refused: /127.9.9.1:9042
Caused by: java.net.ConnectException: Connection refused{code}
So I guess seeing those pop up was a matter of time.
My understanding of CASSANDRA-9054 is that tools do not have many of the
services initialized during regular startup and the test verifies that we do
not do that when using the BulkLoader and nothing new unneeded has popped up.
I suggest we add an exclusion as [this
|https://github.com/ekaterinadimitrova2/cassandra/commit/f14c92889eaeb9432e2ec716495b5bd313e969fe].
(I also fixed some indentation that was off)
Tested locally, same patch for 5.0 and trunk.
> Test Failure:
> org.apache.cassandra.tools.BulkLoaderTest.testBulkLoader_WithArgs2
> --------------------------------------------------------------------------------
>
> Key: CASSANDRA-19000
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19000
> Project: Cassandra
> Issue Type: Bug
> Components: CI
> Reporter: Ekaterina Dimitrova
> Assignee: Ekaterina Dimitrova
> Priority: Normal
> Fix For: 4.0.x, 4.1.x, 5.0-rc, 5.x
>
>
> h3.
> https://ci-cassandra.apache.org/job/Cassandra-trunk/1766/testReport/org.apache.cassandra.tools/BulkLoaderTest/testBulkLoader_WithArgs2_cdc_jdk17_arch_x86_64_python2_7/
> {code:java}
> Error Message
> Wrong thread status, active threads unaccounted for: [cluster3-nio-worker-0]
> Stacktrace
> junit.framework.AssertionFailedError: Wrong thread status, active threads
> unaccounted for: [cluster3-nio-worker-0] at
> org.apache.cassandra.tools.OfflineToolUtils.assertNoUnexpectedThreadsStarted(OfflineToolUtils.java:120)
> at
> org.apache.cassandra.tools.BulkLoaderTest.testBulkLoader_WithArgs2(BulkLoaderTest.java:129)
> at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
