SEPURI-SAI-KRISHNA opened a new issue, #12757: URL: https://github.com/apache/seatunnel/issues/12757
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Bug%22) and found no similar issues. ### What happened `CassandraIT.testCassandra` and `CassandraIT.testCassandraMultiTableSource` fail on `dev` on the Flink 1.20.1 container only. Both fail on their **first** assertion, which is the job exit code rather than anything about the data: ```java CassandraIT.java:134 Assertions.assertEquals(0, execResult.getExitCode()); // testCassandra CassandraIT.java:144 Assertions.assertEquals(0, execResult.getExitCode()); // testCassandraMultiTableSource ``` so the reported `expected: <0> but was: <1>` is the job exiting non-zero, and the tests never reach Cassandra. Only one template invocation fails; the other three pass in the same run. The invocation provider names each container as it starts, in a fixed order, so the mapping is not a guess: ```text TestContainer[1] Flink:1.18.0 pass TestContainer[2] Flink:1.20.1 FAIL TestContainer[3] SeaTunnel:dev pass TestContainer[4] Spark:3.3.0 pass ``` **Flink 1.18.0 passes and Flink 1.20.1 fails in the same job**, which is the most useful part of the signal: it is specific to the 1.20.1 image rather than to Flink. Zeta and Spark are unaffected. The failing invocations take 31 to 36 seconds rather than hitting a timeout. ### Root cause from the job log Inside the Flink taskmanager container `seatunnel-e2e-jdk11-tyrantlucifer-flink-1.20.1-scala_2.12_hadoop27`, the DataStax driver cannot allocate a direct buffer while sending its very first CQL frame: ```text WARN com.datastax.oss.driver.internal.core.pool.ChannelPool [s0|cassandra/172.18.0.2:9042] Error while opening new channel (ConnectionInitException: [s0|connecting...] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.14.0, ...}): failed to send request (io.netty.handler.codec.EncoderException: java.lang.OutOfMemoryError: Direct buffer memory)) ``` which then becomes, 20 times in the one job (counting thrown exceptions, not the matching `at ...AllNodesFailedException.copy` stack frames): ```text com.datastax.oss.driver.api.core.AllNodesFailedException: All 1 node(s) tried for query failed ``` So the chain is: Netty cannot get direct buffer memory inside the Flink 1.20.1 taskmanager, the driver's STARTUP frame never goes out, every node is marked failed, the job exits 1, and the exit-code assertion fails. This is a direct-memory budget problem in that container rather than a Cassandra data or timing problem. ### Reproduction - `dev` head `b0f6feddf`, run https://github.com/apache/seatunnel/actions/runs/38340962420, job `all-connectors-it-2 (11, ubuntu-latest)`: the same two assertions fail, `:134` and `:144`. That run's summary labels the failing invocation `[4]`. - A fork run at a PR head, https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/38150594554, jobs `all-connectors-it-2 (11, ubuntu-latest)` and `(17, ubuntu-latest)`: same two assertions, labelled `[2]` on both legs. The `[N]` label differs between runs, so treat it as a labelling detail. The container named in the log is Flink 1.20.1 in each case. Two different runners failing the same container identically is why this reads as a configuration limit rather than a race. ### Relationship to #12732, which needs reconciling rather than dismissing #12732 reports the **same message**, `expected: <0> but was: <1>`, and attributes it to a `TRUNCATE` visibility race. Two things are worth putting on the record, and a maintainer may want to reconcile them: - On current `dev` the only two assertions in `CassandraIT` that can print `expected: <0>` are the exit-code ones at `:134` and `:144`. There is no `assertEquals(0, count)` anywhere in the file, and #12732's own patch does not add one either; its polling sits inside `Awaitility.until(...)`. So the message #12732 quotes appears to come from the same exit-code assertion rather than from a row-count check. - #12732 does more than the TRUNCATE poll: it also adds Cassandra container readiness waits (`Wait.forListeningPort()` with a 180 s startup timeout, plus an advertised-ports wait). Those would plausibly fix a job that exits 1 because the driver could not reach Cassandra at all. They would not, however, fix the failure described here, because Cassandra was reachable (`cassandra/172.18.0.2:9042` answered) and the driver failed while allocating a direct buffer to send its own STARTUP frame. Waiting longer for the server does not give the client more direct memory. I have not read the job log referenced by #12732, so I am not asserting that their run hit this same cause; I am only noting that the assertion they quote can only be the exit-code one on the current file. ### SeaTunnel Version dev (`b0f6feddf`) ### SeaTunnel Config Not applicable: the existing `CassandraIT` configs, unchanged. ### Running Command ```shell ./mvnw -pl seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cassandra-e2e verify -DskipUT -DskipIT=false ``` ### Error Exception ```log expected: <0> but was: <1> at org.apache.seatunnel.connectors.seatunnel.cassandra.CassandraIT.testCassandra(CassandraIT.java:134) at org.apache.seatunnel.connectors.seatunnel.cassandra.CassandraIT.testCassandraMultiTableSource(CassandraIT.java:144) io.netty.handler.codec.EncoderException: java.lang.OutOfMemoryError: Direct buffer memory com.datastax.oss.driver.api.core.AllNodesFailedException: All 1 node(s) tried for query failed ``` ### Zeta or Flink or Spark Version Flink 1.20.1 only. Flink 1.18.0, Spark 3.3.0 and Zeta (`SeaTunnel:dev`) all pass the same two tests in the same job. ### Java or Scala Version Reproduced on the JDK 11 and JDK 17 connector E2E legs. ### Screenshots Not applicable. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! I have not established the right fix, so I would rather agree the direction first. Raising the taskmanager's direct memory for this e2e container and constraining the driver's Netty allocator are both plausible, and they belong to different owners, so a maintainer's view on which is appropriate would help before I write anything. ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
