SEPURI-SAI-KRISHNA opened a new issue, #13253: URL: https://github.com/apache/seatunnel/issues/13253
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Bug%22) and found no similar issues. No open PR changes `SingleSplitEnumerator` or `FlinkSourceEnumerator.addSplitsBack`. ### What happened On Flink, a source built on `SingleSplitEnumerator` (`connector-common`) fails the job with an `IndexOutOfBoundsException` on its second restart, and this exception replaces the error that caused the restarts. 29 connector modules use it, through `AbstractSingleSplitSource` or directly, among them `connector-http-base` (and the HTTP sources built on it, such as GraphQL), Cassandra, Redis, MQTT, Socket, Salesforce, Google Sheets and Google Pub/Sub. `SingleSplitEnumerator` on dev: ```java @Override public void addSplitsBack(List<SingleSplit> splits, int subtaskId) { pendingSplit = splits.get(0); assignSplit(); } protected void assignSplit() { if (assigned || pendingSplit == null) { return; } ... } ``` 1. On the first restart, Flink passes back the split that was assigned since the last checkpoint. `addSplitsBack` stores it, but `assigned` is still `true`, so `assignSplit()` returns and the restarted reader never gets the split again. 2. On the second restart, nothing was assigned since the last checkpoint, so Flink calls `addSplitsBack` with an empty list, and `splits.get(0)` throws. `SourceCoordinator` then triggers a global failover with this exception. Seen in the CI of #13222 (GraphQL source, fork run https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/42574864549, job `updated-modules-integration-test-part-1`): a job that fails on purpose is restarted twice by the Flink 1.13, 1.15 and 1.18 images, and on each of them the JobManager logs: ```log ERROR org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - Uncaught exception in the SplitEnumerator for Source Source: GraphQL-Source -> Sink Writer: MultiTableSink-Sink while handling subtask 0 recovery to checkpoint -1. Triggering job failover. java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0 at java.util.ArrayList.get(ArrayList.java:459) ~[?:?] at org.apache.seatunnel.connectors.seatunnel.common.source.SingleSplitEnumerator.addSplitsBack(SingleSplitEnumerator.java:58) at org.apache.seatunnel.translation.flink.source.FlinkSourceEnumerator.addSplitsBack(FlinkSourceEnumerator.java:91) at org.apache.flink.runtime.source.coordinator.SourceCoordinator.lambda$subtaskReset$3(SourceCoordinator.java:232) ~[flink-dist_2.11-1.13.6.jar:1.13.6] ``` The job result then shows this exception instead of the GraphQL error, so `testRejectedQueryFailsTheJob` failed on those three images. The Flink 1.20 image does not restart the job, and Zeta and Spark passed. #13222 works around it in its test by setting `execution.restart.strategy = "no"`. I replayed the restart sequence against the class with a mocked `SourceSplitEnumerator.Context` (start, `addSplitsBack([split])` and `registerReader` for restart 1, `addSplitsBack([])` for restart 2): | | assignments after restart 1 | restart 2 | | --- | --- | --- | | dev | 1 (the split is not assigned again) | `IndexOutOfBoundsException: Index: 0` | | fix below | 2 | no exception | Fix: ignore an empty list, and reset `assigned` when a split comes back, so it is assigned again when the reader registers: ```java public void addSplitsBack(List<SingleSplit> splits, int subtaskId) { if (splits.isEmpty()) { return; } pendingSplit = splits.get(0); assigned = false; assignSplit(); } ``` The same empty list should also reach the enumerator on the first restart after a completed checkpoint, when the split was assigned before that checkpoint, which would affect streaming jobs with checkpointing. I have not run that case. ### SeaTunnel Version dev (466d4168a5); `SingleSplitEnumerator` has not changed since #4101. ### SeaTunnel Config `graphql_rejected_query_to_assert.conf` from #13222 without its `execution.restart.strategy = "no"` line: a GraphQL source whose query the server rejects, so every attempt fails and Flink restarts the job. ```conf env { parallelism = 1 job.mode = "BATCH" } source { GraphQL { plugin_output = "http" url = "http://graphql:8080/v1/graphql" format = "json" content_field = "$.data.missing_table" query = """ query Rejected { missing_table { id } } """ schema = { fields { id = "int" } } } } sink { Assert { plugin_input = http rules { row_rules = [ { rule_type = MIN_ROW rule_value = 0 } ] } } } ``` ### Running Command ```shell ./mvnw -B verify -DskipUT=true -DskipIT=false -Dit.test=GraphQLIT -pl :connector-graphql-e2e -am -Pci ``` on the branch of #13222 (the test `testRejectedQueryFailsTheJob` is added there), with the `execution.restart.strategy = "no"` line removed from `graphql_rejected_query_to_assert.conf`; it fails on the Flink 1.13, 1.15 and 1.18 containers. ### Error Exception ```log java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0 at org.apache.seatunnel.connectors.seatunnel.common.source.SingleSplitEnumerator.addSplitsBack(SingleSplitEnumerator.java:58) ``` ### Zeta or Flink or Spark Version Flink 1.13.6, 1.15.3, 1.18.0 (the e2e images). ### Java or Scala Version JDK 11 ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
