SEPURI-SAI-KRISHNA opened a new issue, #13253:
URL: https://github.com/apache/seatunnel/issues/13253

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Bug%22)
 and found no similar issues. No open PR changes `SingleSplitEnumerator` or 
`FlinkSourceEnumerator.addSplitsBack`.
   
   ### What happened
   
   On Flink, a source built on `SingleSplitEnumerator` (`connector-common`) 
fails the job with an `IndexOutOfBoundsException` on its second restart, and 
this exception replaces the error that caused the restarts. 29 connector 
modules use it, through `AbstractSingleSplitSource` or directly, among them 
`connector-http-base` (and the HTTP sources built on it, such as GraphQL), 
Cassandra, Redis, MQTT, Socket, Salesforce, Google Sheets and Google Pub/Sub.
   
   `SingleSplitEnumerator` on dev:
   
   ```java
   @Override
   public void addSplitsBack(List<SingleSplit> splits, int subtaskId) {
       pendingSplit = splits.get(0);
       assignSplit();
   }
   
   protected void assignSplit() {
       if (assigned || pendingSplit == null) {
           return;
       }
       ...
   }
   ```
   
   1. On the first restart, Flink passes back the split that was assigned since 
the last checkpoint. `addSplitsBack` stores it, but `assigned` is still `true`, 
so `assignSplit()` returns and the restarted reader never gets the split again.
   2. On the second restart, nothing was assigned since the last checkpoint, so 
Flink calls `addSplitsBack` with an empty list, and `splits.get(0)` throws. 
`SourceCoordinator` then triggers a global failover with this exception.
   
   Seen in the CI of #13222 (GraphQL source, fork run 
https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/42574864549, job 
`updated-modules-integration-test-part-1`): a job that fails on purpose is 
restarted twice by the Flink 1.13, 1.15 and 1.18 images, and on each of them 
the JobManager logs:
   
   ```log
   ERROR org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - 
Uncaught exception in the SplitEnumerator for Source Source: GraphQL-Source -> 
Sink Writer: MultiTableSink-Sink while handling subtask 0 recovery to 
checkpoint -1. Triggering job failover.
   java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
        at java.util.ArrayList.get(ArrayList.java:459) ~[?:?]
        at 
org.apache.seatunnel.connectors.seatunnel.common.source.SingleSplitEnumerator.addSplitsBack(SingleSplitEnumerator.java:58)
        at 
org.apache.seatunnel.translation.flink.source.FlinkSourceEnumerator.addSplitsBack(FlinkSourceEnumerator.java:91)
        at 
org.apache.flink.runtime.source.coordinator.SourceCoordinator.lambda$subtaskReset$3(SourceCoordinator.java:232)
 ~[flink-dist_2.11-1.13.6.jar:1.13.6]
   ```
   
   The job result then shows this exception instead of the GraphQL error, so 
`testRejectedQueryFailsTheJob` failed on those three images. The Flink 1.20 
image does not restart the job, and Zeta and Spark passed. #13222 works around 
it in its test by setting `execution.restart.strategy = "no"`.
   
   I replayed the restart sequence against the class with a mocked 
`SourceSplitEnumerator.Context` (start, `addSplitsBack([split])` and 
`registerReader` for restart 1, `addSplitsBack([])` for restart 2):
   
   | | assignments after restart 1 | restart 2 |
   | --- | --- | --- |
   | dev | 1 (the split is not assigned again) | `IndexOutOfBoundsException: 
Index: 0` |
   | fix below | 2 | no exception |
   
   Fix: ignore an empty list, and reset `assigned` when a split comes back, so 
it is assigned again when the reader registers:
   
   ```java
   public void addSplitsBack(List<SingleSplit> splits, int subtaskId) {
       if (splits.isEmpty()) {
           return;
       }
       pendingSplit = splits.get(0);
       assigned = false;
       assignSplit();
   }
   ```
   
   The same empty list should also reach the enumerator on the first restart 
after a completed checkpoint, when the split was assigned before that 
checkpoint, which would affect streaming jobs with checkpointing. I have not 
run that case.
   
   ### SeaTunnel Version
   
   dev (466d4168a5); `SingleSplitEnumerator` has not changed since #4101.
   
   ### SeaTunnel Config
   
   `graphql_rejected_query_to_assert.conf` from #13222 without its 
`execution.restart.strategy = "no"` line: a GraphQL source whose query the 
server rejects, so every attempt fails and Flink restarts the job.
   
   ```conf
   env {
     parallelism = 1
     job.mode = "BATCH"
   }
   
   source {
     GraphQL {
       plugin_output = "http"
       url = "http://graphql:8080/v1/graphql";
       format = "json"
       content_field = "$.data.missing_table"
       query = """
         query Rejected {
           missing_table {
             id
           }
         }
       """
       schema = {
         fields {
           id = "int"
         }
       }
     }
   }
   
   sink {
     Assert {
       plugin_input = http
       rules {
         row_rules = [
           {
             rule_type = MIN_ROW
             rule_value = 0
           }
         ]
       }
     }
   }
   ```
   
   ### Running Command
   
   ```shell
   ./mvnw -B verify -DskipUT=true -DskipIT=false -Dit.test=GraphQLIT -pl 
:connector-graphql-e2e -am -Pci
   ```
   
   on the branch of #13222 (the test `testRejectedQueryFailsTheJob` is added 
there), with the `execution.restart.strategy = "no"` line removed from 
`graphql_rejected_query_to_assert.conf`; it fails on the Flink 1.13, 1.15 and 
1.18 containers.
   
   ### Error Exception
   
   ```log
   java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
        at 
org.apache.seatunnel.connectors.seatunnel.common.source.SingleSplitEnumerator.addSplitsBack(SingleSplitEnumerator.java:58)
   ```
   
   ### Zeta or Flink or Spark Version
   
   Flink 1.13.6, 1.15.3, 1.18.0 (the e2e images).
   
   ### Java or Scala Version
   
   JDK 11
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to