github-actions[bot] commented on code in PR #62238:
URL: https://github.com/apache/doris/pull/62238#discussion_r3055231189
##########
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingJobSchedulerTask.java:
##########
@@ -83,28 +86,41 @@ private void handleRunningState() throws JobException {
streamingInsertJob.fetchMeta();
}
- private void autoResumeHandler() throws JobException {
- final FailureReason failureReason =
streamingInsertJob.getFailureReason();
+ private void handleRetryingState() throws JobException {
final long latestAutoResumeTimestamp =
streamingInsertJob.getLatestAutoResumeTimestamp();
final long autoResumeCount = streamingInsertJob.getAutoResumeCount();
final long current = System.currentTimeMillis();
- if (failureReason != null
- && failureReason.getCode() !=
InternalErrorCode.MANUAL_PAUSE_ERR
- && failureReason.getCode() !=
InternalErrorCode.TOO_MANY_FAILURE_ROWS_ERR
- && failureReason.getCode() !=
InternalErrorCode.CANNOT_RESUME_ERR) {
- long autoResumeIntervalTimeSec = autoResumeCount < 5
- ? Math.min((long) Math.pow(2, autoResumeCount) *
BACK_OFF_BASIC_TIME_SEC,
- MAX_BACK_OFF_TIME_SEC) : MAX_BACK_OFF_TIME_SEC;
- if (current - latestAutoResumeTimestamp >
autoResumeIntervalTimeSec * 1000L) {
- streamingInsertJob.setLatestAutoResumeTimestamp(current);
- if (autoResumeCount < Long.MAX_VALUE) {
- streamingInsertJob.setAutoResumeCount(autoResumeCount + 1);
- }
- streamingInsertJob.updateJobStatus(JobStatus.PENDING);
+ long autoResumeIntervalTimeSec = autoResumeCount < 5
+ ? Math.min((long) Math.pow(2, autoResumeCount) *
BACK_OFF_BASIC_TIME_SEC,
+ MAX_BACK_OFF_TIME_SEC) : MAX_BACK_OFF_TIME_SEC;
+ if (current - latestAutoResumeTimestamp <= autoResumeIntervalTimeSec *
1000L) {
+ // backoff not reached, wait
+ return;
+ }
+
+ streamingInsertJob.setLatestAutoResumeTimestamp(current);
+ if (autoResumeCount < Long.MAX_VALUE) {
+ streamingInsertJob.setAutoResumeCount(autoResumeCount + 1);
+ }
+
+ if (Config.isCloudMode()) {
+ try {
+ streamingInsertJob.replayOnCloudMode();
+ } catch (JobException e) {
+ streamingInsertJob.setFailureReason(
+ new FailureReason(InternalErrorCode.INTERNAL_ERR,
e.getMessage()));
+ // stay in RETRYING, will retry next round
return;
}
}
+ if (streamingInsertJob.hasReachedEnd()) {
+ streamingInsertJob.updateJobStatus(JobStatus.FINISHED);
+ return;
+ }
+ streamingInsertJob.createStreamingTask();
+ streamingInsertJob.setSampleStartTime(System.currentTimeMillis());
Review Comment:
`handleRetryingState()` now creates a new streaming task but keeps the job
in `RETRYING`. Because `needScheduleTask()` returns `true` for `RETRYING` and
there is no guard here for an already-active `runningStreamTask`, the next
timer-driven `StreamingJobSchedulerTask` can come back through this branch
again before the first retry attempt finishes and call `createStreamingTask()`
a second time. That opens the door to two concurrent streaming tasks for the
same job. The old flow moved back to `PENDING`, which avoided this
duplicate-task window.
##########
fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingInsertJob.java:
##########
@@ -646,6 +646,11 @@ public void onStreamTaskSuccess(AbstractStreamingTask
task) throws JobException
}
Env.getCurrentEnv().getJobManager().getStreamingTaskManager().removeRunningTask(task);
+ // If the job was retrying due to previous failures, transition
back to RUNNING
+ if (JobStatus.RETRYING.equals(getJobStatus())) {
+ setAutoResumeCount(0);
+ updateJobStatus(JobStatus.RUNNING);
+ }
if (offsetProvider.hasReachedEnd()) {
// offset provider has reached a natural end, mark job as
finished
Review Comment:
`fetchMeta()` explicitly preserves `MANUAL_PAUSE_ERR` before switching to
`RETRYING`, but this task-failure path now unconditionally flips the job to
`RETRYING`. If the user runs `PAUSE JOB` while a streaming task is failing,
`onStreamTaskFail()` overwrites the manual pause reason/status and the job
starts auto-retrying again. That violates the new contract that `PAUSED` is
reserved for user-initiated pause and unrecoverable errors. This path needs the
same manual-pause guard as `fetchMeta()`; the scheduling-failure path has the
same race.
##########
regression-test/suites/job_p0/streaming_job/cdc/tvf/test_streaming_job_cdc_stream_postgres_latest_alter_cred.groovy:
##########
@@ -173,7 +173,7 @@
suite("test_streaming_job_cdc_stream_postgres_latest_alter_cred",
Awaitility.await().atMost(120, SECONDS).pollInterval(2,
SECONDS).until({
def s = sql """select status from jobs("type"="insert") where
Name='${jobName}'"""
log.info("status after wrong cred resume: " + s)
- s.size() == 1 && s.get(0).get(0) == "PAUSED"
+ s.size() == 1 && s.get(0).get(0) == "RETRYING"
Review Comment:
Once this assertion is updated to wait for `RETRYING`, the later `ALTER JOB`
in scenario C no longer works: `AlterJobCommand.validate()` still rejects
anything except `PAUSED`. Before this patch the auto-error state was `PAUSED`,
so the alter step was reachable; now this test needs an explicit `PAUSE JOB`
before the alter, or the product code needs to allow altering a `RETRYING` job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]