DanielLeens commented on issue #13313:
URL: https://github.com/apache/seatunnel/issues/13313#issuecomment-6983599115
I reproduced the restore/error-handling chain with a focused Java 11
fault-injection probe on `89f91a89af15f33d97a942d8b65f4fc1b0c80bf5`.
Steps and observed results:
1. Invoke real `cleanPendingCheckpoint(CHECKPOINT_COORDINATOR_RESET)` to put
the coordinator in `shutdown=true`.
2. Make the ready-to-close IMap key throw an injected
`IllegalStateException` on read; call the real coordinator restore method
through the `CheckpointManager` dependency used by
`SubPlan.prepareRestorePipeline`.
3. The real SubPlan reset changes `FAILED → CREATED`; the coordinator
restore throws while loading ready-to-close state. The SubPlan catches it and
returns `true`, and coordinator `shutdown` remains `true`.
4. Mark all tasks ready and call real
`tryTriggerPendingCheckpoint(CHECKPOINT_TYPE)`. It creates no pending
checkpoint and emits the production warning reporting `shutdown (true)`.
The probe asserts that the manager call was actually reached, so an
unrelated fixture failure cannot satisfy the assertion. This confirms the
error-handling/trigger-guard chain, **not a live streaming/failover run or
measured data replay/loss**. The injected read exception is non-retryable; this
probe does not measure retry exhaustion.
The self-contained probe below also covers the adjacent checkpoint issues
#13321, #13322 and #13324. It calls unmodified production methods, using
Mockito/reflection to initialize narrow fixtures and mock external IMap/storage
dependencies.
### How I ran the focused probe
Baseline: `89f91a89af15f33d97a942d8b65f4fc1b0c80bf5`, macOS ARM64, JDK
11.0.27. The production source is unmodified. These are **in-process
fault-injection probes**, not end-to-end cluster/failover tests; passing the
probe means its asserted failure signature was observed.
Save the Java source below outside the checkout. From a clean checkout of
that commit:
```bash
export JAVA_HOME=/path/to/jdk-11
export PATH="$JAVA_HOME/bin:$PATH"
mvn -B -pl seatunnel-engine/seatunnel-engine-server -am \
-DskipTests -DskipUT=true -DskipIT=true -Dskip.ui=true \
-Dskip.spotless=true -Dmaven.gitcommitid.skip=true \
-Dmaven.javadoc.skip=true -Drat.skip=true package
mvn -X -Dstyle.color=never -pl seatunnel-engine/seatunnel-engine-server -am \
-DskipTests -DskipUT=true -DskipIT=true -Dskip.ui=true \
-Dskip.spotless=true -Dmaven.gitcommitid.skip=true \
-Dmaven.javadoc.skip=true -Drat.skip=true test-compile > classpath.log 2>&1
CP=$(sed -n 's/^.*(f) testPath = \[\(.*\)\]$/\1/p' classpath.log | tail -1 |
sed 's/, /:/g')
# The last reactor module is engine-server; CP uses these freshly compiled
classes.
```
This builds the required reactor modules, not the full project's test suite.
No Docker or live database is needed. I also compared the tested revision with
`dev` at `94adef596685ee37e506c96a52baa6ae13de5da3`; the production classes
exercised here are unchanged between them. That comparison is not a separate
runtime test of the newer commit.
```bash
mkdir -p probe-classes
javac -proc:none -cp "$CP" -d probe-classes CheckpointIssuesRepro.java
java -cp "probe-classes:$CP"
org.apache.seatunnel.engine.server.checkpoint.CheckpointIssuesRepro
```
<details>
<summary>Executable Java probe: CheckpointIssuesRepro.java</summary>
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.seatunnel.engine.server.checkpoint;
import com.hazelcast.map.IMap;
import org.apache.seatunnel.engine.server.dag.physical.SubPlan;
import org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan;
import org.apache.seatunnel.engine.server.master.JobMaster;
import org.apache.seatunnel.engine.core.job.PipelineStatus;
import org.apache.seatunnel.engine.checkpoint.storage.PipelineState;
import org.apache.seatunnel.engine.checkpoint.storage.api.CheckpointStorage;
import org.apache.seatunnel.engine.core.checkpoint.CheckpointType;
import org.apache.seatunnel.engine.core.job.RestoreMode;
import org.apache.seatunnel.engine.common.config.server.CheckpointConfig;
import org.apache.seatunnel.engine.serializer.protobuf.ProtoStuffSerializer;
import java.lang.reflect.*;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;
import static org.mockito.Mockito.*;
/** Executes unmodified production methods; mock only external storage/IMap
dependencies. */
public class CheckpointIssuesRepro {
static void set(Object o,String n,Object v)throws Exception {Field
f=null;Class<?>
c=o.getClass();while(c!=null){try{f=c.getDeclaredField(n);break;}catch(NoSuchFieldException
e){c=c.getSuperclass();}}if(f==null)throw new
NoSuchFieldException(n);f.setAccessible(true);f.set(o,v);}
static Object get(Object o,String n)throws Exception {Class<?>
c=o.getClass();while(c!=null){try{Field
f=c.getDeclaredField(n);f.setAccessible(true);return
f.get(o);}catch(NoSuchFieldException e){c=c.getSuperclass();}}throw new
NoSuchFieldException(n);}
static Object invoke(Object o,Class<?> owner,String n,Class<?>[]
types,Object...args)throws Throwable {Method
m=owner.getDeclaredMethod(n,types);m.setAccessible(true);try{return
m.invoke(o,args);}catch(InvocationTargetException e){throw e.getCause();}}
static void check(boolean b,String s){if(!b)throw new AssertionError(s);}
static CompletedCheckpoint cp(long id,CheckpointType t){return new
CompletedCheckpoint(7L,1,id,1L,t,2L,Collections.emptyMap(),Collections.emptyMap());}
static CheckpointCoordinator coordinator()throws Exception {
CheckpointCoordinator
c=mock(CheckpointCoordinator.class,CALLS_REAL_METHODS);
set(c,"jobId",7L);set(c,"pipelineId",1);set(c,"checkpointStateImapKey","status");set(c,"readyToCloseImapKey","ready");
set(c,"lock",new Object());set(c,"isAllTaskReady",new
AtomicBoolean());set(c,"schemaChanging",new
AtomicBoolean());set(c,"pendingCounter",new
AtomicInteger());set(c,"latestTriggerTimestamp",new AtomicLong());
set(c,"pendingCheckpoints",new
ConcurrentHashMap<>());set(c,"pipelineTaskStatus",new
HashMap<>());set(c,"readyToCloseStartingTask",new
HashSet<>());set(c,"readyToCloseIdleTask",new
HashSet<>());set(c,"closedIdleTask",new HashSet<>());
set(c,"completedCheckpointIds",new
ArrayDeque<String>());set(c,"scheduler",Executors.newScheduledThreadPool(1));set(c,"serializer",new
ProtoStuffSerializer());set(c,"coordinatorConfig",new CheckpointConfig());
return c;
}
static void close(CheckpointCoordinator c)throws Exception
{((ScheduledExecutorService)get(c,"scheduler")).shutdownNow();}
static void issue13321()throws Throwable {
CheckpointManager
m=mock(CheckpointManager.class,CALLS_REAL_METHODS);CheckpointStorage
storage=mock(CheckpointStorage.class);ProtoStuffSerializer serializer=new
ProtoStuffSerializer();set(m,"checkpointStorage",storage);set(m,"serializer",serializer);
PipelineState
good=PipelineState.builder().jobId("7").pipelineId(1).checkpointId(1).states(serializer.serialize(cp(1,CheckpointType.CHECKPOINT_TYPE))).build();
PipelineState
bad=PipelineState.builder().jobId("7").pipelineId(1).checkpointId(2).states(new
byte[]{10,(byte)255}).build();
when(storage.getCheckpointsByJobIdAndPipelineId("7","1")).thenReturn(Collections.singletonList(good));
Object
healthy=invoke(m,CheckpointManager.class,"getLatestCheckpointStateByType",new
Class[]{String.class,String.class,RestoreMode.class},"7","1",RestoreMode.CHECKPOINT);check(healthy==good,"healthy
control failed");
when(storage.getCheckpointsByJobIdAndPipelineId("7","1")).thenReturn(Arrays.asList(good,bad));
Throwable
failure=null;try{invoke(m,CheckpointManager.class,"getLatestCheckpointStateByType",new
Class[]{String.class,String.class,RestoreMode.class},"7","1",RestoreMode.CHECKPOINT);}catch(Throwable
e){failure=e;}
check(failure instanceof RuntimeException,"corrupt candidate did not
abort");System.out.println("REPRODUCED #13321: healthy checkpoint alone
selected; same candidate plus truncated checkpoint throws "+failure);
}
static void issue13322()throws Throwable {
CheckpointCoordinator c=coordinator();try{
CheckpointStorage
storage=mock(CheckpointStorage.class);set(c,"checkpointStorage",storage);doThrow(new
RuntimeException("injected storage write
failure")).when(storage).storeCheckPoint(any(PipelineState.class));
Throwable
failure=null;try{c.completePendingCheckpoint(cp(9,CheckpointType.CHECKPOINT_TYPE));}catch(Throwable
e){failure=e;}
check(failure!=null&&failure.getMessage().contains("injected
storage"),"did not reach failing
store");verify(storage).storeCheckPoint(any(PipelineState.class));
ArrayDeque<String>
q=(ArrayDeque<String>)get(c,"completedCheckpointIds");check(q.contains("9"),"failed
ID absent");
c.cleanPendingCheckpoint(CheckpointCloseReason.CHECKPOINT_COORDINATOR_RESET);check(q.contains("9"),"reset
unexpectedly cleared ID");
System.out.println("REPRODUCED #13322: failed store leaves ID 9 queued;
real RESET cleanup preserves it. Queue="+q);
}finally{close(c);}
}
static void issue13324()throws Throwable {
CheckpointCoordinator c=coordinator();try{
IMap<Object,Object>
map=mock(IMap.class);set(c,"runningJobStateIMap",map);when(map.get("status")).thenReturn(CheckpointCoordinatorStatus.RUNNING);doThrow(new
IllegalStateException("injected IMap set
failure")).when(map).set(eq("status"),eq(CheckpointCoordinatorStatus.FINISHED));
set(c,"latestCompletedCheckpoint",cp(4,CheckpointType.COMPLETED_POINT_TYPE));check(c.isCompleted(),"final
checkpoint control false");
invoke(c,CheckpointCoordinator.class,"updateStatus",new
Class[]{CheckpointCoordinatorStatus.class},CheckpointCoordinatorStatus.FINISHED);verify(map).set("status",CheckpointCoordinatorStatus.FINISHED);check(!c.isNoErrorCompleted(),"stale
state did not affect completion");
when(map.get("status")).thenReturn(CheckpointCoordinatorStatus.FINISHED);check(c.isNoErrorCompleted(),"persisted
status control failed");
System.out.println("REPRODUCED #13324: terminal set throws but
updateStatus returns normally; stale RUNNING yields isNoErrorCompleted=false;
FINISHED control yields true. No whole-cluster failover claimed.");
}finally{close(c);}
}
static void issue13313()throws Throwable {
CheckpointCoordinator c=coordinator();try{
IMap<Object,Object>
map=mock(IMap.class);set(c,"runningJobStateIMap",map);when(map.get("status")).thenReturn(CheckpointCoordinatorStatus.RUNNING);when(map.get("ready")).thenThrow(new
IllegalStateException("injected ready-state read failure"));
c.cleanPendingCheckpoint(CheckpointCloseReason.CHECKPOINT_COORDINATOR_RESET);check(Boolean.TRUE.equals(get(c,"shutdown")),"failed
generation setup not shut down");
Throwable failure=null;try{c.restoreCoordinator(false);}catch(Throwable
e){failure=e;}
check(failure!=null&&failure.getMessage().contains("Failed to
load"),"restore did not fail at ready-state
load");check(Boolean.TRUE.equals(get(c,"shutdown")),"failed restore cleared
shutdown");
SubPlan sub=mock(SubPlan.class,CALLS_REAL_METHODS);
set(sub,"restoreLock",new Object());set(sub,"pipelineRestoreNum",new
AtomicInteger());set(sub,"errorByPhysicalVertex",new
AtomicReference<String>("prior generation failed"));set(sub,"restoreCauses",new
ConcurrentLinkedQueue<String>());set(sub,"currPipelineStatus",PipelineStatus.FAILED);set(sub,"pipelineId",1);
set(sub,"finishedTaskNum",new
AtomicInteger());set(sub,"canceledTaskNum",new
AtomicInteger());set(sub,"failedTaskNum",new
AtomicInteger());set(sub,"coordinatorVertexList",Collections.emptyList());set(sub,"physicalVertexList",Collections.emptyList());set(sub,"runningJobStateIMap",mock(IMap.class));set(sub,"runningJobStateTimestampsIMap",mock(IMap.class));
JobMaster master=mock(JobMaster.class);CheckpointManager
manager=mock(CheckpointManager.class);PhysicalPlan
physical=mock(PhysicalPlan.class);set(sub,"jobMaster",master);when(master.getCheckpointManager()).thenReturn(manager);when(master.getPhysicalPlan()).thenReturn(physical);
doAnswer(inv->{c.restoreCoordinator(false);return
null;}).when(manager).reportedPipelineRunning(1,false);
Object proceed=invoke(sub,SubPlan.class,"prepareRestorePipeline",new
Class[]{});
verify(manager).reportedPipelineRunning(1,false);check(Boolean.TRUE.equals(proceed),"SubPlan
did not swallow coordinator
failure");check(get(sub,"currPipelineStatus")==PipelineStatus.CREATED,"SubPlan
reset did not run");check(Boolean.TRUE.equals(get(c,"shutdown")),"shutdown
changed");
((AtomicBoolean)get(c,"isAllTaskReady")).set(true);
c.tryTriggerPendingCheckpoint(CheckpointType.CHECKPOINT_TYPE);
check(((Map<?,?>)get(c,"pendingCheckpoints")).isEmpty(),"checkpoint
created despite shutdown");
System.out.println("REPRODUCED #13313: actual trigger called with all
tasks ready creates no checkpoint; actual SubPlan reset reaches CREATED, actual
coordinator restore throws at ready-state read, prepareRestorePipeline returns
true, coordinator shutdown remains true. No live streaming/data-flow claim.");
}finally{close(c);}
}
public static void main(String[] args)throws Throwable
{System.out.println("CheckpointCoordinator loaded from
"+CheckpointCoordinator.class.getProtectionDomain().getCodeSource().getLocation());System.out.println("SubPlan
loaded from
"+SubPlan.class.getProtectionDomain().getCodeSource().getLocation());issue13321();issue13322();issue13324();issue13313();System.out.println("ALL
FOUR FOCUSED CHECKS PASSED");}
}
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]