DanielLeens commented on issue #13313:
URL: https://github.com/apache/seatunnel/issues/13313#issuecomment-6983599115

   I reproduced the restore/error-handling chain with a focused Java 11 
fault-injection probe on `89f91a89af15f33d97a942d8b65f4fc1b0c80bf5`.
   
   Steps and observed results:
   1. Invoke real `cleanPendingCheckpoint(CHECKPOINT_COORDINATOR_RESET)` to put 
the coordinator in `shutdown=true`.
   2. Make the ready-to-close IMap key throw an injected 
`IllegalStateException` on read; call the real coordinator restore method 
through the `CheckpointManager` dependency used by 
`SubPlan.prepareRestorePipeline`.
   3. The real SubPlan reset changes `FAILED → CREATED`; the coordinator 
restore throws while loading ready-to-close state. The SubPlan catches it and 
returns `true`, and coordinator `shutdown` remains `true`.
   4. Mark all tasks ready and call real 
`tryTriggerPendingCheckpoint(CHECKPOINT_TYPE)`. It creates no pending 
checkpoint and emits the production warning reporting `shutdown (true)`.
   
   The probe asserts that the manager call was actually reached, so an 
unrelated fixture failure cannot satisfy the assertion. This confirms the 
error-handling/trigger-guard chain, **not a live streaming/failover run or 
measured data replay/loss**. The injected read exception is non-retryable; this 
probe does not measure retry exhaustion.
   
   The self-contained probe below also covers the adjacent checkpoint issues 
#13321, #13322 and #13324. It calls unmodified production methods, using 
Mockito/reflection to initialize narrow fixtures and mock external IMap/storage 
dependencies.
   
   ### How I ran the focused probe
   
   Baseline: `89f91a89af15f33d97a942d8b65f4fc1b0c80bf5`, macOS ARM64, JDK 
11.0.27. The production source is unmodified. These are **in-process 
fault-injection probes**, not end-to-end cluster/failover tests; passing the 
probe means its asserted failure signature was observed.
   
   Save the Java source below outside the checkout. From a clean checkout of 
that commit:
   
   ```bash
   export JAVA_HOME=/path/to/jdk-11
   export PATH="$JAVA_HOME/bin:$PATH"
   mvn -B -pl seatunnel-engine/seatunnel-engine-server -am \
     -DskipTests -DskipUT=true -DskipIT=true -Dskip.ui=true \
     -Dskip.spotless=true -Dmaven.gitcommitid.skip=true \
     -Dmaven.javadoc.skip=true -Drat.skip=true package
   mvn -X -Dstyle.color=never -pl seatunnel-engine/seatunnel-engine-server -am \
     -DskipTests -DskipUT=true -DskipIT=true -Dskip.ui=true \
     -Dskip.spotless=true -Dmaven.gitcommitid.skip=true \
     -Dmaven.javadoc.skip=true -Drat.skip=true test-compile > classpath.log 2>&1
   CP=$(sed -n 's/^.*(f) testPath = \[\(.*\)\]$/\1/p' classpath.log | tail -1 | 
sed 's/, /:/g')
   # The last reactor module is engine-server; CP uses these freshly compiled 
classes.
   ```
   
   This builds the required reactor modules, not the full project's test suite. 
No Docker or live database is needed. I also compared the tested revision with 
`dev` at `94adef596685ee37e506c96a52baa6ae13de5da3`; the production classes 
exercised here are unchanged between them. That comparison is not a separate 
runtime test of the newer commit.
   
   ```bash
   mkdir -p probe-classes
   javac -proc:none -cp "$CP" -d probe-classes CheckpointIssuesRepro.java
   java -cp "probe-classes:$CP" 
org.apache.seatunnel.engine.server.checkpoint.CheckpointIssuesRepro
   ```
   
   <details>
   <summary>Executable Java probe: CheckpointIssuesRepro.java</summary>
   
   ```java
   /*
    * Licensed to the Apache Software Foundation (ASF) under one or more
    * contributor license agreements.  See the NOTICE file distributed with
    * this work for additional information regarding copyright ownership.
    * The ASF licenses this file to You under the Apache License, Version 2.0
    * (the "License"); you may not use this file except in compliance with
    * the License.  You may obtain a copy of the License at
    *
    *    http://www.apache.org/licenses/LICENSE-2.0
    *
    * Unless required by applicable law or agreed to in writing, software
    * distributed under the License is distributed on an "AS IS" BASIS,
    * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    * See the License for the specific language governing permissions and
    * limitations under the License.
    */
   
   package org.apache.seatunnel.engine.server.checkpoint;
   
   import com.hazelcast.map.IMap;
   import org.apache.seatunnel.engine.server.dag.physical.SubPlan;
   import org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan;
   import org.apache.seatunnel.engine.server.master.JobMaster;
   import org.apache.seatunnel.engine.core.job.PipelineStatus;
   import org.apache.seatunnel.engine.checkpoint.storage.PipelineState;
   import org.apache.seatunnel.engine.checkpoint.storage.api.CheckpointStorage;
   import org.apache.seatunnel.engine.core.checkpoint.CheckpointType;
   import org.apache.seatunnel.engine.core.job.RestoreMode;
   import org.apache.seatunnel.engine.common.config.server.CheckpointConfig;
   import org.apache.seatunnel.engine.serializer.protobuf.ProtoStuffSerializer;
   import java.lang.reflect.*;
   import java.util.*;
   import java.util.concurrent.*;
   import java.util.concurrent.atomic.*;
   import static org.mockito.Mockito.*;
   
   /** Executes unmodified production methods; mock only external storage/IMap 
dependencies. */
   public class CheckpointIssuesRepro {
    static void set(Object o,String n,Object v)throws Exception {Field 
f=null;Class<?> 
c=o.getClass();while(c!=null){try{f=c.getDeclaredField(n);break;}catch(NoSuchFieldException
 e){c=c.getSuperclass();}}if(f==null)throw new 
NoSuchFieldException(n);f.setAccessible(true);f.set(o,v);}
    static Object get(Object o,String n)throws Exception {Class<?> 
c=o.getClass();while(c!=null){try{Field 
f=c.getDeclaredField(n);f.setAccessible(true);return 
f.get(o);}catch(NoSuchFieldException e){c=c.getSuperclass();}}throw new 
NoSuchFieldException(n);}
    static Object invoke(Object o,Class<?> owner,String n,Class<?>[] 
types,Object...args)throws Throwable {Method 
m=owner.getDeclaredMethod(n,types);m.setAccessible(true);try{return 
m.invoke(o,args);}catch(InvocationTargetException e){throw e.getCause();}}
    static void check(boolean b,String s){if(!b)throw new AssertionError(s);}
    static CompletedCheckpoint cp(long id,CheckpointType t){return new 
CompletedCheckpoint(7L,1,id,1L,t,2L,Collections.emptyMap(),Collections.emptyMap());}
    static CheckpointCoordinator coordinator()throws Exception {
     CheckpointCoordinator 
c=mock(CheckpointCoordinator.class,CALLS_REAL_METHODS);
     
set(c,"jobId",7L);set(c,"pipelineId",1);set(c,"checkpointStateImapKey","status");set(c,"readyToCloseImapKey","ready");
     set(c,"lock",new Object());set(c,"isAllTaskReady",new 
AtomicBoolean());set(c,"schemaChanging",new 
AtomicBoolean());set(c,"pendingCounter",new 
AtomicInteger());set(c,"latestTriggerTimestamp",new AtomicLong());
     set(c,"pendingCheckpoints",new 
ConcurrentHashMap<>());set(c,"pipelineTaskStatus",new 
HashMap<>());set(c,"readyToCloseStartingTask",new 
HashSet<>());set(c,"readyToCloseIdleTask",new 
HashSet<>());set(c,"closedIdleTask",new HashSet<>());
     set(c,"completedCheckpointIds",new 
ArrayDeque<String>());set(c,"scheduler",Executors.newScheduledThreadPool(1));set(c,"serializer",new
 ProtoStuffSerializer());set(c,"coordinatorConfig",new CheckpointConfig());
     return c;
    }
    static void close(CheckpointCoordinator c)throws Exception 
{((ScheduledExecutorService)get(c,"scheduler")).shutdownNow();}
    static void issue13321()throws Throwable {
     CheckpointManager 
m=mock(CheckpointManager.class,CALLS_REAL_METHODS);CheckpointStorage 
storage=mock(CheckpointStorage.class);ProtoStuffSerializer serializer=new 
ProtoStuffSerializer();set(m,"checkpointStorage",storage);set(m,"serializer",serializer);
     PipelineState 
good=PipelineState.builder().jobId("7").pipelineId(1).checkpointId(1).states(serializer.serialize(cp(1,CheckpointType.CHECKPOINT_TYPE))).build();
     PipelineState 
bad=PipelineState.builder().jobId("7").pipelineId(1).checkpointId(2).states(new 
byte[]{10,(byte)255}).build();
     
when(storage.getCheckpointsByJobIdAndPipelineId("7","1")).thenReturn(Collections.singletonList(good));
     Object 
healthy=invoke(m,CheckpointManager.class,"getLatestCheckpointStateByType",new 
Class[]{String.class,String.class,RestoreMode.class},"7","1",RestoreMode.CHECKPOINT);check(healthy==good,"healthy
 control failed");
     
when(storage.getCheckpointsByJobIdAndPipelineId("7","1")).thenReturn(Arrays.asList(good,bad));
     Throwable 
failure=null;try{invoke(m,CheckpointManager.class,"getLatestCheckpointStateByType",new
 
Class[]{String.class,String.class,RestoreMode.class},"7","1",RestoreMode.CHECKPOINT);}catch(Throwable
 e){failure=e;}
     check(failure instanceof RuntimeException,"corrupt candidate did not 
abort");System.out.println("REPRODUCED #13321: healthy checkpoint alone 
selected; same candidate plus truncated checkpoint throws "+failure);
    }
    static void issue13322()throws Throwable {
     CheckpointCoordinator c=coordinator();try{
      CheckpointStorage 
storage=mock(CheckpointStorage.class);set(c,"checkpointStorage",storage);doThrow(new
 RuntimeException("injected storage write 
failure")).when(storage).storeCheckPoint(any(PipelineState.class));
      Throwable 
failure=null;try{c.completePendingCheckpoint(cp(9,CheckpointType.CHECKPOINT_TYPE));}catch(Throwable
 e){failure=e;}
      check(failure!=null&&failure.getMessage().contains("injected 
storage"),"did not reach failing 
store");verify(storage).storeCheckPoint(any(PipelineState.class));
      ArrayDeque<String> 
q=(ArrayDeque<String>)get(c,"completedCheckpointIds");check(q.contains("9"),"failed
 ID absent");
      
c.cleanPendingCheckpoint(CheckpointCloseReason.CHECKPOINT_COORDINATOR_RESET);check(q.contains("9"),"reset
 unexpectedly cleared ID");
      System.out.println("REPRODUCED #13322: failed store leaves ID 9 queued; 
real RESET cleanup preserves it. Queue="+q);
     }finally{close(c);}
    }
    static void issue13324()throws Throwable {
     CheckpointCoordinator c=coordinator();try{
      IMap<Object,Object> 
map=mock(IMap.class);set(c,"runningJobStateIMap",map);when(map.get("status")).thenReturn(CheckpointCoordinatorStatus.RUNNING);doThrow(new
 IllegalStateException("injected IMap set 
failure")).when(map).set(eq("status"),eq(CheckpointCoordinatorStatus.FINISHED));
      
set(c,"latestCompletedCheckpoint",cp(4,CheckpointType.COMPLETED_POINT_TYPE));check(c.isCompleted(),"final
 checkpoint control false");
      invoke(c,CheckpointCoordinator.class,"updateStatus",new 
Class[]{CheckpointCoordinatorStatus.class},CheckpointCoordinatorStatus.FINISHED);verify(map).set("status",CheckpointCoordinatorStatus.FINISHED);check(!c.isNoErrorCompleted(),"stale
 state did not affect completion");
      
when(map.get("status")).thenReturn(CheckpointCoordinatorStatus.FINISHED);check(c.isNoErrorCompleted(),"persisted
 status control failed");
      System.out.println("REPRODUCED #13324: terminal set throws but 
updateStatus returns normally; stale RUNNING yields isNoErrorCompleted=false; 
FINISHED control yields true. No whole-cluster failover claimed.");
     }finally{close(c);}
    }
    static void issue13313()throws Throwable {
     CheckpointCoordinator c=coordinator();try{
      IMap<Object,Object> 
map=mock(IMap.class);set(c,"runningJobStateIMap",map);when(map.get("status")).thenReturn(CheckpointCoordinatorStatus.RUNNING);when(map.get("ready")).thenThrow(new
 IllegalStateException("injected ready-state read failure"));
      
c.cleanPendingCheckpoint(CheckpointCloseReason.CHECKPOINT_COORDINATOR_RESET);check(Boolean.TRUE.equals(get(c,"shutdown")),"failed
 generation setup not shut down");
      Throwable failure=null;try{c.restoreCoordinator(false);}catch(Throwable 
e){failure=e;}
      check(failure!=null&&failure.getMessage().contains("Failed to 
load"),"restore did not fail at ready-state 
load");check(Boolean.TRUE.equals(get(c,"shutdown")),"failed restore cleared 
shutdown");
      SubPlan sub=mock(SubPlan.class,CALLS_REAL_METHODS);
      set(sub,"restoreLock",new Object());set(sub,"pipelineRestoreNum",new 
AtomicInteger());set(sub,"errorByPhysicalVertex",new 
AtomicReference<String>("prior generation failed"));set(sub,"restoreCauses",new 
ConcurrentLinkedQueue<String>());set(sub,"currPipelineStatus",PipelineStatus.FAILED);set(sub,"pipelineId",1);
      set(sub,"finishedTaskNum",new 
AtomicInteger());set(sub,"canceledTaskNum",new 
AtomicInteger());set(sub,"failedTaskNum",new 
AtomicInteger());set(sub,"coordinatorVertexList",Collections.emptyList());set(sub,"physicalVertexList",Collections.emptyList());set(sub,"runningJobStateIMap",mock(IMap.class));set(sub,"runningJobStateTimestampsIMap",mock(IMap.class));
      JobMaster master=mock(JobMaster.class);CheckpointManager 
manager=mock(CheckpointManager.class);PhysicalPlan 
physical=mock(PhysicalPlan.class);set(sub,"jobMaster",master);when(master.getCheckpointManager()).thenReturn(manager);when(master.getPhysicalPlan()).thenReturn(physical);
      doAnswer(inv->{c.restoreCoordinator(false);return 
null;}).when(manager).reportedPipelineRunning(1,false);
      Object proceed=invoke(sub,SubPlan.class,"prepareRestorePipeline",new 
Class[]{});
      
verify(manager).reportedPipelineRunning(1,false);check(Boolean.TRUE.equals(proceed),"SubPlan
 did not swallow coordinator 
failure");check(get(sub,"currPipelineStatus")==PipelineStatus.CREATED,"SubPlan 
reset did not run");check(Boolean.TRUE.equals(get(c,"shutdown")),"shutdown 
changed");
      ((AtomicBoolean)get(c,"isAllTaskReady")).set(true);
      c.tryTriggerPendingCheckpoint(CheckpointType.CHECKPOINT_TYPE);
      check(((Map<?,?>)get(c,"pendingCheckpoints")).isEmpty(),"checkpoint 
created despite shutdown");
      System.out.println("REPRODUCED #13313: actual trigger called with all 
tasks ready creates no checkpoint; actual SubPlan reset reaches CREATED, actual 
coordinator restore throws at ready-state read, prepareRestorePipeline returns 
true, coordinator shutdown remains true. No live streaming/data-flow claim.");
     }finally{close(c);}
    }
    public static void main(String[] args)throws Throwable 
{System.out.println("CheckpointCoordinator loaded from 
"+CheckpointCoordinator.class.getProtectionDomain().getCodeSource().getLocation());System.out.println("SubPlan
 loaded from 
"+SubPlan.class.getProtectionDomain().getCodeSource().getLocation());issue13321();issue13322();issue13324();issue13313();System.out.println("ALL
 FOUR FOCUSED CHECKS PASSED");}
   }
   
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to