DanielLeens commented on issue #13316:
URL: https://github.com/apache/seatunnel/issues/13316#issuecomment-6983695881

   I reproduced the unresolved-completion-future chain on 
`89f91a89af15f33d97a942d8b65f4fc1b0c80bf5` (Java 11.0.27), using actual 
`JobMaster.initStateFuture()` and `cleanJob()`.
   
   Steps:
   1. Give the PhysicalPlan an externally completable result future and status 
`FINISHED`.
   2. Make the checkpoint-manager dependency throw from 
`clearCheckpointIfNeed(FINISHED)`.
   3. Register the real JobMaster callback with `initStateFuture()`, then 
complete the PhysicalPlan result.
   4. The actual `cleanJob()` throws before completing 
`jobMasterCompleteFuture`. That future is not done, and a thread calling 
`join()` is still blocked after a bounded 200 ms observation.
   5. Control: when cleanup is stubbed as a successful no-op, the same callback 
completes the JobMaster future. The harness explicitly resolves the stuck 
future afterward only to clean up its own join thread.
   
   This is a method-level injected cleanup-failure reproduction, not an 
observed coordinator/client hang in a live cluster. The successful control is a 
no-op cleanup stub, not a full successful production cleanup run.
   
   The executable probe below also covers #13317 and #13320. It uses the 
`StateRepros.java` fixture helpers already [published with 
#13314](https://github.com/apache/seatunnel/issues/13314#issuecomment-6983602245);
 save that file alongside `LifecycleRepros.java`.
   
   ### How I ran the focused probe
   
   Baseline: `89f91a89af15f33d97a942d8b65f4fc1b0c80bf5`, macOS ARM64, JDK 
11.0.27. The production source is unmodified. These are **in-process 
fault-injection probes**, not end-to-end cluster/failover tests; passing the 
probe means its asserted failure signature was observed.
   
   Save the Java source below outside the checkout. From a clean checkout of 
that commit:
   
   ```bash
   export JAVA_HOME=/path/to/jdk-11
   export PATH="$JAVA_HOME/bin:$PATH"
   mvn -B -pl seatunnel-engine/seatunnel-engine-server -am \
     -DskipTests -DskipUT=true -DskipIT=true -Dskip.ui=true \
     -Dskip.spotless=true -Dmaven.gitcommitid.skip=true \
     -Dmaven.javadoc.skip=true -Drat.skip=true package
   mvn -X -Dstyle.color=never -pl seatunnel-engine/seatunnel-engine-server -am \
     -DskipTests -DskipUT=true -DskipIT=true -Dskip.ui=true \
     -Dskip.spotless=true -Dmaven.gitcommitid.skip=true \
     -Dmaven.javadoc.skip=true -Drat.skip=true test-compile > classpath.log 2>&1
   CP=$(sed -n 's/^.*(f) testPath = \[\(.*\)\]$/\1/p' classpath.log | tail -1 | 
sed 's/, /:/g')
   # The last reactor module is engine-server; CP uses these freshly compiled 
classes.
   ```
   
   This builds the required reactor modules, not the full project's test suite. 
No Docker or live database is needed. I also compared the tested revision with 
`dev` at `94adef596685ee37e506c96a52baa6ae13de5da3`; the production classes 
exercised here are unchanged between them. That comparison is not a separate 
runtime test of the newer commit.
   
   ```bash
   mkdir -p probe-classes
   javac -proc:none -cp "$CP" -d probe-classes StateRepros.java 
LifecycleRepros.java
   java -cp "probe-classes:$CP" 
org.apache.seatunnel.engine.server.dag.physical.LifecycleRepros
   ```
   
   <details>
   <summary>Executable Java probe: LifecycleRepros.java</summary>
   
   ```java
   /* Licensed to the Apache Software Foundation (ASF) under one or more
    * contributor license agreements. See the NOTICE file distributed with
    * this work for additional information regarding copyright ownership.
    * The ASF licenses this file to You under the Apache License, Version 2.0
    * (the "License"); you may not use this file except in compliance with
    * the License. You may obtain a copy at 
http://www.apache.org/licenses/LICENSE-2.0
    * Unless required by applicable law or agreed to in writing, software
    * distributed under the License is distributed on an "AS IS" BASIS,
    * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND. See the License for the
    * specific language governing permissions and limitations under the License.
    */
   package org.apache.seatunnel.engine.server.dag.physical;
   import org.apache.seatunnel.engine.common.Constant;
   import org.apache.seatunnel.engine.common.config.JobConfig;
   import org.apache.seatunnel.engine.core.job.JobImmutableInformation;
   import com.hazelcast.flakeidgen.FlakeIdGenerator;
   import org.apache.seatunnel.engine.common.job.*;
   import org.apache.seatunnel.engine.common.utils.PassiveCompletableFuture;
   import org.apache.seatunnel.engine.server.execution.*;
   import org.apache.seatunnel.engine.server.master.JobMaster;
   import org.apache.seatunnel.engine.server.checkpoint.CheckpointManager;
   import org.apache.seatunnel.engine.server.resourcemanager.ResourceManager;
   import 
org.apache.seatunnel.engine.server.resourcemanager.resource.SlotProfile;
   import com.hazelcast.cluster.Address;
   import com.hazelcast.map.IMap;
   import com.hazelcast.spi.impl.NodeEngine;
   import com.hazelcast.spi.impl.operationservice.impl.InvocationFuture;
   import java.lang.reflect.*;
   import java.util.*;
   import java.util.concurrent.*;
   import static org.mockito.Mockito.*;
   import static org.mockito.ArgumentMatchers.*;
   public class LifecycleRepros {
    static IMap<Object,Object> stateMap(Map<Object,Object> 
m){IMap<Object,Object> 
im=mock(IMap.class);when(im.get(any())).thenAnswer(a->m.get(a.getArgument(0)));when(im.put(any(),any())).thenAnswer(a->m.put(a.getArgument(0),a.getArgument(1)));doAnswer(a->{m.put(a.getArgument(0),a.getArgument(1));return
 null;}).when(im).set(any(),any());return im;}
    static IMap<Object,Long[]> times(){IMap<Object,Long[]> 
im=mock(IMap.class);Map<Object,Long[]> m=new 
HashMap<>();when(im.get(any())).thenAnswer(a->m.get(a.getArgument(0)));when(im.put(any(),any())).thenAnswer(a->m.put(a.getArgument(0),a.getArgument(1)));doAnswer(a->{m.put(a.getArgument(0),a.getArgument(1));return
 null;}).when(im).set(any(),any());return im;}
    static JobImmutableInformation info(){JobImmutableInformation 
i=mock(JobImmutableInformation.class);JobConfig cfg=new 
JobConfig();cfg.getEnvOptions().put("job.retry.times",3);when(i.getJobConfig()).thenReturn(cfg);when(i.getJobId()).thenReturn(1L);return
 i;}
    static PhysicalVertex vertex(IMap<Object,Object> map,NodeEngine 
node,TaskGroupLocation loc){TaskGroupDefaultImpl 
group=mock(TaskGroupDefaultImpl.class);when(group.getTaskGroupLocation()).thenReturn(loc);when(group.getTaskGroupName()).thenReturn("repro");return
 new 
PhysicalVertex(0,1,group,mock(FlakeIdGenerator.class),1,1,Collections.emptyList(),Collections.emptyList(),info(),System.currentTimeMillis(),node,map,times());}
    static void set(Object o,String n,Object v)throws Exception {Field 
f=JobMaster.class.getDeclaredField(n);f.setAccessible(true);f.set(o,v);}
    static void require(boolean b,String m){if(!b)throw new AssertionError(m);}
    static void completion(boolean fail)throws Exception {
     JobMaster jm=mock(JobMaster.class);PhysicalPlan 
plan=mock(PhysicalPlan.class);CompletableFuture<JobResult> result=new 
CompletableFuture<>();
     when(plan.initStateFuture()).thenReturn(new 
PassiveCompletableFuture<>(result));when(plan.getJobStatus()).thenReturn(JobStatus.FINISHED);set(jm,"physicalPlan",plan);
     
doCallRealMethod().when(jm).initStateFuture();doCallRealMethod().when(jm).getJobMasterCompleteFuture();
     if(fail){CheckpointManager 
checkpoint=mock(CheckpointManager.class);doThrow(new RuntimeException("injected 
checkpoint delete 
failure")).when(checkpoint).clearCheckpointIfNeed(JobStatus.FINISHED);set(jm,"checkpointManager",checkpoint);doCallRealMethod().when(jm).cleanJob();}
     jm.initStateFuture();result.complete(new 
JobResult(JobStatus.FINISHED,null));
     Field 
f=JobMaster.class.getDeclaredField("jobMasterCompleteFuture");f.setAccessible(true);CompletableFuture<JobResult>
 complete=(CompletableFuture<JobResult>)f.get(jm);
     if(fail){require(!complete.isDone(),"failure must leave future 
incomplete");Thread join=new 
Thread(complete::join);join.setDaemon(true);join.start();join.join(200);require(join.isAlive(),"join
 should remain blocked");System.out.println("13316 REPRODUCED: real 
initStateFuture + real cleanJob; checkpoint delete throws; completion=false; 
join thread remains blocked after 200ms. Cleanup completion below is 
harness-only.");complete.complete(new 
JobResult(JobStatus.FINISHED,null));join.join(1000);require(!join.isAlive(),"harness
 cleanup");}
     else {require(complete.isDone(),"normal cleanup 
completion");System.out.println("13316 CONTROL: no-op successful cleanup 
completes future.");}
    }
    static void probe(Object response)throws Exception {
     TaskGroupLocation loc=new TaskGroupLocation(1,1,1);Map<Object,Object> 
states=new HashMap<>();states.put(loc,ExecutionState.RUNNING);
     NodeEngine node=mock(NodeEngine.class,RETURNS_DEEP_STUBS);Address addr=new 
Address("127.0.0.1",5801);com.hazelcast.cluster.impl.MemberImpl 
member=mock(com.hazelcast.cluster.impl.MemberImpl.class);when(member.getAddress()).thenReturn(addr);when(node.getClusterService().getMembers()).thenReturn(Collections.singleton(member));
     IMap owned=mock(IMap.class);SlotProfile 
slot=mock(SlotProfile.class);when(slot.getWorker()).thenReturn(addr);when(owned.get(any())).thenReturn(Collections.singletonMap(loc,slot));when(node.getHazelcastInstance().getMap(Constant.IMAP_OWNED_SLOT_PROFILES)).thenReturn(owned);
     InvocationFuture<Object> 
invocation=mock(InvocationFuture.class);if(response instanceof 
Exception)when(invocation.get()).thenThrow((Exception)response);else 
when(invocation.get()).thenReturn(response);
     
when(node.getOperationService().createInvocationBuilder(eq(Constant.SEATUNNEL_SERVICE_NAME),any(org.apache.seatunnel.engine.server.task.operation.CheckTaskGroupIsExecutingOperation.class),eq(addr)).invoke()).thenReturn(invocation);
     PhysicalVertex 
v=vertex(stateMap(states),node,loc);v.initStateFuture();ExecutionState 
expected=Boolean.TRUE.equals(response)?ExecutionState.RUNNING:ExecutionState.FAILING;
     require(v.getExecutionState()==expected,"probe 
transition");verify(invocation,times(1)).get();System.out.println("13320 
"+(response instanceof Exception?"REPRODUCED":"CONTROL")+": probe="+response+" 
-> vertex="+v.getExecutionState()+", IMap="+states.get(loc)+", one invocation. 
Worker membership and allocated slot present; actual remote task is mocked.");
    }
    static void staleEvent()throws Exception {
     TaskGroupLocation loc=new TaskGroupLocation(1,1,1);Map<Object,Object> 
states=new HashMap<>();states.put(loc,ExecutionState.FAILED);PhysicalVertex 
v=vertex(stateMap(states),mock(NodeEngine.class),loc);
     TaskExecutionState old=new 
TaskExecutionState(loc,ExecutionState.FAILED,"old attempt terminal 
event");v.updateStateByExecutionService(old);require(v.getExecutionState()==ExecutionState.FAILED,"terminal
 guard 
control");v.reset();v.updateTaskState(ExecutionState.RUNNING);require(v.getExecutionState()==ExecutionState.RUNNING,"new
 attempt simulated running");
     JobMaster jm=mock(JobMaster.class);PhysicalPlan 
plan=mock(PhysicalPlan.class);SubPlan 
sub=mock(SubPlan.class);when(sub.getPipelineLocation()).thenReturn(loc.getPipelineLocation());when(sub.getCoordinatorVertexList()).thenReturn(Collections.emptyList());when(sub.getPhysicalVertexList()).thenReturn(Collections.singletonList(v));when(plan.getPipelineList()).thenReturn(Collections.singletonList(sub));set(jm,"physicalPlan",plan);
     SlotProfile currentSlot=mock(SlotProfile.class);IMap 
owned=mock(IMap.class);when(owned.get(loc.getPipelineLocation())).thenReturn(Collections.singletonMap(loc,currentSlot));set(jm,"ownedSlotProfilesIMap",owned);ResourceManager
 
rm=mock(ResourceManager.class);when(rm.releaseResources(anyLong(),anyList())).thenReturn(org.apache.seatunnel.engine.common.utils.concurrent.CompletableFuture.completedFuture(null));set(jm,"resourceManager",rm);set(jm,"jobImmutableInformation",info());set(jm,"releasedSlotWhenTaskGroupFinished",new
 ConcurrentHashMap<>());
     
doCallRealMethod().when(jm).updateTaskExecutionState(any());doCallRealMethod().when(jm).releaseTaskGroupResource(any(),any());jm.updateTaskExecutionState(old);
     require(v.getExecutionState()==ExecutionState.FAILED,"old event changed 
new 
attempt");verify(rm).releaseResources(eq(1L),eq(Collections.singletonList(currentSlot)));System.out.println("13317
 REPRODUCED: reset FAILED->CREATED, advance to RUNNING, inject prior FAILED 
report with same TaskGroupLocation -> FAILED; real JobMaster 
releaseTaskGroupResource passes CURRENT slot to mocked ResourceManager. No 
actual worker rejoin, retry delivery, redeployment or double assignment 
executed.");
    }
    public static void main(String[]args)throws 
Exception{completion(false);completion(true);probe(true);probe(false);probe(new 
ExecutionException(new RuntimeException("injected transient probe 
failure")));staleEvent();System.out.println("ALL LIFECYCLE ASSERTIONS PASSED");}
   }
   
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to