DanielLeens commented on issue #13316: URL: https://github.com/apache/seatunnel/issues/13316#issuecomment-6983695881
I reproduced the unresolved-completion-future chain on `89f91a89af15f33d97a942d8b65f4fc1b0c80bf5` (Java 11.0.27), using actual `JobMaster.initStateFuture()` and `cleanJob()`. Steps: 1. Give the PhysicalPlan an externally completable result future and status `FINISHED`. 2. Make the checkpoint-manager dependency throw from `clearCheckpointIfNeed(FINISHED)`. 3. Register the real JobMaster callback with `initStateFuture()`, then complete the PhysicalPlan result. 4. The actual `cleanJob()` throws before completing `jobMasterCompleteFuture`. That future is not done, and a thread calling `join()` is still blocked after a bounded 200 ms observation. 5. Control: when cleanup is stubbed as a successful no-op, the same callback completes the JobMaster future. The harness explicitly resolves the stuck future afterward only to clean up its own join thread. This is a method-level injected cleanup-failure reproduction, not an observed coordinator/client hang in a live cluster. The successful control is a no-op cleanup stub, not a full successful production cleanup run. The executable probe below also covers #13317 and #13320. It uses the `StateRepros.java` fixture helpers already [published with #13314](https://github.com/apache/seatunnel/issues/13314#issuecomment-6983602245); save that file alongside `LifecycleRepros.java`. ### How I ran the focused probe Baseline: `89f91a89af15f33d97a942d8b65f4fc1b0c80bf5`, macOS ARM64, JDK 11.0.27. The production source is unmodified. These are **in-process fault-injection probes**, not end-to-end cluster/failover tests; passing the probe means its asserted failure signature was observed. Save the Java source below outside the checkout. From a clean checkout of that commit: ```bash export JAVA_HOME=/path/to/jdk-11 export PATH="$JAVA_HOME/bin:$PATH" mvn -B -pl seatunnel-engine/seatunnel-engine-server -am \ -DskipTests -DskipUT=true -DskipIT=true -Dskip.ui=true \ -Dskip.spotless=true -Dmaven.gitcommitid.skip=true \ -Dmaven.javadoc.skip=true -Drat.skip=true package mvn -X -Dstyle.color=never -pl seatunnel-engine/seatunnel-engine-server -am \ -DskipTests -DskipUT=true -DskipIT=true -Dskip.ui=true \ -Dskip.spotless=true -Dmaven.gitcommitid.skip=true \ -Dmaven.javadoc.skip=true -Drat.skip=true test-compile > classpath.log 2>&1 CP=$(sed -n 's/^.*(f) testPath = \[\(.*\)\]$/\1/p' classpath.log | tail -1 | sed 's/, /:/g') # The last reactor module is engine-server; CP uses these freshly compiled classes. ``` This builds the required reactor modules, not the full project's test suite. No Docker or live database is needed. I also compared the tested revision with `dev` at `94adef596685ee37e506c96a52baa6ae13de5da3`; the production classes exercised here are unchanged between them. That comparison is not a separate runtime test of the newer commit. ```bash mkdir -p probe-classes javac -proc:none -cp "$CP" -d probe-classes StateRepros.java LifecycleRepros.java java -cp "probe-classes:$CP" org.apache.seatunnel.engine.server.dag.physical.LifecycleRepros ``` <details> <summary>Executable Java probe: LifecycleRepros.java</summary> ```java /* Licensed to the Apache Software Foundation (ASF) under one or more * contributor license agreements. See the NOTICE file distributed with * this work for additional information regarding copyright ownership. * The ASF licenses this file to You under the Apache License, Version 2.0 * (the "License"); you may not use this file except in compliance with * the License. You may obtain a copy at http://www.apache.org/licenses/LICENSE-2.0 * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND. See the License for the * specific language governing permissions and limitations under the License. */ package org.apache.seatunnel.engine.server.dag.physical; import org.apache.seatunnel.engine.common.Constant; import org.apache.seatunnel.engine.common.config.JobConfig; import org.apache.seatunnel.engine.core.job.JobImmutableInformation; import com.hazelcast.flakeidgen.FlakeIdGenerator; import org.apache.seatunnel.engine.common.job.*; import org.apache.seatunnel.engine.common.utils.PassiveCompletableFuture; import org.apache.seatunnel.engine.server.execution.*; import org.apache.seatunnel.engine.server.master.JobMaster; import org.apache.seatunnel.engine.server.checkpoint.CheckpointManager; import org.apache.seatunnel.engine.server.resourcemanager.ResourceManager; import org.apache.seatunnel.engine.server.resourcemanager.resource.SlotProfile; import com.hazelcast.cluster.Address; import com.hazelcast.map.IMap; import com.hazelcast.spi.impl.NodeEngine; import com.hazelcast.spi.impl.operationservice.impl.InvocationFuture; import java.lang.reflect.*; import java.util.*; import java.util.concurrent.*; import static org.mockito.Mockito.*; import static org.mockito.ArgumentMatchers.*; public class LifecycleRepros { static IMap<Object,Object> stateMap(Map<Object,Object> m){IMap<Object,Object> im=mock(IMap.class);when(im.get(any())).thenAnswer(a->m.get(a.getArgument(0)));when(im.put(any(),any())).thenAnswer(a->m.put(a.getArgument(0),a.getArgument(1)));doAnswer(a->{m.put(a.getArgument(0),a.getArgument(1));return null;}).when(im).set(any(),any());return im;} static IMap<Object,Long[]> times(){IMap<Object,Long[]> im=mock(IMap.class);Map<Object,Long[]> m=new HashMap<>();when(im.get(any())).thenAnswer(a->m.get(a.getArgument(0)));when(im.put(any(),any())).thenAnswer(a->m.put(a.getArgument(0),a.getArgument(1)));doAnswer(a->{m.put(a.getArgument(0),a.getArgument(1));return null;}).when(im).set(any(),any());return im;} static JobImmutableInformation info(){JobImmutableInformation i=mock(JobImmutableInformation.class);JobConfig cfg=new JobConfig();cfg.getEnvOptions().put("job.retry.times",3);when(i.getJobConfig()).thenReturn(cfg);when(i.getJobId()).thenReturn(1L);return i;} static PhysicalVertex vertex(IMap<Object,Object> map,NodeEngine node,TaskGroupLocation loc){TaskGroupDefaultImpl group=mock(TaskGroupDefaultImpl.class);when(group.getTaskGroupLocation()).thenReturn(loc);when(group.getTaskGroupName()).thenReturn("repro");return new PhysicalVertex(0,1,group,mock(FlakeIdGenerator.class),1,1,Collections.emptyList(),Collections.emptyList(),info(),System.currentTimeMillis(),node,map,times());} static void set(Object o,String n,Object v)throws Exception {Field f=JobMaster.class.getDeclaredField(n);f.setAccessible(true);f.set(o,v);} static void require(boolean b,String m){if(!b)throw new AssertionError(m);} static void completion(boolean fail)throws Exception { JobMaster jm=mock(JobMaster.class);PhysicalPlan plan=mock(PhysicalPlan.class);CompletableFuture<JobResult> result=new CompletableFuture<>(); when(plan.initStateFuture()).thenReturn(new PassiveCompletableFuture<>(result));when(plan.getJobStatus()).thenReturn(JobStatus.FINISHED);set(jm,"physicalPlan",plan); doCallRealMethod().when(jm).initStateFuture();doCallRealMethod().when(jm).getJobMasterCompleteFuture(); if(fail){CheckpointManager checkpoint=mock(CheckpointManager.class);doThrow(new RuntimeException("injected checkpoint delete failure")).when(checkpoint).clearCheckpointIfNeed(JobStatus.FINISHED);set(jm,"checkpointManager",checkpoint);doCallRealMethod().when(jm).cleanJob();} jm.initStateFuture();result.complete(new JobResult(JobStatus.FINISHED,null)); Field f=JobMaster.class.getDeclaredField("jobMasterCompleteFuture");f.setAccessible(true);CompletableFuture<JobResult> complete=(CompletableFuture<JobResult>)f.get(jm); if(fail){require(!complete.isDone(),"failure must leave future incomplete");Thread join=new Thread(complete::join);join.setDaemon(true);join.start();join.join(200);require(join.isAlive(),"join should remain blocked");System.out.println("13316 REPRODUCED: real initStateFuture + real cleanJob; checkpoint delete throws; completion=false; join thread remains blocked after 200ms. Cleanup completion below is harness-only.");complete.complete(new JobResult(JobStatus.FINISHED,null));join.join(1000);require(!join.isAlive(),"harness cleanup");} else {require(complete.isDone(),"normal cleanup completion");System.out.println("13316 CONTROL: no-op successful cleanup completes future.");} } static void probe(Object response)throws Exception { TaskGroupLocation loc=new TaskGroupLocation(1,1,1);Map<Object,Object> states=new HashMap<>();states.put(loc,ExecutionState.RUNNING); NodeEngine node=mock(NodeEngine.class,RETURNS_DEEP_STUBS);Address addr=new Address("127.0.0.1",5801);com.hazelcast.cluster.impl.MemberImpl member=mock(com.hazelcast.cluster.impl.MemberImpl.class);when(member.getAddress()).thenReturn(addr);when(node.getClusterService().getMembers()).thenReturn(Collections.singleton(member)); IMap owned=mock(IMap.class);SlotProfile slot=mock(SlotProfile.class);when(slot.getWorker()).thenReturn(addr);when(owned.get(any())).thenReturn(Collections.singletonMap(loc,slot));when(node.getHazelcastInstance().getMap(Constant.IMAP_OWNED_SLOT_PROFILES)).thenReturn(owned); InvocationFuture<Object> invocation=mock(InvocationFuture.class);if(response instanceof Exception)when(invocation.get()).thenThrow((Exception)response);else when(invocation.get()).thenReturn(response); when(node.getOperationService().createInvocationBuilder(eq(Constant.SEATUNNEL_SERVICE_NAME),any(org.apache.seatunnel.engine.server.task.operation.CheckTaskGroupIsExecutingOperation.class),eq(addr)).invoke()).thenReturn(invocation); PhysicalVertex v=vertex(stateMap(states),node,loc);v.initStateFuture();ExecutionState expected=Boolean.TRUE.equals(response)?ExecutionState.RUNNING:ExecutionState.FAILING; require(v.getExecutionState()==expected,"probe transition");verify(invocation,times(1)).get();System.out.println("13320 "+(response instanceof Exception?"REPRODUCED":"CONTROL")+": probe="+response+" -> vertex="+v.getExecutionState()+", IMap="+states.get(loc)+", one invocation. Worker membership and allocated slot present; actual remote task is mocked."); } static void staleEvent()throws Exception { TaskGroupLocation loc=new TaskGroupLocation(1,1,1);Map<Object,Object> states=new HashMap<>();states.put(loc,ExecutionState.FAILED);PhysicalVertex v=vertex(stateMap(states),mock(NodeEngine.class),loc); TaskExecutionState old=new TaskExecutionState(loc,ExecutionState.FAILED,"old attempt terminal event");v.updateStateByExecutionService(old);require(v.getExecutionState()==ExecutionState.FAILED,"terminal guard control");v.reset();v.updateTaskState(ExecutionState.RUNNING);require(v.getExecutionState()==ExecutionState.RUNNING,"new attempt simulated running"); JobMaster jm=mock(JobMaster.class);PhysicalPlan plan=mock(PhysicalPlan.class);SubPlan sub=mock(SubPlan.class);when(sub.getPipelineLocation()).thenReturn(loc.getPipelineLocation());when(sub.getCoordinatorVertexList()).thenReturn(Collections.emptyList());when(sub.getPhysicalVertexList()).thenReturn(Collections.singletonList(v));when(plan.getPipelineList()).thenReturn(Collections.singletonList(sub));set(jm,"physicalPlan",plan); SlotProfile currentSlot=mock(SlotProfile.class);IMap owned=mock(IMap.class);when(owned.get(loc.getPipelineLocation())).thenReturn(Collections.singletonMap(loc,currentSlot));set(jm,"ownedSlotProfilesIMap",owned);ResourceManager rm=mock(ResourceManager.class);when(rm.releaseResources(anyLong(),anyList())).thenReturn(org.apache.seatunnel.engine.common.utils.concurrent.CompletableFuture.completedFuture(null));set(jm,"resourceManager",rm);set(jm,"jobImmutableInformation",info());set(jm,"releasedSlotWhenTaskGroupFinished",new ConcurrentHashMap<>()); doCallRealMethod().when(jm).updateTaskExecutionState(any());doCallRealMethod().when(jm).releaseTaskGroupResource(any(),any());jm.updateTaskExecutionState(old); require(v.getExecutionState()==ExecutionState.FAILED,"old event changed new attempt");verify(rm).releaseResources(eq(1L),eq(Collections.singletonList(currentSlot)));System.out.println("13317 REPRODUCED: reset FAILED->CREATED, advance to RUNNING, inject prior FAILED report with same TaskGroupLocation -> FAILED; real JobMaster releaseTaskGroupResource passes CURRENT slot to mocked ResourceManager. No actual worker rejoin, retry delivery, redeployment or double assignment executed."); } public static void main(String[]args)throws Exception{completion(false);completion(true);probe(true);probe(false);probe(new ExecutionException(new RuntimeException("injected transient probe failure")));staleEvent();System.out.println("ALL LIFECYCLE ASSERTIONS PASSED");} } ``` </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
