Xiao-zhen-Liu opened a new pull request, #8733: URL: https://github.com/apache/texera/pull/8733
### What changes were proposed in this PR? This PR is part of cache reuse (#5881), where a run reads the saved results of an earlier run instead of computing them again. When a run has saved results it can use, the scheduler now plans only the part of the workflow that runs and puts the skipped operators in skip regions. Nothing sets `WorkflowContext.matchedResults` yet (a later PR adds the lookup), so every run on main takes the old path and behaves exactly as before. This PR sits on #8731 (`feat(amber): decide what a run skips for cache reuse`) and #8732 (`feat(amber): schedule saved-result reads for cache reuse`). Until they merge, its diff also shows their commits; its own change is the last commit. - `WorkflowScheduler.updateSchedule` runs main's code when `matchedResults` is empty. Otherwise it asks `SkeletonGenerator` for the run's skeleton and logs each unusable saved result with the reason. - The schedule generator plans the retained part (`RunSkeleton.retainedPart`): the operators that run, taken from the full plan with `getSubPlan` as the plan's own operator objects, and the links between them except those that read saved results. Each input port such a link feeds gets the link's saved location as a reader URI (`RunSkeleton.cacheReadInputs`), in a fixed order when one port reads several. - The skipped operators become skip regions (`RunSkeleton.skipRegions`, marked by the new `Region.skipped`), one per connected group, all in a new level 0 ahead of the generated levels, with ids above the largest generated id. A skip region's resource config holds each saved result it reads as the port's location, with the saved row count in the new `OutputPortConfig.cachedTupleCount`. They start no workers, so the limit on concurrent regions does not apply. A run that skips every operator gets only level 0. Running a skip region comes in a follow-up PR. - The scheduler's `physicalPlan` keeps the full plan, skipped operators included, so code that looks up an operator still finds it. ### Any related issues, documentation, discussions? Part of #5881 (cache reuse). Advances #5884. Design discussion: #5880. ### How was this PR tested? New unit tests. `WorkflowSchedulerSpec` checks that with no saved results, or only unusable ones, the schedule is main's; that skip regions come first, with higher ids; a run that skips every operator; a hash join whose build input or build operator is saved; that regions hold the plan's own operators; and a Split whose saved port is read while Split runs. `SkeletonGeneratorSpec` gains checks of the retained part, the reader URIs and the skip regions, in its cases and on its 400 random plans. All specs of the scheduling, coordinator and common packages and `scalafmtCheckAll` pass. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude (Claude Code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
