Xiao-zhen-Liu opened a new pull request, #8731: URL: https://github.com/apache/texera/pull/8731
### What changes were proposed in this PR? This PR is part of cache reuse (#5881): a run reads the saved results of an earlier run instead of computing them again. It adds the step that decides, before a run is scheduled, which operators the run can skip because saved results of an earlier run stand in for their outputs, and which links read a saved result. It is the first step of #5884. Nothing calls it yet (the PR that schedules a run from its skeleton does), and nothing provides saved results (a later PR adds the lookup), so every run on main behaves exactly as before. > [!IMPORTANT] > **Assumption: full reuse.** The planner always reuses a usable saved result. Every retained operator that reads a port with a usable saved result reads that saved result, and an operator is skipped whenever usable saved results cover every port the run needs from it. There is no cost-based choice between reading a saved result and computing it again, even where computing would be cheaper; that choice (cost-based reuse planning) is future work. - `WorkflowContext.matchedResults` holds the saved results found for a run's output ports, as `CachedResult` (the port's storage location and row count). Only the server sets it: it is not on `WorkflowSettings`, which the client sends with the run request, and no request type can carry it. - `SkeletonGenerator.generate` returns a `RunSkeleton` that marks what the run does with each operator and link. An output port is needed if the run must store it (the user views its operator's results, or the operator has no outgoing links), if it has no link, or if it feeds an operator that runs. An operator is retained, meaning it runs in this execution, if it has no output ports or if one of its needed ports has no usable saved result. Every other operator is skipped, and so is every link into it. - A port with a usable saved result is read from storage by every retained operator it feeds, even when its own operator runs. If the run must store that port, it still stores and shows its own result. - The skeleton holds only these marks, the saved results the run reads, and the unusable ones with the reason. It holds no plan. - A saved result is not used when its port is not in the plan or its location is not that port's in this workflow and warehouse. A plan with a loop reuses nothing, since a loop runs its regions again. ### Any related issues, documentation, discussions? Part of #5881 (cache reuse). Advances #5884. Design discussion: #5880. ### How was this PR tested? New unit tests. `SkeletonGeneratorSpec` covers the rules above case by case: a viewed port, a join, operators with several outputs, unconnected ports, an operator with no output ports, a saved port read while its operator runs, unusable results and loops. It also compares the marks of 400 random plans with a slow reference computation of the same rules. `WorkflowContextSpec` checks that `matchedResults` is empty by default. Both specs and `scalafmtCheckAll` pass. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude (Claude Code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
