Xiao-zhen-Liu opened a new pull request, #8732: URL: https://github.com/apache/texera/pull/8732
### What changes were proposed in this PR? This PR is part of cache reuse (#5881), where a run reads the saved results of an earlier run instead of computing them again. It lets the caller of `CostBasedScheduleGenerator` say, before scheduling, that some input ports read saved results, and where those results are stored. When part of a run is skipped, the operators that still run read the skipped operators' outputs this way. With no such input, which is every caller on main, the schedule is exactly as before. - `CacheReadInputs` maps input ports to the storage locations they read (`readerUris`). It is a new last parameter of `CostBasedScheduleGenerator` and `ScheduleGenerator`, empty by default. - In its second pass, the generator adds these locations to the ones an input port gets from materialized links. A port can get both: for example a Union with one input from a saved result and one from an operator in an earlier region. When the other input is a pipelined link, its rows arrive over the link, and only the saved result is read from storage. - `DefaultResourceAllocator` is told which operators read saved results. In the plan it is given, such an operator can have no upstream link, so it would count as a source and use its own input requirement (say, a hash partition) as its output partitioning; the next link would then get a shuffle it never gets in the whole plan. It is partitioned like an operator behind a materialized link instead. - The plan given to the generator for a partly skipped run leaves out the links that read saved results, but operators keep their own link lists, which still name those links. So `Region.getSourceOperators` and the reconfiguration scope in `FriesReconfigurationAlgorithm` now use only the region's own links. On main every link between two operators of a region is one of its links, so their answers do not change. ### Any related issues, documentation, discussions? Part of #5881 (cache reuse). Advances #5884. Design discussion: #5880. ### How was this PR tested? New unit tests. `CostBasedScheduleGeneratorSpec` checks that saved-result locations reach the input port configs (alone, next to a materialized link, next to a pipelined link), that new output locations carry the context's warehouse, that the output of an operator reading a saved result is partitioned as behind a materialized link, and that reader locations keyed by an output port are refused. `ResourcePoliciesSpec` checks the same partitioning in the allocator alone. `RegionSpec` and `FriesReconfigurationAlgorithmSpec` check that a link only the operators list is not counted. All specs of the scheduling, coordinator and common packages and `scalafmtCheckAll` pass. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude (Claude Code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
