aglinxinyuan commented on code in PR #5707:
URL: https://github.com/apache/texera/pull/5707#discussion_r3410909081
##########
common/workflow-core/src/main/scala/org/apache/texera/amber/core/workflow/PhysicalOp.scala:
##########
@@ -198,6 +198,12 @@ case class PhysicalOp(
// schema propagation function
propagateSchema: SchemaPropagationFunc = SchemaPropagationFunc(schemas =>
schemas),
isOneToManyOp: Boolean = false,
+ // Whether to reuse this operator's existing output storage instead of
+ // recreating it when its region re-executes, so output accumulated by
+ // earlier runs (e.g. across loop iterations) survives. Named after the
+ // behavior the scheduler checks, not the operator that sets it, so any
+ // future operator needing the same treatment can reuse it.
+ reusesOutputStorageOnReExecution: Boolean = false,
Review Comment:
You're right, it is doable now — done in `6f85e6b`.
`OutputPortReuseFlagSpec` iterates every registered operator (via
`OperatorMetadataGenerator.operatorTypeMap`) and asserts none of their output
ports set `reusesOutputStorage`. That currently holds for all operators (none
need it). When the loop operators land I'll update the guard to allow Loop
End's output port (and only it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
