Yes, DAGScheduler is dealing with it at a stage level - and so individual
RDD’s DeterministicLevel would be handled in order to determine the
stage’s level.
Regards,
Mridul
On Fri, Nov 3, 2023 at 9:45 AM Keyong Zhou wrote:
> I checked RDD#getOutputDeterministicLevel and find that if an RDD's
I checked RDD#getOutputDeterministicLevel and find that if an RDD's
upstream is INDETERMINATE,
then it's also INDETERMINATE.
Thanks,
Keyong Zhou
Keyong Zhou 于2023年11月3日周五 19:57写道：
> Hi Mridul,
>
> I still have a question. DAGScheduler#submitMissingTasks will
> only
Hi Mridul,
I still have a question. DAGScheduler#submitMissingTasks will
only unregisterAllMapAndMergeOutput
if the current ShuffleMapStage is Indeterminate. What if the current stage
is determinate, but its
upstream stage is Indeterminate, and its upstream stage is rerun?
Thanks,
Keyong Zhou
