Yes, sorry I meant DAG. I fixed it in my message but not the subject. The
terminology of "leaf" wasn't helpful I know so hopefully my visual example
was enough. Anyway, I noticed what you said in a local-mode test. I can try
that in a cluster, too. Thank you!
On Thu, Sep 18, 2014 at 10:28 PM, Tobi
Hi,
On Thu, Sep 18, 2014 at 8:55 PM, Victor Tso-Guillen wrote:
>
>> Is it possible to express a diamond DAG and have the leaf dependency
>> evaluate only once?
>>
>
Well, strictly speaking your graph is not a "tree", and also the meaning of
"leaf" is not totally clear, I'd say.
> So say data fl
Caveat: all arrows are shuffle dependencies.
On Thu, Sep 18, 2014 at 8:55 PM, Victor Tso-Guillen wrote:
> Is it possible to express a diamond DAG and have the leaf dependency
> evaluate only once? So say data flows left to right (and the dependencies
> are oriented right to left):
>
> [image: In
Is it possible to express a diamond DAG and have the leaf dependency
evaluate only once? So say data flows left to right (and the dependencies
are oriented right to left):
[image: Inline image 1]
Is it possible to run d.collect() and have a evaluate its iterator only
once?
