Yes, sorry I meant DAG. I fixed it in my message but not the subject. The
terminology of leaf wasn't helpful I know so hopefully my visual example
was enough. Anyway, I noticed what you said in a local-mode test. I can try
that in a cluster, too. Thank you!
On Thu, Sep 18, 2014 at 10:28 PM,
Is it possible to express a diamond DAG and have the leaf dependency
evaluate only once? So say data flows left to right (and the dependencies
are oriented right to left):
[image: Inline image 1]
Is it possible to run d.collect() and have a evaluate its iterator only
once?
Caveat: all arrows are shuffle dependencies.
On Thu, Sep 18, 2014 at 8:55 PM, Victor Tso-Guillen v...@paxata.com wrote:
Is it possible to express a diamond DAG and have the leaf dependency
evaluate only once? So say data flows left to right (and the dependencies
are oriented right to left):
Hi,
On Thu, Sep 18, 2014 at 8:55 PM, Victor Tso-Guillen v...@paxata.com wrote:
Is it possible to express a diamond DAG and have the leaf dependency
evaluate only once?
Well, strictly speaking your graph is not a tree, and also the meaning of
leaf is not totally clear, I'd say.
So say data
