Re: Transform plan with scope

2018-04-24 Thread Marco Gaido
Hi Joseph, Herman, thanks for your answers. The specific rule I was looking at is FoldablePropagation. If you look at it, what is done is that first a AttributeMap containing all the possible foldable alias is collected, then they are replace in the whole plan (it is a bit more complex than this,

Re: Transform plan with scope

2018-04-24 Thread Joseph Torres
Is there some transformation we'd want to apply to that tree, but can't because we have no concept of scope? It's already possible for a plan rule to traverse each node's subtree if it wants. On Tue, Apr 24, 2018 at 10:18 AM, Marco Gaido wrote: > Hi all, > > working on SPARK-24051 I realized tha

Transform plan with scope

2018-04-24 Thread Marco Gaido
Hi all, working on SPARK-24051 I realized that currently in the Optimizer and in all the places where we are transforming a query plan, we are lacking the context information of what is in scope and what is not. Coming back to the ticket, the bug reported in the ticket is caused mainly by two rea