a little bit of additional information is that for rewriting rules stage optimizer does 3 passes over semantic tree, each pass matching a tree fragment using Scala case class matching and rewriting. This allows to match and rewrite pretty elaborate tree structure fragments, although at the moment i don't think we dig farther than immediate children, and perhaps some their known attributes, in most cases.
More detailed description that that i think is only in reading the source. On Wed, Jun 18, 2014 at 6:19 PM, Dmitriy Lyubimov <[email protected]> wrote: > E.g. i know for sure A %.% B is legal where A is string-keyed and b is > int-keyed. > > This is kind of not the point. the point is that you can easily modify > rewriting rules and operators to cover misses. (there shouldn't be many, > since we've already written quite a bit of expressions out there). > > > On Wed, Jun 18, 2014 at 6:15 PM, Dmitriy Lyubimov <[email protected]> > wrote: > >> I am not sure. There are more rewriting rules than i can remember, and i >> did not write an algorithm ( i think) that would involve this combination. >> I guess the best thing is to try in a shell or a unit test. if it falls >> thru, perhaps a new plan element needs to be added (although I am not very >> sure there isn't already). I know that there are join-based multiplicative >> operators there. >> >> >> On Wed, Jun 18, 2014 at 6:11 PM, Ted Dunning <[email protected]> >> wrote: >> >>> On Wed, Jun 18, 2014 at 6:07 PM, Dmitriy Lyubimov <[email protected]> >>> wrote: >>> >>> > in simple terms, if non-integer row keying is used anywhere, it tries >>> to >>> > rewrite pipelines so that product orientations never require non-int >>> keys >>> > to denote columns. In case pipeline makes it impossible, optimizer will >>> > refuse to produce a plan. >>> > >>> > e.g. suppose A is distributed string-keyed. >>> > >>> > (A.t %.% A) collect // ok >>> > >>> >>> What happens with the important case of B.t %.% A where both A and B are >>> string keyed? >>> >> >> >
