Re: Bucketing and catalyst

2019-05-02 Thread Ryan Blue
Andrew, Here's an umbrella issue that is a good starting point for looking at the project to add Hive bucketing support: https://issues.apache.org/jira/browse/SPARK-19256 rb On Thu, May 2, 2019 at 11:40 AM Long, Andrew wrote: > Hey Friends, > > > > How aware of bucketing is Catalyst? I’ve been

Bucketing and catalyst

2019-05-02 Thread Long, Andrew
Hey Friends, How aware of bucketing is Catalyst? I’ve been trying to piece together how Catalyst knows that it can remove a sort and shuffle given that both tables are bucketed and sorted the same way. Is there any classes in particular I should look at? Cheers Andrew