Custom Partitioning in Catalyst

2017-06-16 Thread Russell Spitzer
I've been trying to work with making Catalyst Cassandra partitioning aware. There seem to be two major blocks on this. The first is that DataSourceScanExec is unable to learn what the underlying partitioning should be from the BaseRelation it comes from. I'm currently able to get around this by us

Re: Custom Partitioning in Catalyst

2017-06-16 Thread Reynold Xin
Perhaps we should extend the data source API to support that. On Fri, Jun 16, 2017 at 11:37 AM, Russell Spitzer wrote: > I've been trying to work with making Catalyst Cassandra partitioning > aware. There seem to be two major blocks on this. > > The first is that DataSourceScanExec is unable to

Re: Custom Partitioning in Catalyst

2017-06-16 Thread Russell Spitzer
I considered adding this to DataSource APIV2 ticket but I didn't want to be first :P Do you think there will be any issues with opening up the partitioning as well? On Fri, Jun 16, 2017 at 11:58 AM Reynold Xin wrote: > Perhaps we should extend the data source API to support that. > > > On Fri, J

Re: Custom Partitioning in Catalyst

2017-06-16 Thread Reynold Xin
Seems like a great idea to do? On Fri, Jun 16, 2017 at 12:03 PM, Russell Spitzer wrote: > I considered adding this to DataSource APIV2 ticket but I didn't want to > be first :P Do you think there will be any issues with opening up the > partitioning as well? > > On Fri, Jun 16, 2017 at 11:58 AM