Re: DataSourceV2 write input requirements

Yea it is for read-side only. I think for the write-side, implementations can provide some options to allow users to set partitioning/ordering, or the data source has a natural partitioning/ordering which doesn't require any interface. On Mon, Mar 26, 2018 at 7:59 PM, Patrick Woody

Re: DataSourceV2 write input requirements

Hey Ryan, Ted, Wenchen Thanks for the quick replies. @Ryan - the sorting portion makes sense, but I think we'd have to ensure something similar to requiredChildDistribution in SparkPlan where we have the number of partitions as well if we'd want to further report to SupportsReportPartitioning,

Re: DataSourceV2 write input requirements

Hmm. Ryan seems to be right. Looking at sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsReportPartitioning.java : import org.apache.spark.sql.sources.v2.reader.partitioning.Partitioning; ... Partitioning outputPartitioning(); On Mon, Mar 26, 2018 at 6:58 PM, Wenchen Fan

Re: DataSourceV2 write input requirements

Wenchen, I thought SupportsReportPartitioning was for the read side. It works with the write side as well? On Mon, Mar 26, 2018 at 6:58 PM, Wenchen Fan wrote: > Actually clustering is already supported, please take a look at > SupportsReportPartitioning > > Ordering is not

Re: DataSourceV2 write input requirements

Actually clustering is already supported, please take a look at SupportsReportPartitioning Ordering is not proposed yet, might be similar to what Ryan proposed. On Mon, Mar 26, 2018 at 6:11 PM, Ted Yu wrote: > Interesting. > > Should requiredClustering return a Set of

Re: DataSourceV2 write input requirements

Interesting. Should requiredClustering return a Set of Expression's ? This way, we can determine the order of Expression's by looking at what requiredOrdering() returns. On Mon, Mar 26, 2018 at 5:45 PM, Ryan Blue wrote: > Hi Pat, > > Thanks for starting the

Re: DataSourceV2 write input requirements

Hi Pat, Thanks for starting the discussion on this, we’re really interested in it as well. I don’t think there is a proposed API yet, but I was thinking something like this: interface RequiresClustering { List requiredClustering(); } interface RequiresSort { List requiredOrdering(); } The

subscribe

DataSourceV2 write input requirements

Hey all, I saw in some of the discussions around DataSourceV2 writes that we might have the data source inform Spark of requirements for the input data's ordering and partitioning. Has there been a proposed API for that yet? Even one level up it would be helpful to understand how I should be

Re: Re: the issue about the + in column,can we support the string please?

Hi, Using concat is one of the way. But the + is more intuitive and easy to understand. 1427357...@qq.com From: Shmuel Blitz Date: 2018-03-26 15:31 To: 1427357...@qq.com CC: spark?users; dev Subject: Re: the issue about the + in column,can we support the string please? Hi, you can get the