[ 
https://issues.apache.org/jira/browse/SPARK-61646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61646:
-----------------------------------
    Labels: pull-request-available  (was: )

> Support write distribution and ordering in the Python Data Source API
> ---------------------------------------------------------------------
>
>                 Key: SPARK-61646
>                 URL: https://issues.apache.org/jira/browse/SPARK-61646
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>    Affects Versions: 4.4.0
>            Reporter: Ganesha S
>            Priority: Major
>              Labels: pull-request-available
>
> Python data source writers have no way to tell Spark how their input rows 
> should be grouped or sorted. Scala/Java (DSv2) sinks can already do this 
> through RequiresDistributionAndOrdering, and Spark adds the needed shuffle 
> and sort before the write.
> Without this, a Python sink that writes partitioned or sorted output gets 
> rows in whatever layout the query produced. The result is many small files or 
> requests, or the sink has to ask users to call repartition() / 
> sortWithinPartitions() themselves.
> Proposal: add optional methods to DataSourceWriter so a writer can declare 
> its requirements:
>  
> {code:java}
> class MyWriter(DataSourceWriter):
>     def requiredDistribution(self):
>         return ClusteredDistribution(["region"])
>     def requiredOrdering(self):
>         return [SortOrder("event_time")]{code}
>  
> Spark asks the writer once while planning the query and adds the repartition 
> and sort automatically, using the same planner path as DSv2 sinks.
> - Optional methods: requiredDistribution, requiredOrdering, 
> requiredNumPartitions, distributionStrictlyRequired, 
> advisoryPartitionSizeInBytes.
> - Every method defaults to "no requirement", so existing writers are 
> unaffected.
> - Gated by an internal config, 
> spark.sql.python.writeDistributionAndOrdering.enabled, off by default.
> Related: SPARK-56635 (read side: letting a Python source report its existing 
> partitioning).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to