Ganesha S created SPARK-61646:
---------------------------------

             Summary: Support write distribution and ordering in the Python 
Data Source API
                 Key: SPARK-61646
                 URL: https://issues.apache.org/jira/browse/SPARK-61646
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 4.4.0
            Reporter: Ganesha S


Python data source writers have no way to tell Spark how their input rows 
should be grouped or sorted. Scala/Java (DSv2) sinks can already do this 
through RequiresDistributionAndOrdering, and Spark adds the needed shuffle and 
sort before the write.

Without this, a Python sink that writes partitioned or sorted output gets rows 
in whatever layout the query produced. The result is many small files or 
requests, or the sink has to ask users to call repartition() / 
sortWithinPartitions() themselves.

Proposal: add optional methods to DataSourceWriter so a writer can declare its 
requirements:

 
{code:java}
class MyWriter(DataSourceWriter):
    def requiredDistribution(self):
        return ClusteredDistribution(["region"])
    def requiredOrdering(self):
        return [SortOrder("event_time")]{code}
 

Spark asks the writer once while planning the query and adds the repartition 
and sort automatically, using the same planner path as DSv2 sinks.

- Optional methods: requiredDistribution, requiredOrdering, 
requiredNumPartitions, distributionStrictlyRequired, 
advisoryPartitionSizeInBytes.
- Every method defaults to "no requirement", so existing writers are unaffected.
- Gated by an internal config, 
spark.sql.python.writeDistributionAndOrdering.enabled, off by default.

Related: SPARK-56635 (read side: letting a Python source report its existing 
partitioning).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to