Ganesha S created SPARK-61646:
---------------------------------
Summary: Support write distribution and ordering in the Python
Data Source API
Key: SPARK-61646
URL: https://issues.apache.org/jira/browse/SPARK-61646
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 4.4.0
Reporter: Ganesha S
Python data source writers have no way to tell Spark how their input rows
should be grouped or sorted. Scala/Java (DSv2) sinks can already do this
through RequiresDistributionAndOrdering, and Spark adds the needed shuffle and
sort before the write.
Without this, a Python sink that writes partitioned or sorted output gets rows
in whatever layout the query produced. The result is many small files or
requests, or the sink has to ask users to call repartition() /
sortWithinPartitions() themselves.
Proposal: add optional methods to DataSourceWriter so a writer can declare its
requirements:
{code:java}
class MyWriter(DataSourceWriter):
def requiredDistribution(self):
return ClusteredDistribution(["region"])
def requiredOrdering(self):
return [SortOrder("event_time")]{code}
Spark asks the writer once while planning the query and adds the repartition
and sort automatically, using the same planner path as DSv2 sinks.
- Optional methods: requiredDistribution, requiredOrdering,
requiredNumPartitions, distributionStrictlyRequired,
advisoryPartitionSizeInBytes.
- Every method defaults to "no requirement", so existing writers are unaffected.
- Gated by an internal config,
spark.sql.python.writeDistributionAndOrdering.enabled, off by default.
Related: SPARK-56635 (read side: letting a Python source report its existing
partitioning).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]