[jira] [Created] (SPARK-39630) Allow all Reader or Writer settings to be provided as options

[Nicholas Chammas (Jira)]

Title: Message Title

Nicholas Chammas created an issue   Spark / SPARK-39630 Allow all Reader or Writer settings to be provided as options Issue Type: Improvement Affects Versions: 3.3.0 Assignee: Unassigned Components: SQL Created: 28/Jun/22 21:03 Priority: Minor Reporter: Nicholas Chammas Almost all Reader or Writer settings can be provided via individual calls to `.option()` or by providing a map to `.options()`. There are notable exceptions, though, like: read/write format write mode write partitionBy, bucketBy, and sortBy These settings must be provided via dedicated method calls. Why not make it so that all settings can be provided as options? Is there a design reason not to do this? Any given DataFrameReader or DataFrameWriter (along with the streaming equivalents) should be able to "export" all of its settings as a map of options, and then in turn be reconstituted entirely from that map of options. reader1 = spark.read.option("format", "parquet").option("path", "/data") options = reader.getOptions() reader2 = spark.read.options(options) # reader1 and reader2 are configured identically data1 = reader1.load() data2 = reader2.load() data1.collect() == data2.collect() Some of these special directives like `bucketBy` will need to be broken up to fit the structure of an option. # instead of this data.write.bucketBy(42, "name") # someone would be able to do this data.write.option("bucketByCount", "42").option("bucketByColumn", "name") Complex read or write options aren't new to Spark. We already use them to configure detailed JDBC connections, for example. So why would someone care about this? Mainly, because they want to capture read or write settings declaratively somewhere, like in a YAML file, and use them to drive a Spark job's behavior. Right now we are 80% of the way there. Unless there is a design obstacle in the way, it seems good to go the last 20% and just make it standard for every and any read or write directive to be supported via the option interface. Add Comment   This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)