[
https://issues.apache.org/jira/browse/SPARK-22387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan updated SPARK-22387:
--------------------------------
Description:
This is an open discussion. The general idea is we should allow users to set
some common configs in session conf so that they don't need to type them again
and again for each data source operations.
Proposal 1:
propagate every session config which starts with {{spark.datasource.config.}}
to data source options. The downside is, users may only want to set some common
configs for a specific data source.
Proposal 2:
propagate session config which starts with
{{spark.datasource.config.myDataSource.}} only to {{myDataSource}} operations.
One downside is, some data source may not have a short name and makes the
config key pretty long, e.g.
{{spark.datasource.config.com.company.foo.bar.key1}}.
Proposal 3:
Introduce a trait `WithSessionConfig` which defines session config key prefix.
Then we can pick session configs with this key-prefix and propagate it to this
particular data source.
One another thing also worth to think: sometimes it's really awful if users
have a type in the config key and spent
was:
This is an open discussion. The general idea is we should allow users to set
some common configs in session conf so that they don't need to type them again
and again for each data source operations.
Proposal 1:
propagate every session config which starts with {{spark.datasource.config.}}
to data source options. The downside is, users may only want to set some common
configs for a specific data source.
Proposal 2:
propagate session config which starts with
{{spark.datasource.config.myDataSource.}} only to {{myDataSource}} operations.
One downside is, some data source may not have a short name and makes the
config key pretty long, e.g.
{{spark.datasource.config.com.company.foo.bar.key1}}.
Proposal 3:
Introduce a trait `WithSessionConfig` which defines session config key prefix.
Then we can pick session configs with this key-prefix and propagate it to this
particular data sourcde.
> propagate session configs to data source read/write options
> -----------------------------------------------------------
>
> Key: SPARK-22387
> URL: https://issues.apache.org/jira/browse/SPARK-22387
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Wenchen Fan
>
> This is an open discussion. The general idea is we should allow users to set
> some common configs in session conf so that they don't need to type them
> again and again for each data source operations.
> Proposal 1:
> propagate every session config which starts with {{spark.datasource.config.}}
> to data source options. The downside is, users may only want to set some
> common configs for a specific data source.
> Proposal 2:
> propagate session config which starts with
> {{spark.datasource.config.myDataSource.}} only to {{myDataSource}}
> operations. One downside is, some data source may not have a short name and
> makes the config key pretty long, e.g.
> {{spark.datasource.config.com.company.foo.bar.key1}}.
> Proposal 3:
> Introduce a trait `WithSessionConfig` which defines session config key
> prefix. Then we can pick session configs with this key-prefix and propagate
> it to this particular data source.
> One another thing also worth to think: sometimes it's really awful if users
> have a type in the config key and spent
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
