[
https://issues.apache.org/jira/browse/SPARK-22387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan updated SPARK-22387:
--------------------------------
Description:
This is an open discussion. The general idea is we should allow users to set
some common configs in session conf so that they don't need to type them again
and again for each data source operations.
Proposal 1:
propagate every session config which starts with {{spark.datasource.config.}}
to data source options. The downside is, users may only want to set some common
configs for a specific data source.
Proposal 2:
propagate session config which starts with
{{spark.datasource.config.myDataSource.}} only to {{myDataSource}} operations.
One downside is, some data source may not have a short name and makes the
config key pretty long, e.g.
{{spark.datasource.config.com.company.foo.bar.key1}}.
Proposal 3:
Introduce a trait `WithSessionConfig` which defines session config key prefix.
Then we can pick session configs with this key-prefix and propagate it to this
particular data source.
One another thing also worth to think: sometimes it's really annoying if users
have a typo in the config key and spend a lot of time to figure out why things
don't work as expected. We should allow data source to validate the given
options and throw exception if an option can't be recognized.
was:
This is an open discussion. The general idea is we should allow users to set
some common configs in session conf so that they don't need to type them again
and again for each data source operations.
Proposal 1:
propagate every session config which starts with {{spark.datasource.config.}}
to data source options. The downside is, users may only want to set some common
configs for a specific data source.
Proposal 2:
propagate session config which starts with
{{spark.datasource.config.myDataSource.}} only to {{myDataSource}} operations.
One downside is, some data source may not have a short name and makes the
config key pretty long, e.g.
{{spark.datasource.config.com.company.foo.bar.key1}}.
Proposal 3:
Introduce a trait `WithSessionConfig` which defines session config key prefix.
Then we can pick session configs with this key-prefix and propagate it to this
particular data source.
One another thing also worth to think: sometimes it's really annoying if users
have a type in the config key and spent
> propagate session configs to data source read/write options
> -----------------------------------------------------------
>
> Key: SPARK-22387
> URL: https://issues.apache.org/jira/browse/SPARK-22387
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 2.3.0
> Reporter: Wenchen Fan
>
> This is an open discussion. The general idea is we should allow users to set
> some common configs in session conf so that they don't need to type them
> again and again for each data source operations.
> Proposal 1:
> propagate every session config which starts with {{spark.datasource.config.}}
> to data source options. The downside is, users may only want to set some
> common configs for a specific data source.
> Proposal 2:
> propagate session config which starts with
> {{spark.datasource.config.myDataSource.}} only to {{myDataSource}}
> operations. One downside is, some data source may not have a short name and
> makes the config key pretty long, e.g.
> {{spark.datasource.config.com.company.foo.bar.key1}}.
> Proposal 3:
> Introduce a trait `WithSessionConfig` which defines session config key
> prefix. Then we can pick session configs with this key-prefix and propagate
> it to this particular data source.
> One another thing also worth to think: sometimes it's really annoying if
> users have a typo in the config key and spend a lot of time to figure out why
> things don't work as expected. We should allow data source to validate the
> given options and throw exception if an option can't be recognized.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
