Allison Wang created SPARK-46568:
------------------------------------
Summary: Python data source options should be a case insensitive
dictionary
Key: SPARK-46568
URL: https://issues.apache.org/jira/browse/SPARK-46568
Project: Spark
Issue Type: Sub-task
Components: PySpark
Affects Versions: 4.0.0
Reporter: Allison Wang
Data source options are stored as a `CaseInsensitiveStringMap` in Scala,
however, its behavior is inconsistent in Python:
{code:java}
class MyDataSource(DataSource):
def __init__(self, options):
self.api_key = options.get("API_KEY") # <- This is None
spark.read.format(..).option("API_KEY", my_key).load(...){code}
Currently, options will not have this "API_KEY" as everything is converted to
lowercase on the Scala side. This can be confusing to users.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
