JyunSheng Kao created SPARK-61343:
-------------------------------------
Summary: Remove type: [arg-type] in pyspark.sql.connect.session
Key: SPARK-61343
URL: https://issues.apache.org/jira/browse/SPARK-61343
Project: Spark
Issue Type: Improvement
Components: Connect, PySpark
Affects Versions: 4.4.0
Reporter: JyunSheng Kao
In {{def createDataFrame(...)}} ({*}python/pyspark/sql/connect/session.py{*}),
{{configs}} comes from {{{}get_config_dict{}}}.
The comment (line 810) clarifies why values are never None or
{{SQL_CONF_NOT_FOUND}} is raised in line 574.
{code:java}
line 573 # Get all related configs in a batch
line 574 configs = self._client.get_config_dict(
line 810 # get_config_dict throws [SQL_CONF_NOT_FOUND] if the key is not found.
line 811 cache_threshold = int(
{code}
So, it's safe to remove {{# type: ignore[arg-type]}} for these config load
* spark.sql.session.localRelationCacheThreshold
* spark.sql.session.localRelationSizeLimit
* spark.sql.session.localRelationChunkSizeRows
* spark.sql.session.localRelationChunkSizeBytes
* spark.sql.session.localRelationBatchOfChunksSizeBytes
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]