[
https://issues.apache.org/jira/browse/SPARK-61343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JyunSheng Kao resolved SPARK-61343.
-----------------------------------
Resolution: Won't Fix
> Remove type: [arg-type] in pyspark.sql.connect.session
> ------------------------------------------------------
>
> Key: SPARK-61343
> URL: https://issues.apache.org/jira/browse/SPARK-61343
> Project: Spark
> Issue Type: Improvement
> Components: Connect, PySpark
> Affects Versions: 4.4.0
> Reporter: JyunSheng Kao
> Priority: Minor
> Labels: pull-request-available
>
> In {{def createDataFrame(...)}}
> ({*}python/pyspark/sql/connect/session.py{*}), {{configs}} comes from
> {{{}get_config_dict{}}}.
> The comment (line 810) clarifies why values are never None or
> {{SQL_CONF_NOT_FOUND}} is raised in line 574.
> {code:java}
> line 573 # Get all related configs in a batch
> line 574 configs = self._client.get_config_dict(
> line 810 # get_config_dict throws [SQL_CONF_NOT_FOUND] if the key is not
> found.
> line 811 cache_threshold = int(
> {code}
> So, it's safe to remove {{# type: ignore[arg-type]}} for these config load
> * spark.sql.session.localRelationCacheThreshold
> * spark.sql.session.localRelationSizeLimit
> * spark.sql.session.localRelationChunkSizeRows
> * spark.sql.session.localRelationChunkSizeBytes
> * spark.sql.session.localRelationBatchOfChunksSizeBytes
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]