[jira] [Updated] (SPARK-42663) Fix `default_session ` to work properly
[ https://issues.apache.org/jira/browse/SPARK-42663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-42663: Summary: Fix `default_session ` to work properly (was: Fix `SparkSession.conf.get` to work properly) > Fix `default_session ` to work properly > --- > > Key: SPARK-42663 > URL: https://issues.apache.org/jira/browse/SPARK-42663 > Project: Spark > Issue Type: Sub-task > Components: Connect, ps >Affects Versions: 3.5.0 >Reporter: Haejoon Lee >Priority: Major > > Currently, default_session is not working properly in Spark Connect as below > since `SparkSession.conf.get` is nor working as expected: > {code:java} > >>> spark = default_session() > >>> spark.conf.set("default_index_type", "sequence") > >>> spark.conf.get("default_index_type") > 'sequence' > >>> > >>> spark = default_session() > >>> spark.conf.get("default_index_type") > Traceback (most recent call last): > ... > pyspark.errors.exceptions.connect.SparkConnectGrpcException: > (java.util.NoSuchElementException) default_index_type > {code} > It should work as expected in regular PySpark as below: > {code:java} > >>> spark = default_session() > >>> spark.conf.set("default_index_type", "sequence") > >>> spark.conf.get("default_index_type") > 'sequence' > >>> > >>> spark = default_session() > >>> spark.conf.get("default_index_type") > 'sequence'{code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-42663) Fix `default_session` to work properly
[ https://issues.apache.org/jira/browse/SPARK-42663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-42663: Summary: Fix `default_session` to work properly (was: Fix `default_session ` to work properly) > Fix `default_session` to work properly > -- > > Key: SPARK-42663 > URL: https://issues.apache.org/jira/browse/SPARK-42663 > Project: Spark > Issue Type: Sub-task > Components: Connect, ps >Affects Versions: 3.5.0 >Reporter: Haejoon Lee >Priority: Major > > Currently, default_session is not working properly in Spark Connect as below > since `SparkSession.conf.get` is nor working as expected: > {code:java} > >>> spark = default_session() > >>> spark.conf.set("default_index_type", "sequence") > >>> spark.conf.get("default_index_type") > 'sequence' > >>> > >>> spark = default_session() > >>> spark.conf.get("default_index_type") > Traceback (most recent call last): > ... > pyspark.errors.exceptions.connect.SparkConnectGrpcException: > (java.util.NoSuchElementException) default_index_type > {code} > It should work as expected in regular PySpark as below: > {code:java} > >>> spark = default_session() > >>> spark.conf.set("default_index_type", "sequence") > >>> spark.conf.get("default_index_type") > 'sequence' > >>> > >>> spark = default_session() > >>> spark.conf.get("default_index_type") > 'sequence'{code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-42663) Fix `default_session` to work properly
[ https://issues.apache.org/jira/browse/SPARK-42663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-42663: Description: Currently, default_session is not working properly in Spark Connect as below: {code:java} >>> spark = default_session() >>> spark.conf.set("default_index_type", "sequence") >>> spark.conf.get("default_index_type") 'sequence' >>> >>> spark = default_session() >>> spark.conf.get("default_index_type") Traceback (most recent call last): ... pyspark.errors.exceptions.connect.SparkConnectGrpcException: (java.util.NoSuchElementException) default_index_type {code} It should work as expected in regular PySpark as below: {code:java} >>> spark = default_session() >>> spark.conf.set("default_index_type", "sequence") >>> spark.conf.get("default_index_type") 'sequence' >>> >>> spark = default_session() >>> spark.conf.get("default_index_type") 'sequence'{code} was: Currently, default_session is not working properly in Spark Connect as below since `SparkSession.conf.get` is nor working as expected: {code:java} >>> spark = default_session() >>> spark.conf.set("default_index_type", "sequence") >>> spark.conf.get("default_index_type") 'sequence' >>> >>> spark = default_session() >>> spark.conf.get("default_index_type") Traceback (most recent call last): ... pyspark.errors.exceptions.connect.SparkConnectGrpcException: (java.util.NoSuchElementException) default_index_type {code} It should work as expected in regular PySpark as below: {code:java} >>> spark = default_session() >>> spark.conf.set("default_index_type", "sequence") >>> spark.conf.get("default_index_type") 'sequence' >>> >>> spark = default_session() >>> spark.conf.get("default_index_type") 'sequence'{code} > Fix `default_session` to work properly > -- > > Key: SPARK-42663 > URL: https://issues.apache.org/jira/browse/SPARK-42663 > Project: Spark > Issue Type: Sub-task > Components: Connect, ps >Affects Versions: 3.5.0 >Reporter: Haejoon Lee >Priority: Major > > Currently, default_session is not working properly in Spark Connect as below: > {code:java} > >>> spark = default_session() > >>> spark.conf.set("default_index_type", "sequence") > >>> spark.conf.get("default_index_type") > 'sequence' > >>> > >>> spark = default_session() > >>> spark.conf.get("default_index_type") > Traceback (most recent call last): > ... > pyspark.errors.exceptions.connect.SparkConnectGrpcException: > (java.util.NoSuchElementException) default_index_type > {code} > It should work as expected in regular PySpark as below: > {code:java} > >>> spark = default_session() > >>> spark.conf.set("default_index_type", "sequence") > >>> spark.conf.get("default_index_type") > 'sequence' > >>> > >>> spark = default_session() > >>> spark.conf.get("default_index_type") > 'sequence'{code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-42663) Fix default_session to work properly in Spark Connect
[ https://issues.apache.org/jira/browse/SPARK-42663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-42663: Description: Currently, default_session is not working properly in Spark Connect: {code:java} >>> spark = default_session() >>> spark.conf.set("default_index_type", "sequence") >>> spark.conf.get("default_index_type") 'sequence' >>> >>> spark = default_session() >>> spark.conf.get("default_index_type") Traceback (most recent call last): ... pyspark.errors.exceptions.connect.SparkConnectGrpcException: (java.util.NoSuchElementException) default_index_type {code} It should work as expected in regular PySpark as below: {code:java} >>> spark = default_session() >>> spark.conf.set("default_index_type", "sequence") >>> spark.conf.get("default_index_type") 'sequence' >>> >>> spark = default_session() >>> spark.conf.get("default_index_type") 'sequence'{code} was: Currently, default_session is not working properly in Spark Connect: >>> spark = default_session() >>> spark.conf.set("default_index_type", "sequence") >>> spark.conf.get("default_index_type") 'sequence' >>> >>> spark = default_session() >>> spark.conf.get("default_index_type") Traceback (most recent call last): ... pyspark.errors.exceptions.connect.SparkConnectGrpcException: (java.util.NoSuchElementException) default_index_type It should work as expected in regular PySpark as below: >>> spark = default_session() >>> spark.conf.set("default_index_type", "sequence") >>> spark.conf.get("default_index_type") 'sequence' >>> >>> spark = default_session() >>> spark.conf.get("default_index_type") 'sequence' > Fix default_session to work properly in Spark Connect > - > > Key: SPARK-42663 > URL: https://issues.apache.org/jira/browse/SPARK-42663 > Project: Spark > Issue Type: Sub-task > Components: Connect, ps >Affects Versions: 3.5.0 >Reporter: Haejoon Lee >Priority: Major > > Currently, default_session is not working properly in Spark Connect: > > {code:java} > >>> spark = default_session() > >>> spark.conf.set("default_index_type", "sequence") > >>> spark.conf.get("default_index_type") > 'sequence' > >>> > >>> spark = default_session() > >>> spark.conf.get("default_index_type") > Traceback (most recent call last): > ... > pyspark.errors.exceptions.connect.SparkConnectGrpcException: > (java.util.NoSuchElementException) default_index_type > {code} > It should work as expected in regular PySpark as below: > {code:java} > >>> spark = default_session() > >>> spark.conf.set("default_index_type", "sequence") > >>> spark.conf.get("default_index_type") > 'sequence' > >>> > >>> spark = default_session() > >>> spark.conf.get("default_index_type") > 'sequence'{code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-42663) Fix default_session to work properly in Spark Connect
[ https://issues.apache.org/jira/browse/SPARK-42663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haejoon Lee updated SPARK-42663: Description: Currently, default_session is not working properly in Spark Connect: {code:java} >>> spark = default_session() >>> spark.conf.set("default_index_type", "sequence") >>> spark.conf.get("default_index_type") 'sequence' >>> >>> spark = default_session() >>> spark.conf.get("default_index_type") Traceback (most recent call last): ... pyspark.errors.exceptions.connect.SparkConnectGrpcException: (java.util.NoSuchElementException) default_index_type {code} It should work as expected in regular PySpark as below: {code:java} >>> spark = default_session() >>> spark.conf.set("default_index_type", "sequence") >>> spark.conf.get("default_index_type") 'sequence' >>> >>> spark = default_session() >>> spark.conf.get("default_index_type") 'sequence'{code} was: Currently, default_session is not working properly in Spark Connect: {code:java} >>> spark = default_session() >>> spark.conf.set("default_index_type", "sequence") >>> spark.conf.get("default_index_type") 'sequence' >>> >>> spark = default_session() >>> spark.conf.get("default_index_type") Traceback (most recent call last): ... pyspark.errors.exceptions.connect.SparkConnectGrpcException: (java.util.NoSuchElementException) default_index_type {code} It should work as expected in regular PySpark as below: {code:java} >>> spark = default_session() >>> spark.conf.set("default_index_type", "sequence") >>> spark.conf.get("default_index_type") 'sequence' >>> >>> spark = default_session() >>> spark.conf.get("default_index_type") 'sequence'{code} > Fix default_session to work properly in Spark Connect > - > > Key: SPARK-42663 > URL: https://issues.apache.org/jira/browse/SPARK-42663 > Project: Spark > Issue Type: Sub-task > Components: Connect, ps >Affects Versions: 3.5.0 >Reporter: Haejoon Lee >Priority: Major > > Currently, default_session is not working properly in Spark Connect: > {code:java} > >>> spark = default_session() > >>> spark.conf.set("default_index_type", "sequence") > >>> spark.conf.get("default_index_type") > 'sequence' > >>> > >>> spark = default_session() > >>> spark.conf.get("default_index_type") > Traceback (most recent call last): > ... > pyspark.errors.exceptions.connect.SparkConnectGrpcException: > (java.util.NoSuchElementException) default_index_type > {code} > It should work as expected in regular PySpark as below: > {code:java} > >>> spark = default_session() > >>> spark.conf.set("default_index_type", "sequence") > >>> spark.conf.get("default_index_type") > 'sequence' > >>> > >>> spark = default_session() > >>> spark.conf.get("default_index_type") > 'sequence'{code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org