[jira] [Resolved] (SPARK-42007) Reuse pyspark.sql.tests.test_group test cases
[ https://issues.apache.org/jira/browse/SPARK-42007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42007. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39525 [https://github.com/apache/spark/pull/39525] > Reuse pyspark.sql.tests.test_group test cases > - > > Key: SPARK-42007 > URL: https://issues.apache.org/jira/browse/SPARK-42007 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-42007) Reuse pyspark.sql.tests.test_group test cases
[ https://issues.apache.org/jira/browse/SPARK-42007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-42007: Assignee: Hyukjin Kwon > Reuse pyspark.sql.tests.test_group test cases > - > > Key: SPARK-42007 > URL: https://issues.apache.org/jira/browse/SPARK-42007 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-42025) Optimize logs for removeBlocks and removeShuffleMerge PRC
Wan Kun created SPARK-42025: --- Summary: Optimize logs for removeBlocks and removeShuffleMerge PRC Key: SPARK-42025 URL: https://issues.apache.org/jira/browse/SPARK-42025 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 3.4.0 Reporter: Wan Kun 1. Change the error logs level for RemoveBlocks RPC. 2. Add some error logs for RemoveShuffleMerge RPC. Discuss link: https://github.com/apache/spark/pull/37922#discussion_r1067551485 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-42026) Protobuf serializer for AppSummary and PoolData
Yang Jie created SPARK-42026: Summary: Protobuf serializer for AppSummary and PoolData Key: SPARK-42026 URL: https://issues.apache.org/jira/browse/SPARK-42026 Project: Spark Issue Type: Sub-task Components: Spark Core Affects Versions: 3.4.0 Reporter: Yang Jie -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-42026) Protobuf serializer for AppSummary and PoolData
[ https://issues.apache.org/jira/browse/SPARK-42026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-42026: Assignee: (was: Apache Spark) > Protobuf serializer for AppSummary and PoolData > --- > > Key: SPARK-42026 > URL: https://issues.apache.org/jira/browse/SPARK-42026 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.4.0 >Reporter: Yang Jie >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-42026) Protobuf serializer for AppSummary and PoolData
[ https://issues.apache.org/jira/browse/SPARK-42026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-42026: Assignee: Apache Spark > Protobuf serializer for AppSummary and PoolData > --- > > Key: SPARK-42026 > URL: https://issues.apache.org/jira/browse/SPARK-42026 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.4.0 >Reporter: Yang Jie >Assignee: Apache Spark >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-42026) Protobuf serializer for AppSummary and PoolData
[ https://issues.apache.org/jira/browse/SPARK-42026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17675861#comment-17675861 ] Apache Spark commented on SPARK-42026: -- User 'LuciferYang' has created a pull request for this issue: https://github.com/apache/spark/pull/39530 > Protobuf serializer for AppSummary and PoolData > --- > > Key: SPARK-42026 > URL: https://issues.apache.org/jira/browse/SPARK-42026 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.4.0 >Reporter: Yang Jie >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-42008) Reuse pyspark.sql.tests.test_datasources test cases
[ https://issues.apache.org/jira/browse/SPARK-42008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-42008: Assignee: Hyukjin Kwon > Reuse pyspark.sql.tests.test_datasources test cases > > > Key: SPARK-42008 > URL: https://issues.apache.org/jira/browse/SPARK-42008 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-42008) Reuse pyspark.sql.tests.test_datasources test cases
[ https://issues.apache.org/jira/browse/SPARK-42008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42008. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39526 [https://github.com/apache/spark/pull/39526] > Reuse pyspark.sql.tests.test_datasources test cases > > > Key: SPARK-42008 > URL: https://issues.apache.org/jira/browse/SPARK-42008 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-42003) Reduce duplicate code in ResolveGroupByAll
[ https://issues.apache.org/jira/browse/SPARK-42003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-42003. - Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39523 [https://github.com/apache/spark/pull/39523] > Reduce duplicate code in ResolveGroupByAll > -- > > Key: SPARK-42003 > URL: https://issues.apache.org/jira/browse/SPARK-42003 > Project: Spark > Issue Type: Task > Components: SQL >Affects Versions: 3.4.0 >Reporter: Gengliang Wang >Assignee: Gengliang Wang >Priority: Minor > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-42009) Reuse pyspark.sql.tests.test_serde test cases
[ https://issues.apache.org/jira/browse/SPARK-42009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42009. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39527 [https://github.com/apache/spark/pull/39527] > Reuse pyspark.sql.tests.test_serde test cases > -- > > Key: SPARK-42009 > URL: https://issues.apache.org/jira/browse/SPARK-42009 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-42009) Reuse pyspark.sql.tests.test_serde test cases
[ https://issues.apache.org/jira/browse/SPARK-42009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-42009: Assignee: Hyukjin Kwon > Reuse pyspark.sql.tests.test_serde test cases > -- > > Key: SPARK-42009 > URL: https://issues.apache.org/jira/browse/SPARK-42009 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-41989) PYARROW_IGNORE_TIMEZONE warning can break application logging setup
[ https://issues.apache.org/jira/browse/SPARK-41989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-41989. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39516 [https://github.com/apache/spark/pull/39516] > PYARROW_IGNORE_TIMEZONE warning can break application logging setup > --- > > Key: SPARK-41989 > URL: https://issues.apache.org/jira/browse/SPARK-41989 > Project: Spark > Issue Type: Bug > Components: PySpark >Affects Versions: 3.2.3 > Environment: python 3.9 env with pyspark installed >Reporter: Stefaan Lippens >Assignee: Stefaan Lippens >Priority: Major > Fix For: 3.4.0 > > > in {code}python/pyspark/pandas/__init__.py{code} there is currently a > warning when {{PYARROW_IGNORE_TIMEZONE}} env var is not set > (https://github.com/apache/spark/blob/187c4a9c66758e973633c5c309b551b1d9094e6e/python/pyspark/pandas/__init__.py#L44-L59): > {code:python} > import logging > logging.warning( > "'PYARROW_IGNORE_TIMEZONE' environment variable was not set. It is > required to "... > {code} > The {{logging.warning()}} call will silently do a {{logging.basicConfig()}} > call (at least in python 3.9, which I tried). > (FYI: Something like {{logging.getLogger(...).warning()}} would not do this > silent call) > This has the following very hard to figure out side-effect: > importing `pyspark.pandas` (directly or indirectly somewhere) might break > your logging setup (if PYARROW_IGNORE_TIMEZONE is not set). > Very basic example (assuming PYARROW_IGNORE_TIMEZONE is not set): > {code:python} > import logging > import pyspark.pandas > logging.basicConfig(level=logging.DEBUG) > logger = logging.getLogger("test") > logger.warning("I warn you") > logger.debug("I debug you") > {code} > Will only produce the warning, not the debug line. > By removing the {{import pyspark.pandas}}, the debug line is produced -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-41989) PYARROW_IGNORE_TIMEZONE warning can break application logging setup
[ https://issues.apache.org/jira/browse/SPARK-41989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-41989: Assignee: Stefaan Lippens > PYARROW_IGNORE_TIMEZONE warning can break application logging setup > --- > > Key: SPARK-41989 > URL: https://issues.apache.org/jira/browse/SPARK-41989 > Project: Spark > Issue Type: Bug > Components: PySpark >Affects Versions: 3.2.3 > Environment: python 3.9 env with pyspark installed >Reporter: Stefaan Lippens >Assignee: Stefaan Lippens >Priority: Major > > in {code}python/pyspark/pandas/__init__.py{code} there is currently a > warning when {{PYARROW_IGNORE_TIMEZONE}} env var is not set > (https://github.com/apache/spark/blob/187c4a9c66758e973633c5c309b551b1d9094e6e/python/pyspark/pandas/__init__.py#L44-L59): > {code:python} > import logging > logging.warning( > "'PYARROW_IGNORE_TIMEZONE' environment variable was not set. It is > required to "... > {code} > The {{logging.warning()}} call will silently do a {{logging.basicConfig()}} > call (at least in python 3.9, which I tried). > (FYI: Something like {{logging.getLogger(...).warning()}} would not do this > silent call) > This has the following very hard to figure out side-effect: > importing `pyspark.pandas` (directly or indirectly somewhere) might break > your logging setup (if PYARROW_IGNORE_TIMEZONE is not set). > Very basic example (assuming PYARROW_IGNORE_TIMEZONE is not set): > {code:python} > import logging > import pyspark.pandas > logging.basicConfig(level=logging.DEBUG) > logger = logging.getLogger("test") > logger.warning("I warn you") > logger.debug("I debug you") > {code} > Will only produce the warning, not the debug line. > By removing the {{import pyspark.pandas}}, the debug line is produced -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-41989) PYARROW_IGNORE_TIMEZONE warning can break application logging setup
[ https://issues.apache.org/jira/browse/SPARK-41989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-41989: - Fix Version/s: 3.2.4 3.3.2 > PYARROW_IGNORE_TIMEZONE warning can break application logging setup > --- > > Key: SPARK-41989 > URL: https://issues.apache.org/jira/browse/SPARK-41989 > Project: Spark > Issue Type: Bug > Components: PySpark >Affects Versions: 3.2.3 > Environment: python 3.9 env with pyspark installed >Reporter: Stefaan Lippens >Assignee: Stefaan Lippens >Priority: Major > Fix For: 3.2.4, 3.3.2, 3.4.0 > > > in {code}python/pyspark/pandas/__init__.py{code} there is currently a > warning when {{PYARROW_IGNORE_TIMEZONE}} env var is not set > (https://github.com/apache/spark/blob/187c4a9c66758e973633c5c309b551b1d9094e6e/python/pyspark/pandas/__init__.py#L44-L59): > {code:python} > import logging > logging.warning( > "'PYARROW_IGNORE_TIMEZONE' environment variable was not set. It is > required to "... > {code} > The {{logging.warning()}} call will silently do a {{logging.basicConfig()}} > call (at least in python 3.9, which I tried). > (FYI: Something like {{logging.getLogger(...).warning()}} would not do this > silent call) > This has the following very hard to figure out side-effect: > importing `pyspark.pandas` (directly or indirectly somewhere) might break > your logging setup (if PYARROW_IGNORE_TIMEZONE is not set). > Very basic example (assuming PYARROW_IGNORE_TIMEZONE is not set): > {code:python} > import logging > import pyspark.pandas > logging.basicConfig(level=logging.DEBUG) > logger = logging.getLogger("test") > logger.warning("I warn you") > logger.debug("I debug you") > {code} > Will only produce the warning, not the debug line. > By removing the {{import pyspark.pandas}}, the debug line is produced -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-42027) CreateDataframe from Pandas with Struct and Timestamp
Martin Grund created SPARK-42027: Summary: CreateDataframe from Pandas with Struct and Timestamp Key: SPARK-42027 URL: https://issues.apache.org/jira/browse/SPARK-42027 Project: Spark Issue Type: Sub-task Components: Connect Affects Versions: 3.4.0 Reporter: Martin Grund The following should be supported and correctly truncate the nanosecond timestamps. {code:python} from datetime import datetime, timezone, timedelta from pandas import Timestamp ts=Timestamp(year=2019, month=1, day=1, nanosecond=500, tz=timezone(timedelta(hours=-8))) d = pd.DataFrame({"col1": [1], "col2": [{"a":1, "b":2.32, "c":ts}]}) spark.createDataFrame(d).collect() {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-42028) Support Pandas DF to Spark DF with Nanosecond Timestamps
Martin Grund created SPARK-42028: Summary: Support Pandas DF to Spark DF with Nanosecond Timestamps Key: SPARK-42028 URL: https://issues.apache.org/jira/browse/SPARK-42028 Project: Spark Issue Type: Sub-task Components: Connect Affects Versions: 3.4.0 Reporter: Martin Grund -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-42029) Distribution build for Spark Connect does not work with Spark Shell
Martin Grund created SPARK-42029: Summary: Distribution build for Spark Connect does not work with Spark Shell Key: SPARK-42029 URL: https://issues.apache.org/jira/browse/SPARK-42029 Project: Spark Issue Type: Sub-task Components: Connect Affects Versions: 3.4.0 Reporter: Martin Grund -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-42028) Support Pandas DF to Spark DF with Nanosecond Timestamps
[ https://issues.apache.org/jira/browse/SPARK-42028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17675909#comment-17675909 ] Apache Spark commented on SPARK-42028: -- User 'grundprinzip' has created a pull request for this issue: https://github.com/apache/spark/pull/39469 > Support Pandas DF to Spark DF with Nanosecond Timestamps > > > Key: SPARK-42028 > URL: https://issues.apache.org/jira/browse/SPARK-42028 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Martin Grund >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-42028) Support Pandas DF to Spark DF with Nanosecond Timestamps
[ https://issues.apache.org/jira/browse/SPARK-42028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-42028: Assignee: Apache Spark > Support Pandas DF to Spark DF with Nanosecond Timestamps > > > Key: SPARK-42028 > URL: https://issues.apache.org/jira/browse/SPARK-42028 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Martin Grund >Assignee: Apache Spark >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-42028) Support Pandas DF to Spark DF with Nanosecond Timestamps
[ https://issues.apache.org/jira/browse/SPARK-42028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-42028: Assignee: (was: Apache Spark) > Support Pandas DF to Spark DF with Nanosecond Timestamps > > > Key: SPARK-42028 > URL: https://issues.apache.org/jira/browse/SPARK-42028 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Martin Grund >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-42028) Support Pandas DF to Spark DF with Nanosecond Timestamps
[ https://issues.apache.org/jira/browse/SPARK-42028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17675910#comment-17675910 ] Apache Spark commented on SPARK-42028: -- User 'grundprinzip' has created a pull request for this issue: https://github.com/apache/spark/pull/39469 > Support Pandas DF to Spark DF with Nanosecond Timestamps > > > Key: SPARK-42028 > URL: https://issues.apache.org/jira/browse/SPARK-42028 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Martin Grund >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-42029) Distribution build for Spark Connect does not work with Spark Shell
[ https://issues.apache.org/jira/browse/SPARK-42029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17675912#comment-17675912 ] Apache Spark commented on SPARK-42029: -- User 'grundprinzip' has created a pull request for this issue: https://github.com/apache/spark/pull/39531 > Distribution build for Spark Connect does not work with Spark Shell > --- > > Key: SPARK-42029 > URL: https://issues.apache.org/jira/browse/SPARK-42029 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Martin Grund >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-42029) Distribution build for Spark Connect does not work with Spark Shell
[ https://issues.apache.org/jira/browse/SPARK-42029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-42029: Assignee: (was: Apache Spark) > Distribution build for Spark Connect does not work with Spark Shell > --- > > Key: SPARK-42029 > URL: https://issues.apache.org/jira/browse/SPARK-42029 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Martin Grund >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-42029) Distribution build for Spark Connect does not work with Spark Shell
[ https://issues.apache.org/jira/browse/SPARK-42029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-42029: Assignee: Apache Spark > Distribution build for Spark Connect does not work with Spark Shell > --- > > Key: SPARK-42029 > URL: https://issues.apache.org/jira/browse/SPARK-42029 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Martin Grund >Assignee: Apache Spark >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-42029) Distribution build for Spark Connect does not work with Spark Shell
[ https://issues.apache.org/jira/browse/SPARK-42029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-42029: Assignee: Martin Grund > Distribution build for Spark Connect does not work with Spark Shell > --- > > Key: SPARK-42029 > URL: https://issues.apache.org/jira/browse/SPARK-42029 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Martin Grund >Assignee: Martin Grund >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-42029) Distribution build for Spark Connect does not work with Spark Shell
[ https://issues.apache.org/jira/browse/SPARK-42029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42029. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39531 [https://github.com/apache/spark/pull/39531] > Distribution build for Spark Connect does not work with Spark Shell > --- > > Key: SPARK-42029 > URL: https://issues.apache.org/jira/browse/SPARK-42029 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Martin Grund >Assignee: Martin Grund >Priority: Major > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-42010) Reuse pyspark.sql.tests.test_column test cases
[ https://issues.apache.org/jira/browse/SPARK-42010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-42010: Assignee: Hyukjin Kwon > Reuse pyspark.sql.tests.test_column test cases > -- > > Key: SPARK-42010 > URL: https://issues.apache.org/jira/browse/SPARK-42010 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-42010) Reuse pyspark.sql.tests.test_column test cases
[ https://issues.apache.org/jira/browse/SPARK-42010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42010. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39528 [https://github.com/apache/spark/pull/39528] > Reuse pyspark.sql.tests.test_column test cases > -- > > Key: SPARK-42010 > URL: https://issues.apache.org/jira/browse/SPARK-42010 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-42028) Support Pandas DF to Spark DF with Nanosecond Timestamps
[ https://issues.apache.org/jira/browse/SPARK-42028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-42028: Assignee: Martin Grund > Support Pandas DF to Spark DF with Nanosecond Timestamps > > > Key: SPARK-42028 > URL: https://issues.apache.org/jira/browse/SPARK-42028 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Martin Grund >Assignee: Martin Grund >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-42028) Support Pandas DF to Spark DF with Nanosecond Timestamps
[ https://issues.apache.org/jira/browse/SPARK-42028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42028. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39469 [https://github.com/apache/spark/pull/39469] > Support Pandas DF to Spark DF with Nanosecond Timestamps > > > Key: SPARK-42028 > URL: https://issues.apache.org/jira/browse/SPARK-42028 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Martin Grund >Assignee: Martin Grund >Priority: Major > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-42019) Reuse pyspark.sql.tests.test_types test cases
[ https://issues.apache.org/jira/browse/SPARK-42019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-42019. -- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 39529 [https://github.com/apache/spark/pull/39529] > Reuse pyspark.sql.tests.test_types test cases > - > > Key: SPARK-42019 > URL: https://issues.apache.org/jira/browse/SPARK-42019 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-42030) Remove unused Constructor from RocksDB.TypeAliases and LevelDB.TypeAliases
Yang Jie created SPARK-42030: Summary: Remove unused Constructor from RocksDB.TypeAliases and LevelDB.TypeAliases Key: SPARK-42030 URL: https://issues.apache.org/jira/browse/SPARK-42030 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 3.4.0 Reporter: Yang Jie -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-42019) Reuse pyspark.sql.tests.test_types test cases
[ https://issues.apache.org/jira/browse/SPARK-42019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-42019: Assignee: Hyukjin Kwon > Reuse pyspark.sql.tests.test_types test cases > - > > Key: SPARK-42019 > URL: https://issues.apache.org/jira/browse/SPARK-42019 > Project: Spark > Issue Type: Sub-task > Components: Connect, Tests >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Assignee: Hyukjin Kwon >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-42030) Remove unused Constructor from RocksDB.TypeAliases and LevelDB.TypeAliases
[ https://issues.apache.org/jira/browse/SPARK-42030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17675963#comment-17675963 ] Apache Spark commented on SPARK-42030: -- User 'LuciferYang' has created a pull request for this issue: https://github.com/apache/spark/pull/39532 > Remove unused Constructor from RocksDB.TypeAliases and LevelDB.TypeAliases > -- > > Key: SPARK-42030 > URL: https://issues.apache.org/jira/browse/SPARK-42030 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.4.0 >Reporter: Yang Jie >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-42030) Remove unused Constructor from RocksDB.TypeAliases and LevelDB.TypeAliases
[ https://issues.apache.org/jira/browse/SPARK-42030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-42030: Assignee: (was: Apache Spark) > Remove unused Constructor from RocksDB.TypeAliases and LevelDB.TypeAliases > -- > > Key: SPARK-42030 > URL: https://issues.apache.org/jira/browse/SPARK-42030 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.4.0 >Reporter: Yang Jie >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-42030) Remove unused Constructor from RocksDB.TypeAliases and LevelDB.TypeAliases
[ https://issues.apache.org/jira/browse/SPARK-42030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-42030: Assignee: Apache Spark > Remove unused Constructor from RocksDB.TypeAliases and LevelDB.TypeAliases > -- > > Key: SPARK-42030 > URL: https://issues.apache.org/jira/browse/SPARK-42030 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.4.0 >Reporter: Yang Jie >Assignee: Apache Spark >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-40307) Introduce Arrow-optimized Python UDFs
[ https://issues.apache.org/jira/browse/SPARK-40307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17675983#comment-17675983 ] Xinrong Meng commented on SPARK-40307: -- Resolved by https://github.com/apache/spark/pull/39384. > Introduce Arrow-optimized Python UDFs > - > > Key: SPARK-40307 > URL: https://issues.apache.org/jira/browse/SPARK-40307 > Project: Spark > Issue Type: Umbrella > Components: PySpark >Affects Versions: 3.4.0 >Reporter: Xinrong Meng >Assignee: Xinrong Meng >Priority: Major > > Python user-defined function (UDF) enables users to run arbitrary code > against PySpark columns. It uses Pickle for (de)serialization and executes > row by row. > One major performance bottleneck of Python UDFs is (de)serialization, that > is, the data interchanging between the worker JVM and the spawned Python > subprocess which actually executes the UDF. We should seek an alternative to > handle the (de)serialization: Arrow, which is used in the (de)serialization > of Pandas UDF already. > There should be two ways to enable/disable the Arrow optimization for Python > UDFs: > - the Spark configuration `spark.sql.execution.pythonUDF.arrow.enabled`, > disabled by default. > - the `useArrow` parameter of the `udf` function, None by default. > The Spark configuration takes effect only when `useArrow` is None. Otherwise, > `useArrow` decides whether a specific user-defined function is optimized by > Arrow or not. > The reason why we introduce these two ways is to provide both a convenient, > per-Spark-session control and a finer-grained, per-UDF control of the Arrow > optimization for Python UDFs. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-40307) Introduce Arrow-optimized Python UDFs
[ https://issues.apache.org/jira/browse/SPARK-40307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng resolved SPARK-40307. -- Assignee: Xinrong Meng Resolution: Resolved > Introduce Arrow-optimized Python UDFs > - > > Key: SPARK-40307 > URL: https://issues.apache.org/jira/browse/SPARK-40307 > Project: Spark > Issue Type: Umbrella > Components: PySpark >Affects Versions: 3.4.0 >Reporter: Xinrong Meng >Assignee: Xinrong Meng >Priority: Major > > Python user-defined function (UDF) enables users to run arbitrary code > against PySpark columns. It uses Pickle for (de)serialization and executes > row by row. > One major performance bottleneck of Python UDFs is (de)serialization, that > is, the data interchanging between the worker JVM and the spawned Python > subprocess which actually executes the UDF. We should seek an alternative to > handle the (de)serialization: Arrow, which is used in the (de)serialization > of Pandas UDF already. > There should be two ways to enable/disable the Arrow optimization for Python > UDFs: > - the Spark configuration `spark.sql.execution.pythonUDF.arrow.enabled`, > disabled by default. > - the `useArrow` parameter of the `udf` function, None by default. > The Spark configuration takes effect only when `useArrow` is None. Otherwise, > `useArrow` decides whether a specific user-defined function is optimized by > Arrow or not. > The reason why we introduce these two ways is to provide both a convenient, > per-Spark-session control and a finer-grained, per-UDF control of the Arrow > optimization for Python UDFs. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-42031) Clean up remove methods that do not need override
Yang Jie created SPARK-42031: Summary: Clean up remove methods that do not need override Key: SPARK-42031 URL: https://issues.apache.org/jira/browse/SPARK-42031 Project: Spark Issue Type: Improvement Components: Spark Core, SQL Affects Versions: 3.4.0 Reporter: Yang Jie Java 8 began to provide the default remove method implementation for the `java.util.Iterator` interface. https://github.com/openjdk/jdk/blob/9a9add8825a040565051a09010b29b099c2e7d49/jdk/src/share/classes/java/util/Iterator.java#L92-L94 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-42031) Clean up remove methods that do not need override
[ https://issues.apache.org/jira/browse/SPARK-42031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-42031: Assignee: Apache Spark > Clean up remove methods that do not need override > - > > Key: SPARK-42031 > URL: https://issues.apache.org/jira/browse/SPARK-42031 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 3.4.0 >Reporter: Yang Jie >Assignee: Apache Spark >Priority: Minor > > Java 8 began to provide the default remove method implementation for the > `java.util.Iterator` interface. > https://github.com/openjdk/jdk/blob/9a9add8825a040565051a09010b29b099c2e7d49/jdk/src/share/classes/java/util/Iterator.java#L92-L94 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-42031) Clean up remove methods that do not need override
[ https://issues.apache.org/jira/browse/SPARK-42031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676009#comment-17676009 ] Apache Spark commented on SPARK-42031: -- User 'LuciferYang' has created a pull request for this issue: https://github.com/apache/spark/pull/39533 > Clean up remove methods that do not need override > - > > Key: SPARK-42031 > URL: https://issues.apache.org/jira/browse/SPARK-42031 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 3.4.0 >Reporter: Yang Jie >Priority: Minor > > Java 8 began to provide the default remove method implementation for the > `java.util.Iterator` interface. > https://github.com/openjdk/jdk/blob/9a9add8825a040565051a09010b29b099c2e7d49/jdk/src/share/classes/java/util/Iterator.java#L92-L94 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-42031) Clean up remove methods that do not need override
[ https://issues.apache.org/jira/browse/SPARK-42031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-42031: Assignee: (was: Apache Spark) > Clean up remove methods that do not need override > - > > Key: SPARK-42031 > URL: https://issues.apache.org/jira/browse/SPARK-42031 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 3.4.0 >Reporter: Yang Jie >Priority: Minor > > Java 8 began to provide the default remove method implementation for the > `java.util.Iterator` interface. > https://github.com/openjdk/jdk/blob/9a9add8825a040565051a09010b29b099c2e7d49/jdk/src/share/classes/java/util/Iterator.java#L92-L94 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-42032) transform_key, transform_values doctest output have different order
Ruifeng Zheng created SPARK-42032: - Summary: transform_key, transform_values doctest output have different order Key: SPARK-42032 URL: https://issues.apache.org/jira/browse/SPARK-42032 Project: Spark Issue Type: Sub-task Components: Connect, PySpark Affects Versions: 3.4.0 Reporter: Ruifeng Zheng not sure whether this should be fixed: {code:java} ** File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1623, in pyspark.sql.connect.functions.transform_keys Failed example: df.select(transform_keys( "data", lambda k, _: upper(k)).alias("data_upper") ).show(truncate=False) Expected: +-+ |data_upper | +-+ |{BAR -> 2.0, FOO -> -2.0}| +-+ Got: +-+ |data_upper | +-+ |{FOO -> -2.0, BAR -> 2.0}| +-+ ** File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1630, in pyspark.sql.connect.functions.transform_values Failed example: df.select(transform_values( "data", lambda k, v: when(k.isin("IT", "OPS"), v + 10.0).otherwise(v) ).alias("new_data")).show(truncate=False) Expected: +---+ |new_data | +---+ |{OPS -> 34.0, IT -> 20.0, SALES -> 2.0}| +---+ Got: +---+ |new_data | +---+ |{IT -> 20.0, SALES -> 2.0, OPS -> 34.0}| +---+ ** 1 of 2 in pyspark.sql.connect.functions.transform_keys 1 of 2 in pyspark.sql.connect.functions.transform_values {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] (SPARK-41746) SparkSession.createDataFrame does not support nested datatypes
[ https://issues.apache.org/jira/browse/SPARK-41746 ] Ruifeng Zheng deleted comment on SPARK-41746: --- was (Author: podongfeng): {code:java} ** File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1423, in pyspark.sql.connect.functions.map_filter Failed example: df.select(map_filter( "data", lambda _, v: v > 30.0).alias("data_filtered") ).show(truncate=False) Expected: +--+ |data_filtered | +--+ |{baz -> 32.0, foo -> 42.0}| +--+ Got: +--+ |data_filtered | +--+ |{foo -> 42.0, baz -> 32.0}| +--+ ** File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1465, in pyspark.sql.connect.functions.map_zip_with Failed example: df.select(map_zip_with( "base", "ratio", lambda k, v1, v2: round(v1 * v2, 2)).alias("updated_data") ).show(truncate=False) Expected: +---+ |updated_data | +---+ |{SALES -> 16.8, IT -> 48.0}| +---+ Got: +---+ |updated_data | +---+ |{IT -> 48.0, SALES -> 16.8}| +---+ ** 1 of 2 in pyspark.sql.connect.functions.map_filter 1 of 2 in pyspark.sql.connect.functions.map_zip_with {code} > SparkSession.createDataFrame does not support nested datatypes > -- > > Key: SPARK-41746 > URL: https://issues.apache.org/jira/browse/SPARK-41746 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Priority: Major > > {code} > File "/.../spark/python/pyspark/sql/connect/group.py", line 183, in > pyspark.sql.connect.group.GroupedData.pivot > Failed example: > df2 = spark.createDataFrame([ > Row(training="expert", sales=Row(course="dotNET", year=2012, > earnings=1)), > Row(training="junior", sales=Row(course="Java", year=2012, > earnings=2)), > Row(training="expert", sales=Row(course="dotNET", year=2012, > earnings=5000)), > Row(training="junior", sales=Row(course="dotNET", year=2013, > earnings=48000)), > Row(training="expert", sales=Row(course="Java", year=2013, > earnings=3)), > ]) > Exception raised: > Traceback (most recent call last): > File "/.../miniconda3/envs/python3.9/lib/python3.9/doctest.py", line > 1336, in __run > exec(compile(example.source, filename, "single", > File "", line > 1, in > df2 = spark.createDataFrame([ > File > "/.../workspace/forked/spark/python/pyspark/sql/connect/session.py", line > 196, in createDataFrame > table = pa.Table.from_pandas(pdf) > File "pyarrow/table.pxi", line 3475, in pyarrow.lib.Table.from_pandas > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 611, in dataframe_to_arrays > arrays = [convert_column(c, f) > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 611, in > arrays = [convert_column(c, f) > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 598, in convert_column > raise e > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 592, in convert_column > result = pa.array(col, type=type_, from_pandas=True, safe=safe) > File "pyarrow/array.pxi", line 316, in pyarrow.lib.array > File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array > File "pyarrow/error.pxi", line 123, in pyarrow.lib.check_status > pyarrow.lib.ArrowTypeError: ("Expected bytes, got a 'int' object", > 'Conversion failed for column 1 with type object') > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41746) SparkSession.createDataFrame does not support nested datatypes
[ https://issues.apache.org/jira/browse/SPARK-41746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676059#comment-17676059 ] Ruifeng Zheng commented on SPARK-41746: --- {code:java} ** File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1423, in pyspark.sql.connect.functions.map_filter Failed example: df.select(map_filter( "data", lambda _, v: v > 30.0).alias("data_filtered") ).show(truncate=False) Expected: +--+ |data_filtered | +--+ |{baz -> 32.0, foo -> 42.0}| +--+ Got: +--+ |data_filtered | +--+ |{foo -> 42.0, baz -> 32.0}| +--+ ** File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1465, in pyspark.sql.connect.functions.map_zip_with Failed example: df.select(map_zip_with( "base", "ratio", lambda k, v1, v2: round(v1 * v2, 2)).alias("updated_data") ).show(truncate=False) Expected: +---+ |updated_data | +---+ |{SALES -> 16.8, IT -> 48.0}| +---+ Got: +---+ |updated_data | +---+ |{IT -> 48.0, SALES -> 16.8}| +---+ ** 1 of 2 in pyspark.sql.connect.functions.map_filter 1 of 2 in pyspark.sql.connect.functions.map_zip_with {code} > SparkSession.createDataFrame does not support nested datatypes > -- > > Key: SPARK-41746 > URL: https://issues.apache.org/jira/browse/SPARK-41746 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Priority: Major > > {code} > File "/.../spark/python/pyspark/sql/connect/group.py", line 183, in > pyspark.sql.connect.group.GroupedData.pivot > Failed example: > df2 = spark.createDataFrame([ > Row(training="expert", sales=Row(course="dotNET", year=2012, > earnings=1)), > Row(training="junior", sales=Row(course="Java", year=2012, > earnings=2)), > Row(training="expert", sales=Row(course="dotNET", year=2012, > earnings=5000)), > Row(training="junior", sales=Row(course="dotNET", year=2013, > earnings=48000)), > Row(training="expert", sales=Row(course="Java", year=2013, > earnings=3)), > ]) > Exception raised: > Traceback (most recent call last): > File "/.../miniconda3/envs/python3.9/lib/python3.9/doctest.py", line > 1336, in __run > exec(compile(example.source, filename, "single", > File "", line > 1, in > df2 = spark.createDataFrame([ > File > "/.../workspace/forked/spark/python/pyspark/sql/connect/session.py", line > 196, in createDataFrame > table = pa.Table.from_pandas(pdf) > File "pyarrow/table.pxi", line 3475, in pyarrow.lib.Table.from_pandas > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 611, in dataframe_to_arrays > arrays = [convert_column(c, f) > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 611, in > arrays = [convert_column(c, f) > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 598, in convert_column > raise e > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 592, in convert_column > result = pa.array(col, type=type_, from_pandas=True, safe=safe) > File "pyarrow/array.pxi", line 316, in pyarrow.lib.array > File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array > File "pyarrow/error.pxi", line 123, in pyarrow.lib.check_status > pyarrow.lib.ArrowTypeError: ("Expected bytes, got a 'int' object", > 'Conversion failed for column 1 with type object') > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-42032) transform_key, transform_values doctest output have different order
[ https://issues.apache.org/jira/browse/SPARK-42032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676060#comment-17676060 ] Ruifeng Zheng commented on SPARK-42032: --- {code:java} ** File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1423, in pyspark.sql.connect.functions.map_filter Failed example: df.select(map_filter( "data", lambda _, v: v > 30.0).alias("data_filtered") ).show(truncate=False) Expected: +--+ |data_filtered | +--+ |{baz -> 32.0, foo -> 42.0}| +--+ Got: +--+ |data_filtered | +--+ |{foo -> 42.0, baz -> 32.0}| +--+ ** File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1465, in pyspark.sql.connect.functions.map_zip_with Failed example: df.select(map_zip_with( "base", "ratio", lambda k, v1, v2: round(v1 * v2, 2)).alias("updated_data") ).show(truncate=False) Expected: +---+ |updated_data | +---+ |{SALES -> 16.8, IT -> 48.0}| +---+ Got: +---+ |updated_data | +---+ |{IT -> 48.0, SALES -> 16.8}| +---+ ** 1 of 2 in pyspark.sql.connect.functions.map_filter 1 of 2 in pyspark.sql.connect.functions.map_zip_with {code} > transform_key, transform_values doctest output have different order > --- > > Key: SPARK-42032 > URL: https://issues.apache.org/jira/browse/SPARK-42032 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 3.4.0 >Reporter: Ruifeng Zheng >Priority: Major > > not sure whether this should be fixed: > {code:java} > ** > File > "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", > line 1623, in pyspark.sql.connect.functions.transform_keys > Failed example: > df.select(transform_keys( > "data", lambda k, _: upper(k)).alias("data_upper") > ).show(truncate=False) > Expected: > +-+ > |data_upper | > +-+ > |{BAR -> 2.0, FOO -> -2.0}| > +-+ > Got: > +-+ > |data_upper | > +-+ > |{FOO -> -2.0, BAR -> 2.0}| > +-+ > > ** > File > "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", > line 1630, in pyspark.sql.connect.functions.transform_values > Failed example: > df.select(transform_values( > "data", lambda k, v: when(k.isin("IT", "OPS"), v + 10.0).otherwise(v) > ).alias("new_data")).show(truncate=False) > Expected: > +---+ > |new_data | > +---+ > |{OPS -> 34.0, IT -> 20.0, SALES -> 2.0}| > +---+ > Got: > +---+ > |new_data | > +---+ > |{IT -> 20.0, SALES -> 2.0, OPS -> 34.0}| > +---+ > > ** >1 of 2 in pyspark.sql.connect.functions.transform_keys >1 of 2 in pyspark.sql.connect.functions.transform_values > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-42032) Map data show in different order
[ https://issues.apache.org/jira/browse/SPARK-42032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-42032: -- Summary: Map data show in different order (was: transform_key, transform_values doctest output have different order) > Map data show in different order > > > Key: SPARK-42032 > URL: https://issues.apache.org/jira/browse/SPARK-42032 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 3.4.0 >Reporter: Ruifeng Zheng >Priority: Major > > not sure whether this should be fixed: > {code:java} > ** > File > "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", > line 1623, in pyspark.sql.connect.functions.transform_keys > Failed example: > df.select(transform_keys( > "data", lambda k, _: upper(k)).alias("data_upper") > ).show(truncate=False) > Expected: > +-+ > |data_upper | > +-+ > |{BAR -> 2.0, FOO -> -2.0}| > +-+ > Got: > +-+ > |data_upper | > +-+ > |{FOO -> -2.0, BAR -> 2.0}| > +-+ > > ** > File > "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", > line 1630, in pyspark.sql.connect.functions.transform_values > Failed example: > df.select(transform_values( > "data", lambda k, v: when(k.isin("IT", "OPS"), v + 10.0).otherwise(v) > ).alias("new_data")).show(truncate=False) > Expected: > +---+ > |new_data | > +---+ > |{OPS -> 34.0, IT -> 20.0, SALES -> 2.0}| > +---+ > Got: > +---+ > |new_data | > +---+ > |{IT -> 20.0, SALES -> 2.0, OPS -> 34.0}| > +---+ > > ** >1 of 2 in pyspark.sql.connect.functions.transform_keys >1 of 2 in pyspark.sql.connect.functions.transform_values > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-42032) Map data show in different order
[ https://issues.apache.org/jira/browse/SPARK-42032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruifeng Zheng updated SPARK-42032: -- Description: not sure whether this needs to be fixed: {code:java} ** File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1623, in pyspark.sql.connect.functions.transform_keys Failed example: df.select(transform_keys( "data", lambda k, _: upper(k)).alias("data_upper") ).show(truncate=False) Expected: +-+ |data_upper | +-+ |{BAR -> 2.0, FOO -> -2.0}| +-+ Got: +-+ |data_upper | +-+ |{FOO -> -2.0, BAR -> 2.0}| +-+ ** File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1630, in pyspark.sql.connect.functions.transform_values Failed example: df.select(transform_values( "data", lambda k, v: when(k.isin("IT", "OPS"), v + 10.0).otherwise(v) ).alias("new_data")).show(truncate=False) Expected: +---+ |new_data | +---+ |{OPS -> 34.0, IT -> 20.0, SALES -> 2.0}| +---+ Got: +---+ |new_data | +---+ |{IT -> 20.0, SALES -> 2.0, OPS -> 34.0}| +---+ ** 1 of 2 in pyspark.sql.connect.functions.transform_keys 1 of 2 in pyspark.sql.connect.functions.transform_values {code} was: not sure whether this should be fixed: {code:java} ** File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1623, in pyspark.sql.connect.functions.transform_keys Failed example: df.select(transform_keys( "data", lambda k, _: upper(k)).alias("data_upper") ).show(truncate=False) Expected: +-+ |data_upper | +-+ |{BAR -> 2.0, FOO -> -2.0}| +-+ Got: +-+ |data_upper | +-+ |{FOO -> -2.0, BAR -> 2.0}| +-+ ** File "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", line 1630, in pyspark.sql.connect.functions.transform_values Failed example: df.select(transform_values( "data", lambda k, v: when(k.isin("IT", "OPS"), v + 10.0).otherwise(v) ).alias("new_data")).show(truncate=False) Expected: +---+ |new_data | +---+ |{OPS -> 34.0, IT -> 20.0, SALES -> 2.0}| +---+ Got: +---+ |new_data | +---+ |{IT -> 20.0, SALES -> 2.0, OPS -> 34.0}| +---+ ** 1 of 2 in pyspark.sql.connect.functions.transform_keys 1 of 2 in pyspark.sql.connect.functions.transform_values {code} > Map data show in different order > > > Key: SPARK-42032 > URL: https://issues.apache.org/jira/browse/SPARK-42032 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 3.4.0 >Reporter: Ruifeng Zheng >Priority: Major > > not sure whether this needs to be fixed: > {code:java} > ** > File > "/Users/ruifeng.zheng/Dev/spark/python/pyspark/sql/connect/functions.py", > line 1623, in pyspark.sql.connect.functions.transform_keys > Failed example: > df.select(transform_keys( > "data", lambda k, _: upper(k)).alias("data_upper") > ).show(truncate=False) > Expected: > +-+ > |data_upper | > +-+ > |{BAR -> 2.0, FOO -> -2.0}| > +-+ > Got: > +-+ > |data_upper | > +-+ > |{FOO -> -2.0, BAR -> 2.0}| > +-+ > > *
[jira] [Assigned] (SPARK-41746) SparkSession.createDataFrame does not support nested datatypes
[ https://issues.apache.org/jira/browse/SPARK-41746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41746: Assignee: (was: Apache Spark) > SparkSession.createDataFrame does not support nested datatypes > -- > > Key: SPARK-41746 > URL: https://issues.apache.org/jira/browse/SPARK-41746 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Priority: Major > > {code} > File "/.../spark/python/pyspark/sql/connect/group.py", line 183, in > pyspark.sql.connect.group.GroupedData.pivot > Failed example: > df2 = spark.createDataFrame([ > Row(training="expert", sales=Row(course="dotNET", year=2012, > earnings=1)), > Row(training="junior", sales=Row(course="Java", year=2012, > earnings=2)), > Row(training="expert", sales=Row(course="dotNET", year=2012, > earnings=5000)), > Row(training="junior", sales=Row(course="dotNET", year=2013, > earnings=48000)), > Row(training="expert", sales=Row(course="Java", year=2013, > earnings=3)), > ]) > Exception raised: > Traceback (most recent call last): > File "/.../miniconda3/envs/python3.9/lib/python3.9/doctest.py", line > 1336, in __run > exec(compile(example.source, filename, "single", > File "", line > 1, in > df2 = spark.createDataFrame([ > File > "/.../workspace/forked/spark/python/pyspark/sql/connect/session.py", line > 196, in createDataFrame > table = pa.Table.from_pandas(pdf) > File "pyarrow/table.pxi", line 3475, in pyarrow.lib.Table.from_pandas > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 611, in dataframe_to_arrays > arrays = [convert_column(c, f) > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 611, in > arrays = [convert_column(c, f) > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 598, in convert_column > raise e > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 592, in convert_column > result = pa.array(col, type=type_, from_pandas=True, safe=safe) > File "pyarrow/array.pxi", line 316, in pyarrow.lib.array > File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array > File "pyarrow/error.pxi", line 123, in pyarrow.lib.check_status > pyarrow.lib.ArrowTypeError: ("Expected bytes, got a 'int' object", > 'Conversion failed for column 1 with type object') > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-41746) SparkSession.createDataFrame does not support nested datatypes
[ https://issues.apache.org/jira/browse/SPARK-41746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41746: Assignee: Apache Spark > SparkSession.createDataFrame does not support nested datatypes > -- > > Key: SPARK-41746 > URL: https://issues.apache.org/jira/browse/SPARK-41746 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Assignee: Apache Spark >Priority: Major > > {code} > File "/.../spark/python/pyspark/sql/connect/group.py", line 183, in > pyspark.sql.connect.group.GroupedData.pivot > Failed example: > df2 = spark.createDataFrame([ > Row(training="expert", sales=Row(course="dotNET", year=2012, > earnings=1)), > Row(training="junior", sales=Row(course="Java", year=2012, > earnings=2)), > Row(training="expert", sales=Row(course="dotNET", year=2012, > earnings=5000)), > Row(training="junior", sales=Row(course="dotNET", year=2013, > earnings=48000)), > Row(training="expert", sales=Row(course="Java", year=2013, > earnings=3)), > ]) > Exception raised: > Traceback (most recent call last): > File "/.../miniconda3/envs/python3.9/lib/python3.9/doctest.py", line > 1336, in __run > exec(compile(example.source, filename, "single", > File "", line > 1, in > df2 = spark.createDataFrame([ > File > "/.../workspace/forked/spark/python/pyspark/sql/connect/session.py", line > 196, in createDataFrame > table = pa.Table.from_pandas(pdf) > File "pyarrow/table.pxi", line 3475, in pyarrow.lib.Table.from_pandas > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 611, in dataframe_to_arrays > arrays = [convert_column(c, f) > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 611, in > arrays = [convert_column(c, f) > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 598, in convert_column > raise e > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 592, in convert_column > result = pa.array(col, type=type_, from_pandas=True, safe=safe) > File "pyarrow/array.pxi", line 316, in pyarrow.lib.array > File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array > File "pyarrow/error.pxi", line 123, in pyarrow.lib.check_status > pyarrow.lib.ArrowTypeError: ("Expected bytes, got a 'int' object", > 'Conversion failed for column 1 with type object') > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41746) SparkSession.createDataFrame does not support nested datatypes
[ https://issues.apache.org/jira/browse/SPARK-41746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676065#comment-17676065 ] Apache Spark commented on SPARK-41746: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > SparkSession.createDataFrame does not support nested datatypes > -- > > Key: SPARK-41746 > URL: https://issues.apache.org/jira/browse/SPARK-41746 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Priority: Major > > {code} > File "/.../spark/python/pyspark/sql/connect/group.py", line 183, in > pyspark.sql.connect.group.GroupedData.pivot > Failed example: > df2 = spark.createDataFrame([ > Row(training="expert", sales=Row(course="dotNET", year=2012, > earnings=1)), > Row(training="junior", sales=Row(course="Java", year=2012, > earnings=2)), > Row(training="expert", sales=Row(course="dotNET", year=2012, > earnings=5000)), > Row(training="junior", sales=Row(course="dotNET", year=2013, > earnings=48000)), > Row(training="expert", sales=Row(course="Java", year=2013, > earnings=3)), > ]) > Exception raised: > Traceback (most recent call last): > File "/.../miniconda3/envs/python3.9/lib/python3.9/doctest.py", line > 1336, in __run > exec(compile(example.source, filename, "single", > File "", line > 1, in > df2 = spark.createDataFrame([ > File > "/.../workspace/forked/spark/python/pyspark/sql/connect/session.py", line > 196, in createDataFrame > table = pa.Table.from_pandas(pdf) > File "pyarrow/table.pxi", line 3475, in pyarrow.lib.Table.from_pandas > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 611, in dataframe_to_arrays > arrays = [convert_column(c, f) > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 611, in > arrays = [convert_column(c, f) > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 598, in convert_column > raise e > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 592, in convert_column > result = pa.array(col, type=type_, from_pandas=True, safe=safe) > File "pyarrow/array.pxi", line 316, in pyarrow.lib.array > File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array > File "pyarrow/error.pxi", line 123, in pyarrow.lib.check_status > pyarrow.lib.ArrowTypeError: ("Expected bytes, got a 'int' object", > 'Conversion failed for column 1 with type object') > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41838) DataFrame.show() fix map printing
[ https://issues.apache.org/jira/browse/SPARK-41838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676067#comment-17676067 ] Apache Spark commented on SPARK-41838: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > DataFrame.show() fix map printing > - > > Key: SPARK-41838 > URL: https://issues.apache.org/jira/browse/SPARK-41838 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1472, in pyspark.sql.connect.functions.posexplode_outer > Failed example: > df.select("id", "a_map", posexplode_outer("an_array")).show() > Expected: > +---+--+++ > | id| a_map| pos| col| > +---+--+++ > | 1|{x -> 1.0}| 0| foo| > | 1|{x -> 1.0}| 1| bar| > | 2| {}|null|null| > | 3| null|null|null| > +---+--+++ > Got: > +---+--+++ > | id| a_map| pos| col| > +---+--+++ > | 1| {1.0}| 0| foo| > | 1| {1.0}| 1| bar| > | 2|{null}|null|null| > | 3| null|null|null| > +---+--+++ > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-41838) DataFrame.show() fix map printing
[ https://issues.apache.org/jira/browse/SPARK-41838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41838: Assignee: Apache Spark > DataFrame.show() fix map printing > - > > Key: SPARK-41838 > URL: https://issues.apache.org/jira/browse/SPARK-41838 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Assignee: Apache Spark >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1472, in pyspark.sql.connect.functions.posexplode_outer > Failed example: > df.select("id", "a_map", posexplode_outer("an_array")).show() > Expected: > +---+--+++ > | id| a_map| pos| col| > +---+--+++ > | 1|{x -> 1.0}| 0| foo| > | 1|{x -> 1.0}| 1| bar| > | 2| {}|null|null| > | 3| null|null|null| > +---+--+++ > Got: > +---+--+++ > | id| a_map| pos| col| > +---+--+++ > | 1| {1.0}| 0| foo| > | 1| {1.0}| 1| bar| > | 2|{null}|null|null| > | 3| null|null|null| > +---+--+++ > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41746) SparkSession.createDataFrame does not support nested datatypes
[ https://issues.apache.org/jira/browse/SPARK-41746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676066#comment-17676066 ] Apache Spark commented on SPARK-41746: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > SparkSession.createDataFrame does not support nested datatypes > -- > > Key: SPARK-41746 > URL: https://issues.apache.org/jira/browse/SPARK-41746 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Hyukjin Kwon >Priority: Major > > {code} > File "/.../spark/python/pyspark/sql/connect/group.py", line 183, in > pyspark.sql.connect.group.GroupedData.pivot > Failed example: > df2 = spark.createDataFrame([ > Row(training="expert", sales=Row(course="dotNET", year=2012, > earnings=1)), > Row(training="junior", sales=Row(course="Java", year=2012, > earnings=2)), > Row(training="expert", sales=Row(course="dotNET", year=2012, > earnings=5000)), > Row(training="junior", sales=Row(course="dotNET", year=2013, > earnings=48000)), > Row(training="expert", sales=Row(course="Java", year=2013, > earnings=3)), > ]) > Exception raised: > Traceback (most recent call last): > File "/.../miniconda3/envs/python3.9/lib/python3.9/doctest.py", line > 1336, in __run > exec(compile(example.source, filename, "single", > File "", line > 1, in > df2 = spark.createDataFrame([ > File > "/.../workspace/forked/spark/python/pyspark/sql/connect/session.py", line > 196, in createDataFrame > table = pa.Table.from_pandas(pdf) > File "pyarrow/table.pxi", line 3475, in pyarrow.lib.Table.from_pandas > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 611, in dataframe_to_arrays > arrays = [convert_column(c, f) > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 611, in > arrays = [convert_column(c, f) > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 598, in convert_column > raise e > File > "/.../miniconda3/envs/python3.9/lib/python3.9/site-packages/pyarrow/pandas_compat.py", > line 592, in convert_column > result = pa.array(col, type=type_, from_pandas=True, safe=safe) > File "pyarrow/array.pxi", line 316, in pyarrow.lib.array > File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array > File "pyarrow/error.pxi", line 123, in pyarrow.lib.check_status > pyarrow.lib.ArrowTypeError: ("Expected bytes, got a 'int' object", > 'Conversion failed for column 1 with type object') > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-41838) DataFrame.show() fix map printing
[ https://issues.apache.org/jira/browse/SPARK-41838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41838: Assignee: (was: Apache Spark) > DataFrame.show() fix map printing > - > > Key: SPARK-41838 > URL: https://issues.apache.org/jira/browse/SPARK-41838 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1472, in pyspark.sql.connect.functions.posexplode_outer > Failed example: > df.select("id", "a_map", posexplode_outer("an_array")).show() > Expected: > +---+--+++ > | id| a_map| pos| col| > +---+--+++ > | 1|{x -> 1.0}| 0| foo| > | 1|{x -> 1.0}| 1| bar| > | 2| {}|null|null| > | 3| null|null|null| > +---+--+++ > Got: > +---+--+++ > | id| a_map| pos| col| > +---+--+++ > | 1| {1.0}| 0| foo| > | 1| {1.0}| 1| bar| > | 2|{null}|null|null| > | 3| null|null|null| > +---+--+++ > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41838) DataFrame.show() fix map printing
[ https://issues.apache.org/jira/browse/SPARK-41838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676068#comment-17676068 ] Apache Spark commented on SPARK-41838: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > DataFrame.show() fix map printing > - > > Key: SPARK-41838 > URL: https://issues.apache.org/jira/browse/SPARK-41838 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1472, in pyspark.sql.connect.functions.posexplode_outer > Failed example: > df.select("id", "a_map", posexplode_outer("an_array")).show() > Expected: > +---+--+++ > | id| a_map| pos| col| > +---+--+++ > | 1|{x -> 1.0}| 0| foo| > | 1|{x -> 1.0}| 1| bar| > | 2| {}|null|null| > | 3| null|null|null| > +---+--+++ > Got: > +---+--+++ > | id| a_map| pos| col| > +---+--+++ > | 1| {1.0}| 0| foo| > | 1| {1.0}| 1| bar| > | 2|{null}|null|null| > | 3| null|null|null| > +---+--+++ > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41837) DataFrame.createDataFrame datatype conversion error
[ https://issues.apache.org/jira/browse/SPARK-41837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676071#comment-17676071 ] Apache Spark commented on SPARK-41837: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > DataFrame.createDataFrame datatype conversion error > --- > > Key: SPARK-41837 > URL: https://issues.apache.org/jira/browse/SPARK-41837 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1594, in pyspark.sql.connect.functions.to_json > Failed example: > df = spark.createDataFrame(data, ("key", "value")) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, in > > df = spark.createDataFrame(data, ("key", "value")) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/session.py", > line 252, in createDataFrame > table = pa.Table.from_pandas(pdf) > File "pyarrow/table.pxi", line 3475, in pyarrow.lib.Table.from_pandas > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 611, > in dataframe_to_arrays > arrays = [convert_column(c, f) > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 611, > in > arrays = [convert_column(c, f) > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 598, > in convert_column > raise e > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 592, > in convert_column > result = pa.array(col, type=type_, from_pandas=True, safe=safe) > File "pyarrow/array.pxi", line 316, in pyarrow.lib.array > File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array > File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status > pyarrow.lib.ArrowInvalid: ("Could not convert 'Alice' with type str: > tried to convert to int64", 'Conversion failed for column 1 with type > object'){code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41837) DataFrame.createDataFrame datatype conversion error
[ https://issues.apache.org/jira/browse/SPARK-41837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676072#comment-17676072 ] Apache Spark commented on SPARK-41837: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > DataFrame.createDataFrame datatype conversion error > --- > > Key: SPARK-41837 > URL: https://issues.apache.org/jira/browse/SPARK-41837 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1594, in pyspark.sql.connect.functions.to_json > Failed example: > df = spark.createDataFrame(data, ("key", "value")) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, in > > df = spark.createDataFrame(data, ("key", "value")) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/session.py", > line 252, in createDataFrame > table = pa.Table.from_pandas(pdf) > File "pyarrow/table.pxi", line 3475, in pyarrow.lib.Table.from_pandas > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 611, > in dataframe_to_arrays > arrays = [convert_column(c, f) > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 611, > in > arrays = [convert_column(c, f) > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 598, > in convert_column > raise e > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 592, > in convert_column > result = pa.array(col, type=type_, from_pandas=True, safe=safe) > File "pyarrow/array.pxi", line 316, in pyarrow.lib.array > File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array > File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status > pyarrow.lib.ArrowInvalid: ("Could not convert 'Alice' with type str: > tried to convert to int64", 'Conversion failed for column 1 with type > object'){code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41837) DataFrame.createDataFrame datatype conversion error
[ https://issues.apache.org/jira/browse/SPARK-41837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676073#comment-17676073 ] Apache Spark commented on SPARK-41837: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > DataFrame.createDataFrame datatype conversion error > --- > > Key: SPARK-41837 > URL: https://issues.apache.org/jira/browse/SPARK-41837 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1594, in pyspark.sql.connect.functions.to_json > Failed example: > df = spark.createDataFrame(data, ("key", "value")) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, in > > df = spark.createDataFrame(data, ("key", "value")) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/session.py", > line 252, in createDataFrame > table = pa.Table.from_pandas(pdf) > File "pyarrow/table.pxi", line 3475, in pyarrow.lib.Table.from_pandas > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 611, > in dataframe_to_arrays > arrays = [convert_column(c, f) > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 611, > in > arrays = [convert_column(c, f) > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 598, > in convert_column > raise e > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 592, > in convert_column > result = pa.array(col, type=type_, from_pandas=True, safe=safe) > File "pyarrow/array.pxi", line 316, in pyarrow.lib.array > File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array > File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status > pyarrow.lib.ArrowInvalid: ("Could not convert 'Alice' with type str: > tried to convert to int64", 'Conversion failed for column 1 with type > object'){code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41838) DataFrame.show() fix map printing
[ https://issues.apache.org/jira/browse/SPARK-41838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676070#comment-17676070 ] Apache Spark commented on SPARK-41838: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > DataFrame.show() fix map printing > - > > Key: SPARK-41838 > URL: https://issues.apache.org/jira/browse/SPARK-41838 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1472, in pyspark.sql.connect.functions.posexplode_outer > Failed example: > df.select("id", "a_map", posexplode_outer("an_array")).show() > Expected: > +---+--+++ > | id| a_map| pos| col| > +---+--+++ > | 1|{x -> 1.0}| 0| foo| > | 1|{x -> 1.0}| 1| bar| > | 2| {}|null|null| > | 3| null|null|null| > +---+--+++ > Got: > +---+--+++ > | id| a_map| pos| col| > +---+--+++ > | 1| {1.0}| 0| foo| > | 1| {1.0}| 1| bar| > | 2|{null}|null|null| > | 3| null|null|null| > +---+--+++ > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-41837) DataFrame.createDataFrame datatype conversion error
[ https://issues.apache.org/jira/browse/SPARK-41837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41837: Assignee: Apache Spark > DataFrame.createDataFrame datatype conversion error > --- > > Key: SPARK-41837 > URL: https://issues.apache.org/jira/browse/SPARK-41837 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Assignee: Apache Spark >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1594, in pyspark.sql.connect.functions.to_json > Failed example: > df = spark.createDataFrame(data, ("key", "value")) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, in > > df = spark.createDataFrame(data, ("key", "value")) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/session.py", > line 252, in createDataFrame > table = pa.Table.from_pandas(pdf) > File "pyarrow/table.pxi", line 3475, in pyarrow.lib.Table.from_pandas > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 611, > in dataframe_to_arrays > arrays = [convert_column(c, f) > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 611, > in > arrays = [convert_column(c, f) > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 598, > in convert_column > raise e > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 592, > in convert_column > result = pa.array(col, type=type_, from_pandas=True, safe=safe) > File "pyarrow/array.pxi", line 316, in pyarrow.lib.array > File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array > File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status > pyarrow.lib.ArrowInvalid: ("Could not convert 'Alice' with type str: > tried to convert to int64", 'Conversion failed for column 1 with type > object'){code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-41837) DataFrame.createDataFrame datatype conversion error
[ https://issues.apache.org/jira/browse/SPARK-41837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41837: Assignee: (was: Apache Spark) > DataFrame.createDataFrame datatype conversion error > --- > > Key: SPARK-41837 > URL: https://issues.apache.org/jira/browse/SPARK-41837 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1594, in pyspark.sql.connect.functions.to_json > Failed example: > df = spark.createDataFrame(data, ("key", "value")) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, in > > df = spark.createDataFrame(data, ("key", "value")) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/session.py", > line 252, in createDataFrame > table = pa.Table.from_pandas(pdf) > File "pyarrow/table.pxi", line 3475, in pyarrow.lib.Table.from_pandas > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 611, > in dataframe_to_arrays > arrays = [convert_column(c, f) > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 611, > in > arrays = [convert_column(c, f) > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 598, > in convert_column > raise e > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 592, > in convert_column > result = pa.array(col, type=type_, from_pandas=True, safe=safe) > File "pyarrow/array.pxi", line 316, in pyarrow.lib.array > File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array > File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status > pyarrow.lib.ArrowInvalid: ("Could not convert 'Alice' with type str: > tried to convert to int64", 'Conversion failed for column 1 with type > object'){code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41837) DataFrame.createDataFrame datatype conversion error
[ https://issues.apache.org/jira/browse/SPARK-41837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676074#comment-17676074 ] Apache Spark commented on SPARK-41837: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > DataFrame.createDataFrame datatype conversion error > --- > > Key: SPARK-41837 > URL: https://issues.apache.org/jira/browse/SPARK-41837 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1594, in pyspark.sql.connect.functions.to_json > Failed example: > df = spark.createDataFrame(data, ("key", "value")) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, in > > df = spark.createDataFrame(data, ("key", "value")) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/session.py", > line 252, in createDataFrame > table = pa.Table.from_pandas(pdf) > File "pyarrow/table.pxi", line 3475, in pyarrow.lib.Table.from_pandas > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 611, > in dataframe_to_arrays > arrays = [convert_column(c, f) > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 611, > in > arrays = [convert_column(c, f) > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 598, > in convert_column > raise e > File > "/usr/local/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 592, > in convert_column > result = pa.array(col, type=type_, from_pandas=True, safe=safe) > File "pyarrow/array.pxi", line 316, in pyarrow.lib.array > File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array > File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status > pyarrow.lib.ArrowInvalid: ("Could not convert 'Alice' with type str: > tried to convert to int64", 'Conversion failed for column 1 with type > object'){code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-41835) Implement `transform_keys` function
[ https://issues.apache.org/jira/browse/SPARK-41835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41835: Assignee: (was: Apache Spark) > Implement `transform_keys` function > --- > > Key: SPARK-41835 > URL: https://issues.apache.org/jira/browse/SPARK-41835 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1611, in pyspark.sql.connect.functions.transform_keys > Failed example: > df.select(transform_keys( > "data", lambda k, _: upper(k)).alias("data_upper") > ).show(truncate=False) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line > 1, in > df.select(transform_keys( > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve > "transform_keys(data, lambdafunction(upper(x_11), x_11, y_12))" due to data > type mismatch: Parameter 1 requires the "MAP" type, however "data" has the > type "STRUCT". > Plan: 'Project [transform_keys(data#4493, lambdafunction('upper(lambda > 'x_11), lambda 'x_11, lambda 'y_12, false)) AS data_upper#4496] > +- Project [0#4488L AS id#4492L, 1#4489 AS data#4493] > +- LocalRelation [0#4488L, 1#4489] {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41835) Implement `transform_keys` function
[ https://issues.apache.org/jira/browse/SPARK-41835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676075#comment-17676075 ] Apache Spark commented on SPARK-41835: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > Implement `transform_keys` function > --- > > Key: SPARK-41835 > URL: https://issues.apache.org/jira/browse/SPARK-41835 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1611, in pyspark.sql.connect.functions.transform_keys > Failed example: > df.select(transform_keys( > "data", lambda k, _: upper(k)).alias("data_upper") > ).show(truncate=False) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line > 1, in > df.select(transform_keys( > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve > "transform_keys(data, lambdafunction(upper(x_11), x_11, y_12))" due to data > type mismatch: Parameter 1 requires the "MAP" type, however "data" has the > type "STRUCT". > Plan: 'Project [transform_keys(data#4493, lambdafunction('upper(lambda > 'x_11), lambda 'x_11, lambda 'y_12, false)) AS data_upper#4496] > +- Project [0#4488L AS id#4492L, 1#4489 AS data#4493] > +- LocalRelation [0#4488L, 1#4489] {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-41835) Implement `transform_keys` function
[ https://issues.apache.org/jira/browse/SPARK-41835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41835: Assignee: Apache Spark > Implement `transform_keys` function > --- > > Key: SPARK-41835 > URL: https://issues.apache.org/jira/browse/SPARK-41835 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Assignee: Apache Spark >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1611, in pyspark.sql.connect.functions.transform_keys > Failed example: > df.select(transform_keys( > "data", lambda k, _: upper(k)).alias("data_upper") > ).show(truncate=False) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line > 1, in > df.select(transform_keys( > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve > "transform_keys(data, lambdafunction(upper(x_11), x_11, y_12))" due to data > type mismatch: Parameter 1 requires the "MAP" type, however "data" has the > type "STRUCT". > Plan: 'Project [transform_keys(data#4493, lambdafunction('upper(lambda > 'x_11), lambda 'x_11, lambda 'y_12, false)) AS data_upper#4496] > +- Project [0#4488L AS id#4492L, 1#4489 AS data#4493] > +- LocalRelation [0#4488L, 1#4489] {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41835) Implement `transform_keys` function
[ https://issues.apache.org/jira/browse/SPARK-41835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676076#comment-17676076 ] Apache Spark commented on SPARK-41835: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > Implement `transform_keys` function > --- > > Key: SPARK-41835 > URL: https://issues.apache.org/jira/browse/SPARK-41835 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1611, in pyspark.sql.connect.functions.transform_keys > Failed example: > df.select(transform_keys( > "data", lambda k, _: upper(k)).alias("data_upper") > ).show(truncate=False) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line > 1, in > df.select(transform_keys( > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve > "transform_keys(data, lambdafunction(upper(x_11), x_11, y_12))" due to data > type mismatch: Parameter 1 requires the "MAP" type, however "data" has the > type "STRUCT". > Plan: 'Project [transform_keys(data#4493, lambdafunction('upper(lambda > 'x_11), lambda 'x_11, lambda 'y_12, false)) AS data_upper#4496] > +- Project [0#4488L AS id#4492L, 1#4489 AS data#4493] > +- LocalRelation [0#4488L, 1#4489] {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41835) Implement `transform_keys` function
[ https://issues.apache.org/jira/browse/SPARK-41835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676078#comment-17676078 ] Apache Spark commented on SPARK-41835: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > Implement `transform_keys` function > --- > > Key: SPARK-41835 > URL: https://issues.apache.org/jira/browse/SPARK-41835 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1611, in pyspark.sql.connect.functions.transform_keys > Failed example: > df.select(transform_keys( > "data", lambda k, _: upper(k)).alias("data_upper") > ).show(truncate=False) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line > 1, in > df.select(transform_keys( > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve > "transform_keys(data, lambdafunction(upper(x_11), x_11, y_12))" due to data > type mismatch: Parameter 1 requires the "MAP" type, however "data" has the > type "STRUCT". > Plan: 'Project [transform_keys(data#4493, lambdafunction('upper(lambda > 'x_11), lambda 'x_11, lambda 'y_12, false)) AS data_upper#4496] > +- Project [0#4488L AS id#4492L, 1#4489 AS data#4493] > +- LocalRelation [0#4488L, 1#4489] {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-41836) Implement `transform_values` function
[ https://issues.apache.org/jira/browse/SPARK-41836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41836: Assignee: Apache Spark (was: Ruifeng Zheng) > Implement `transform_values` function > - > > Key: SPARK-41836 > URL: https://issues.apache.org/jira/browse/SPARK-41836 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Assignee: Apache Spark >Priority: Major > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41835) Implement `transform_keys` function
[ https://issues.apache.org/jira/browse/SPARK-41835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676082#comment-17676082 ] Apache Spark commented on SPARK-41835: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > Implement `transform_keys` function > --- > > Key: SPARK-41835 > URL: https://issues.apache.org/jira/browse/SPARK-41835 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1611, in pyspark.sql.connect.functions.transform_keys > Failed example: > df.select(transform_keys( > "data", lambda k, _: upper(k)).alias("data_upper") > ).show(truncate=False) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line > 1, in > df.select(transform_keys( > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve > "transform_keys(data, lambdafunction(upper(x_11), x_11, y_12))" due to data > type mismatch: Parameter 1 requires the "MAP" type, however "data" has the > type "STRUCT". > Plan: 'Project [transform_keys(data#4493, lambdafunction('upper(lambda > 'x_11), lambda 'x_11, lambda 'y_12, false)) AS data_upper#4496] > +- Project [0#4488L AS id#4492L, 1#4489 AS data#4493] > +- LocalRelation [0#4488L, 1#4489] {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-41836) Implement `transform_values` function
[ https://issues.apache.org/jira/browse/SPARK-41836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41836: Assignee: Ruifeng Zheng (was: Apache Spark) > Implement `transform_values` function > - > > Key: SPARK-41836 > URL: https://issues.apache.org/jira/browse/SPARK-41836 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Assignee: Ruifeng Zheng >Priority: Major > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41835) Implement `transform_keys` function
[ https://issues.apache.org/jira/browse/SPARK-41835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676083#comment-17676083 ] Apache Spark commented on SPARK-41835: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > Implement `transform_keys` function > --- > > Key: SPARK-41835 > URL: https://issues.apache.org/jira/browse/SPARK-41835 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1611, in pyspark.sql.connect.functions.transform_keys > Failed example: > df.select(transform_keys( > "data", lambda k, _: upper(k)).alias("data_upper") > ).show(truncate=False) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line > 1, in > df.select(transform_keys( > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] Cannot resolve > "transform_keys(data, lambdafunction(upper(x_11), x_11, y_12))" due to data > type mismatch: Parameter 1 requires the "MAP" type, however "data" has the > type "STRUCT". > Plan: 'Project [transform_keys(data#4493, lambdafunction('upper(lambda > 'x_11), lambda 'x_11, lambda 'y_12, false)) AS data_upper#4496] > +- Project [0#4488L AS id#4492L, 1#4489 AS data#4493] > +- LocalRelation [0#4488L, 1#4489] {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-41847) DataFrame mapfield,structlist invalid type
[ https://issues.apache.org/jira/browse/SPARK-41847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41847: Assignee: (was: Apache Spark) > DataFrame mapfield,structlist invalid type > -- > > Key: SPARK-41847 > URL: https://issues.apache.org/jira/browse/SPARK-41847 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1270, in pyspark.sql.connect.functions.explode > Failed example: > eDF.select(explode(eDF.mapfield).alias("key", "value")).show() > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, in > > eDF.select(explode(eDF.mapfield).alias("key", "value")).show() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [INVALID_COLUMN_OR_FIELD_DATA_TYPE] Column or field `mapfield` is of type > "STRUCT" while it's required to be "MAP". > Plan: {code} > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1364, in pyspark.sql.connect.functions.inline > Failed example: > df.select(inline(df.structlist)).show() > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, in > > df.select(inline(df.structlist)).show() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [INVALID_COLUMN_OR_FIELD_DATA_TYPE] Column or field `structlist`.`element` is > of type "ARRAY" while it's required to be "STRUCT BIGINT>". > Plan: {code} > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1411, in pyspark.sql.connect.functions.map_filter > Failed example: > df.select(map_filter( > "data", lambda _, v: v > 30.0).alias("data_filtered") > ).show(truncate=False) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, > in > df.select(map_filter( > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n,
[jira] [Assigned] (SPARK-41847) DataFrame mapfield,structlist invalid type
[ https://issues.apache.org/jira/browse/SPARK-41847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-41847: Assignee: Apache Spark > DataFrame mapfield,structlist invalid type > -- > > Key: SPARK-41847 > URL: https://issues.apache.org/jira/browse/SPARK-41847 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Assignee: Apache Spark >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1270, in pyspark.sql.connect.functions.explode > Failed example: > eDF.select(explode(eDF.mapfield).alias("key", "value")).show() > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, in > > eDF.select(explode(eDF.mapfield).alias("key", "value")).show() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [INVALID_COLUMN_OR_FIELD_DATA_TYPE] Column or field `mapfield` is of type > "STRUCT" while it's required to be "MAP". > Plan: {code} > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1364, in pyspark.sql.connect.functions.inline > Failed example: > df.select(inline(df.structlist)).show() > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, in > > df.select(inline(df.structlist)).show() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [INVALID_COLUMN_OR_FIELD_DATA_TYPE] Column or field `structlist`.`element` is > of type "ARRAY" while it's required to be "STRUCT BIGINT>". > Plan: {code} > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1411, in pyspark.sql.connect.functions.map_filter > Failed example: > df.select(map_filter( > "data", lambda _, v: v > 30.0).alias("data_filtered") > ).show(truncate=False) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, > in > df.select(map_filter( > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > p
[jira] [Commented] (SPARK-41836) Implement `transform_values` function
[ https://issues.apache.org/jira/browse/SPARK-41836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676080#comment-17676080 ] Apache Spark commented on SPARK-41836: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > Implement `transform_values` function > - > > Key: SPARK-41836 > URL: https://issues.apache.org/jira/browse/SPARK-41836 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Assignee: Ruifeng Zheng >Priority: Major > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41836) Implement `transform_values` function
[ https://issues.apache.org/jira/browse/SPARK-41836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676085#comment-17676085 ] Apache Spark commented on SPARK-41836: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > Implement `transform_values` function > - > > Key: SPARK-41836 > URL: https://issues.apache.org/jira/browse/SPARK-41836 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Assignee: Ruifeng Zheng >Priority: Major > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41847) DataFrame mapfield,structlist invalid type
[ https://issues.apache.org/jira/browse/SPARK-41847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676090#comment-17676090 ] Apache Spark commented on SPARK-41847: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > DataFrame mapfield,structlist invalid type > -- > > Key: SPARK-41847 > URL: https://issues.apache.org/jira/browse/SPARK-41847 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1270, in pyspark.sql.connect.functions.explode > Failed example: > eDF.select(explode(eDF.mapfield).alias("key", "value")).show() > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, in > > eDF.select(explode(eDF.mapfield).alias("key", "value")).show() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [INVALID_COLUMN_OR_FIELD_DATA_TYPE] Column or field `mapfield` is of type > "STRUCT" while it's required to be "MAP". > Plan: {code} > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1364, in pyspark.sql.connect.functions.inline > Failed example: > df.select(inline(df.structlist)).show() > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, in > > df.select(inline(df.structlist)).show() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [INVALID_COLUMN_OR_FIELD_DATA_TYPE] Column or field `structlist`.`element` is > of type "ARRAY" while it's required to be "STRUCT BIGINT>". > Plan: {code} > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1411, in pyspark.sql.connect.functions.map_filter > Failed example: > df.select(map_filter( > "data", lambda _, v: v > 30.0).alias("data_filtered") > ).show(truncate=False) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, > in > df.select(map_filter( > File > "/Users/s.singh
[jira] [Commented] (SPARK-41847) DataFrame mapfield,structlist invalid type
[ https://issues.apache.org/jira/browse/SPARK-41847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676094#comment-17676094 ] Apache Spark commented on SPARK-41847: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > DataFrame mapfield,structlist invalid type > -- > > Key: SPARK-41847 > URL: https://issues.apache.org/jira/browse/SPARK-41847 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1270, in pyspark.sql.connect.functions.explode > Failed example: > eDF.select(explode(eDF.mapfield).alias("key", "value")).show() > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, in > > eDF.select(explode(eDF.mapfield).alias("key", "value")).show() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [INVALID_COLUMN_OR_FIELD_DATA_TYPE] Column or field `mapfield` is of type > "STRUCT" while it's required to be "MAP". > Plan: {code} > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1364, in pyspark.sql.connect.functions.inline > Failed example: > df.select(inline(df.structlist)).show() > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, in > > df.select(inline(df.structlist)).show() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [INVALID_COLUMN_OR_FIELD_DATA_TYPE] Column or field `structlist`.`element` is > of type "ARRAY" while it's required to be "STRUCT BIGINT>". > Plan: {code} > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1411, in pyspark.sql.connect.functions.map_filter > Failed example: > df.select(map_filter( > "data", lambda _, v: v > 30.0).alias("data_filtered") > ).show(truncate=False) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, > in > df.select(map_filter( > File > "/Users/s.singh
[jira] [Commented] (SPARK-41847) DataFrame mapfield,structlist invalid type
[ https://issues.apache.org/jira/browse/SPARK-41847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676099#comment-17676099 ] Apache Spark commented on SPARK-41847: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > DataFrame mapfield,structlist invalid type > -- > > Key: SPARK-41847 > URL: https://issues.apache.org/jira/browse/SPARK-41847 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1270, in pyspark.sql.connect.functions.explode > Failed example: > eDF.select(explode(eDF.mapfield).alias("key", "value")).show() > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, in > > eDF.select(explode(eDF.mapfield).alias("key", "value")).show() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [INVALID_COLUMN_OR_FIELD_DATA_TYPE] Column or field `mapfield` is of type > "STRUCT" while it's required to be "MAP". > Plan: {code} > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1364, in pyspark.sql.connect.functions.inline > Failed example: > df.select(inline(df.structlist)).show() > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, in > > df.select(inline(df.structlist)).show() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [INVALID_COLUMN_OR_FIELD_DATA_TYPE] Column or field `structlist`.`element` is > of type "ARRAY" while it's required to be "STRUCT BIGINT>". > Plan: {code} > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1411, in pyspark.sql.connect.functions.map_filter > Failed example: > df.select(map_filter( > "data", lambda _, v: v > 30.0).alias("data_filtered") > ).show(truncate=False) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, > in > df.select(map_filter( > File > "/Users/s.singh
[jira] [Commented] (SPARK-41836) Implement `transform_values` function
[ https://issues.apache.org/jira/browse/SPARK-41836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676096#comment-17676096 ] Apache Spark commented on SPARK-41836: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > Implement `transform_values` function > - > > Key: SPARK-41836 > URL: https://issues.apache.org/jira/browse/SPARK-41836 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Assignee: Ruifeng Zheng >Priority: Major > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41836) Implement `transform_values` function
[ https://issues.apache.org/jira/browse/SPARK-41836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676093#comment-17676093 ] Apache Spark commented on SPARK-41836: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > Implement `transform_values` function > - > > Key: SPARK-41836 > URL: https://issues.apache.org/jira/browse/SPARK-41836 > Project: Spark > Issue Type: Sub-task > Components: Connect, PySpark >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Assignee: Ruifeng Zheng >Priority: Major > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41847) DataFrame mapfield,structlist invalid type
[ https://issues.apache.org/jira/browse/SPARK-41847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676100#comment-17676100 ] Apache Spark commented on SPARK-41847: -- User 'zhengruifeng' has created a pull request for this issue: https://github.com/apache/spark/pull/39535 > DataFrame mapfield,structlist invalid type > -- > > Key: SPARK-41847 > URL: https://issues.apache.org/jira/browse/SPARK-41847 > Project: Spark > Issue Type: Sub-task > Components: Connect >Affects Versions: 3.4.0 >Reporter: Sandeep Singh >Priority: Major > > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1270, in pyspark.sql.connect.functions.explode > Failed example: > eDF.select(explode(eDF.mapfield).alias("key", "value")).show() > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, in > > eDF.select(explode(eDF.mapfield).alias("key", "value")).show() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [INVALID_COLUMN_OR_FIELD_DATA_TYPE] Column or field `mapfield` is of type > "STRUCT" while it's required to be "MAP". > Plan: {code} > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1364, in pyspark.sql.connect.functions.inline > Failed example: > df.select(inline(df.structlist)).show() > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, in > > df.select(inline(df.structlist)).show() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 534, in show > print(self._show_string(n, truncate, vertical)) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 423, in _show_string > ).toPandas() > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/dataframe.py", > line 1031, in toPandas > return self._session.client.to_pandas(query) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 413, in to_pandas > return self._execute_and_fetch(req) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 573, in _execute_and_fetch > self._handle_error(rpc_error) > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/client.py", > line 619, in _handle_error > raise SparkConnectAnalysisException( > pyspark.sql.connect.client.SparkConnectAnalysisException: > [INVALID_COLUMN_OR_FIELD_DATA_TYPE] Column or field `structlist`.`element` is > of type "ARRAY" while it's required to be "STRUCT BIGINT>". > Plan: {code} > {code:java} > File > "/Users/s.singh/personal/spark-oss/python/pyspark/sql/connect/functions.py", > line 1411, in pyspark.sql.connect.functions.map_filter > Failed example: > df.select(map_filter( > "data", lambda _, v: v > 30.0).alias("data_filtered") > ).show(truncate=False) > Exception raised: > Traceback (most recent call last): > File > "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/doctest.py", > line 1350, in __run > exec(compile(example.source, filename, "single", > File "", line 1, > in > df.select(map_filter( > File > "/Users/s.singh
[jira] [Created] (SPARK-42033) Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline
Pankaj Nagla created SPARK-42033: Summary: Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline Key: SPARK-42033 URL: https://issues.apache.org/jira/browse/SPARK-42033 Project: Spark Issue Type: Bug Components: Deploy Affects Versions: 2.1.1 Reporter: Pankaj Nagla Fix For: 1.6.2 I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training|[https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/]] follow the page. Thanks -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-42033) Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline
[ https://issues.apache.org/jira/browse/SPARK-42033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Nagla updated SPARK-42033: - Description: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [#Aws Sysops Training][https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/] follow the page. Thanks was: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training|[https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/]] follow the page. Thanks > Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline > > > Key: SPARK-42033 > URL: https://issues.apache.org/jira/browse/SPARK-42033 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 2.1.1 >Reporter: Pankaj Nagla >Priority: Major > Fix For: 1.6.2 > > > I'm going through the "Scalable FastAPI Application on AWS" course. My > gitlab-ci.yml file is below. > stages: > - docker > variables: > DOCKER_DRIVER: overlay2 > DOCKER_TLS_CERTDIR: "/certs" > cache: > key: ${CI_JOB_NAME} > paths: > - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ > build-python-ci-image: > image: docker:19.03.0 > services: > - docker:19.03.0-dind > stage: docker > before_script: > - cd ci_cd/python/ > script: > - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > $CI_REGISTRY > - docker build -t > registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . > - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > My Pipeline fails with this error: > See > [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] > Login Succeeded > $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > . > invalid argument > "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" > flag: invalid reference format > See 'docker build --help'. > Cleaning up project directory and file based variables > ERROR: Job failed: exit code 125 > It may or may not be relevant but the Container Registry for the GitLab > project says there's a Docker connection error. All these problems have been > discussed in this [#Aws Sysops > Training][https://www.igmg
[jira] [Updated] (SPARK-42033) Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline
[ https://issues.apache.org/jira/browse/SPARK-42033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Nagla updated SPARK-42033: - Description: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [[Aws Sysops Training||#Aws Sysops Training] [https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/] []|#Aws Sysops Training] follow the page. Thanks was: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [#Aws Sysops Training] follow the page. Thanks > Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline > > > Key: SPARK-42033 > URL: https://issues.apache.org/jira/browse/SPARK-42033 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 2.1.1 >Reporter: Pankaj Nagla >Priority: Major > Fix For: 1.6.2 > > > I'm going through the "Scalable FastAPI Application on AWS" course. My > gitlab-ci.yml file is below. > stages: > - docker > variables: > DOCKER_DRIVER: overlay2 > DOCKER_TLS_CERTDIR: "/certs" > cache: > key: ${CI_JOB_NAME} > paths: > - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ > build-python-ci-image: > image: docker:19.03.0 > services: > - docker:19.03.0-dind > stage: docker > before_script: > - cd ci_cd/python/ > script: > - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > $CI_REGISTRY > - docker build -t > registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . > - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > My Pipeline fails with this error: > See > [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] > Login Succeeded > $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > . > invalid argument > "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" > flag: invalid reference format > See 'docker build --help'. > Cleaning up project directory and file based variables > ERROR: Job failed: exit code 125 > It may or may not be relevant but the Container Registry for the GitLab > project says there's a Docker connection error. All these problems have been > discussed in this [[Aws Sysops Training||#Aws Sysops Training] > [https://www.igmguru.co
[jira] [Updated] (SPARK-42033) Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline
[ https://issues.apache.org/jira/browse/SPARK-42033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Nagla updated SPARK-42033: - Description: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [#Aws Sysops Training] follow the page. Thanks was: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [#Aws Sysops Training][https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/] follow the page. Thanks > Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline > > > Key: SPARK-42033 > URL: https://issues.apache.org/jira/browse/SPARK-42033 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 2.1.1 >Reporter: Pankaj Nagla >Priority: Major > Fix For: 1.6.2 > > > I'm going through the "Scalable FastAPI Application on AWS" course. My > gitlab-ci.yml file is below. > stages: > - docker > variables: > DOCKER_DRIVER: overlay2 > DOCKER_TLS_CERTDIR: "/certs" > cache: > key: ${CI_JOB_NAME} > paths: > - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ > build-python-ci-image: > image: docker:19.03.0 > services: > - docker:19.03.0-dind > stage: docker > before_script: > - cd ci_cd/python/ > script: > - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > $CI_REGISTRY > - docker build -t > registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . > - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > My Pipeline fails with this error: > See > [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] > Login Succeeded > $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > . > invalid argument > "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" > flag: invalid reference format > See 'docker build --help'. > Cleaning up project directory and file based variables > ERROR: Job failed: exit code 125 > It may or may not be relevant but the Container Registry for the GitLab > project says there's a Docker connection error. All these problems have been > discussed in this [#Aws Sysops Training] follow the page. > Thanks -- This message was sent by Atlassian Jira (v8.20.10#820010) -
[jira] [Updated] (SPARK-42033) Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline
[ https://issues.apache.org/jira/browse/SPARK-42033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Nagla updated SPARK-42033: - Description: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training|[https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/]] follow the page. Thanks was: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [[Aws Sysops Training||#Aws Sysops Training] [https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/] []|#Aws Sysops Training] follow the page. Thanks > Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline > > > Key: SPARK-42033 > URL: https://issues.apache.org/jira/browse/SPARK-42033 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 2.1.1 >Reporter: Pankaj Nagla >Priority: Major > Fix For: 1.6.2 > > > I'm going through the "Scalable FastAPI Application on AWS" course. My > gitlab-ci.yml file is below. > stages: > - docker > variables: > DOCKER_DRIVER: overlay2 > DOCKER_TLS_CERTDIR: "/certs" > cache: > key: ${CI_JOB_NAME} > paths: > - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ > build-python-ci-image: > image: docker:19.03.0 > services: > - docker:19.03.0-dind > stage: docker > before_script: > - cd ci_cd/python/ > script: > - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > $CI_REGISTRY > - docker build -t > registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . > - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > My Pipeline fails with this error: > See > [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] > Login Succeeded > $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > . > invalid argument > "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" > flag: invalid reference format > See 'docker build --help'. > Cleaning up project directory and file based variables > ERROR: Job failed: exit code 125 > It may or may not be relevant but the Container Registry for the GitLab > project says there's a Docker connection error. All these problems have been > discussed i
[jira] [Updated] (SPARK-42033) Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline
[ https://issues.apache.org/jira/browse/SPARK-42033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Nagla updated SPARK-42033: - Description: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training][[https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/]] follow the page. Thanks was: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training|[https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/]] follow the page. Thanks > Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline > > > Key: SPARK-42033 > URL: https://issues.apache.org/jira/browse/SPARK-42033 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 2.1.1 >Reporter: Pankaj Nagla >Priority: Major > Fix For: 1.6.2 > > > I'm going through the "Scalable FastAPI Application on AWS" course. My > gitlab-ci.yml file is below. > stages: > - docker > variables: > DOCKER_DRIVER: overlay2 > DOCKER_TLS_CERTDIR: "/certs" > cache: > key: ${CI_JOB_NAME} > paths: > - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ > build-python-ci-image: > image: docker:19.03.0 > services: > - docker:19.03.0-dind > stage: docker > before_script: > - cd ci_cd/python/ > script: > - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > $CI_REGISTRY > - docker build -t > registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . > - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > My Pipeline fails with this error: > See > [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] > Login Succeeded > $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > . > invalid argument > "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" > flag: invalid reference format > See 'docker build --help'. > Cleaning up project directory and file based variables > ERROR: Job failed: exit code 125 > It may or may not be relevant but the Container Registry for the GitLab > project says there's a Docker connection error. All these problems have been > discussed in this [Aws Sysops > Training][[https://www.ig
[jira] [Updated] (SPARK-42033) Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline
[ https://issues.apache.org/jira/browse/SPARK-42033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Nagla updated SPARK-42033: - Description: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training|[https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/]] follow the page. Thanks was: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training][[https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/]] follow the page. Thanks > Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline > > > Key: SPARK-42033 > URL: https://issues.apache.org/jira/browse/SPARK-42033 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 2.1.1 >Reporter: Pankaj Nagla >Priority: Major > Fix For: 1.6.2 > > > I'm going through the "Scalable FastAPI Application on AWS" course. My > gitlab-ci.yml file is below. > stages: > - docker > variables: > DOCKER_DRIVER: overlay2 > DOCKER_TLS_CERTDIR: "/certs" > cache: > key: ${CI_JOB_NAME} > paths: > - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ > build-python-ci-image: > image: docker:19.03.0 > services: > - docker:19.03.0-dind > stage: docker > before_script: > - cd ci_cd/python/ > script: > - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > $CI_REGISTRY > - docker build -t > registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . > - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > My Pipeline fails with this error: > See > [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] > Login Succeeded > $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > . > invalid argument > "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" > flag: invalid reference format > See 'docker build --help'. > Cleaning up project directory and file based variables > ERROR: Job failed: exit code 125 > It may or may not be relevant but the Container Registry for the GitLab > project says there's a Docker connection error. All these problems have been > discussed in this [Aws Sysops > Training|[https://www.igm
[jira] [Updated] (SPARK-42033) Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline
[ https://issues.apache.org/jira/browse/SPARK-42033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Nagla updated SPARK-42033: - Description: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training|[https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/]] follow the page. Thanks [url=https://www.igmguru.com/digital-marketing-programming/mulesoft-training/]Mule 4 Training[/url] https://www.igmguru.com/salesforce/salesforce-commerce-cloud-certification-training/";>Salesforce Demandware Training was: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training|[https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/]] follow the page. Thanks > Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline > > > Key: SPARK-42033 > URL: https://issues.apache.org/jira/browse/SPARK-42033 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 2.1.1 >Reporter: Pankaj Nagla >Priority: Major > Fix For: 1.6.2 > > > I'm going through the "Scalable FastAPI Application on AWS" course. My > gitlab-ci.yml file is below. > stages: > - docker > variables: > DOCKER_DRIVER: overlay2 > DOCKER_TLS_CERTDIR: "/certs" > cache: > key: ${CI_JOB_NAME} > paths: > - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ > build-python-ci-image: > image: docker:19.03.0 > services: > - docker:19.03.0-dind > stage: docker > before_script: > - cd ci_cd/python/ > script: > - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > $CI_REGISTRY > - docker build -t > registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . > - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > My Pipeline fails with this error: > See > [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] > Login Succeeded > $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > . > invalid argument > "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" > flag: invalid reference format > See 'docker build --help'. > Cleaning up project directory and file based variables > ERROR: Job failed: exit cod
[jira] [Commented] (SPARK-36263) Add Dataset.observe(Observation, Column, Column*) to PySpark
[ https://issues.apache.org/jira/browse/SPARK-36263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676265#comment-17676265 ] Nick Hryhoriev commented on SPARK-36263: Hi, I find this feature does not work for `foreach` or `foreachPartition` actions. Maybe because they use `rdd.foreach` under the hood. Look like it's because `foreach` or `foreachPartition` actions do not work with `QueryExecutionListener`. Example to reproduce [https://gist.github.com/GrigorievNick/e7cf9ec5584b417d9719e2812722e6d3] Do I miss something or is it's a know issue? > Add Dataset.observe(Observation, Column, Column*) to PySpark > > > Key: SPARK-36263 > URL: https://issues.apache.org/jira/browse/SPARK-36263 > Project: Spark > Issue Type: New Feature > Components: PySpark >Affects Versions: 3.3.0 >Reporter: Enrico Minack >Assignee: Enrico Minack >Priority: Major > Fix For: 3.3.0 > > > With SPARK-34806 we now have a way to use the `Dataset.observe` method > without the need to interact with > `org.apache.spark.sql.util.QueryExecutionListener`. This allows us to easily > retrieve observations in PySpark. > Adding a `Dataset.observe(Observation, Column, Column*)` equivalent to > PySpark's `DataFrame` is straightforward while it allows to utilise > observations from Python. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-42033) Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline
[ https://issues.apache.org/jira/browse/SPARK-42033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Nagla updated SPARK-42033: - Description: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training|[https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/] follow the page. Thanks was: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training|[https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/]] follow the page. Thanks [url=https://www.igmguru.com/digital-marketing-programming/mulesoft-training/]Mule 4 Training[/url] https://www.igmguru.com/salesforce/salesforce-commerce-cloud-certification-training/";>Salesforce Demandware Training > Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline > > > Key: SPARK-42033 > URL: https://issues.apache.org/jira/browse/SPARK-42033 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 2.1.1 >Reporter: Pankaj Nagla >Priority: Major > Fix For: 1.6.2 > > > I'm going through the "Scalable FastAPI Application on AWS" course. My > gitlab-ci.yml file is below. > stages: > - docker > variables: > DOCKER_DRIVER: overlay2 > DOCKER_TLS_CERTDIR: "/certs" > cache: > key: ${CI_JOB_NAME} > paths: > - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ > build-python-ci-image: > image: docker:19.03.0 > services: > - docker:19.03.0-dind > stage: docker > before_script: > - cd ci_cd/python/ > script: > - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > $CI_REGISTRY > - docker build -t > registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . > - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > My Pipeline fails with this error: > See > [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] > Login Succeeded > $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > . > invalid argument > "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" > flag: invalid reference format > See 'docker build --help'. > Cleaning up project directory and file based variables > ERROR: Job failed: exit c
[jira] [Updated] (SPARK-42033) Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline
[ https://issues.apache.org/jira/browse/SPARK-42033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Nagla updated SPARK-42033: - Description: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training]([https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/)|https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/] follow the page. Thanks was: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training|[https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/] follow the page. Thanks > Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline > > > Key: SPARK-42033 > URL: https://issues.apache.org/jira/browse/SPARK-42033 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 2.1.1 >Reporter: Pankaj Nagla >Priority: Major > Fix For: 1.6.2 > > > I'm going through the "Scalable FastAPI Application on AWS" course. My > gitlab-ci.yml file is below. > stages: > - docker > variables: > DOCKER_DRIVER: overlay2 > DOCKER_TLS_CERTDIR: "/certs" > cache: > key: ${CI_JOB_NAME} > paths: > - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ > build-python-ci-image: > image: docker:19.03.0 > services: > - docker:19.03.0-dind > stage: docker > before_script: > - cd ci_cd/python/ > script: > - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > $CI_REGISTRY > - docker build -t > registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . > - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > My Pipeline fails with this error: > See > [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] > Login Succeeded > $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > . > invalid argument > "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" > flag: invalid reference format > See 'docker build --help'. > Cleaning up project directory and file based variables > ERROR: Job failed: exit code 125 > It may or may not be relevant but the Container Registry for the GitLab > project says there's a Docker connection error. All these
[jira] [Updated] (SPARK-42033) Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline
[ https://issues.apache.org/jira/browse/SPARK-42033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Nagla updated SPARK-42033: - Description: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [#Aws Sysops Training][https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/] follow the page. Thanks was: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training]([https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/)|https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/] follow the page. Thanks > Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline > > > Key: SPARK-42033 > URL: https://issues.apache.org/jira/browse/SPARK-42033 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 2.1.1 >Reporter: Pankaj Nagla >Priority: Major > Fix For: 1.6.2 > > > I'm going through the "Scalable FastAPI Application on AWS" course. My > gitlab-ci.yml file is below. > stages: > - docker > variables: > DOCKER_DRIVER: overlay2 > DOCKER_TLS_CERTDIR: "/certs" > cache: > key: ${CI_JOB_NAME} > paths: > - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ > build-python-ci-image: > image: docker:19.03.0 > services: > - docker:19.03.0-dind > stage: docker > before_script: > - cd ci_cd/python/ > script: > - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > $CI_REGISTRY > - docker build -t > registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . > - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > My Pipeline fails with this error: > See > [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] > Login Succeeded > $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > . > invalid argument > "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" > flag: invalid reference format > See 'docker build --help'. > Cleaning up project directory and file based variables > ERROR: Job failed: exit code 125 > It may or may not be relevant but the Container Registry for the GitLab > project says there's a Docker connection error. All these
[jira] [Updated] (SPARK-42033) Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline
[ https://issues.apache.org/jira/browse/SPARK-42033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Nagla updated SPARK-42033: - Description: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [[Aws Sysops Training||#Aws Sysops Training] [https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/] []|#Aws Sysops Training] follow the page. Thanks was: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [#Aws Sysops Training][https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/] follow the page. Thanks > Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline > > > Key: SPARK-42033 > URL: https://issues.apache.org/jira/browse/SPARK-42033 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 2.1.1 >Reporter: Pankaj Nagla >Priority: Major > Fix For: 1.6.2 > > > I'm going through the "Scalable FastAPI Application on AWS" course. My > gitlab-ci.yml file is below. > stages: > - docker > variables: > DOCKER_DRIVER: overlay2 > DOCKER_TLS_CERTDIR: "/certs" > cache: > key: ${CI_JOB_NAME} > paths: > - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ > build-python-ci-image: > image: docker:19.03.0 > services: > - docker:19.03.0-dind > stage: docker > before_script: > - cd ci_cd/python/ > script: > - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > $CI_REGISTRY > - docker build -t > registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . > - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > My Pipeline fails with this error: > See > [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] > Login Succeeded > $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > . > invalid argument > "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" > flag: invalid reference format > See 'docker build --help'. > Cleaning up project directory and file based variables > ERROR: Job failed: exit code 125 > It may or may not be relevant but the Container Registry for the GitLab > project says there's a Docker connection error. All these problems have been > discu
[jira] [Updated] (SPARK-42033) Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline
[ https://issues.apache.org/jira/browse/SPARK-42033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Nagla updated SPARK-42033: - Description: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training| [https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/] []#Aws Sysops Training] follow the page. Thanks was: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [[Aws Sysops Training||#Aws Sysops Training] [https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/] []|#Aws Sysops Training] follow the page. Thanks > Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline > > > Key: SPARK-42033 > URL: https://issues.apache.org/jira/browse/SPARK-42033 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 2.1.1 >Reporter: Pankaj Nagla >Priority: Major > Fix For: 1.6.2 > > > I'm going through the "Scalable FastAPI Application on AWS" course. My > gitlab-ci.yml file is below. > stages: > - docker > variables: > DOCKER_DRIVER: overlay2 > DOCKER_TLS_CERTDIR: "/certs" > cache: > key: ${CI_JOB_NAME} > paths: > - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ > build-python-ci-image: > image: docker:19.03.0 > services: > - docker:19.03.0-dind > stage: docker > before_script: > - cd ci_cd/python/ > script: > - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > $CI_REGISTRY > - docker build -t > registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . > - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > My Pipeline fails with this error: > See > [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] > Login Succeeded > $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > . > invalid argument > "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" > flag: invalid reference format > See 'docker build --help'. > Cleaning up project directory and file based variables > ERROR: Job failed: exit code 125 > It may or may not be relevant but the Container Registry for the GitLab > project says there's a Docker connection error. All these pro
[jira] [Updated] (SPARK-42033) Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline
[ https://issues.apache.org/jira/browse/SPARK-42033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Nagla updated SPARK-42033: - Description: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training|[https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/ |https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/]follow the page. Thanks was: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training| [https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/] []#Aws Sysops Training] follow the page. Thanks > Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline > > > Key: SPARK-42033 > URL: https://issues.apache.org/jira/browse/SPARK-42033 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 2.1.1 >Reporter: Pankaj Nagla >Priority: Major > Fix For: 1.6.2 > > > I'm going through the "Scalable FastAPI Application on AWS" course. My > gitlab-ci.yml file is below. > stages: > - docker > variables: > DOCKER_DRIVER: overlay2 > DOCKER_TLS_CERTDIR: "/certs" > cache: > key: ${CI_JOB_NAME} > paths: > - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ > build-python-ci-image: > image: docker:19.03.0 > services: > - docker:19.03.0-dind > stage: docker > before_script: > - cd ci_cd/python/ > script: > - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > $CI_REGISTRY > - docker build -t > registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . > - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > My Pipeline fails with this error: > See > [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] > Login Succeeded > $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > . > invalid argument > "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" > flag: invalid reference format > See 'docker build --help'. > Cleaning up project directory and file based variables > ERROR: Job failed: exit code 125 > It may or may not be relevant but the Container Registry for the GitLab > project says there's a Docker con
[jira] [Updated] (SPARK-42033) Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline
[ https://issues.apache.org/jira/browse/SPARK-42033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Nagla updated SPARK-42033: - Description: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training|[https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/ ] |https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/]follow the page. Thanks was: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training|[https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/ |https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/]follow the page. Thanks > Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline > > > Key: SPARK-42033 > URL: https://issues.apache.org/jira/browse/SPARK-42033 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 2.1.1 >Reporter: Pankaj Nagla >Priority: Major > Fix For: 1.6.2 > > > I'm going through the "Scalable FastAPI Application on AWS" course. My > gitlab-ci.yml file is below. > stages: > - docker > variables: > DOCKER_DRIVER: overlay2 > DOCKER_TLS_CERTDIR: "/certs" > cache: > key: ${CI_JOB_NAME} > paths: > - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ > build-python-ci-image: > image: docker:19.03.0 > services: > - docker:19.03.0-dind > stage: docker > before_script: > - cd ci_cd/python/ > script: > - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > $CI_REGISTRY > - docker build -t > registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . > - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > My Pipeline fails with this error: > See > [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] > Login Succeeded > $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > . > invalid argument > "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" > flag: invalid reference format > See 'docker build --help'. > Cleaning up project directory and file based variables > ERROR: Job failed: exit code 125 > It may or may not be relevant but the Container Regist
[jira] [Updated] (SPARK-42033) Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline
[ https://issues.apache.org/jira/browse/SPARK-42033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pankaj Nagla updated SPARK-42033: - Description: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training|[https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/ |https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/]follow the page. Thanks was: I'm going through the "Scalable FastAPI Application on AWS" course. My gitlab-ci.yml file is below. stages: - docker variables: DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "/certs" cache: key: ${CI_JOB_NAME} paths: - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ build-python-ci-image: image: docker:19.03.0 services: - docker:19.03.0-dind stage: docker before_script: - cd ci_cd/python/ script: - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY - docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim My Pipeline fails with this error: See [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] Login Succeeded $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . invalid argument "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" flag: invalid reference format See 'docker build --help'. Cleaning up project directory and file based variables ERROR: Job failed: exit code 125 It may or may not be relevant but the Container Registry for the GitLab project says there's a Docker connection error. All these problems have been discussed in this [Aws Sysops Training|[https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/ ] |https://www.igmguru.com/cloud-computing/aws-sysops-certification-training/]follow the page. Thanks > Docker Tag Error 25 on gitlab-ci.yml trying to start GitLab Pipeline > > > Key: SPARK-42033 > URL: https://issues.apache.org/jira/browse/SPARK-42033 > Project: Spark > Issue Type: Bug > Components: Deploy >Affects Versions: 2.1.1 >Reporter: Pankaj Nagla >Priority: Major > Fix For: 1.6.2 > > > I'm going through the "Scalable FastAPI Application on AWS" course. My > gitlab-ci.yml file is below. > stages: > - docker > variables: > DOCKER_DRIVER: overlay2 > DOCKER_TLS_CERTDIR: "/certs" > cache: > key: ${CI_JOB_NAME} > paths: > - ${CI_PROJECT_DIR}/services/talk_booking/.venv/ > build-python-ci-image: > image: docker:19.03.0 > services: > - docker:19.03.0-dind > stage: docker > before_script: > - cd ci_cd/python/ > script: > - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" > $CI_REGISTRY > - docker build -t > registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim . > - docker push registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > My Pipeline fails with this error: > See > [https://docs.docker.com/engine/reference/commandline/login/#credentials-store] > Login Succeeded > $ docker build -t registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim > . > invalid argument > "registry.gitlab.com/chris_/talk-booking:cicd-python3.9-slim" for "-t, --tag" > flag: invalid reference format > See 'docker build --help'. > Cleaning up project directory and file based variables > ERROR: Job failed: exit code 125 > It may or may not be relevant but the Container Regist