[jira] [Assigned] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled
[ https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-36892: -- Assignee: Ye Zhou > Disable batch fetch for a shuffle when push based shuffle is enabled > > > Key: SPARK-36892 > URL: https://issues.apache.org/jira/browse/SPARK-36892 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 3.2.0 >Reporter: Mridul Muralidharan >Assignee: Ye Zhou >Priority: Blocker > > When push based shuffle is enabled, efficient fetch of merged mapper shuffle > output happens. > Unfortunately, this currently interacts badly with > spark.sql.adaptive.fetchShuffleBlocksInBatch, potentially causing shuffle > fetch to hang and/or duplicate data to be fetched, causing correctness issues. > Given batch fetch does not benefit spark stages reading merged blocks when > push based shuffle is enabled, ShuffleBlockFetcherIterator.doBatchFetch can > be disabled when push based shuffle is enabled. > Thx to [~Ngone51] for surfacing this issue. > +CC [~Gengliang.Wang] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled
[ https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-36892. Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 34156 [https://github.com/apache/spark/pull/34156] > Disable batch fetch for a shuffle when push based shuffle is enabled > > > Key: SPARK-36892 > URL: https://issues.apache.org/jira/browse/SPARK-36892 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 3.2.0 >Reporter: Mridul Muralidharan >Assignee: Ye Zhou >Priority: Blocker > Fix For: 3.2.0 > > > When push based shuffle is enabled, efficient fetch of merged mapper shuffle > output happens. > Unfortunately, this currently interacts badly with > spark.sql.adaptive.fetchShuffleBlocksInBatch, potentially causing shuffle > fetch to hang and/or duplicate data to be fetched, causing correctness issues. > Given batch fetch does not benefit spark stages reading merged blocks when > push based shuffle is enabled, ShuffleBlockFetcherIterator.doBatchFetch can > be disabled when push based shuffle is enabled. > Thx to [~Ngone51] for surfacing this issue. > +CC [~Gengliang.Wang] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled
[ https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424825#comment-17424825 ] Gengliang Wang commented on SPARK-36892: [~mridulm80] [~mshen] [~zhouyejoe] [~apatnam] Again, thanks for testing Spark 3.2 with real workloads. Now that all the blockers are resolved. I will have the new RC soon. > Disable batch fetch for a shuffle when push based shuffle is enabled > > > Key: SPARK-36892 > URL: https://issues.apache.org/jira/browse/SPARK-36892 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 3.2.0 >Reporter: Mridul Muralidharan >Assignee: Ye Zhou >Priority: Blocker > Fix For: 3.2.0 > > > When push based shuffle is enabled, efficient fetch of merged mapper shuffle > output happens. > Unfortunately, this currently interacts badly with > spark.sql.adaptive.fetchShuffleBlocksInBatch, potentially causing shuffle > fetch to hang and/or duplicate data to be fetched, causing correctness issues. > Given batch fetch does not benefit spark stages reading merged blocks when > push based shuffle is enabled, ShuffleBlockFetcherIterator.doBatchFetch can > be disabled when push based shuffle is enabled. > Thx to [~Ngone51] for surfacing this issue. > +CC [~Gengliang.Wang] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36934) Timestamp are written as array bytes.
[
https://issues.apache.org/jira/browse/SPARK-36934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424860#comment-17424860
]
Bjørn Jørgensen commented on SPARK-36934:
-
.config("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MICROS")
now in Apache drill it prints
year
14230080
14571360
> Timestamp are written as array bytes.
> -
>
> Key: SPARK-36934
> URL: https://issues.apache.org/jira/browse/SPARK-36934
> Project: Spark
> Issue Type: Bug
> Components: PySpark
>Affects Versions: 3.3.0
>Reporter: Bjørn Jørgensen
>Priority: Major
>
> This is tested with master build 04.10.21
> {code}
> df = ps.DataFrame({'year': ['2015-2-4', '2016-3-5'],
> 'month': [2, 3],
> 'day': [4, 5],
> 'test': [1, 2]})
> df["year"] = ps.to_datetime(df["year"])
> df.info()
> Int64Index: 2 entries, 0 to 1 Data
> columns (total 4 columns): # Column Non-Null Count Dtype --- --
> -- - 0 year 2 non-null datetime64 1 month 2 non-null int64 2
> day 2 non-null int64 3 test 2 non-null int64 dtypes: datetime64(1), int64(3)
> spark_df_date = df.to_spark()
> spark_df_date.printSchema()
> root
> |-- year: timestamp (nullable = true)
> |-- month: long (nullable = false)
> |-- day: long (nullable = false)
> |-- test: long (nullable = false)
> spark_df_date.write.parquet("s3a://falk0509/spark_df_date.parquet")
> {code}
> Load the files in to Apache drill I use docker apache/drill:master-openjdk-14
>
> SELECT * FROM cp.`/data/spark_df_date.*`
> It print's
> year
> {code}
> \x00\x00\x00\x00\x00\x00\x00\x00\xE2}%\x00
> \x00\x00\x00\x00\x00\x00\x00\x00m\x7F%\x00
> {code}
>
> The rest of the columns are ok.
> So is this a spark problem or Apache drill?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-36934) Timestamp are written as array bytes.
[
https://issues.apache.org/jira/browse/SPARK-36934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bjørn Jørgensen updated SPARK-36934:
Component/s: Spark Core
> Timestamp are written as array bytes.
> -
>
> Key: SPARK-36934
> URL: https://issues.apache.org/jira/browse/SPARK-36934
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Spark Core
>Affects Versions: 3.3.0
>Reporter: Bjørn Jørgensen
>Priority: Major
>
> This is tested with master build 04.10.21
> {code}
> df = ps.DataFrame({'year': ['2015-2-4', '2016-3-5'],
> 'month': [2, 3],
> 'day': [4, 5],
> 'test': [1, 2]})
> df["year"] = ps.to_datetime(df["year"])
> df.info()
> Int64Index: 2 entries, 0 to 1 Data
> columns (total 4 columns): # Column Non-Null Count Dtype --- --
> -- - 0 year 2 non-null datetime64 1 month 2 non-null int64 2
> day 2 non-null int64 3 test 2 non-null int64 dtypes: datetime64(1), int64(3)
> spark_df_date = df.to_spark()
> spark_df_date.printSchema()
> root
> |-- year: timestamp (nullable = true)
> |-- month: long (nullable = false)
> |-- day: long (nullable = false)
> |-- test: long (nullable = false)
> spark_df_date.write.parquet("s3a://falk0509/spark_df_date.parquet")
> {code}
> Load the files in to Apache drill I use docker apache/drill:master-openjdk-14
>
> SELECT * FROM cp.`/data/spark_df_date.*`
> It print's
> year
> {code}
> \x00\x00\x00\x00\x00\x00\x00\x00\xE2}%\x00
> \x00\x00\x00\x00\x00\x00\x00\x00m\x7F%\x00
> {code}
>
> The rest of the columns are ok.
> So is this a spark problem or Apache drill?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36934) Timestamp are written as array bytes.
[
https://issues.apache.org/jira/browse/SPARK-36934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424885#comment-17424885
]
Hyukjin Kwon commented on SPARK-36934:
--
what about TIMESTAMP_MILLIS?
> Timestamp are written as array bytes.
> -
>
> Key: SPARK-36934
> URL: https://issues.apache.org/jira/browse/SPARK-36934
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Spark Core
>Affects Versions: 3.3.0
>Reporter: Bjørn Jørgensen
>Priority: Major
>
> This is tested with master build 04.10.21
> {code}
> df = ps.DataFrame({'year': ['2015-2-4', '2016-3-5'],
> 'month': [2, 3],
> 'day': [4, 5],
> 'test': [1, 2]})
> df["year"] = ps.to_datetime(df["year"])
> df.info()
> Int64Index: 2 entries, 0 to 1 Data
> columns (total 4 columns): # Column Non-Null Count Dtype --- --
> -- - 0 year 2 non-null datetime64 1 month 2 non-null int64 2
> day 2 non-null int64 3 test 2 non-null int64 dtypes: datetime64(1), int64(3)
> spark_df_date = df.to_spark()
> spark_df_date.printSchema()
> root
> |-- year: timestamp (nullable = true)
> |-- month: long (nullable = false)
> |-- day: long (nullable = false)
> |-- test: long (nullable = false)
> spark_df_date.write.parquet("s3a://falk0509/spark_df_date.parquet")
> {code}
> Load the files in to Apache drill I use docker apache/drill:master-openjdk-14
>
> SELECT * FROM cp.`/data/spark_df_date.*`
> It print's
> year
> {code}
> \x00\x00\x00\x00\x00\x00\x00\x00\xE2}%\x00
> \x00\x00\x00\x00\x00\x00\x00\x00m\x7F%\x00
> {code}
>
> The rest of the columns are ok.
> So is this a spark problem or Apache drill?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-36927) Inline type hints for python/pyspark/sql/window.py
[ https://issues.apache.org/jira/browse/SPARK-36927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36927. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34173 [https://github.com/apache/spark/pull/34173] > Inline type hints for python/pyspark/sql/window.py > -- > > Key: SPARK-36927 > URL: https://issues.apache.org/jira/browse/SPARK-36927 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 1.3.0 >Reporter: Xinrong Meng >Assignee: Xinrong Meng >Priority: Major > Fix For: 3.3.0 > > > Inline type hints for python/pyspark/sql/window.py. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36927) Inline type hints for python/pyspark/sql/window.py
[ https://issues.apache.org/jira/browse/SPARK-36927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-36927: Assignee: Xinrong Meng > Inline type hints for python/pyspark/sql/window.py > -- > > Key: SPARK-36927 > URL: https://issues.apache.org/jira/browse/SPARK-36927 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 1.3.0 >Reporter: Xinrong Meng >Assignee: Xinrong Meng >Priority: Major > > Inline type hints for python/pyspark/sql/window.py. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-36919) Make BadRecordException serializable
[
https://issues.apache.org/jira/browse/SPARK-36919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-36919.
--
Fix Version/s: 3.1.3
3.2.0
3.0.4
Resolution: Fixed
Issue resolved by pull request 34167
[https://github.com/apache/spark/pull/34167]
> Make BadRecordException serializable
>
>
> Key: SPARK-36919
> URL: https://issues.apache.org/jira/browse/SPARK-36919
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 3.2.0, 3.3.0, 3.2.1
>Reporter: Tianhan Hu
>Assignee: Tianhan Hu
>Priority: Minor
> Fix For: 3.0.4, 3.2.0, 3.1.3
>
>
> Migrating a Spark application from 2.4.x to 3.1.x and finding a difference in
> the exception chaining behavior. In a case of parsing a malformed CSV, where
> the root cause exception should be {{Caused by: java.lang.RuntimeException:
> Malformed CSV record}}, only the top level exception is kept, and all lower
> level exceptions and root cause are lost. Thus, when we call
> {{ExceptionUtils.getRootCause}} on the exception, we still get itself.
> The reason for the difference is that {{RuntimeException}} is wrapped in
> {{BadRecordException}}, which has unserializable fields. When we try to
> serialize the exception from tasks and deserialize from scheduler, the
> exception is lost.
> This PR makes unserializable fields of {{BadRecordException}} transient, so
> the rest of the exception could be serialized and deserialized properly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36919) Make BadRecordException serializable
[
https://issues.apache.org/jira/browse/SPARK-36919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-36919:
Assignee: Tianhan Hu
> Make BadRecordException serializable
>
>
> Key: SPARK-36919
> URL: https://issues.apache.org/jira/browse/SPARK-36919
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 3.2.0, 3.3.0, 3.2.1
>Reporter: Tianhan Hu
>Assignee: Tianhan Hu
>Priority: Minor
>
> Migrating a Spark application from 2.4.x to 3.1.x and finding a difference in
> the exception chaining behavior. In a case of parsing a malformed CSV, where
> the root cause exception should be {{Caused by: java.lang.RuntimeException:
> Malformed CSV record}}, only the top level exception is kept, and all lower
> level exceptions and root cause are lost. Thus, when we call
> {{ExceptionUtils.getRootCause}} on the exception, we still get itself.
> The reason for the difference is that {{RuntimeException}} is wrapped in
> {{BadRecordException}}, which has unserializable fields. When we try to
> serialize the exception from tasks and deserialize from scheduler, the
> exception is lost.
> This PR makes unserializable fields of {{BadRecordException}} transient, so
> the rest of the exception could be serialized and deserialized properly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36934) Timestamp are written as array bytes.
[
https://issues.apache.org/jira/browse/SPARK-36934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424901#comment-17424901
]
Bjørn Jørgensen commented on SPARK-36934:
-
With .config("spark.sql.parquet.outputTimestampType", "TIMESTAMP_MILLIS")
In Apache drill now
year
2015-02-04T00:00
|2016-03-05T00:00|
> Timestamp are written as array bytes.
> -
>
> Key: SPARK-36934
> URL: https://issues.apache.org/jira/browse/SPARK-36934
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Spark Core
>Affects Versions: 3.3.0
>Reporter: Bjørn Jørgensen
>Priority: Major
>
> This is tested with master build 04.10.21
> {code}
> df = ps.DataFrame({'year': ['2015-2-4', '2016-3-5'],
> 'month': [2, 3],
> 'day': [4, 5],
> 'test': [1, 2]})
> df["year"] = ps.to_datetime(df["year"])
> df.info()
> Int64Index: 2 entries, 0 to 1 Data
> columns (total 4 columns): # Column Non-Null Count Dtype --- --
> -- - 0 year 2 non-null datetime64 1 month 2 non-null int64 2
> day 2 non-null int64 3 test 2 non-null int64 dtypes: datetime64(1), int64(3)
> spark_df_date = df.to_spark()
> spark_df_date.printSchema()
> root
> |-- year: timestamp (nullable = true)
> |-- month: long (nullable = false)
> |-- day: long (nullable = false)
> |-- test: long (nullable = false)
> spark_df_date.write.parquet("s3a://falk0509/spark_df_date.parquet")
> {code}
> Load the files in to Apache drill I use docker apache/drill:master-openjdk-14
>
> SELECT * FROM cp.`/data/spark_df_date.*`
> It print's
> year
> {code}
> \x00\x00\x00\x00\x00\x00\x00\x00\xE2}%\x00
> \x00\x00\x00\x00\x00\x00\x00\x00m\x7F%\x00
> {code}
>
> The rest of the columns are ok.
> So is this a spark problem or Apache drill?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36934) Timestamp are written as array bytes.
[
https://issues.apache.org/jira/browse/SPARK-36934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424917#comment-17424917
]
Hyukjin Kwon commented on SPARK-36934:
--
Looks like Apache Drill only implements TIMESTAMP_MILLIS in Parquet.
TIMESTAMP_MICROS is also Parquet standard but looks like the read path for
this type seems missing in Drill.
You will have to use TIMESTAMP_MILLIS for now
> Timestamp are written as array bytes.
> -
>
> Key: SPARK-36934
> URL: https://issues.apache.org/jira/browse/SPARK-36934
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Spark Core
>Affects Versions: 3.3.0
>Reporter: Bjørn Jørgensen
>Priority: Major
>
> This is tested with master build 04.10.21
> {code}
> df = ps.DataFrame({'year': ['2015-2-4', '2016-3-5'],
> 'month': [2, 3],
> 'day': [4, 5],
> 'test': [1, 2]})
> df["year"] = ps.to_datetime(df["year"])
> df.info()
> Int64Index: 2 entries, 0 to 1 Data
> columns (total 4 columns): # Column Non-Null Count Dtype --- --
> -- - 0 year 2 non-null datetime64 1 month 2 non-null int64 2
> day 2 non-null int64 3 test 2 non-null int64 dtypes: datetime64(1), int64(3)
> spark_df_date = df.to_spark()
> spark_df_date.printSchema()
> root
> |-- year: timestamp (nullable = true)
> |-- month: long (nullable = false)
> |-- day: long (nullable = false)
> |-- test: long (nullable = false)
> spark_df_date.write.parquet("s3a://falk0509/spark_df_date.parquet")
> {code}
> Load the files in to Apache drill I use docker apache/drill:master-openjdk-14
>
> SELECT * FROM cp.`/data/spark_df_date.*`
> It print's
> year
> {code}
> \x00\x00\x00\x00\x00\x00\x00\x00\xE2}%\x00
> \x00\x00\x00\x00\x00\x00\x00\x00m\x7F%\x00
> {code}
>
> The rest of the columns are ok.
> So is this a spark problem or Apache drill?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-36887) Inline type hints for python/pyspark/sql/conf.py
[ https://issues.apache.org/jira/browse/SPARK-36887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dch nguyen resolved SPARK-36887. Resolution: Resolved This issue is resolved by https://issues.apache.org/jira/browse/SPARK-36906 > Inline type hints for python/pyspark/sql/conf.py > > > Key: SPARK-36887 > URL: https://issues.apache.org/jira/browse/SPARK-36887 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dgd_contributor >Priority: Major > > Inline type hints for python/pyspark/sql/session.py from Inline type hints > for python/pyspark/sql/conf.pyi. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36938) Inline type hints for group.py in python/pyspark/sql
[ https://issues.apache.org/jira/browse/SPARK-36938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424929#comment-17424929 ] dch nguyen commented on SPARK-36938: i am working on this > Inline type hints for group.py in python/pyspark/sql > - > > Key: SPARK-36938 > URL: https://issues.apache.org/jira/browse/SPARK-36938 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-36938) Inline type hints for group.py in python/pyspark/sql
[ https://issues.apache.org/jira/browse/SPARK-36938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dch nguyen updated SPARK-36938: --- Summary: Inline type hints for group.py in python/pyspark/sql (was: nline type hints for group.py in python/pyspark/sql ) > Inline type hints for group.py in python/pyspark/sql > - > > Key: SPARK-36938 > URL: https://issues.apache.org/jira/browse/SPARK-36938 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-36938) nline type hints for group.py in python/pyspark/sql
dch nguyen created SPARK-36938: -- Summary: nline type hints for group.py in python/pyspark/sql Key: SPARK-36938 URL: https://issues.apache.org/jira/browse/SPARK-36938 Project: Spark Issue Type: Sub-task Components: PySpark Affects Versions: 3.3.0 Reporter: dch nguyen -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-36939) Add orphan migration page into list in PySpark documentation
Hyukjin Kwon created SPARK-36939:
Summary: Add orphan migration page into list in PySpark
documentation
Key: SPARK-36939
URL: https://issues.apache.org/jira/browse/SPARK-36939
Project: Spark
Issue Type: Test
Components: docs, PySpark
Affects Versions: 3.2.0
Reporter: Hyukjin Kwon
SPARK-36618 added a new migration guide page but that's mistakenly not added to
{{spark/python/docs/source/migration_guideindex.rst}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-36939) Add orphan migration page into list in PySpark documentation
[
https://issues.apache.org/jira/browse/SPARK-36939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-36939:
-
Issue Type: Documentation (was: Test)
> Add orphan migration page into list in PySpark documentation
>
>
> Key: SPARK-36939
> URL: https://issues.apache.org/jira/browse/SPARK-36939
> Project: Spark
> Issue Type: Documentation
> Components: docs, PySpark
>Affects Versions: 3.2.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> SPARK-36618 added a new migration guide page but that's mistakenly not added
> to {{spark/python/docs/source/migration_guideindex.rst}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36939) Add orphan migration page into list in PySpark documentation
[
https://issues.apache.org/jira/browse/SPARK-36939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-36939:
Assignee: (was: Apache Spark)
> Add orphan migration page into list in PySpark documentation
>
>
> Key: SPARK-36939
> URL: https://issues.apache.org/jira/browse/SPARK-36939
> Project: Spark
> Issue Type: Documentation
> Components: docs, PySpark
>Affects Versions: 3.2.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> SPARK-36618 added a new migration guide page but that's mistakenly not added
> to {{spark/python/docs/source/migration_guideindex.rst}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36939) Add orphan migration page into list in PySpark documentation
[
https://issues.apache.org/jira/browse/SPARK-36939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424935#comment-17424935
]
Apache Spark commented on SPARK-36939:
--
User 'HyukjinKwon' has created a pull request for this issue:
https://github.com/apache/spark/pull/34195
> Add orphan migration page into list in PySpark documentation
>
>
> Key: SPARK-36939
> URL: https://issues.apache.org/jira/browse/SPARK-36939
> Project: Spark
> Issue Type: Documentation
> Components: docs, PySpark
>Affects Versions: 3.2.0
>Reporter: Hyukjin Kwon
>Priority: Major
>
> SPARK-36618 added a new migration guide page but that's mistakenly not added
> to {{spark/python/docs/source/migration_guideindex.rst}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36939) Add orphan migration page into list in PySpark documentation
[
https://issues.apache.org/jira/browse/SPARK-36939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-36939:
Assignee: Apache Spark
> Add orphan migration page into list in PySpark documentation
>
>
> Key: SPARK-36939
> URL: https://issues.apache.org/jira/browse/SPARK-36939
> Project: Spark
> Issue Type: Documentation
> Components: docs, PySpark
>Affects Versions: 3.2.0
>Reporter: Hyukjin Kwon
>Assignee: Apache Spark
>Priority: Major
>
> SPARK-36618 added a new migration guide page but that's mistakenly not added
> to {{spark/python/docs/source/migration_guideindex.rst}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-36934) Timestamp are written as array bytes.
[
https://issues.apache.org/jira/browse/SPARK-36934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-36934.
--
Resolution: Not A Problem
> Timestamp are written as array bytes.
> -
>
> Key: SPARK-36934
> URL: https://issues.apache.org/jira/browse/SPARK-36934
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Spark Core
>Affects Versions: 3.3.0
>Reporter: Bjørn Jørgensen
>Priority: Major
>
> This is tested with master build 04.10.21
> {code}
> df = ps.DataFrame({'year': ['2015-2-4', '2016-3-5'],
> 'month': [2, 3],
> 'day': [4, 5],
> 'test': [1, 2]})
> df["year"] = ps.to_datetime(df["year"])
> df.info()
> Int64Index: 2 entries, 0 to 1 Data
> columns (total 4 columns): # Column Non-Null Count Dtype --- --
> -- - 0 year 2 non-null datetime64 1 month 2 non-null int64 2
> day 2 non-null int64 3 test 2 non-null int64 dtypes: datetime64(1), int64(3)
> spark_df_date = df.to_spark()
> spark_df_date.printSchema()
> root
> |-- year: timestamp (nullable = true)
> |-- month: long (nullable = false)
> |-- day: long (nullable = false)
> |-- test: long (nullable = false)
> spark_df_date.write.parquet("s3a://falk0509/spark_df_date.parquet")
> {code}
> Load the files in to Apache drill I use docker apache/drill:master-openjdk-14
>
> SELECT * FROM cp.`/data/spark_df_date.*`
> It print's
> year
> {code}
> \x00\x00\x00\x00\x00\x00\x00\x00\xE2}%\x00
> \x00\x00\x00\x00\x00\x00\x00\x00m\x7F%\x00
> {code}
>
> The rest of the columns are ok.
> So is this a spark problem or Apache drill?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36751) octet_length/bit_length API is not implemented on Scala/Python/R
[ https://issues.apache.org/jira/browse/SPARK-36751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424939#comment-17424939 ] Apache Spark commented on SPARK-36751: -- User 'HyukjinKwon' has created a pull request for this issue: https://github.com/apache/spark/pull/34196 > octet_length/bit_length API is not implemented on Scala/Python/R > - > > Key: SPARK-36751 > URL: https://issues.apache.org/jira/browse/SPARK-36751 > Project: Spark > Issue Type: Improvement > Components: PySpark, SparkR, SQL >Affects Versions: 3.3.0 >Reporter: Leona Yoda >Assignee: Leona Yoda >Priority: Major > Fix For: 3.3.0 > > > * octet_length: caliculate the byte length of strings > * bit_length: caliculate the bit length of strings > Those two string related functions are only implemented on SparkSQL, not on > Scala, Python and R. > Those functions would be useful for multi-bytes character users, who mainly > working with those languages. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36858) Spark API to apply same function to multiple columns
[
https://issues.apache.org/jira/browse/SPARK-36858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424988#comment-17424988
]
Armand BERGES commented on SPARK-36858:
---
[~hyukjin.kwon] How would you do this ?
>From my point, if you make a `df.withColumn` in a for loop, it will end in the
>same execution plan (so probably in the same problem at the end, no ?
> Spark API to apply same function to multiple columns
>
>
> Key: SPARK-36858
> URL: https://issues.apache.org/jira/browse/SPARK-36858
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
>Affects Versions: 2.4.8, 3.1.2
>Reporter: Armand BERGES
>Priority: Minor
>
> Hi
> My team and I have regularly need to apply the same function to multiple
> columns at once.
> For example, we want to remove all non alphanumerical characters to each
> columns of our dataframes.
> When we hit this use case first, some people in my team were using this kind
> of code :
> {code:java}
> val colListToClean = ## Generate some list, could be very long.
> val dfToClean: DataFrame = ... ## This is the dataframe we want to clean
> def cleanFunction(colName: String): Column = ... ## Write some function to
> manipulate column based on its name.
> val dfCleaned = colListToClean.foldLeft(dfToClean)((df, colName) =>
> df.withColumn(colName, cleanFunction(colName)){code}
> This kind of code when applied on a large set of columns overloaded our
> driver (because a Dataframe is generated for each column to clean).
> Based on this issue, we developed some code to add two functions :
> * One to apply the same function to multiple columns
> * One to rename multiple columns based on a Map.
>
> I wonder if your ever ask your team to add such kind of API ? If you did, had
> you any kind of issue regarding the implementation ? If you didn't, is this
> any idea you could add to Spark ?
> Best regards,
>
> LvffY
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36938) Inline type hints for group.py in python/pyspark/sql
[ https://issues.apache.org/jira/browse/SPARK-36938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36938: Assignee: Apache Spark > Inline type hints for group.py in python/pyspark/sql > - > > Key: SPARK-36938 > URL: https://issues.apache.org/jira/browse/SPARK-36938 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Assignee: Apache Spark >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36938) Inline type hints for group.py in python/pyspark/sql
[ https://issues.apache.org/jira/browse/SPARK-36938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424989#comment-17424989 ] Apache Spark commented on SPARK-36938: -- User 'dchvn' has created a pull request for this issue: https://github.com/apache/spark/pull/34197 > Inline type hints for group.py in python/pyspark/sql > - > > Key: SPARK-36938 > URL: https://issues.apache.org/jira/browse/SPARK-36938 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36938) Inline type hints for group.py in python/pyspark/sql
[ https://issues.apache.org/jira/browse/SPARK-36938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36938: Assignee: (was: Apache Spark) > Inline type hints for group.py in python/pyspark/sql > - > > Key: SPARK-36938 > URL: https://issues.apache.org/jira/browse/SPARK-36938 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36938) Inline type hints for group.py in python/pyspark/sql
[ https://issues.apache.org/jira/browse/SPARK-36938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424990#comment-17424990 ] Apache Spark commented on SPARK-36938: -- User 'dchvn' has created a pull request for this issue: https://github.com/apache/spark/pull/34197 > Inline type hints for group.py in python/pyspark/sql > - > > Key: SPARK-36938 > URL: https://issues.apache.org/jira/browse/SPARK-36938 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36892) Disable batch fetch for a shuffle when push based shuffle is enabled
[ https://issues.apache.org/jira/browse/SPARK-36892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425016#comment-17425016 ] Mridul Muralidharan commented on SPARK-36892: - Sounds good [~Gengliang.Wang], I am not aware of any other issues. Thanks for driving the process ! > Disable batch fetch for a shuffle when push based shuffle is enabled > > > Key: SPARK-36892 > URL: https://issues.apache.org/jira/browse/SPARK-36892 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 3.2.0 >Reporter: Mridul Muralidharan >Assignee: Ye Zhou >Priority: Blocker > Fix For: 3.2.0 > > > When push based shuffle is enabled, efficient fetch of merged mapper shuffle > output happens. > Unfortunately, this currently interacts badly with > spark.sql.adaptive.fetchShuffleBlocksInBatch, potentially causing shuffle > fetch to hang and/or duplicate data to be fetched, causing correctness issues. > Given batch fetch does not benefit spark stages reading merged blocks when > push based shuffle is enabled, ShuffleBlockFetcherIterator.doBatchFetch can > be disabled when push based shuffle is enabled. > Thx to [~Ngone51] for surfacing this issue. > +CC [~Gengliang.Wang] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36905) Reading Hive view without explicit column names fails in Spark
[
https://issues.apache.org/jira/browse/SPARK-36905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425072#comment-17425072
]
Gengliang Wang commented on SPARK-36905:
[~shardulm] Thanks for reporting the issue.
I don't think this is a release blocker. I will mention this one as a known
issue in the release note if it is not resolved by then.
> Reading Hive view without explicit column names fails in Spark
> ---
>
> Key: SPARK-36905
> URL: https://issues.apache.org/jira/browse/SPARK-36905
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.2.0
>Reporter: Shardul Mahadik
>Priority: Major
>
> Consider a Hive view in which some columns are not explicitly named
> {code:sql}
> CREATE VIEW test_view AS
> SELECT 1
> FROM some_table
> {code}
> Reading this view in Spark leads to an {{AnalysisException}}
> {code:java}
> org.apache.spark.sql.AnalysisException: cannot resolve '`_c0`' given input
> columns: [1]
> at
> org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:188)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:185)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$2(TreeNode.scala:340)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:72)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:340)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$1(TreeNode.scala:337)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:406)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:242)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:404)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:357)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:337)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$1(TreeNode.scala:337)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:406)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:242)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:404)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:357)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.transformUp(TreeNode.scala:337)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$transformExpressionsUp$1(QueryPlan.scala:104)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$1(QueryPlan.scala:116)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:72)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpression$1(QueryPlan.scala:116)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:127)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$3(QueryPlan.scala:132)
> at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> at scala.collection.TraversableLike.map(TraversableLike.scala:238)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> at scala.collection.AbstractTraversable.map(Traversable.scala:108)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.recursiveTransform$1(QueryPlan.scala:132)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.$anonfun$mapExpressions$4(QueryPlan.scala:137)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:242)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.mapExpressions(QueryPlan.scala:137)
> at
> org.apache.spark.sql.catalyst.plans.QueryPlan.transformExpressionsUp(QueryPlan.scala:104)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1(CheckAnalysis.scala:185)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1$adapted(CheckAnalysis.scala:94)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:182)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis(CheckAnalysis.scala:94)
>
[jira] [Assigned] (SPARK-36300) Refactor eleventh set of 20 query execution errors to use error classes
[
https://issues.apache.org/jira/browse/SPARK-36300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-36300:
Assignee: (was: Apache Spark)
> Refactor eleventh set of 20 query execution errors to use error classes
> ---
>
> Key: SPARK-36300
> URL: https://issues.apache.org/jira/browse/SPARK-36300
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core, SQL
>Affects Versions: 3.2.0
>Reporter: Karen Feng
>Priority: Major
>
> Refactor some exceptions in
> [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala]
> to use error classes.
> There are currently ~350 exceptions in this file; so this PR only focuses on
> the eleventh set of 20.
> {code:java}
> expressionDecodingError
> expressionEncodingError
> classHasUnexpectedSerializerError
> cannotGetOuterPointerForInnerClassError
> userDefinedTypeNotAnnotatedAndRegisteredError
> invalidInputSyntaxForBooleanError
> unsupportedOperandTypeForSizeFunctionError
> unexpectedValueForStartInFunctionError
> unexpectedValueForLengthInFunctionError
> sqlArrayIndexNotStartAtOneError
> concatArraysWithElementsExceedLimitError
> flattenArraysWithElementsExceedLimitError
> createArrayWithElementsExceedLimitError
> unionArrayWithElementsExceedLimitError
> initialTypeNotTargetDataTypeError
> initialTypeNotTargetDataTypesError
> cannotConvertColumnToJSONError
> malformedRecordsDetectedInSchemaInferenceError
> malformedJSONError
> malformedRecordsDetectedInSchemaInferenceError
> {code}
> For more detail, see the parent ticket SPARK-36094.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36300) Refactor eleventh set of 20 query execution errors to use error classes
[
https://issues.apache.org/jira/browse/SPARK-36300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-36300:
Assignee: Apache Spark
> Refactor eleventh set of 20 query execution errors to use error classes
> ---
>
> Key: SPARK-36300
> URL: https://issues.apache.org/jira/browse/SPARK-36300
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core, SQL
>Affects Versions: 3.2.0
>Reporter: Karen Feng
>Assignee: Apache Spark
>Priority: Major
>
> Refactor some exceptions in
> [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala]
> to use error classes.
> There are currently ~350 exceptions in this file; so this PR only focuses on
> the eleventh set of 20.
> {code:java}
> expressionDecodingError
> expressionEncodingError
> classHasUnexpectedSerializerError
> cannotGetOuterPointerForInnerClassError
> userDefinedTypeNotAnnotatedAndRegisteredError
> invalidInputSyntaxForBooleanError
> unsupportedOperandTypeForSizeFunctionError
> unexpectedValueForStartInFunctionError
> unexpectedValueForLengthInFunctionError
> sqlArrayIndexNotStartAtOneError
> concatArraysWithElementsExceedLimitError
> flattenArraysWithElementsExceedLimitError
> createArrayWithElementsExceedLimitError
> unionArrayWithElementsExceedLimitError
> initialTypeNotTargetDataTypeError
> initialTypeNotTargetDataTypesError
> cannotConvertColumnToJSONError
> malformedRecordsDetectedInSchemaInferenceError
> malformedJSONError
> malformedRecordsDetectedInSchemaInferenceError
> {code}
> For more detail, see the parent ticket SPARK-36094.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36300) Refactor eleventh set of 20 query execution errors to use error classes
[
https://issues.apache.org/jira/browse/SPARK-36300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425129#comment-17425129
]
Apache Spark commented on SPARK-36300:
--
User 'changvvb' has created a pull request for this issue:
https://github.com/apache/spark/pull/34198
> Refactor eleventh set of 20 query execution errors to use error classes
> ---
>
> Key: SPARK-36300
> URL: https://issues.apache.org/jira/browse/SPARK-36300
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core, SQL
>Affects Versions: 3.2.0
>Reporter: Karen Feng
>Priority: Major
>
> Refactor some exceptions in
> [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala]
> to use error classes.
> There are currently ~350 exceptions in this file; so this PR only focuses on
> the eleventh set of 20.
> {code:java}
> expressionDecodingError
> expressionEncodingError
> classHasUnexpectedSerializerError
> cannotGetOuterPointerForInnerClassError
> userDefinedTypeNotAnnotatedAndRegisteredError
> invalidInputSyntaxForBooleanError
> unsupportedOperandTypeForSizeFunctionError
> unexpectedValueForStartInFunctionError
> unexpectedValueForLengthInFunctionError
> sqlArrayIndexNotStartAtOneError
> concatArraysWithElementsExceedLimitError
> flattenArraysWithElementsExceedLimitError
> createArrayWithElementsExceedLimitError
> unionArrayWithElementsExceedLimitError
> initialTypeNotTargetDataTypeError
> initialTypeNotTargetDataTypesError
> cannotConvertColumnToJSONError
> malformedRecordsDetectedInSchemaInferenceError
> malformedJSONError
> malformedRecordsDetectedInSchemaInferenceError
> {code}
> For more detail, see the parent ticket SPARK-36094.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-36940) Inline type hints for python/pyspark/sql/avro/functions.py
Xinrong Meng created SPARK-36940: Summary: Inline type hints for python/pyspark/sql/avro/functions.py Key: SPARK-36940 URL: https://issues.apache.org/jira/browse/SPARK-36940 Project: Spark Issue Type: Sub-task Components: PySpark Affects Versions: 3.3.0 Reporter: Xinrong Meng Inline type hints for python/pyspark/sql/avro/functions.py -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36940) Inline type hints for python/pyspark/sql/avro/functions.py
[ https://issues.apache.org/jira/browse/SPARK-36940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425132#comment-17425132 ] Xinrong Meng commented on SPARK-36940: -- I'm working on this. > Inline type hints for python/pyspark/sql/avro/functions.py > -- > > Key: SPARK-36940 > URL: https://issues.apache.org/jira/browse/SPARK-36940 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: Xinrong Meng >Priority: Major > > Inline type hints for python/pyspark/sql/avro/functions.py -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-36940) Inline type hints for python/pyspark/sql/avro/functions.py
[ https://issues.apache.org/jira/browse/SPARK-36940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinrong Meng updated SPARK-36940: - Description: Inline type hints for python/pyspark/sql/avro/functions.py. Currently, we use stub files for type annotations, which don't support type checks within function bodies. So we inline type hints to support that. was:Inline type hints for python/pyspark/sql/avro/functions.py > Inline type hints for python/pyspark/sql/avro/functions.py > -- > > Key: SPARK-36940 > URL: https://issues.apache.org/jira/browse/SPARK-36940 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: Xinrong Meng >Priority: Major > > Inline type hints for python/pyspark/sql/avro/functions.py. > > Currently, we use stub files for type annotations, which don't support type > checks within function bodies. So we inline type hints to support that. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36935) Enhance ParquetSchemaConverter to capture Parquet repetition & definition level
[
https://issues.apache.org/jira/browse/SPARK-36935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-36935:
Assignee: Apache Spark
> Enhance ParquetSchemaConverter to capture Parquet repetition & definition
> level
> ---
>
> Key: SPARK-36935
> URL: https://issues.apache.org/jira/browse/SPARK-36935
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.3.0
>Reporter: Chao Sun
>Assignee: Apache Spark
>Priority: Major
>
> In order to support complex type for Parquet vectorized reader, we'll need to
> capture the repetition & definition level information associated with
> Catalyst Spark type converted from Parquet {{MessageType}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36935) Enhance ParquetSchemaConverter to capture Parquet repetition & definition level
[
https://issues.apache.org/jira/browse/SPARK-36935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425140#comment-17425140
]
Apache Spark commented on SPARK-36935:
--
User 'sunchao' has created a pull request for this issue:
https://github.com/apache/spark/pull/34199
> Enhance ParquetSchemaConverter to capture Parquet repetition & definition
> level
> ---
>
> Key: SPARK-36935
> URL: https://issues.apache.org/jira/browse/SPARK-36935
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.3.0
>Reporter: Chao Sun
>Priority: Major
>
> In order to support complex type for Parquet vectorized reader, we'll need to
> capture the repetition & definition level information associated with
> Catalyst Spark type converted from Parquet {{MessageType}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36935) Enhance ParquetSchemaConverter to capture Parquet repetition & definition level
[
https://issues.apache.org/jira/browse/SPARK-36935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-36935:
Assignee: (was: Apache Spark)
> Enhance ParquetSchemaConverter to capture Parquet repetition & definition
> level
> ---
>
> Key: SPARK-36935
> URL: https://issues.apache.org/jira/browse/SPARK-36935
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.3.0
>Reporter: Chao Sun
>Priority: Major
>
> In order to support complex type for Parquet vectorized reader, we'll need to
> capture the repetition & definition level information associated with
> Catalyst Spark type converted from Parquet {{MessageType}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36935) Enhance ParquetSchemaConverter to capture Parquet repetition & definition level
[
https://issues.apache.org/jira/browse/SPARK-36935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425141#comment-17425141
]
Apache Spark commented on SPARK-36935:
--
User 'sunchao' has created a pull request for this issue:
https://github.com/apache/spark/pull/34199
> Enhance ParquetSchemaConverter to capture Parquet repetition & definition
> level
> ---
>
> Key: SPARK-36935
> URL: https://issues.apache.org/jira/browse/SPARK-36935
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
>Affects Versions: 3.3.0
>Reporter: Chao Sun
>Priority: Major
>
> In order to support complex type for Parquet vectorized reader, we'll need to
> capture the repetition & definition level information associated with
> Catalyst Spark type converted from Parquet {{MessageType}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36940) Inline type hints for python/pyspark/sql/avro/functions.py
[ https://issues.apache.org/jira/browse/SPARK-36940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425143#comment-17425143 ] Apache Spark commented on SPARK-36940: -- User 'xinrong-databricks' has created a pull request for this issue: https://github.com/apache/spark/pull/34200 > Inline type hints for python/pyspark/sql/avro/functions.py > -- > > Key: SPARK-36940 > URL: https://issues.apache.org/jira/browse/SPARK-36940 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: Xinrong Meng >Priority: Major > > Inline type hints for python/pyspark/sql/avro/functions.py. > > Currently, we use stub files for type annotations, which don't support type > checks within function bodies. So we inline type hints to support that. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36940) Inline type hints for python/pyspark/sql/avro/functions.py
[ https://issues.apache.org/jira/browse/SPARK-36940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36940: Assignee: Apache Spark > Inline type hints for python/pyspark/sql/avro/functions.py > -- > > Key: SPARK-36940 > URL: https://issues.apache.org/jira/browse/SPARK-36940 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: Xinrong Meng >Assignee: Apache Spark >Priority: Major > > Inline type hints for python/pyspark/sql/avro/functions.py. > > Currently, we use stub files for type annotations, which don't support type > checks within function bodies. So we inline type hints to support that. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36940) Inline type hints for python/pyspark/sql/avro/functions.py
[ https://issues.apache.org/jira/browse/SPARK-36940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36940: Assignee: (was: Apache Spark) > Inline type hints for python/pyspark/sql/avro/functions.py > -- > > Key: SPARK-36940 > URL: https://issues.apache.org/jira/browse/SPARK-36940 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: Xinrong Meng >Priority: Major > > Inline type hints for python/pyspark/sql/avro/functions.py. > > Currently, we use stub files for type annotations, which don't support type > checks within function bodies. So we inline type hints to support that. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-36937) Change OrcSourceSuite to test both V1 and V2 sources.
[ https://issues.apache.org/jira/browse/SPARK-36937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-36937. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34194 [https://github.com/apache/spark/pull/34194] > Change OrcSourceSuite to test both V1 and V2 sources. > - > > Key: SPARK-36937 > URL: https://issues.apache.org/jira/browse/SPARK-36937 > Project: Spark > Issue Type: Improvement > Components: SQL, Tests >Affects Versions: 3.3.0 >Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Major > Fix For: 3.3.0 > > > There is no V2 test for the ORC source which implements > CommonFileDataSourceSuite while the corresponding ones exist for all other > built-in file-based datasources. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36936) spark-hadoop-cloud broken on release and only published via 3rd party repositories
[
https://issues.apache.org/jira/browse/SPARK-36936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425162#comment-17425162
]
Chao Sun commented on SPARK-36936:
--
[~colin.williams] which version of {{spark-hadoop-cloud}} you were using? I
think the above error shouldn't happen if the version is the same as the
Spark's version.
We've already started to publish {{spark-hadoop-cloud}} as part of the Spark
release procedure, see SPARK-35844.
> spark-hadoop-cloud broken on release and only published via 3rd party
> repositories
> --
>
> Key: SPARK-36936
> URL: https://issues.apache.org/jira/browse/SPARK-36936
> Project: Spark
> Issue Type: Bug
> Components: Input/Output
>Affects Versions: 3.1.1, 3.1.2
> Environment: name:=spark-demo
> version := "0.0.1"
> scalaVersion := "2.12.12"
> lazy val app = (project in file("app")).settings(
> assemblyPackageScala / assembleArtifact := false,
> assembly / assemblyJarName := "uber.jar",
> assembly / mainClass := Some("com.example.Main"),
> // more settings here ...
> )
> resolvers += "Cloudera" at
> "https://repository.cloudera.com/artifactory/cloudera-repos/";
> libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.1.2" %
> "provided"
> libraryDependencies += "org.apache.spark" %% "spark-hadoop-cloud" %
> "3.1.1.3.1.7270.0-253"
> libraryDependencies += "org.apache.hadoop" % "hadoop-aws" %
> "3.1.1.7.2.7.0-184"
> libraryDependencies += "com.amazonaws" % "aws-java-sdk-bundle" % "1.11.901"
> libraryDependencies += "org.scalatest" %% "scalatest" % "3.0.1" % "test"
> // test suite settings
> fork in Test := true
> javaOptions ++= Seq("-Xms512M", "-Xmx2048M", "-XX:MaxPermSize=2048M",
> "-XX:+CMSClassUnloadingEnabled")
> // Show runtime of tests
> testOptions in Test += Tests.Argument(TestFrameworks.ScalaTest, "-oD")
> ___
>
> import org.apache.spark.sql.SparkSession
> object SparkApp {
> def main(args: Array[String]){
> val spark = SparkSession.builder().master("local")
> //.config("spark.jars.repositories",
> "https://repository.cloudera.com/artifactory/cloudera-repos/";)
> //.config("spark.jars.packages",
> "org.apache.spark:spark-hadoop-cloud_2.12:3.1.1.3.1.7270.0-253")
> .appName("spark session").getOrCreate
> val jsonDF = spark.read.json("s3a://path-to-bucket/compact.json")
> val csvDF = spark.read.format("csv").load("s3a://path-to-bucket/some.csv")
> jsonDF.show()
> csvDF.show()
> }
> }
>Reporter: Colin Williams
>Priority: Major
>
> The spark docmentation suggests using `spark-hadoop-cloud` to read / write
> from S3 in [https://spark.apache.org/docs/latest/cloud-integration.html] .
> However artifacts are currently published via only 3rd party resolvers in
> [https://mvnrepository.com/artifact/org.apache.spark/spark-hadoop-cloud]
> including Cloudera and Palantir.
>
> Then apache spark documentation is providing a 3rd party solution for object
> stores including S3. Furthermore, if you follow the instructions and include
> one of the 3rd party jars IE the Cloudera jar with the spark 3.1.2 release
> and try to access object store, the following exception is returned.
>
> ```
> Exception in thread "main" java.lang.NoSuchMethodError: 'void
> com.google.common.base.Preconditions.checkArgument(boolean, java.lang.String,
> java.lang.Object, java.lang.Object)'
> at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:894)
> at org.apache.hadoop.fs.s3a.S3AUtils.lookupPassword(S3AUtils.java:870)
> at
> org.apache.hadoop.fs.s3a.S3AUtils.getEncryptionAlgorithm(S3AUtils.java:1605)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:363)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124)
> at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
> at
> org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:46)
> at
> org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:377)
> at
> org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:325)
> at
> org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:307)
> at scala.Option.getOrElse(Option.scala:189)
> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:307)
> at org.apache.spark.sql.DataFrameReader.json(DataFrameReader.scala:519)
> at org.apache.spark.sql.DataFrameRead
[jira] [Commented] (SPARK-32929) StreamSuite failure on IBM Z: - SPARK-20432: union one stream with itself
[
https://issues.apache.org/jira/browse/SPARK-32929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425179#comment-17425179
]
Kun Lu commented on SPARK-32929:
This issue has been addressed in commit
[3a299aa|https://github.com/apache/spark/commit/3a299aa6480ac22501512cd0310d31a441d7dfdc].
It does not happen on SPARK 3.1.2 version on IBM Z.
> StreamSuite failure on IBM Z: - SPARK-20432: union one stream with itself
> -
>
> Key: SPARK-32929
> URL: https://issues.apache.org/jira/browse/SPARK-32929
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.1
> Environment: openjdk version "11.0.8" 2020-07-14
> OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.8+10)
> OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.8+10, mixed mode)
> Linux 4.15.0-117-generic #118-Ubuntu SMP Fri Sep 4 20:00:20 UTC 2020 s390x
> s390x s390x GNU/Linux
>Reporter: Michael Munday
>Priority: Minor
> Labels: big-endian
>
> I am getting zeros in the output of this test on IBM Z. This is a big-endian
> system. See error below.
> I think this issue is related to the use of {{IntegerType}} in the schema for
> {{FakeDefaultSource}}. Modifying the schema to use {{LongType}} fixes the
> issue. Another workaround is to remove {{.select("a")}} (see patch below).
> My working theory is that long data (longs are generated by Range) is being
> read using unsafe int operations (as specified in the schema). This would
> 'work' on little-endian systems but not big-endian systems. I'm still working
> to figure out what the mechanism is and I'd appreciate any hints or insights.
> The error looks like this:
> {noformat}
> - SPARK-20432: union one stream with itself *** FAILED ***
> Decoded objects do not match expected objects:
> expected: WrappedArray(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 0, 1, 2, 3, 4, 5,
> 6, 7, 8, 9, 10)
> actual: WrappedArray(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0, 0)
> assertnotnull(upcast(getcolumnbyordinal(0, LongType), LongType, - root
> class: "scala.Long"))
> +- upcast(getcolumnbyordinal(0, LongType), LongType, - root class:
> "scala.Long")
> +- getcolumnbyordinal(0, LongType) (QueryTest.scala:88)
> {noformat}
> This change fixes the issue:
> {code:java}
> --- a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala
> +++ b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala
> @@ -45,7 +45,7 @@ import org.apache.spark.sql.functions._
> import org.apache.spark.sql.internal.SQLConf
> import org.apache.spark.sql.sources.StreamSourceProvider
> import org.apache.spark.sql.streaming.util.{BlockOnStopSourceProvider,
> StreamManualClock}
> -import org.apache.spark.sql.types.{IntegerType, StructField, StructType}
> +import org.apache.spark.sql.types.{IntegerType, LongType, StructField,
> StructType}
> import org.apache.spark.util.Utils
> class StreamSuite extends StreamTest {
> @@ -1265,7 +1265,7 @@ class StreamSuite extends StreamTest {
> }
> abstract class FakeSource extends StreamSourceProvider {
> - private val fakeSchema = StructType(StructField("a", IntegerType) :: Nil)
> + private val fakeSchema = StructType(StructField("a", LongType) :: Nil)
>override def sourceSchema(
>spark: SQLContext,
> @@ -1287,7 +1287,7 @@ class FakeDefaultSource extends FakeSource {
> new Source {
>private var offset = -1L
> - override def schema: StructType = StructType(StructField("a",
> IntegerType) :: Nil)
> + override def schema: StructType = StructType(StructField("a",
> LongType) :: Nil)
>override def getOffset: Option[Offset] = {
> if (offset >= 10) {
> {code}
> Alternatively, this change also fixes the issue:
> {code:java}
> --- a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala
> +++ b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala
> @@ -154,7 +154,7 @@ class StreamSuite extends StreamTest {
>}
>
>test("SPARK-20432: union one stream with itself") {
> -val df =
> spark.readStream.format(classOf[FakeDefaultSource].getName).load().select("a")
> +val df =
> spark.readStream.format(classOf[FakeDefaultSource].getName).load()
> val unioned = df.union(df)
> withTempDir { outputDir =>
>withTempDir { checkpointDir =>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-32952) Test failure on IBM Z: CoalesceShufflePartitionsSuite: - determining the number of reducers: complex query 1
[
https://issues.apache.org/jira/browse/SPARK-32952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425187#comment-17425187
]
Kun Lu commented on SPARK-32952:
This issue still exists on Spark v3.1.2 on IBM Z. Any updates from the
community would be greatly appreciated.
> Test failure on IBM Z: CoalesceShufflePartitionsSuite: - determining the
> number of reducers: complex query 1
>
>
> Key: SPARK-32952
> URL: https://issues.apache.org/jira/browse/SPARK-32952
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.0.1
> Environment: Linux on IBM Z (s390x).
>Reporter: Michael Munday
>Priority: Minor
> Labels: big-endian
>
> I'm seeing the test 'CoalesceShufflePartitionsSuite: - determining the number
> of reducers: complex query 1' fail on IBM Z with the wrong number of
> partitions (1 instead of 2). It's strange because none of the other tests
> fail.
> I'd be grateful for any hints as to how the number of partitions is
> calculated. Could that calculation be affected by incorrect unsafe code? Is
> there a way to trace the calculation?
> {noformat}
> CoalesceShufflePartitionsSuite:
> - determining the number of reducers: aggregate
> operator(minNumPostShufflePartitions: 5)
> - determining the number of reducers: join
> operator(minNumPostShufflePartitions: 5)
> - determining the number of reducers: complex query
> 1(minNumPostShufflePartitions: 5)
> - determining the number of reducers: complex query
> 2(minNumPostShufflePartitions: 5)
> - determining the number of reducers: plan already
> partitioned(minNumPostShufflePartitions: 5)
> - determining the number of reducers: aggregate operator
> - determining the number of reducers: join operator
> - determining the number of reducers: complex query 1 *** FAILED ***
> 1 did not equal 2 (CoalesceShufflePartitionsSuite.scala:221)
> - determining the number of reducers: complex query 2
> - determining the number of reducers: plan already partitioned
> - SPARK-24705 adaptive query execution works correctly when exchange reuse
> enabled
> - Do not reduce the number of shuffle partition for repartition
> - Union two datasets with different pre-shuffle partition number{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-36941) Check saving of a dataframe with ANSI intervals to a Hive parquet table
Max Gekk created SPARK-36941: Summary: Check saving of a dataframe with ANSI intervals to a Hive parquet table Key: SPARK-36941 URL: https://issues.apache.org/jira/browse/SPARK-36941 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 3.3.0 Reporter: Max Gekk Assignee: Max Gekk Add a test which checks saving of a dataframe with ANSI intervals to a Hive table using parquet datasource. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36941) Check saving of a dataframe with ANSI intervals to a Hive parquet table
[ https://issues.apache.org/jira/browse/SPARK-36941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425196#comment-17425196 ] Max Gekk commented on SPARK-36941: -- I am working on this. > Check saving of a dataframe with ANSI intervals to a Hive parquet table > --- > > Key: SPARK-36941 > URL: https://issues.apache.org/jira/browse/SPARK-36941 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Max Gekk >Assignee: Max Gekk >Priority: Major > > Add a test which checks saving of a dataframe with ANSI intervals to a Hive > table using parquet datasource. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36941) Check saving of a dataframe with ANSI intervals to a Hive parquet table
[ https://issues.apache.org/jira/browse/SPARK-36941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425201#comment-17425201 ] Apache Spark commented on SPARK-36941: -- User 'MaxGekk' has created a pull request for this issue: https://github.com/apache/spark/pull/34201 > Check saving of a dataframe with ANSI intervals to a Hive parquet table > --- > > Key: SPARK-36941 > URL: https://issues.apache.org/jira/browse/SPARK-36941 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Max Gekk >Assignee: Max Gekk >Priority: Major > > Add a test which checks saving of a dataframe with ANSI intervals to a Hive > table using parquet datasource. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36941) Check saving of a dataframe with ANSI intervals to a Hive parquet table
[ https://issues.apache.org/jira/browse/SPARK-36941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425202#comment-17425202 ] Apache Spark commented on SPARK-36941: -- User 'MaxGekk' has created a pull request for this issue: https://github.com/apache/spark/pull/34201 > Check saving of a dataframe with ANSI intervals to a Hive parquet table > --- > > Key: SPARK-36941 > URL: https://issues.apache.org/jira/browse/SPARK-36941 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Max Gekk >Assignee: Max Gekk >Priority: Major > > Add a test which checks saving of a dataframe with ANSI intervals to a Hive > table using parquet datasource. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36941) Check saving of a dataframe with ANSI intervals to a Hive parquet table
[ https://issues.apache.org/jira/browse/SPARK-36941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36941: Assignee: Apache Spark (was: Max Gekk) > Check saving of a dataframe with ANSI intervals to a Hive parquet table > --- > > Key: SPARK-36941 > URL: https://issues.apache.org/jira/browse/SPARK-36941 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Max Gekk >Assignee: Apache Spark >Priority: Major > > Add a test which checks saving of a dataframe with ANSI intervals to a Hive > table using parquet datasource. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36941) Check saving of a dataframe with ANSI intervals to a Hive parquet table
[ https://issues.apache.org/jira/browse/SPARK-36941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36941: Assignee: Max Gekk (was: Apache Spark) > Check saving of a dataframe with ANSI intervals to a Hive parquet table > --- > > Key: SPARK-36941 > URL: https://issues.apache.org/jira/browse/SPARK-36941 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Max Gekk >Assignee: Max Gekk >Priority: Major > > Add a test which checks saving of a dataframe with ANSI intervals to a Hive > table using parquet datasource. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36941) Check saving of a dataframe with ANSI intervals to a Hive parquet table
[ https://issues.apache.org/jira/browse/SPARK-36941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36941: Assignee: Apache Spark (was: Max Gekk) > Check saving of a dataframe with ANSI intervals to a Hive parquet table > --- > > Key: SPARK-36941 > URL: https://issues.apache.org/jira/browse/SPARK-36941 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Max Gekk >Assignee: Apache Spark >Priority: Major > > Add a test which checks saving of a dataframe with ANSI intervals to a Hive > table using parquet datasource. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35520) Spark-SQL test fails on IBM Z for certain config combinations.
[
https://issues.apache.org/jira/browse/SPARK-35520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425205#comment-17425205
]
Kun Lu commented on SPARK-35520:
I've also observed this issue on Spark v3.1.2 on IBM Z. Any comments from the
community would be greatly appreciated.
> Spark-SQL test fails on IBM Z for certain config combinations.
> --
>
> Key: SPARK-35520
> URL: https://issues.apache.org/jira/browse/SPARK-35520
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 3.1.1
>Reporter: Simrit Kaur
>Priority: Major
>
> Some queries of SQL related test cases: in-joins.sql, in-order-by.sql,
> not-in-group-by.sql and SubquerySuite.scala are failing with specific
> configuration combinations on IBM Z(s390x).
> For example:
> sql("select * from l where a = 6 and a not in (select c from r where c is not
> null)") query from SubquerySuite.scala fails for following config
> combinations:
> |enableNAAJ|enableAQE|enableCodegen|
> |TRUE|FALSE|FALSE|
> |TRUE|TRUE|FALSE|
> The above combination is also causing 2 other queries in in-joins.sql and
> in-order-by.sql failing.
> Another query:
> SELECT Count(*)
> FROM (SELECT *
> FROM t2
> WHERE t2a NOT IN (SELECT t3a
> FROM t3
> WHERE t3h != t2h)) t2
> WHERE t2b NOT IN (SELECT Min(t2b)
> FROM t2
> WHERE t2b = t2b
> GROUP BY t2c);
> from not-in-group-by.sql is failing for following combinations:
> |enableAQE|enableCodegen|
> |FALSE|TRUE|
> |FALSE|FALSE|
>
> These Test cases are not failing for 3.0.1 release and I believe might have
> been introduced with
> [SPARK-32290|https://issues.apache.org/jira/browse/SPARK-32290] .
> There is another strange behaviour observed, if expected output is 1,3 , I am
> getting 1, 3, 9. If I update the Golden file to expect 1, 3, 9, the output
> will be 1, 3.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-36942) Inline type hints for python/pyspark/sql/readwriter.py
Xinrong Meng created SPARK-36942: Summary: Inline type hints for python/pyspark/sql/readwriter.py Key: SPARK-36942 URL: https://issues.apache.org/jira/browse/SPARK-36942 Project: Spark Issue Type: Sub-task Components: PySpark Affects Versions: 3.3.0 Reporter: Xinrong Meng Inline type hints for python/pyspark/sql/readwriter.py. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36942) Inline type hints for python/pyspark/sql/readwriter.py
[ https://issues.apache.org/jira/browse/SPARK-36942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425213#comment-17425213 ] Xinrong Meng commented on SPARK-36942: -- I'm working on that. > Inline type hints for python/pyspark/sql/readwriter.py > -- > > Key: SPARK-36942 > URL: https://issues.apache.org/jira/browse/SPARK-36942 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: Xinrong Meng >Priority: Major > > Inline type hints for python/pyspark/sql/readwriter.py. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36114) Support subqueries with correlated non-equality predicates
[ https://issues.apache.org/jira/browse/SPARK-36114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425275#comment-17425275 ] Allison Wang commented on SPARK-36114: -- Support non-equality predicates can lift the restrictions added in SPARK-35080. > Support subqueries with correlated non-equality predicates > -- > > Key: SPARK-36114 > URL: https://issues.apache.org/jira/browse/SPARK-36114 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.2.0 >Reporter: Allison Wang >Priority: Major > > The new decorrelation framework is able to support subqueries with > non-equality predicates. For example: > SELECT * FROM t1 WHERE c1 = (SELECT SUM(c1) FROM t2 WHERE t1.c2 > t2.c2) > The restrictions in CheckAnlysis can be removed. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-36943) Improve error message for missing column
Karen Feng created SPARK-36943: -- Summary: Improve error message for missing column Key: SPARK-36943 URL: https://issues.apache.org/jira/browse/SPARK-36943 Project: Spark Issue Type: Improvement Components: Spark Core, SQL Affects Versions: 3.3.0 Reporter: Karen Feng Improve the error message for the case that a user asks for a column that does not exist. Today, the message is "cannot resolve 'foo' given input columns [bar, baz, froo]". We should sort the suggestion list by similarity and improve the grammar to remove lingo like "resolve." -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36943) Improve error message for missing column
[ https://issues.apache.org/jira/browse/SPARK-36943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36943: Assignee: (was: Apache Spark) > Improve error message for missing column > > > Key: SPARK-36943 > URL: https://issues.apache.org/jira/browse/SPARK-36943 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 3.3.0 >Reporter: Karen Feng >Priority: Major > > Improve the error message for the case that a user asks for a column that > does not exist. > Today, the message is "cannot resolve 'foo' given input columns [bar, baz, > froo]". > We should sort the suggestion list by similarity and improve the grammar to > remove lingo like "resolve." -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36943) Improve error message for missing column
[ https://issues.apache.org/jira/browse/SPARK-36943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425289#comment-17425289 ] Apache Spark commented on SPARK-36943: -- User 'karenfeng' has created a pull request for this issue: https://github.com/apache/spark/pull/34202 > Improve error message for missing column > > > Key: SPARK-36943 > URL: https://issues.apache.org/jira/browse/SPARK-36943 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 3.3.0 >Reporter: Karen Feng >Priority: Major > > Improve the error message for the case that a user asks for a column that > does not exist. > Today, the message is "cannot resolve 'foo' given input columns [bar, baz, > froo]". > We should sort the suggestion list by similarity and improve the grammar to > remove lingo like "resolve." -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36943) Improve error message for missing column
[ https://issues.apache.org/jira/browse/SPARK-36943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36943: Assignee: Apache Spark > Improve error message for missing column > > > Key: SPARK-36943 > URL: https://issues.apache.org/jira/browse/SPARK-36943 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 3.3.0 >Reporter: Karen Feng >Assignee: Apache Spark >Priority: Major > > Improve the error message for the case that a user asks for a column that > does not exist. > Today, the message is "cannot resolve 'foo' given input columns [bar, baz, > froo]". > We should sort the suggestion list by similarity and improve the grammar to > remove lingo like "resolve." -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36943) Improve error message for missing column
[ https://issues.apache.org/jira/browse/SPARK-36943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425290#comment-17425290 ] Apache Spark commented on SPARK-36943: -- User 'karenfeng' has created a pull request for this issue: https://github.com/apache/spark/pull/34202 > Improve error message for missing column > > > Key: SPARK-36943 > URL: https://issues.apache.org/jira/browse/SPARK-36943 > Project: Spark > Issue Type: Improvement > Components: Spark Core, SQL >Affects Versions: 3.3.0 >Reporter: Karen Feng >Priority: Major > > Improve the error message for the case that a user asks for a column that > does not exist. > Today, the message is "cannot resolve 'foo' given input columns [bar, baz, > froo]". > We should sort the suggestion list by similarity and improve the grammar to > remove lingo like "resolve." -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-36939) Add orphan migration page into list in PySpark documentation
[
https://issues.apache.org/jira/browse/SPARK-36939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-36939.
--
Fix Version/s: 3.2.0
Resolution: Fixed
Issue resolved by pull request 34195
[https://github.com/apache/spark/pull/34195]
> Add orphan migration page into list in PySpark documentation
>
>
> Key: SPARK-36939
> URL: https://issues.apache.org/jira/browse/SPARK-36939
> Project: Spark
> Issue Type: Documentation
> Components: docs, PySpark
>Affects Versions: 3.2.0
>Reporter: Hyukjin Kwon
>Assignee: Hyukjin Kwon
>Priority: Major
> Fix For: 3.2.0
>
>
> SPARK-36618 added a new migration guide page but that's mistakenly not added
> to {{spark/python/docs/source/migration_guideindex.rst}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36939) Add orphan migration page into list in PySpark documentation
[
https://issues.apache.org/jira/browse/SPARK-36939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-36939:
Assignee: Hyukjin Kwon
> Add orphan migration page into list in PySpark documentation
>
>
> Key: SPARK-36939
> URL: https://issues.apache.org/jira/browse/SPARK-36939
> Project: Spark
> Issue Type: Documentation
> Components: docs, PySpark
>Affects Versions: 3.2.0
>Reporter: Hyukjin Kwon
>Assignee: Hyukjin Kwon
>Priority: Major
>
> SPARK-36618 added a new migration guide page but that's mistakenly not added
> to {{spark/python/docs/source/migration_guideindex.rst}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-36939) Add orphan migration page into list in PySpark documentation
[
https://issues.apache.org/jira/browse/SPARK-36939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-36939:
-
Fix Version/s: (was: 3.2.0)
3.2.1
> Add orphan migration page into list in PySpark documentation
>
>
> Key: SPARK-36939
> URL: https://issues.apache.org/jira/browse/SPARK-36939
> Project: Spark
> Issue Type: Documentation
> Components: docs, PySpark
>Affects Versions: 3.2.0
>Reporter: Hyukjin Kwon
>Assignee: Hyukjin Kwon
>Priority: Major
> Fix For: 3.2.1
>
>
> SPARK-36618 added a new migration guide page but that's mistakenly not added
> to {{spark/python/docs/source/migration_guideindex.rst}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-36944) Remove unused python/pyspark/sql/__init__.pyi
dch nguyen created SPARK-36944: -- Summary: Remove unused python/pyspark/sql/__init__.pyi Key: SPARK-36944 URL: https://issues.apache.org/jira/browse/SPARK-36944 Project: Spark Issue Type: Sub-task Components: PySpark Affects Versions: 3.3.0 Reporter: dch nguyen -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36858) Spark API to apply same function to multiple columns
[
https://issues.apache.org/jira/browse/SPARK-36858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425297#comment-17425297
]
Hyukjin Kwon commented on SPARK-36858:
--
you could use var. e.g.)
{code}
var df = ...
colListToClean.foreach { c => df = df.withColumn(c, func(...)) }
{code}
or actually what you did with foldLeft looks making sense too. What API do you
have on your mind on this?
> Spark API to apply same function to multiple columns
>
>
> Key: SPARK-36858
> URL: https://issues.apache.org/jira/browse/SPARK-36858
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
>Affects Versions: 2.4.8, 3.1.2
>Reporter: Armand BERGES
>Priority: Minor
>
> Hi
> My team and I have regularly need to apply the same function to multiple
> columns at once.
> For example, we want to remove all non alphanumerical characters to each
> columns of our dataframes.
> When we hit this use case first, some people in my team were using this kind
> of code :
> {code:java}
> val colListToClean = ## Generate some list, could be very long.
> val dfToClean: DataFrame = ... ## This is the dataframe we want to clean
> def cleanFunction(colName: String): Column = ... ## Write some function to
> manipulate column based on its name.
> val dfCleaned = colListToClean.foldLeft(dfToClean)((df, colName) =>
> df.withColumn(colName, cleanFunction(colName)){code}
> This kind of code when applied on a large set of columns overloaded our
> driver (because a Dataframe is generated for each column to clean).
> Based on this issue, we developed some code to add two functions :
> * One to apply the same function to multiple columns
> * One to rename multiple columns based on a Map.
>
> I wonder if your ever ask your team to add such kind of API ? If you did, had
> you any kind of issue regarding the implementation ? If you didn't, is this
> any idea you could add to Spark ?
> Best regards,
>
> LvffY
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36944) Remove unused python/pyspark/sql/__init__.pyi
[ https://issues.apache.org/jira/browse/SPARK-36944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36944: Assignee: Apache Spark > Remove unused python/pyspark/sql/__init__.pyi > - > > Key: SPARK-36944 > URL: https://issues.apache.org/jira/browse/SPARK-36944 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Assignee: Apache Spark >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36944) Remove unused python/pyspark/sql/__init__.pyi
[ https://issues.apache.org/jira/browse/SPARK-36944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425298#comment-17425298 ] Apache Spark commented on SPARK-36944: -- User 'dchvn' has created a pull request for this issue: https://github.com/apache/spark/pull/34203 > Remove unused python/pyspark/sql/__init__.pyi > - > > Key: SPARK-36944 > URL: https://issues.apache.org/jira/browse/SPARK-36944 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36944) Remove unused python/pyspark/sql/__init__.pyi
[ https://issues.apache.org/jira/browse/SPARK-36944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36944: Assignee: (was: Apache Spark) > Remove unused python/pyspark/sql/__init__.pyi > - > > Key: SPARK-36944 > URL: https://issues.apache.org/jira/browse/SPARK-36944 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36944) Remove unused python/pyspark/sql/__init__.pyi
[ https://issues.apache.org/jira/browse/SPARK-36944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425299#comment-17425299 ] Apache Spark commented on SPARK-36944: -- User 'dchvn' has created a pull request for this issue: https://github.com/apache/spark/pull/34203 > Remove unused python/pyspark/sql/__init__.pyi > - > > Key: SPARK-36944 > URL: https://issues.apache.org/jira/browse/SPARK-36944 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-36945) Inline type hints for python/pyspark/sql/udf.py
dch nguyen created SPARK-36945: -- Summary: Inline type hints for python/pyspark/sql/udf.py Key: SPARK-36945 URL: https://issues.apache.org/jira/browse/SPARK-36945 Project: Spark Issue Type: Sub-task Components: PySpark Affects Versions: 3.3.0 Reporter: dch nguyen -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36945) Inline type hints for python/pyspark/sql/udf.py
[ https://issues.apache.org/jira/browse/SPARK-36945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425301#comment-17425301 ] dch nguyen commented on SPARK-36945: working on this > Inline type hints for python/pyspark/sql/udf.py > --- > > Key: SPARK-36945 > URL: https://issues.apache.org/jira/browse/SPARK-36945 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dch nguyen >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-36884) Inline type hints for python/pyspark/sql/session.py
[ https://issues.apache.org/jira/browse/SPARK-36884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36884. -- Fix Version/s: 3.3.0 Assignee: Takuya Ueshin Resolution: Fixed Fixed in https://github.com/apache/spark/pull/34136 > Inline type hints for python/pyspark/sql/session.py > --- > > Key: SPARK-36884 > URL: https://issues.apache.org/jira/browse/SPARK-36884 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: Takuya Ueshin >Assignee: Takuya Ueshin >Priority: Major > Fix For: 3.3.0 > > > Inline type hints for python/pyspark/sql/session.py from Inline type hints > for python/pyspark/sql/session.pyi. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-36918) unionByName shouldn't consider types when comparing structs
[ https://issues.apache.org/jira/browse/SPARK-36918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-36918. - Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34166 [https://github.com/apache/spark/pull/34166] > unionByName shouldn't consider types when comparing structs > --- > > Key: SPARK-36918 > URL: https://issues.apache.org/jira/browse/SPARK-36918 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.2.0 >Reporter: Adam Binford >Assignee: Adam Binford >Priority: Major > Fix For: 3.3.0 > > > Improvement/follow-on of https://issues.apache.org/jira/browse/SPARK-35290. > We use StructType.sameType to see if we need to recreate the struct, but this > can lead to false positives if the structure is the same but the types are > different, and will lead to simply creating a new struct that's exactly the > same as the original. This can cause significant overhead when unioning > multiple deeply nested nullable structs, as each time it's recreated it gets > wrapped in a If(IsNull()). Only comparing the field names can lead to more > efficient plans. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36918) unionByName shouldn't consider types when comparing structs
[ https://issues.apache.org/jira/browse/SPARK-36918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-36918: --- Assignee: Adam Binford > unionByName shouldn't consider types when comparing structs > --- > > Key: SPARK-36918 > URL: https://issues.apache.org/jira/browse/SPARK-36918 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.2.0 >Reporter: Adam Binford >Assignee: Adam Binford >Priority: Major > > Improvement/follow-on of https://issues.apache.org/jira/browse/SPARK-35290. > We use StructType.sameType to see if we need to recreate the struct, but this > can lead to false positives if the structure is the same but the types are > different, and will lead to simply creating a new struct that's exactly the > same as the original. This can cause significant overhead when unioning > multiple deeply nested nullable structs, as each time it's recreated it gets > wrapped in a If(IsNull()). Only comparing the field names can lead to more > efficient plans. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-36742) Fix ps.to_datetime with plurals of keys like years, months, days
[ https://issues.apache.org/jira/browse/SPARK-36742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-36742. -- Fix Version/s: 3.3.0 Assignee: dch nguyen Resolution: Fixed Fixed in https://github.com/apache/spark/pull/34182 > Fix ps.to_datetime with plurals of keys like years, months, days > > > Key: SPARK-36742 > URL: https://issues.apache.org/jira/browse/SPARK-36742 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: dgd_contributor >Assignee: dch nguyen >Priority: Major > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36874) Ambiguous Self-Join detected only on right dataframe
[
https://issues.apache.org/jira/browse/SPARK-36874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425329#comment-17425329
]
Apache Spark commented on SPARK-36874:
--
User 'sarutak' has created a pull request for this issue:
https://github.com/apache/spark/pull/34205
> Ambiguous Self-Join detected only on right dataframe
>
>
> Key: SPARK-36874
> URL: https://issues.apache.org/jira/browse/SPARK-36874
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.1.2
>Reporter: Vincent Doba
>Assignee: Kousuke Saruta
>Priority: Major
> Labels: correctness
> Fix For: 3.2.0
>
>
> When joining two dataframes, if they share the same lineage and one dataframe
> is a transformation of the other, Ambiguous Self Join detection only works
> when transformed dataframe is the right dataframe.
> For instance {{df1}} and {{df2}} where {{df2}} is a filtered {{df1}},
> Ambiguous Self Join detection only works when {{df2}} is the right dataframe:
> - {{df1.join(df2, ...)}} correctly fails with Ambiguous Self Join error
> - {{df2.join(df1, ...)}} returns a valid dataframe
> h1. Minimum Reproducible example
> h2. Code
> {code:scala}
> import sparkSession.implicit._
> val df1 = Seq((1, 2, "A1"),(2, 1, "A2")).toDF("key1", "key2", "value")
> val df2 = df1.filter($"value" === "A2")
> df2.join(df1, df1("key1") === df2("key2")).show()
> {code}
> h2. Expected Result
> Throw the following exception:
> {code}
> Exception in thread "main" org.apache.spark.sql.AnalysisException: Column
> key2#11 are ambiguous. It's probably because you joined several Datasets
> together, and some of these Datasets are the same. This column points to one
> of the Datasets but Spark is unable to figure out which one. Please alias the
> Datasets with different names via `Dataset.as` before joining them, and
> specify the column using qualified name, e.g. `df.as("a").join(df.as("b"),
> $"a.id" > $"b.id")`. You can also set
> spark.sql.analyzer.failAmbiguousSelfJoin to false to disable this check.
> at
> org.apache.spark.sql.execution.analysis.DetectAmbiguousSelfJoin$.apply(DetectAmbiguousSelfJoin.scala:157)
> at
> org.apache.spark.sql.execution.analysis.DetectAmbiguousSelfJoin$.apply(DetectAmbiguousSelfJoin.scala:43)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:216)
> at
> scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
> at
> scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
> at scala.collection.immutable.List.foldLeft(List.scala:91)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:213)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:205)
> at scala.collection.immutable.List.foreach(List.scala:431)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:205)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:196)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:190)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:155)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:183)
> at
> org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:88)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:183)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:174)
> at
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:228)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:173)
> at
> org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:73)
> at
> org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
> at
> org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:143)
> at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
> at
> org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:143)
> at
> org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:73)
> at
> org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:71)
> at
> org.apache.spark.sql.execution.QueryExecution.assertAnalyzed
[jira] [Commented] (SPARK-36874) Ambiguous Self-Join detected only on right dataframe
[
https://issues.apache.org/jira/browse/SPARK-36874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425330#comment-17425330
]
Apache Spark commented on SPARK-36874:
--
User 'sarutak' has created a pull request for this issue:
https://github.com/apache/spark/pull/34205
> Ambiguous Self-Join detected only on right dataframe
>
>
> Key: SPARK-36874
> URL: https://issues.apache.org/jira/browse/SPARK-36874
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.1.2
>Reporter: Vincent Doba
>Assignee: Kousuke Saruta
>Priority: Major
> Labels: correctness
> Fix For: 3.2.0
>
>
> When joining two dataframes, if they share the same lineage and one dataframe
> is a transformation of the other, Ambiguous Self Join detection only works
> when transformed dataframe is the right dataframe.
> For instance {{df1}} and {{df2}} where {{df2}} is a filtered {{df1}},
> Ambiguous Self Join detection only works when {{df2}} is the right dataframe:
> - {{df1.join(df2, ...)}} correctly fails with Ambiguous Self Join error
> - {{df2.join(df1, ...)}} returns a valid dataframe
> h1. Minimum Reproducible example
> h2. Code
> {code:scala}
> import sparkSession.implicit._
> val df1 = Seq((1, 2, "A1"),(2, 1, "A2")).toDF("key1", "key2", "value")
> val df2 = df1.filter($"value" === "A2")
> df2.join(df1, df1("key1") === df2("key2")).show()
> {code}
> h2. Expected Result
> Throw the following exception:
> {code}
> Exception in thread "main" org.apache.spark.sql.AnalysisException: Column
> key2#11 are ambiguous. It's probably because you joined several Datasets
> together, and some of these Datasets are the same. This column points to one
> of the Datasets but Spark is unable to figure out which one. Please alias the
> Datasets with different names via `Dataset.as` before joining them, and
> specify the column using qualified name, e.g. `df.as("a").join(df.as("b"),
> $"a.id" > $"b.id")`. You can also set
> spark.sql.analyzer.failAmbiguousSelfJoin to false to disable this check.
> at
> org.apache.spark.sql.execution.analysis.DetectAmbiguousSelfJoin$.apply(DetectAmbiguousSelfJoin.scala:157)
> at
> org.apache.spark.sql.execution.analysis.DetectAmbiguousSelfJoin$.apply(DetectAmbiguousSelfJoin.scala:43)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:216)
> at
> scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
> at
> scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
> at scala.collection.immutable.List.foldLeft(List.scala:91)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:213)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:205)
> at scala.collection.immutable.List.foreach(List.scala:431)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:205)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:196)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:190)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:155)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:183)
> at
> org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:88)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:183)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:174)
> at
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:228)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:173)
> at
> org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:73)
> at
> org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
> at
> org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:143)
> at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
> at
> org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:143)
> at
> org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:73)
> at
> org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:71)
> at
> org.apache.spark.sql.execution.QueryExecution.assertAnalyzed
[jira] [Commented] (SPARK-34634) Self-join with script transformation failed to resolve attribute correctly
[
https://issues.apache.org/jira/browse/SPARK-34634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425332#comment-17425332
]
Apache Spark commented on SPARK-34634:
--
User 'sarutak' has created a pull request for this issue:
https://github.com/apache/spark/pull/34205
> Self-join with script transformation failed to resolve attribute correctly
> --
>
> Key: SPARK-34634
> URL: https://issues.apache.org/jira/browse/SPARK-34634
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 2.4.6, 2.4.7,
> 3.0.0, 3.0.1, 3.0.2, 3.1.0, 3.1.1
>Reporter: EdisonWang
>Assignee: EdisonWang
>Priority: Minor
> Fix For: 3.2.0
>
>
> To reproduce,
> {code:java}
> // code placeholder
> create temporary view t as select * from values 0, 1, 2 as t(a);
> WITH temp AS (
> SELECT TRANSFORM(a) USING 'cat' AS (b string) FROM t
> )
> SELECT t1.b FROM temp t1 JOIN temp t2 ON t1.b = t2.b
> {code}
>
> Spark will throw AnalysisException
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-34634) Self-join with script transformation failed to resolve attribute correctly
[
https://issues.apache.org/jira/browse/SPARK-34634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425331#comment-17425331
]
Apache Spark commented on SPARK-34634:
--
User 'sarutak' has created a pull request for this issue:
https://github.com/apache/spark/pull/34205
> Self-join with script transformation failed to resolve attribute correctly
> --
>
> Key: SPARK-34634
> URL: https://issues.apache.org/jira/browse/SPARK-34634
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 2.4.6, 2.4.7,
> 3.0.0, 3.0.1, 3.0.2, 3.1.0, 3.1.1
>Reporter: EdisonWang
>Assignee: EdisonWang
>Priority: Minor
> Fix For: 3.2.0
>
>
> To reproduce,
> {code:java}
> // code placeholder
> create temporary view t as select * from values 0, 1, 2 as t(a);
> WITH temp AS (
> SELECT TRANSFORM(a) USING 'cat' AS (b string) FROM t
> )
> SELECT t1.b FROM temp t1 JOIN temp t2 ON t1.b = t2.b
> {code}
>
> Spark will throw AnalysisException
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-33913) Upgrade Kafka to 2.8.0
[ https://issues.apache.org/jira/browse/SPARK-33913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33913: -- Fix Version/s: 3.2.0 > Upgrade Kafka to 2.8.0 > -- > > Key: SPARK-33913 > URL: https://issues.apache.org/jira/browse/SPARK-33913 > Project: Spark > Issue Type: Sub-task > Components: Build, DStreams >Affects Versions: 3.2.0 >Reporter: dengziming >Assignee: Dongjoon Hyun >Priority: Major > Fix For: 3.2.0 > > > This issue aims to upgrade Kafka client to 2.8.0. > Note that Kafka 2.8.0 uses ZSTD JNI 1.4.9-1 like Apache Spark 3.2.0. > *RELEASE NOTE* > - https://downloads.apache.org/kafka/2.8.0/RELEASE_NOTES.html > - https://downloads.apache.org/kafka/2.7.0/RELEASE_NOTES.html > This will bring the latest client-side improvement and bug fixes like the > following examples. > - KAFKA-10631 ProducerFencedException is not Handled on Offest Commit > - KAFKA-10134 High CPU issue during rebalance in Kafka consumer after > upgrading to 2.5 > - KAFKA-12193 Re-resolve IPs when a client is disconnected > - KAFKA-10090 Misleading warnings: The configuration was supplied but isn't a > known config > - KAFKA-9263 The new hw is added to incorrect log when > ReplicaAlterLogDirsThread is replacing log > - KAFKA-10607 Ensure the error counts contains the NONE > - KAFKA-10458 Need a way to update quota for TokenBucket registered with > Sensor > - KAFKA-10503 MockProducer doesn't throw ClassCastException when no partition > for topic -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36930) Support ps.MultiIndex.dtypes
[
https://issues.apache.org/jira/browse/SPARK-36930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425335#comment-17425335
]
Apache Spark commented on SPARK-36930:
--
User 'dchvn' has created a pull request for this issue:
https://github.com/apache/spark/pull/34206
> Support ps.MultiIndex.dtypes
>
>
> Key: SPARK-36930
> URL: https://issues.apache.org/jira/browse/SPARK-36930
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
>Affects Versions: 3.3.0
>Reporter: dch nguyen
>Assignee: dch nguyen
>Priority: Major
> Fix For: 3.3.0
>
>
> when MultiIndex.dtypes is supported, we can use:
> {code:java}
> >>> idx = pd.MultiIndex.from_arrays([[0, 1, 2, 3, 4, 5, 6, 7, 8], [1, 2, 3,
> >>> 4, 5, 6, 7, 8, 9]], names=("zero", "one"))
> >>> pdf = pd.DataFrame(
> ... {"a": [1, 2, 3, 4, 5, 6, 7, 8, 9], "b": [4, 5, 6, 3, 2, 1, 0, 0, 0]},
> ... index=idx,
> ... )
> >>> psdf = ps.from_pandas(pdf)
> >>> ps.DataFrame[psdf.index.dtypes, psdf.dtypes]
> typing.Tuple[pyspark.pandas.typedef.typehints.IndexNameType,
> pyspark.pandas.typedef.typehints.IndexNameType,
> pyspark.pandas.typedef.typehints.NameType,
> pyspark.pandas.typedef.typehints.NameType]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36711) Support multi-index in new syntax
[ https://issues.apache.org/jira/browse/SPARK-36711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425338#comment-17425338 ] Apache Spark commented on SPARK-36711: -- User 'dchvn' has created a pull request for this issue: https://github.com/apache/spark/pull/34207 > Support multi-index in new syntax > - > > Key: SPARK-36711 > URL: https://issues.apache.org/jira/browse/SPARK-36711 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: Hyukjin Kwon >Assignee: dch nguyen >Priority: Major > Fix For: 3.3.0 > > > Support multi-index in the new syntax SPARK-36709 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36304) Refactor fifteenth set of 20 query execution errors to use error classes
[
https://issues.apache.org/jira/browse/SPARK-36304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425341#comment-17425341
]
Apache Spark commented on SPARK-36304:
--
User 'dchvn' has created a pull request for this issue:
https://github.com/apache/spark/pull/34208
> Refactor fifteenth set of 20 query execution errors to use error classes
>
>
> Key: SPARK-36304
> URL: https://issues.apache.org/jira/browse/SPARK-36304
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core, SQL
>Affects Versions: 3.2.0
>Reporter: Karen Feng
>Priority: Major
>
> Refactor some exceptions in
> [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala]
> to use error classes.
> There are currently ~350 exceptions in this file; so this PR only focuses on
> the fifteenth set of 20.
> {code:java}
> unsupportedOperationExceptionError
> nullLiteralsCannotBeCastedError
> notUserDefinedTypeError
> cannotLoadUserDefinedTypeError
> timeZoneIdNotSpecifiedForTimestampTypeError
> notPublicClassError
> primitiveTypesNotSupportedError
> fieldIndexOnRowWithoutSchemaError
> valueIsNullError
> onlySupportDataSourcesProvidingFileFormatError
> failToSetOriginalPermissionBackError
> failToSetOriginalACLBackError
> multiFailuresInStageMaterializationError
> unrecognizedCompressionSchemaTypeIDError
> getParentLoggerNotImplementedError
> cannotCreateParquetConverterForTypeError
> cannotCreateParquetConverterForDecimalTypeError
> cannotCreateParquetConverterForDataTypeError
> cannotAddMultiPartitionsOnNonatomicPartitionTableError
> userSpecifiedSchemaUnsupportedByDataSourceError
> {code}
> For more detail, see the parent ticket SPARK-36094.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36304) Refactor fifteenth set of 20 query execution errors to use error classes
[
https://issues.apache.org/jira/browse/SPARK-36304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425342#comment-17425342
]
Apache Spark commented on SPARK-36304:
--
User 'dchvn' has created a pull request for this issue:
https://github.com/apache/spark/pull/34208
> Refactor fifteenth set of 20 query execution errors to use error classes
>
>
> Key: SPARK-36304
> URL: https://issues.apache.org/jira/browse/SPARK-36304
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core, SQL
>Affects Versions: 3.2.0
>Reporter: Karen Feng
>Priority: Major
>
> Refactor some exceptions in
> [QueryExecutionErrors|https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala]
> to use error classes.
> There are currently ~350 exceptions in this file; so this PR only focuses on
> the fifteenth set of 20.
> {code:java}
> unsupportedOperationExceptionError
> nullLiteralsCannotBeCastedError
> notUserDefinedTypeError
> cannotLoadUserDefinedTypeError
> timeZoneIdNotSpecifiedForTimestampTypeError
> notPublicClassError
> primitiveTypesNotSupportedError
> fieldIndexOnRowWithoutSchemaError
> valueIsNullError
> onlySupportDataSourcesProvidingFileFormatError
> failToSetOriginalPermissionBackError
> failToSetOriginalACLBackError
> multiFailuresInStageMaterializationError
> unrecognizedCompressionSchemaTypeIDError
> getParentLoggerNotImplementedError
> cannotCreateParquetConverterForTypeError
> cannotCreateParquetConverterForDecimalTypeError
> cannotCreateParquetConverterForDataTypeError
> cannotAddMultiPartitionsOnNonatomicPartitionTableError
> userSpecifiedSchemaUnsupportedByDataSourceError
> {code}
> For more detail, see the parent ticket SPARK-36094.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36713) Document new syntax for specifying index type
[ https://issues.apache.org/jira/browse/SPARK-36713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36713: Assignee: (was: Apache Spark) > Document new syntax for specifying index type > - > > Key: SPARK-36713 > URL: https://issues.apache.org/jira/browse/SPARK-36713 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: Hyukjin Kwon >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36713) Document new syntax for specifying index type
[ https://issues.apache.org/jira/browse/SPARK-36713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425346#comment-17425346 ] Apache Spark commented on SPARK-36713: -- User 'HyukjinKwon' has created a pull request for this issue: https://github.com/apache/spark/pull/34210 > Document new syntax for specifying index type > - > > Key: SPARK-36713 > URL: https://issues.apache.org/jira/browse/SPARK-36713 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: Hyukjin Kwon >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-36713) Document new syntax for specifying index type
[ https://issues.apache.org/jira/browse/SPARK-36713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-36713: Assignee: Apache Spark > Document new syntax for specifying index type > - > > Key: SPARK-36713 > URL: https://issues.apache.org/jira/browse/SPARK-36713 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: Hyukjin Kwon >Assignee: Apache Spark >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36713) Document new syntax for specifying index type
[ https://issues.apache.org/jira/browse/SPARK-36713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425347#comment-17425347 ] Apache Spark commented on SPARK-36713: -- User 'HyukjinKwon' has created a pull request for this issue: https://github.com/apache/spark/pull/34210 > Document new syntax for specifying index type > - > > Key: SPARK-36713 > URL: https://issues.apache.org/jira/browse/SPARK-36713 > Project: Spark > Issue Type: Sub-task > Components: PySpark >Affects Versions: 3.3.0 >Reporter: Hyukjin Kwon >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-36941) Check saving of a dataframe with ANSI intervals to a Hive parquet table
[ https://issues.apache.org/jira/browse/SPARK-36941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Gekk resolved SPARK-36941. -- Fix Version/s: 3.3.0 Resolution: Fixed Issue resolved by pull request 34201 [https://github.com/apache/spark/pull/34201] > Check saving of a dataframe with ANSI intervals to a Hive parquet table > --- > > Key: SPARK-36941 > URL: https://issues.apache.org/jira/browse/SPARK-36941 > Project: Spark > Issue Type: Sub-task > Components: SQL >Affects Versions: 3.3.0 >Reporter: Max Gekk >Assignee: Apache Spark >Priority: Major > Fix For: 3.3.0 > > > Add a test which checks saving of a dataframe with ANSI intervals to a Hive > table using parquet datasource. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-34634) Self-join with script transformation failed to resolve attribute correctly
[
https://issues.apache.org/jira/browse/SPARK-34634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-34634:
-
Fix Version/s: 3.1.3
> Self-join with script transformation failed to resolve attribute correctly
> --
>
> Key: SPARK-34634
> URL: https://issues.apache.org/jira/browse/SPARK-34634
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 2.4.6, 2.4.7,
> 3.0.0, 3.0.1, 3.0.2, 3.1.0, 3.1.1
>Reporter: EdisonWang
>Assignee: EdisonWang
>Priority: Minor
> Fix For: 3.2.0, 3.1.3
>
>
> To reproduce,
> {code:java}
> // code placeholder
> create temporary view t as select * from values 0, 1, 2 as t(a);
> WITH temp AS (
> SELECT TRANSFORM(a) USING 'cat' AS (b string) FROM t
> )
> SELECT t1.b FROM temp t1 JOIN temp t2 ON t1.b = t2.b
> {code}
>
> Spark will throw AnalysisException
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-36874) Ambiguous Self-Join detected only on right dataframe
[
https://issues.apache.org/jira/browse/SPARK-36874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-36874:
-
Fix Version/s: 3.1.3
> Ambiguous Self-Join detected only on right dataframe
>
>
> Key: SPARK-36874
> URL: https://issues.apache.org/jira/browse/SPARK-36874
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.1.2
>Reporter: Vincent Doba
>Assignee: Kousuke Saruta
>Priority: Major
> Labels: correctness
> Fix For: 3.2.0, 3.1.3
>
>
> When joining two dataframes, if they share the same lineage and one dataframe
> is a transformation of the other, Ambiguous Self Join detection only works
> when transformed dataframe is the right dataframe.
> For instance {{df1}} and {{df2}} where {{df2}} is a filtered {{df1}},
> Ambiguous Self Join detection only works when {{df2}} is the right dataframe:
> - {{df1.join(df2, ...)}} correctly fails with Ambiguous Self Join error
> - {{df2.join(df1, ...)}} returns a valid dataframe
> h1. Minimum Reproducible example
> h2. Code
> {code:scala}
> import sparkSession.implicit._
> val df1 = Seq((1, 2, "A1"),(2, 1, "A2")).toDF("key1", "key2", "value")
> val df2 = df1.filter($"value" === "A2")
> df2.join(df1, df1("key1") === df2("key2")).show()
> {code}
> h2. Expected Result
> Throw the following exception:
> {code}
> Exception in thread "main" org.apache.spark.sql.AnalysisException: Column
> key2#11 are ambiguous. It's probably because you joined several Datasets
> together, and some of these Datasets are the same. This column points to one
> of the Datasets but Spark is unable to figure out which one. Please alias the
> Datasets with different names via `Dataset.as` before joining them, and
> specify the column using qualified name, e.g. `df.as("a").join(df.as("b"),
> $"a.id" > $"b.id")`. You can also set
> spark.sql.analyzer.failAmbiguousSelfJoin to false to disable this check.
> at
> org.apache.spark.sql.execution.analysis.DetectAmbiguousSelfJoin$.apply(DetectAmbiguousSelfJoin.scala:157)
> at
> org.apache.spark.sql.execution.analysis.DetectAmbiguousSelfJoin$.apply(DetectAmbiguousSelfJoin.scala:43)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:216)
> at
> scala.collection.LinearSeqOptimized.foldLeft(LinearSeqOptimized.scala:126)
> at
> scala.collection.LinearSeqOptimized.foldLeft$(LinearSeqOptimized.scala:122)
> at scala.collection.immutable.List.foldLeft(List.scala:91)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:213)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:205)
> at scala.collection.immutable.List.foreach(List.scala:431)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:205)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:196)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:190)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:155)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:183)
> at
> org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:88)
> at
> org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:183)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:174)
> at
> org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:228)
> at
> org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:173)
> at
> org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:73)
> at
> org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:111)
> at
> org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:143)
> at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
> at
> org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:143)
> at
> org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:73)
> at
> org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:71)
> at
> org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:63)
> at org.apache.spark.sql.Dataset$.$anonfun$ofRows$1(Dataset.scala:90)
> at org.apache.sp
