[jira] [Commented] (SPARK-35924) Add Java 17 ea build test to GitHub action

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371076#comment-17371076 ] Apache Spark commented on SPARK-35924: -- User 'williamhyun' has created a pull request for this

[jira] [Assigned] (SPARK-35924) Add Java 17 ea build test to GitHub action

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35924: Assignee: Apache Spark > Add Java 17 ea build test to GitHub action >

[jira] [Assigned] (SPARK-35924) Add Java 17 ea build test to GitHub action

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35924: Assignee: (was: Apache Spark) > Add Java 17 ea build test to GitHub action >

[jira] [Updated] (SPARK-35924) Add Java 17 ea build test to GitHub action

2021-06-28 Thread William Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Hyun updated SPARK-35924: - Component/s: Tests > Add Java 17 ea build test to GitHub action >

[jira] [Created] (SPARK-35924) Add Java 17 ea build test to GitHub action

2021-06-28 Thread William Hyun (Jira)
William Hyun created SPARK-35924: Summary: Add Java 17 ea build test to GitHub action Key: SPARK-35924 URL: https://issues.apache.org/jira/browse/SPARK-35924 Project: Spark Issue Type:

[jira] [Commented] (SPARK-35483) Add a new GA test job for the docker integration tests

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371063#comment-17371063 ] Apache Spark commented on SPARK-35483: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-35483) Add a new GA test job for the docker integration tests

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371062#comment-17371062 ] Apache Spark commented on SPARK-35483: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Resolved] (SPARK-35922) Upgrade maven-shade-plugin to 3.2.4

2021-06-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-35922. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33122

[jira] [Assigned] (SPARK-35922) Upgrade maven-shade-plugin to 3.2.4

2021-06-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-35922: - Assignee: Dongjoon Hyun > Upgrade maven-shade-plugin to 3.2.4 >

[jira] [Commented] (SPARK-34302) Migrate ALTER TABLE .. CHANGE COLUMN to new resolution framework

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371041#comment-17371041 ] Apache Spark commented on SPARK-34302: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-34302) Migrate ALTER TABLE .. CHANGE COLUMN to new resolution framework

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371040#comment-17371040 ] Apache Spark commented on SPARK-34302: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Resolved] (SPARK-35920) Upgrade to Chill 0.10.0

2021-06-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-35920. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33119

[jira] [Assigned] (SPARK-35920) Upgrade to Chill 0.10.0

2021-06-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-35920: - Assignee: Dongjoon Hyun > Upgrade to Chill 0.10.0 > --- > >

[jira] [Assigned] (SPARK-35923) Coalesce empty partition with mixed CoalescedPartitionSpec and PartialReducerPartitionSpec

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35923: Assignee: (was: Apache Spark) > Coalesce empty partition with mixed

[jira] [Assigned] (SPARK-35923) Coalesce empty partition with mixed CoalescedPartitionSpec and PartialReducerPartitionSpec

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35923: Assignee: Apache Spark > Coalesce empty partition with mixed CoalescedPartitionSpec and

[jira] [Commented] (SPARK-35923) Coalesce empty partition with mixed CoalescedPartitionSpec and PartialReducerPartitionSpec

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371038#comment-17371038 ] Apache Spark commented on SPARK-35923: -- User 'ulysses-you' has created a pull request for this

[jira] [Created] (SPARK-35923) Coalesce empty partition with mixed CoalescedPartitionSpec and PartialReducerPartitionSpec

2021-06-28 Thread XiDuo You (Jira)
XiDuo You created SPARK-35923: - Summary: Coalesce empty partition with mixed CoalescedPartitionSpec and PartialReducerPartitionSpec Key: SPARK-35923 URL: https://issues.apache.org/jira/browse/SPARK-35923

[jira] [Assigned] (SPARK-35876) array_zip unexpected column names

2021-06-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-35876: Assignee: Kousuke Saruta > array_zip unexpected column names >

[jira] [Resolved] (SPARK-35876) array_zip unexpected column names

2021-06-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-35876. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33106

[jira] [Assigned] (SPARK-35922) Upgrade maven-shade-plugin to 3.2.4

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35922: Assignee: (was: Apache Spark) > Upgrade maven-shade-plugin to 3.2.4 >

[jira] [Assigned] (SPARK-35922) Upgrade maven-shade-plugin to 3.2.4

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35922: Assignee: (was: Apache Spark) > Upgrade maven-shade-plugin to 3.2.4 >

[jira] [Commented] (SPARK-35922) Upgrade maven-shade-plugin to 3.2.4

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371014#comment-17371014 ] Apache Spark commented on SPARK-35922: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-35922) Upgrade maven-shade-plugin to 3.2.4

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35922: Assignee: Apache Spark > Upgrade maven-shade-plugin to 3.2.4 >

[jira] [Created] (SPARK-35922) Upgrade maven-shade-plugin to 3.2.4

2021-06-28 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-35922: - Summary: Upgrade maven-shade-plugin to 3.2.4 Key: SPARK-35922 URL: https://issues.apache.org/jira/browse/SPARK-35922 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-34537) Repartition miss/duplicated data

2021-06-28 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] angerszhu resolved SPARK-34537. --- Resolution: Not A Problem > Repartition miss/duplicated data > > >

[jira] [Commented] (SPARK-35914) Driver can't distribute task to executor because NullPointerException

2021-06-28 Thread Helt Long (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371009#comment-17371009 ] Helt Long commented on SPARK-35914: --- I guess this problem is related to hadoop-version, I use

[jira] [Assigned] (SPARK-35921) ${spark.yarn.isHadoopProvided} in config.properties is not edited if build with SBT

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35921: Assignee: Apache Spark (was: Kousuke Saruta) > ${spark.yarn.isHadoopProvided} in

[jira] [Assigned] (SPARK-35921) ${spark.yarn.isHadoopProvided} in config.properties is not edited if build with SBT

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35921: Assignee: Kousuke Saruta (was: Apache Spark) > ${spark.yarn.isHadoopProvided} in

[jira] [Commented] (SPARK-35921) ${spark.yarn.isHadoopProvided} in config.properties is not edited if build with SBT

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371007#comment-17371007 ] Apache Spark commented on SPARK-35921: -- User 'sarutak' has created a pull request for this issue:

[jira] [Resolved] (SPARK-35802) Error loading the stages/stage/ page in spark UI

2021-06-28 Thread Helt Long (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Helt Long resolved SPARK-35802. --- Resolution: Not A Bug I try hadoop-2.7.5, the problem is not existed. So I confirm it's because I

[jira] [Assigned] (SPARK-34302) Migrate ALTER TABLE .. CHANGE COLUMN to new resolution framework

2021-06-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-34302: --- Assignee: Terry Kim (was: Max Gekk) > Migrate ALTER TABLE .. CHANGE COLUMN to new

[jira] [Resolved] (SPARK-34302) Migrate ALTER TABLE .. CHANGE COLUMN to new resolution framework

2021-06-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-34302. - Resolution: Fixed Issue resolved by pull request 33113

[jira] [Resolved] (SPARK-35888) Add dataSize field in CoalescedPartitionSpec

2021-06-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-35888. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33079

[jira] [Assigned] (SPARK-35888) Add dataSize field in CoalescedPartitionSpec

2021-06-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-35888: --- Assignee: XiDuo You > Add dataSize field in CoalescedPartitionSpec >

[jira] [Updated] (SPARK-35921) ${spark.yarn.isHadoopProvided} in config.properties is not edited if build with SBT

2021-06-28 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-35921: --- Summary: ${spark.yarn.isHadoopProvided} in config.properties is not edited if build with

[jira] [Created] (SPARK-35921) The value of spark.yarn.isHadoopProvided property in config.properties is not edited if build with SBT

2021-06-28 Thread Kousuke Saruta (Jira)
Kousuke Saruta created SPARK-35921: -- Summary: The value of spark.yarn.isHadoopProvided property in config.properties is not edited if build with SBT Key: SPARK-35921 URL:

[jira] [Commented] (SPARK-26764) [SPIP] Spark Relational Cache

2021-06-28 Thread Adrian Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370993#comment-17370993 ] Adrian Wang commented on SPARK-26764: - [~zshao] Thanks for the interest. We created an open-source

[jira] [Commented] (SPARK-35899) Add a utility to convert connector expressions to Catalyst expressions

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370991#comment-17370991 ] Apache Spark commented on SPARK-35899: -- User 'aokolnychyi' has created a pull request for this

[jira] [Commented] (SPARK-35899) Add a utility to convert connector expressions to Catalyst expressions

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370990#comment-17370990 ] Apache Spark commented on SPARK-35899: -- User 'aokolnychyi' has created a pull request for this

[jira] [Resolved] (SPARK-33898) Support SHOW CREATE TABLE in v2

2021-06-28 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao resolved SPARK-33898. -- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32931

[jira] [Assigned] (SPARK-33898) Support SHOW CREATE TABLE in v2

2021-06-28 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao reassigned SPARK-33898: Assignee: PengLei > Support SHOW CREATE TABLE in v2 > --- > >

[jira] [Resolved] (SPARK-35344) Support creating a Column of numpy literal value in pandas-on-Spark

2021-06-28 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin resolved SPARK-35344. --- Fix Version/s: 3.2.0 Assignee: Xinrong Meng Resolution: Fixed Issue

[jira] [Commented] (SPARK-35920) Upgrade to Chill 0.10.0

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370971#comment-17370971 ] Apache Spark commented on SPARK-35920: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-35920) Upgrade to Chill 0.10.0

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35920: Assignee: (was: Apache Spark) > Upgrade to Chill 0.10.0 > --- >

[jira] [Commented] (SPARK-35920) Upgrade to Chill 0.10.0

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370970#comment-17370970 ] Apache Spark commented on SPARK-35920: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-35920) Upgrade to Chill 0.10.0

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35920: Assignee: Apache Spark > Upgrade to Chill 0.10.0 > --- > >

[jira] [Created] (SPARK-35920) Upgrade to Chill 0.10.0

2021-06-28 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-35920: - Summary: Upgrade to Chill 0.10.0 Key: SPARK-35920 URL: https://issues.apache.org/jira/browse/SPARK-35920 Project: Spark Issue Type: Task

[jira] [Created] (SPARK-35919) Support pathlib.PurePath-like objects in DataFrameReader / DataFrameWriter

2021-06-28 Thread Andrew Grigorev (Jira)
Andrew Grigorev created SPARK-35919: --- Summary: Support pathlib.PurePath-like objects in DataFrameReader / DataFrameWriter Key: SPARK-35919 URL: https://issues.apache.org/jira/browse/SPARK-35919

[jira] [Assigned] (SPARK-35917) Disable push-based shuffle until the feature is complete

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35917: Assignee: Apache Spark > Disable push-based shuffle until the feature is complete >

[jira] [Assigned] (SPARK-35917) Disable push-based shuffle until the feature is complete

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35917: Assignee: (was: Apache Spark) > Disable push-based shuffle until the feature is

[jira] [Commented] (SPARK-35917) Disable push-based shuffle until the feature is complete

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370877#comment-17370877 ] Apache Spark commented on SPARK-35917: -- User 'otterc' has created a pull request for this issue:

[jira] [Updated] (SPARK-35881) [SQL] AQE does not support columnar execution for the final query stage

2021-06-28 Thread Andy Grove (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy Grove updated SPARK-35881: --- Description: In AdaptiveSparkPlanExec, a query is broken down into stages and these stages are

[jira] [Created] (SPARK-35918) Consolidate logic between AvroSerializer/AvroDeserializer for schema mismatch handling and error messages

2021-06-28 Thread Erik Krogen (Jira)
Erik Krogen created SPARK-35918: --- Summary: Consolidate logic between AvroSerializer/AvroDeserializer for schema mismatch handling and error messages Key: SPARK-35918 URL:

[jira] [Assigned] (SPARK-35910) Update remoteBlockBytes based on merged block info

2021-06-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-35910: - Assignee: Kent Yao > Update remoteBlockBytes based on merged block info >

[jira] [Resolved] (SPARK-35910) Update remoteBlockBytes based on merged block info

2021-06-28 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-35910. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33109

[jira] [Commented] (SPARK-35859) Cleanup type hints.

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370842#comment-17370842 ] Apache Spark commented on SPARK-35859: -- User 'ueshin' has created a pull request for this issue:

[jira] [Assigned] (SPARK-35859) Cleanup type hints.

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35859: Assignee: Apache Spark > Cleanup type hints. > --- > >

[jira] [Assigned] (SPARK-35859) Cleanup type hints.

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35859: Assignee: (was: Apache Spark) > Cleanup type hints. > --- > >

[jira] [Commented] (SPARK-35859) Cleanup type hints.

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370840#comment-17370840 ] Apache Spark commented on SPARK-35859: -- User 'ueshin' has created a pull request for this issue:

[jira] [Commented] (SPARK-35259) ExternalBlockHandler metrics have misleading unit in the name

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370792#comment-17370792 ] Apache Spark commented on SPARK-35259: -- User 'xkrogen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-35259) ExternalBlockHandler metrics have misleading unit in the name

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35259: Assignee: Apache Spark > ExternalBlockHandler metrics have misleading unit in the name >

[jira] [Commented] (SPARK-35259) ExternalBlockHandler metrics have misleading unit in the name

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370790#comment-17370790 ] Apache Spark commented on SPARK-35259: -- User 'xkrogen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-35259) ExternalBlockHandler metrics have misleading unit in the name

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35259: Assignee: (was: Apache Spark) > ExternalBlockHandler metrics have misleading unit in

[jira] [Updated] (SPARK-35259) ExternalBlockHandler metrics have misleading unit in the name

2021-06-28 Thread Erik Krogen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erik Krogen updated SPARK-35259: Description: Today {{ExternalBlockHandler}} exposes a few {{Timer}} metrics: {code} // Time

[jira] [Commented] (SPARK-26764) [SPIP] Spark Relational Cache

2021-06-28 Thread Zheng Shao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370770#comment-17370770 ] Zheng Shao commented on SPARK-26764: [~adrian-wang] It has been over 2 years since this issue was

[jira] [Commented] (SPARK-29038) SPIP: Support Spark Materialized View

2021-06-28 Thread Zheng Shao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370768#comment-17370768 ] Zheng Shao commented on SPARK-29038: [~cltlfcjin] and [~AidenZhang]. I also recently started to look

[jira] [Updated] (SPARK-35917) Disable push-based shuffle until the feature is complete

2021-06-28 Thread Chandni Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated SPARK-35917: -- Description: Push-based shuffle is partially merged in apache master but some of the tasks

[jira] [Created] (SPARK-35917) Disable push-based shuffle until the feature is complete

2021-06-28 Thread Chandni Singh (Jira)
Chandni Singh created SPARK-35917: - Summary: Disable push-based shuffle until the feature is complete Key: SPARK-35917 URL: https://issues.apache.org/jira/browse/SPARK-35917 Project: Spark

[jira] [Resolved] (SPARK-35898) Converting arrays with RowToColumnConverter triggers assertion

2021-06-28 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-35898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell resolved SPARK-35898. --- Fix Version/s: 3.1.3 3.2.0 Resolution: Fixed > Converting

[jira] [Assigned] (SPARK-35898) Converting arrays with RowToColumnConverter triggers assertion

2021-06-28 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-35898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell reassigned SPARK-35898: - Assignee: Tom van Bussel > Converting arrays with RowToColumnConverter

[jira] [Commented] (SPARK-35802) Error loading the stages/stage/ page in spark UI

2021-06-28 Thread Helt Long (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370626#comment-17370626 ] Helt Long commented on SPARK-35802: --- Sound like it's key point, I will try higher hadoop version and

[jira] [Commented] (SPARK-35802) Error loading the stages/stage/ page in spark UI

2021-06-28 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370590#comment-17370590 ] Kousuke Saruta commented on SPARK-35802: Hadoop version is 2.9.0 on my environment. I ran an

[jira] [Assigned] (SPARK-35916) Support subtraction among Date/Timestamp/TimestampWithoutTZ

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35916: Assignee: Gengliang Wang (was: Apache Spark) > Support subtraction among

[jira] [Commented] (SPARK-35916) Support subtraction among Date/Timestamp/TimestampWithoutTZ

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370578#comment-17370578 ] Apache Spark commented on SPARK-35916: -- User 'gengliangwang' has created a pull request for this

[jira] [Assigned] (SPARK-35916) Support subtraction among Date/Timestamp/TimestampWithoutTZ

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35916: Assignee: Apache Spark (was: Gengliang Wang) > Support subtraction among

[jira] [Created] (SPARK-35916) Support subtraction among Date/Timestamp/TimestampWithoutTZ

2021-06-28 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-35916: -- Summary: Support subtraction among Date/Timestamp/TimestampWithoutTZ Key: SPARK-35916 URL: https://issues.apache.org/jira/browse/SPARK-35916 Project: Spark

[jira] [Commented] (SPARK-35912) [SQL] JSON read behavior is different depending on the cache setting when nullable is false.

2021-06-28 Thread Fu Chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370512#comment-17370512 ] Fu Chen commented on SPARK-35912: - Working on this > [SQL] JSON read behavior is different depending on

[jira] [Created] (SPARK-35915) Kafka doesn't recover from data loss

2021-06-28 Thread Yuval Yellin (Jira)
Yuval Yellin created SPARK-35915: Summary: Kafka doesn't recover from data loss Key: SPARK-35915 URL: https://issues.apache.org/jira/browse/SPARK-35915 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-35914) Driver can't distribute task to executor because NullPointerException

2021-06-28 Thread Helt Long (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Helt Long updated SPARK-35914: -- Attachment: webui stuck.png > Driver can't distribute task to executor because NullPointerException >

[jira] [Updated] (SPARK-35914) Driver can't distribute task to executor because NullPointerException

2021-06-28 Thread Helt Long (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Helt Long updated SPARK-35914: -- Attachment: stuck log.png > Driver can't distribute task to executor because NullPointerException >

[jira] [Assigned] (SPARK-35904) Collapse above RebalancePartitions

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35904: Assignee: Yuming Wang (was: Apache Spark) > Collapse above RebalancePartitions >

[jira] [Assigned] (SPARK-35904) Collapse above RebalancePartitions

2021-06-28 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35904: Assignee: Apache Spark (was: Yuming Wang) > Collapse above RebalancePartitions >

[jira] [Updated] (SPARK-35904) Collapse above RebalancePartitions

2021-06-28 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-35904: Fix Version/s: (was: 3.2.0) > Collapse above RebalancePartitions >

[jira] [Reopened] (SPARK-35904) Collapse above RebalancePartitions

2021-06-28 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reopened SPARK-35904: - Reverted at https://github.com/apache/spark/commit/108635af1708173a72bec0e36bf3f2cea5b088c4 >

[jira] [Updated] (SPARK-35672) Spark fails to launch executors with very large user classpath lists on YARN

2021-06-28 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-35672: - Fix Version/s: 3.1.3 > Spark fails to launch executors with very large user classpath lists on

[jira] [Commented] (SPARK-35802) Error loading the stages/stage/ page in spark UI

2021-06-28 Thread Helt Long (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370491#comment-17370491 ] Helt Long commented on SPARK-35802: --- [~sarutak] I used CDH 5.7.1, and used Spark on Yarn cluster

[jira] [Comment Edited] (SPARK-35802) Error loading the stages/stage/ page in spark UI

2021-06-28 Thread Helt Long (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370491#comment-17370491 ] Helt Long edited comment on SPARK-35802 at 6/28/21, 8:33 AM: - [~sarutak] I

[jira] [Updated] (SPARK-35802) Error loading the stages/stage/ page in spark UI

2021-06-28 Thread Helt Long (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Helt Long updated SPARK-35802: -- Description: I try to load the sparkUI page for a specific stage, I get the following error:

[jira] [Created] (SPARK-35914) Driver can't distribute task to executor because NullPointerException

2021-06-28 Thread Helt Long (Jira)
Helt Long created SPARK-35914: - Summary: Driver can't distribute task to executor because NullPointerException Key: SPARK-35914 URL: https://issues.apache.org/jira/browse/SPARK-35914 Project: Spark

[jira] [Resolved] (SPARK-35258) Enhance ESS ExternalBlockHandler with additional block rate-based metrics and histograms

2021-06-28 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35258. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request

[jira] [Assigned] (SPARK-35258) Enhance ESS ExternalBlockHandler with additional block rate-based metrics and histograms

2021-06-28 Thread Mridul Muralidharan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35258: --- Assignee: Erik Krogen > Enhance ESS ExternalBlockHandler with additional

[jira] [Assigned] (SPARK-35064) Group exception messages in spark/sql (catalyst)

2021-06-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-35064: --- Assignee: dgd_contributor > Group exception messages in spark/sql (catalyst) >

[jira] [Resolved] (SPARK-35064) Group exception messages in spark/sql (catalyst)

2021-06-28 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-35064. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32916

[jira] [Commented] (SPARK-35802) Error loading the stages/stage/ page in spark UI

2021-06-28 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17370448#comment-17370448 ] Kousuke Saruta commented on SPARK-35802: [~Heltman]According to the URL, I guess you run your

[jira] [Reopened] (SPARK-35802) Error loading the stages/stage/ page in spark UI

2021-06-28 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta reopened SPARK-35802: > Error loading the stages/stage/ page in spark UI >