[jira] [Commented] (SPARK-34435) ArrayIndexOutOfBoundsException when select in different case

2021-06-03 Thread dc-heros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356239#comment-17356239 ] dc-heros commented on SPARK-34435: -- I have test this on branch-3.0 and master and this

[jira] [Created] (SPARK-35623) Volcano resource manager for Spark on Kubernetes

2021-06-03 Thread Dipanjan Kailthya (Jira)
Dipanjan Kailthya created SPARK-35623: - Summary: Volcano resource manager for Spark on Kubernetes Key: SPARK-35623 URL: https://issues.apache.org/jira/browse/SPARK-35623 Project: Spark Is

[jira] [Updated] (SPARK-35623) Volcano resource manager for Spark on Kubernetes

2021-06-03 Thread Dipanjan Kailthya (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dipanjan Kailthya updated SPARK-35623: -- Description: Dear Spark Developers,    Hello from the Netherlands! Sending this here

[jira] [Updated] (SPARK-35623) Volcano resource manager for Spark on Kubernetes

2021-06-03 Thread Dipanjan Kailthya (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dipanjan Kailthya updated SPARK-35623: -- Description: Dear Spark Developers,    Hello from the Netherlands! Posting this here

[jira] [Updated] (SPARK-35296) Dataset.observe fails with an assertion

2021-06-03 Thread Tanel Kiis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tanel Kiis updated SPARK-35296: --- Affects Version/s: 3.2.0 > Dataset.observe fails with an assertion > ---

[jira] [Commented] (SPARK-35296) Dataset.observe fails with an assertion

2021-06-03 Thread Tanel Kiis (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356253#comment-17356253 ] Tanel Kiis commented on SPARK-35296: Perhaps someone who knows the internals better,

[jira] [Created] (SPARK-35624) Support reading inputbytes and inputrecords of the stage int spark context interface

2021-06-03 Thread allendang (Jira)
allendang created SPARK-35624: - Summary: Support reading inputbytes and inputrecords of the stage int spark context interface Key: SPARK-35624 URL: https://issues.apache.org/jira/browse/SPARK-35624 Proje

[jira] [Commented] (SPARK-35100) Refactor AFT - support virtual centering

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356288#comment-17356288 ] Apache Spark commented on SPARK-35100: -- User 'zhengruifeng' has created a pull requ

[jira] [Created] (SPARK-35625) Spark on k8s zombie executors

2021-06-03 Thread Liran (Jira)
Liran created SPARK-35625: - Summary: Spark on k8s zombie executors Key: SPARK-35625 URL: https://issues.apache.org/jira/browse/SPARK-35625 Project: Spark Issue Type: Bug Components: Kuberne

[jira] [Updated] (SPARK-35625) Spark on k8s zombie executors

2021-06-03 Thread Liran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liran updated SPARK-35625: -- Attachment: image-2021-06-03-12-15-57-095.png > Spark on k8s zombie executors > -

[jira] [Updated] (SPARK-35625) Spark on k8s zombie executors

2021-06-03 Thread Liran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liran updated SPARK-35625: -- Attachment: image-2021-06-03-12-16-03-621.png > Spark on k8s zombie executors > -

[jira] [Updated] (SPARK-35625) Spark on k8s zombie executors

2021-06-03 Thread Liran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liran updated SPARK-35625: -- Description: We are running a POC of Spark on K8s setup for one of our apps, it's scaling up/down quite a lot

[jira] [Commented] (SPARK-34512) Disable validate default values when parsing Avro schemas

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356298#comment-17356298 ] Apache Spark commented on SPARK-34512: -- User 'wangyum' has created a pull request f

[jira] [Assigned] (SPARK-34512) Disable validate default values when parsing Avro schemas

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34512: Assignee: (was: Apache Spark) > Disable validate default values when parsing Avro sch

[jira] [Assigned] (SPARK-34512) Disable validate default values when parsing Avro schemas

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-34512: Assignee: Apache Spark > Disable validate default values when parsing Avro schemas >

[jira] [Updated] (SPARK-35625) Spark on k8s zombie executors

2021-06-03 Thread Liran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liran updated SPARK-35625: -- Attachment: image-2021-06-03-12-16-12-573.png > Spark on k8s zombie executors > -

[jira] [Updated] (SPARK-35625) Spark on k8s zombie executors

2021-06-03 Thread Liran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liran updated SPARK-35625: -- Description: We are running a POC of Spark on K8s setup for one of our apps, it's scaling up/down quite a lot

[jira] [Updated] (SPARK-35625) Spark on k8s zombie executors

2021-06-03 Thread Liran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liran updated SPARK-35625: -- Description: We are running a POC of Spark on K8s setup for one of our apps, it's scaling up/down quite a lot

[jira] [Resolved] (SPARK-34435) ArrayIndexOutOfBoundsException when select in different case

2021-06-03 Thread dc-heros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dc-heros resolved SPARK-34435. -- Resolution: Cannot Reproduce > ArrayIndexOutOfBoundsException when select in different case >

[jira] [Commented] (SPARK-26868) Duplicate error message for implicit cartesian product in verbose explain

2021-06-03 Thread dc-heros (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356313#comment-17356313 ] dc-heros commented on SPARK-26868: -- anything new?   > Duplicate error message for imp

[jira] [Created] (SPARK-35626) Decommission executors in batches to avoid overloading network by block migrations.

2021-06-03 Thread abhishek kumar tiwari (Jira)
abhishek kumar tiwari created SPARK-35626: - Summary: Decommission executors in batches to avoid overloading network by block migrations. Key: SPARK-35626 URL: https://issues.apache.org/jira/browse/SPARK-35

[jira] [Assigned] (SPARK-35626) Decommission executors in batches to avoid overloading network by block migrations.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35626: Assignee: Apache Spark > Decommission executors in batches to avoid overloading network b

[jira] [Commented] (SPARK-35626) Decommission executors in batches to avoid overloading network by block migrations.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356316#comment-17356316 ] Apache Spark commented on SPARK-35626: -- User 'q2w' has created a pull request for t

[jira] [Assigned] (SPARK-35626) Decommission executors in batches to avoid overloading network by block migrations.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35626: Assignee: (was: Apache Spark) > Decommission executors in batches to avoid overloadin

[jira] [Created] (SPARK-35627) Decommission executors in batches to avoid overloading network by block migrations.

2021-06-03 Thread abhishek kumar tiwari (Jira)
abhishek kumar tiwari created SPARK-35627: - Summary: Decommission executors in batches to avoid overloading network by block migrations. Key: SPARK-35627 URL: https://issues.apache.org/jira/browse/SPARK-35

[jira] [Resolved] (SPARK-35626) Decommission executors in batches to avoid overloading network by block migrations.

2021-06-03 Thread abhishek kumar tiwari (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] abhishek kumar tiwari resolved SPARK-35626. --- Resolution: Duplicate > Decommission executors in batches to avoid overloadi

[jira] [Commented] (SPARK-35627) Decommission executors in batches to avoid overloading network by block migrations.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356318#comment-17356318 ] Apache Spark commented on SPARK-35627: -- User 'q2w' has created a pull request for t

[jira] [Assigned] (SPARK-35627) Decommission executors in batches to avoid overloading network by block migrations.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35627: Assignee: Apache Spark > Decommission executors in batches to avoid overloading network b

[jira] [Assigned] (SPARK-35627) Decommission executors in batches to avoid overloading network by block migrations.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35627: Assignee: (was: Apache Spark) > Decommission executors in batches to avoid overloadin

[jira] [Commented] (SPARK-35627) Decommission executors in batches to avoid overloading network by block migrations.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356320#comment-17356320 ] Apache Spark commented on SPARK-35627: -- User 'q2w' has created a pull request for t

[jira] [Created] (SPARK-35628) RocksDBFileManager - load checkpoint from DFS

2021-06-03 Thread Yuanjian Li (Jira)
Yuanjian Li created SPARK-35628: --- Summary: RocksDBFileManager - load checkpoint from DFS Key: SPARK-35628 URL: https://issues.apache.org/jira/browse/SPARK-35628 Project: Spark Issue Type: Sub-t

[jira] [Assigned] (SPARK-35628) RocksDBFileManager - load checkpoint from DFS

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35628: Assignee: Apache Spark > RocksDBFileManager - load checkpoint from DFS >

[jira] [Commented] (SPARK-35628) RocksDBFileManager - load checkpoint from DFS

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356334#comment-17356334 ] Apache Spark commented on SPARK-35628: -- User 'xuanyuanking' has created a pull requ

[jira] [Assigned] (SPARK-35628) RocksDBFileManager - load checkpoint from DFS

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35628: Assignee: (was: Apache Spark) > RocksDBFileManager - load checkpoint from DFS > -

[jira] [Created] (SPARK-35629) Drop database should check if exists

2021-06-03 Thread XiDuo You (Jira)
XiDuo You created SPARK-35629: - Summary: Drop database should check if exists Key: SPARK-35629 URL: https://issues.apache.org/jira/browse/SPARK-35629 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-35629) Drop database should check if exists

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35629: Assignee: Apache Spark > Drop database should check if exists > -

[jira] [Commented] (SPARK-35629) Drop database should check if exists

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356343#comment-17356343 ] Apache Spark commented on SPARK-35629: -- User 'ulysses-you' has created a pull reque

[jira] [Assigned] (SPARK-35629) Drop database should check if exists

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35629: Assignee: (was: Apache Spark) > Drop database should check if exists > --

[jira] [Commented] (SPARK-35629) Drop database should check if exists

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356344#comment-17356344 ] Apache Spark commented on SPARK-35629: -- User 'ulysses-you' has created a pull reque

[jira] [Created] (SPARK-35630) ExpandExec should not introduce unnecessary exchanges

2021-06-03 Thread Tanel Kiis (Jira)
Tanel Kiis created SPARK-35630: -- Summary: ExpandExec should not introduce unnecessary exchanges Key: SPARK-35630 URL: https://issues.apache.org/jira/browse/SPARK-35630 Project: Spark Issue Type:

[jira] [Commented] (SPARK-35630) ExpandExec should not introduce unnecessary exchanges

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356402#comment-17356402 ] Apache Spark commented on SPARK-35630: -- User 'tanelk' has created a pull request fo

[jira] [Assigned] (SPARK-35630) ExpandExec should not introduce unnecessary exchanges

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35630: Assignee: Apache Spark > ExpandExec should not introduce unnecessary exchanges >

[jira] [Commented] (SPARK-21957) Add current_user function

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356403#comment-17356403 ] Apache Spark commented on SPARK-21957: -- User 'yaooqinn' has created a pull request

[jira] [Assigned] (SPARK-35630) ExpandExec should not introduce unnecessary exchanges

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35630: Assignee: (was: Apache Spark) > ExpandExec should not introduce unnecessary exchanges

[jira] [Commented] (SPARK-21957) Add current_user function

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356404#comment-17356404 ] Apache Spark commented on SPARK-21957: -- User 'yaooqinn' has created a pull request

[jira] [Resolved] (SPARK-35609) Add style rules to prohibit to use a Guava's API which is incompatible with newer versions

2021-06-03 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta resolved SPARK-35609. Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved in https://github.com/apache/

[jira] [Commented] (SPARK-35602) Job crashes with java.io.UTFDataFormatException: encoded string too long

2021-06-03 Thread dejan miljkovic (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356465#comment-17356465 ] dejan miljkovic commented on SPARK-35602: - StateSchemaCompatibilityChecker.scala

[jira] [Created] (SPARK-35631) java.lang.ArithmeticException: integer overflow when SELECT 2147483647 + 1 executed with set spark.sql.ansi.enabled=true

2021-06-03 Thread Chetan Bhat (Jira)
Chetan Bhat created SPARK-35631: --- Summary: java.lang.ArithmeticException: integer overflow when SELECT 2147483647 + 1 executed with set spark.sql.ansi.enabled=true Key: SPARK-35631 URL: https://issues.apache.org/jir

[jira] [Resolved] (SPARK-35631) java.lang.ArithmeticException: integer overflow when SELECT 2147483647 + 1 executed with set spark.sql.ansi.enabled=true

2021-06-03 Thread Chetan Bhat (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Bhat resolved SPARK-35631. - Resolution: Invalid As per the information provided in  [https://spark.apache.org/docs/3.1.2/sql

[jira] [Resolved] (SPARK-35316) UnwrapCastInBinaryComparison support In/InSet predicate

2021-06-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-35316. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32488 [https://gith

[jira] [Created] (SPARK-35632) SELECT rand(cast(3 / 7 AS int)) value gives different resultset in Spark 3.1 as compared to Spark 2.4.5

2021-06-03 Thread Chetan Bhat (Jira)
Chetan Bhat created SPARK-35632: --- Summary: SELECT rand(cast(3 / 7 AS int)) value gives different resultset in Spark 3.1 as compared to Spark 2.4.5 Key: SPARK-35632 URL: https://issues.apache.org/jira/browse/SPARK-35

[jira] [Created] (SPARK-35633) Select nvl query throws parse exception

2021-06-03 Thread Chetan Bhat (Jira)
Chetan Bhat created SPARK-35633: --- Summary: Select nvl query throws parse exception Key: SPARK-35633 URL: https://issues.apache.org/jira/browse/SPARK-35633 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-35634) Select queries with union throw parsing exception

2021-06-03 Thread Chetan Bhat (Jira)
Chetan Bhat created SPARK-35634: --- Summary: Select queries with union throw parsing exception Key: SPARK-35634 URL: https://issues.apache.org/jira/browse/SPARK-35634 Project: Spark Issue Type: B

[jira] [Created] (SPARK-35635) concurrent insert statements from multiple beeline fail with job aborted exception

2021-06-03 Thread Chetan Bhat (Jira)
Chetan Bhat created SPARK-35635: --- Summary: concurrent insert statements from multiple beeline fail with job aborted exception Key: SPARK-35635 URL: https://issues.apache.org/jira/browse/SPARK-35635 Proj

[jira] [Commented] (SPARK-34435) ArrayIndexOutOfBoundsException when select in different case

2021-06-03 Thread Enver Osmanov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356550#comment-17356550 ] Enver Osmanov commented on SPARK-34435: --- I see it was fixed in [this PR|https://gi

[jira] [Assigned] (SPARK-35580) Support subexpression elimination for higher order functions

2021-06-03 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh reassigned SPARK-35580: --- Assignee: L. C. Hsieh > Support subexpression elimination for higher order functions >

[jira] [Resolved] (SPARK-35580) Support subexpression elimination for higher order functions

2021-06-03 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh resolved SPARK-35580. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32735 [https://gith

[jira] [Closed] (SPARK-35631) java.lang.ArithmeticException: integer overflow when SELECT 2147483647 + 1 executed with set spark.sql.ansi.enabled=true

2021-06-03 Thread Chetan Bhat (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chetan Bhat closed SPARK-35631. ---  this is not a issue. Closed without handling. > java.lang.ArithmeticException: integer overflow when S

[jira] [Updated] (SPARK-35589) BlockManagerMasterEndpoint should not ignore index-only shuffle file during updating

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35589: -- Fix Version/s: 3.1.3 > BlockManagerMasterEndpoint should not ignore index-only shuffle file du

[jira] [Commented] (SPARK-35316) UnwrapCastInBinaryComparison support In/InSet predicate

2021-06-03 Thread Fu Chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356574#comment-17356574 ] Fu Chen commented on SPARK-35316: - Hi, [~cloud_fan], this is my Jira account, what shoul

[jira] [Commented] (SPARK-35406) TaskCompletionListenerException: Premature end of Content-Length delimited message body

2021-06-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356607#comment-17356607 ] Steve Loughran commented on SPARK-35406: This is fixed by HADOOP-17338 and by th

[jira] [Commented] (SPARK-35602) Job crashes with java.io.UTFDataFormatException: encoded string too long

2021-06-03 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356614#comment-17356614 ] Kousuke Saruta commented on SPARK-35602: [~dej...@gmail.com] Thank you for repor

[jira] [Updated] (SPARK-35299) Dataframe overwrite on S3A does not delete old files with S3 object-put to table path/

2021-06-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-35299: --- Summary: Dataframe overwrite on S3A does not delete old files with S3 object-put to table pa

[jira] [Commented] (SPARK-35299) Dataframe overwrite on S3 does not delete old files with S3 object-put to table path

2021-06-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356615#comment-17356615 ] Steve Loughran commented on SPARK-35299: Actually if you are doing a PUT of test

[jira] [Commented] (SPARK-35406) TaskCompletionListenerException: Premature end of Content-Length delimited message body

2021-06-03 Thread Saurabh Chawla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356619#comment-17356619 ] Saurabh Chawla commented on SPARK-35406: [~ste...@apache.org]- This backport of 

[jira] [Commented] (SPARK-34298) SaveMode.Overwrite not usable when using s3a root paths

2021-06-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-34298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356625#comment-17356625 ] Steve Loughran commented on SPARK-34298: Actually, you've just found a bug in Pa

[jira] [Updated] (SPARK-35636) Do not push lambda variables out of lambda functions in NestedColumnAliasing

2021-06-03 Thread Karen Feng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Feng updated SPARK-35636: --- Description: Currently, lambda keys can be referenced outside of the lambda function: Project [tran

[jira] [Created] (SPARK-35636) Do not push lambda variables out of lambda functions in NestedColumnAliasing

2021-06-03 Thread Karen Feng (Jira)
Karen Feng created SPARK-35636: -- Summary: Do not push lambda variables out of lambda functions in NestedColumnAliasing Key: SPARK-35636 URL: https://issues.apache.org/jira/browse/SPARK-35636 Project: Spa

[jira] [Updated] (SPARK-35636) Do not push lambda variables out of lambda functions in NestedColumnAliasing

2021-06-03 Thread Karen Feng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Feng updated SPARK-35636: --- Description: Currently, lambda keys can be referenced outside of the lambda function: {quote}Projec

[jira] [Updated] (SPARK-35636) Do not push down extract value in higher order function that references both sides of a join

2021-06-03 Thread Karen Feng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Feng updated SPARK-35636: --- Summary: Do not push down extract value in higher order function that references both sides of a joi

[jira] [Updated] (SPARK-35533) Do not drop cached RDD blocks to accommodate blocks from decommissioned block manager if enough memory is not available

2021-06-03 Thread abhishek kumar tiwari (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] abhishek kumar tiwari updated SPARK-35533: -- Fix Version/s: (was: 3.2.0) > Do not drop cached RDD blocks to accommodate

[jira] [Created] (SPARK-35637) 2 active tasks shown in Spark UI but executor core is 1 and task cpu is 1

2021-06-03 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-35637: --- Summary: 2 active tasks shown in Spark UI but executor core is 1 and task cpu is 1 Key: SPARK-35637 URL: https://issues.apache.org/jira/browse/SPARK-35637 Project: Spar

[jira] [Commented] (SPARK-35625) Spark on k8s zombie executors

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356697#comment-17356697 ] Dongjoon Hyun commented on SPARK-35625: --- Thank you for reporting, [~lbuanos]. BTW,

[jira] [Commented] (SPARK-35599) Introduce a way to compare series of array for older pandas

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356699#comment-17356699 ] Apache Spark commented on SPARK-35599: -- User 'xinrong-databricks' has created a pul

[jira] [Assigned] (SPARK-35599) Introduce a way to compare series of array for older pandas

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35599: Assignee: Apache Spark > Introduce a way to compare series of array for older pandas > --

[jira] [Assigned] (SPARK-35599) Introduce a way to compare series of array for older pandas

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35599: Assignee: (was: Apache Spark) > Introduce a way to compare series of array for older

[jira] [Commented] (SPARK-35599) Introduce a way to compare series of array for older pandas

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356701#comment-17356701 ] Apache Spark commented on SPARK-35599: -- User 'xinrong-databricks' has created a pul

[jira] [Updated] (SPARK-35624) Support reading inputbytes and inputrecords of the stage int spark context interface

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35624: -- Fix Version/s: (was: 3.1.2) > Support reading inputbytes and inputrecords of the stage int

[jira] [Updated] (SPARK-35624) Support reading inputbytes and inputrecords of the stage int spark context interface

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35624: -- Affects Version/s: (was: 3.1.2) 3.2.0 > Support reading inputbytes

[jira] [Commented] (SPARK-35624) Support reading inputbytes and inputrecords of the stage int spark context interface

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356703#comment-17356703 ] Dongjoon Hyun commented on SPARK-35624: --- Thank you for filing JIRA, [~allendang].

[jira] [Updated] (SPARK-35624) Support reading inputbytes and inputrecords of the stage int spark context interface

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-35624: -- Target Version/s: (was: 3.1.2) > Support reading inputbytes and inputrecords of the stage in

[jira] [Assigned] (SPARK-35612) Support LZ4 compression in ORC data source

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-35612: - Assignee: Han > Support LZ4 compression in ORC data source > --

[jira] [Resolved] (SPARK-35612) Support LZ4 compression in ORC data source

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-35612. --- Resolution: Fixed Issue resolved by pull request 32751 [https://github.com/apache/spark/pull

[jira] [Assigned] (SPARK-35636) Do not push down extract value in higher order function that references both sides of a join

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35636: Assignee: Apache Spark > Do not push down extract value in higher order function that ref

[jira] [Commented] (SPARK-35636) Do not push down extract value in higher order function that references both sides of a join

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356728#comment-17356728 ] Apache Spark commented on SPARK-35636: -- User 'karenfeng' has created a pull request

[jira] [Assigned] (SPARK-35636) Do not push down extract value in higher order function that references both sides of a join

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35636: Assignee: (was: Apache Spark) > Do not push down extract value in higher order functi

[jira] [Commented] (SPARK-35636) Do not push down extract value in higher order function that references both sides of a join

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356731#comment-17356731 ] Apache Spark commented on SPARK-35636: -- User 'karenfeng' has created a pull request

[jira] [Commented] (SPARK-35589) BlockManagerMasterEndpoint should not ignore index-only shuffle file during updating

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356733#comment-17356733 ] Apache Spark commented on SPARK-35589: -- User 'dongjoon-hyun' has created a pull req

[jira] [Commented] (SPARK-35589) BlockManagerMasterEndpoint should not ignore index-only shuffle file during updating

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356734#comment-17356734 ] Apache Spark commented on SPARK-35589: -- User 'dongjoon-hyun' has created a pull req

[jira] [Assigned] (SPARK-35606) List Python 3.9 installed libraries in build_and_test workflow

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-35606: - Assignee: Xinrong Meng > List Python 3.9 installed libraries in build_and_test workflow

[jira] [Resolved] (SPARK-35606) List Python 3.9 installed libraries in build_and_test workflow

2021-06-03 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-35606. --- Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32737 [https://

[jira] [Created] (SPARK-35638) Introduce Field to manage dtypes and StructField.

2021-06-03 Thread Takuya Ueshin (Jira)
Takuya Ueshin created SPARK-35638: - Summary: Introduce Field to manage dtypes and StructField. Key: SPARK-35638 URL: https://issues.apache.org/jira/browse/SPARK-35638 Project: Spark Issue Typ

[jira] [Updated] (SPARK-35638) Introduce Field to manage dtypes and StructFields.

2021-06-03 Thread Takuya Ueshin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takuya Ueshin updated SPARK-35638: -- Summary: Introduce Field to manage dtypes and StructFields. (was: Introduce Field to manage d

[jira] [Commented] (SPARK-35638) Introduce Field to manage dtypes and StructFields.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356912#comment-17356912 ] Apache Spark commented on SPARK-35638: -- User 'ueshin' has created a pull request fo

[jira] [Assigned] (SPARK-35638) Introduce Field to manage dtypes and StructFields.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35638: Assignee: Apache Spark > Introduce Field to manage dtypes and StructFields. > ---

[jira] [Assigned] (SPARK-35638) Introduce Field to manage dtypes and StructFields.

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35638: Assignee: (was: Apache Spark) > Introduce Field to manage dtypes and StructFields. >

[jira] [Created] (SPARK-35639) Add metrics about coalesced partitions to CustomShuffleReader in AQE

2021-06-03 Thread Eugene Koifman (Jira)
Eugene Koifman created SPARK-35639: -- Summary: Add metrics about coalesced partitions to CustomShuffleReader in AQE Key: SPARK-35639 URL: https://issues.apache.org/jira/browse/SPARK-35639 Project: Spa

[jira] [Commented] (SPARK-35639) Add metrics about coalesced partitions to CustomShuffleReader in AQE

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356918#comment-17356918 ] Apache Spark commented on SPARK-35639: -- User 'ekoifman' has created a pull request

[jira] [Assigned] (SPARK-35639) Add metrics about coalesced partitions to CustomShuffleReader in AQE

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35639: Assignee: Apache Spark > Add metrics about coalesced partitions to CustomShuffleReader in

[jira] [Assigned] (SPARK-35639) Add metrics about coalesced partitions to CustomShuffleReader in AQE

2021-06-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-35639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-35639: Assignee: (was: Apache Spark) > Add metrics about coalesced partitions to CustomShuff

  1   2   >