[jira] [Commented] (SPARK-19256) Hive bucketing support

[ https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185617#comment-17185617 ] Cheng Su commented on SPARK-19256: -- Hi all, after discussion with [~cloud_fan], we decide to support

[jira] [Updated] (SPARK-32693) Compare two dataframes with same schema except nullable property

[ https://issues.apache.org/jira/browse/SPARK-32693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] L. C. Hsieh updated SPARK-32693: Affects Version/s: 3.1.0 > Compare two dataframes with same schema except nullable property >

[jira] [Updated] (SPARK-19256) Hive bucketing support

[ https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Su updated SPARK-19256: - Description: Update (2020 by Cheng Su): We use this JIRA to track progress for Hive bucketing write

[jira] [Comment Edited] (SPARK-32190) Development - Contribution Guide

[ https://issues.apache.org/jira/browse/SPARK-32190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17151885#comment-17151885 ] Hyukjin Kwon edited comment on SPARK-32190 at 8/27/20, 3:56 AM: Common

[jira] [Assigned] (SPARK-32707) Support File Based spark.authenticate secret in Mesos Backend

[ https://issues.apache.org/jira/browse/SPARK-32707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32707: Assignee: (was: Apache Spark) > Support File Based spark.authenticate secret in

[jira] [Assigned] (SPARK-32707) Support File Based spark.authenticate secret in Mesos Backend

[ https://issues.apache.org/jira/browse/SPARK-32707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32707: Assignee: Apache Spark > Support File Based spark.authenticate secret in Mesos Backend >

[jira] [Commented] (SPARK-32707) Support File Based spark.authenticate secret in Mesos Backend

[ https://issues.apache.org/jira/browse/SPARK-32707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185542#comment-17185542 ] Apache Spark commented on SPARK-32707: -- User 'farhan5900' has created a pull request for this

[jira] [Commented] (SPARK-32706) Poor performance when casting invalid decimal string to decimal type

[ https://issues.apache.org/jira/browse/SPARK-32706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185530#comment-17185530 ] Yuming Wang commented on SPARK-32706: - Yes, but not much, valid and invalid data are mixed, such as:

[jira] [Comment Edited] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185523#comment-17185523 ] Mingjia Liu edited comment on SPARK-32708 at 8/27/20, 12:14 AM: This

[jira] [Comment Edited] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185523#comment-17185523 ] Mingjia Liu edited comment on SPARK-32708 at 8/27/20, 12:11 AM: This

[jira] [Comment Edited] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185523#comment-17185523 ] Mingjia Liu edited comment on SPARK-32708 at 8/27/20, 12:11 AM: This

[jira] [Comment Edited] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185523#comment-17185523 ] Mingjia Liu edited comment on SPARK-32708 at 8/27/20, 12:10 AM: This

[jira] [Comment Edited] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185526#comment-17185526 ] Mingjia Liu edited comment on SPARK-32708 at 8/27/20, 12:10 AM: I can't

[jira] [Issue Comment Deleted] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingjia Liu updated SPARK-32708: Comment: was deleted (was: I can't repro the issue at Spark 3.0 *Fix:* 2.4 needs  to partially

[jira] [Commented] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185526#comment-17185526 ] Mingjia Liu commented on SPARK-32708: - I can't repro the issue at Spark 3.0 Fix:  2.4 needs  to

[jira] [Comment Edited] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185523#comment-17185523 ] Mingjia Liu edited comment on SPARK-32708 at 8/27/20, 12:06 AM: This

[jira] [Commented] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185524#comment-17185524 ] Mingjia Liu commented on SPARK-32708: - This bug is brought to the surface by [link

[jira] [Commented] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185523#comment-17185523 ] Mingjia Liu commented on SPARK-32708: - This difference happened at ReuseExchange rule apply.  The

[jira] [Comment Edited] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185523#comment-17185523 ] Mingjia Liu edited comment on SPARK-32708 at 8/27/20, 12:01 AM: This

[jira] [Updated] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingjia Liu updated SPARK-32708: Description: Repro query: {code:java} spark.conf.set("spark.sql.exchange.reuse","true")

[jira] [Updated] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingjia Liu updated SPARK-32708: Description: Repro query: {code:java} spark.conf.set("spark.sql.exchange.reuse","true")

[jira] [Updated] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingjia Liu updated SPARK-32708: Description: Repro query: {code:java} spark.conf.set("spark.sql.exchange.reuse","true")

[jira] [Updated] (SPARK-32708) Query optimization fails to reuse exchange with DataSourceV2

[ https://issues.apache.org/jira/browse/SPARK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingjia Liu updated SPARK-32708: Summary: Query optimization fails to reuse exchange with DataSourceV2 (was: Query optimization

[jira] [Created] (SPARK-32708) Query optimization fail to reuse exchange with DataSourceV2

Mingjia Liu created SPARK-32708: --- Summary: Query optimization fail to reuse exchange with DataSourceV2 Key: SPARK-32708 URL: https://issues.apache.org/jira/browse/SPARK-32708 Project: Spark

[jira] [Commented] (SPARK-32092) CrossvalidatorModel does not save all submodels (it saves only 3)

[ https://issues.apache.org/jira/browse/SPARK-32092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185518#comment-17185518 ] Apache Spark commented on SPARK-32092: -- User 'Louiszr' has created a pull request for this issue:

[jira] [Commented] (SPARK-32092) CrossvalidatorModel does not save all submodels (it saves only 3)

[ https://issues.apache.org/jira/browse/SPARK-32092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185517#comment-17185517 ] Apache Spark commented on SPARK-32092: -- User 'Louiszr' has created a pull request for this issue:

[jira] [Resolved] (SPARK-32643) [Cleanup] Consolidate state kept in ExecutorDecommissionInfo with TaskSetManager.tidToExecutorKillTimeMapping

[ https://issues.apache.org/jira/browse/SPARK-32643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau resolved SPARK-32643. -- Fix Version/s: 3.1.0 Resolution: Fixed > [Cleanup] Consolidate state kept in

[jira] [Assigned] (SPARK-32643) [Cleanup] Consolidate state kept in ExecutorDecommissionInfo with TaskSetManager.tidToExecutorKillTimeMapping

[ https://issues.apache.org/jira/browse/SPARK-32643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Holden Karau reassigned SPARK-32643: Assignee: Devesh Agrawal > [Cleanup] Consolidate state kept in ExecutorDecommissionInfo

[jira] [Commented] (SPARK-27733) Upgrade to Avro 1.10.0

[ https://issues.apache.org/jira/browse/SPARK-27733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185494#comment-17185494 ] Xiao Li commented on SPARK-27733: - BTW, is Hive community will to introduce the version upgrade of Avro

[jira] [Commented] (SPARK-27733) Upgrade to Avro 1.10.0

[ https://issues.apache.org/jira/browse/SPARK-27733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185493#comment-17185493 ] Xiao Li commented on SPARK-27733: - Let us wait for one more month. If no one is complaining about the

[jira] [Commented] (SPARK-27733) Upgrade to Avro 1.10.0

[ https://issues.apache.org/jira/browse/SPARK-27733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185477#comment-17185477 ] Dongjoon Hyun commented on SPARK-27733: --- [~csun]. There was a community discussion on that and

[jira] [Commented] (SPARK-32481) Support truncate table to move the data to trash

[ https://issues.apache.org/jira/browse/SPARK-32481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185429#comment-17185429 ] Apache Spark commented on SPARK-32481: -- User 'Udbhav30' has created a pull request for this issue:

[jira] [Commented] (SPARK-32481) Support truncate table to move the data to trash

[ https://issues.apache.org/jira/browse/SPARK-32481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185430#comment-17185430 ] Apache Spark commented on SPARK-32481: -- User 'Udbhav30' has created a pull request for this issue:

[jira] [Updated] (SPARK-32481) Support truncate table to move the data to trash

[ https://issues.apache.org/jira/browse/SPARK-32481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32481: -- Fix Version/s: (was: 3.1.0) > Support truncate table to move the data to trash >

[jira] [Commented] (SPARK-32481) Support truncate table to move the data to trash

[ https://issues.apache.org/jira/browse/SPARK-32481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185419#comment-17185419 ] Dongjoon Hyun commented on SPARK-32481: --- Apache Spark 3.1 will support Hadoop 2.7 distribution in

[jira] [Assigned] (SPARK-32481) Support truncate table to move the data to trash

[ https://issues.apache.org/jira/browse/SPARK-32481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32481: Assignee: (was: Apache Spark) > Support truncate table to move the data to trash >

[jira] [Assigned] (SPARK-32481) Support truncate table to move the data to trash

[ https://issues.apache.org/jira/browse/SPARK-32481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32481: Assignee: Apache Spark > Support truncate table to move the data to trash >

[jira] [Reopened] (SPARK-32481) Support truncate table to move the data to trash

[ https://issues.apache.org/jira/browse/SPARK-32481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-32481: --- Assignee: (was: Udbhav Agrawal) This is reverted due to the compilation error with

[jira] [Commented] (SPARK-27733) Upgrade to Avro 1.10.0

[ https://issues.apache.org/jira/browse/SPARK-27733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185293#comment-17185293 ] Chao Sun commented on SPARK-27733: -- bq. Apache Spark 3.0 will ship both Hive 1.2.1 built-in and Hive

[jira] [Updated] (SPARK-32412) Unify error handling for spark thrift server operations

[ https://issues.apache.org/jira/browse/SPARK-32412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-32412: - Fix Version/s: (was: 3.1.0) > Unify error handling for spark thrift server operations >

[jira] [Updated] (SPARK-32511) Add dropFields method to Column class

[ https://issues.apache.org/jira/browse/SPARK-32511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-32511: - Fix Version/s: (was: 3.1.0) > Add dropFields method to Column class >

[jira] [Updated] (SPARK-32701) mapreduce.fileoutputcommitter.algorithm.version default depends on runtime environment

[ https://issues.apache.org/jira/browse/SPARK-32701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-32701: - Issue Type: Improvement (was: Bug) Priority: Minor (was: Major) >

[jira] [Commented] (SPARK-32706) Poor performance when casting invalid decimal string to decimal type

[ https://issues.apache.org/jira/browse/SPARK-32706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185247#comment-17185247 ] Sean R. Owen commented on SPARK-32706: -- Do we care much about this path - is it common to have

[jira] [Updated] (SPARK-32706) Poor performance when casting invalid decimal string to decimal type

[ https://issues.apache.org/jira/browse/SPARK-32706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-32706: - Priority: Minor (was: Major) > Poor performance when casting invalid decimal string to decimal

[jira] [Created] (SPARK-32707) Support File Based spark.authenticate secret in Mesos Backend

Farhan Khan created SPARK-32707: --- Summary: Support File Based spark.authenticate secret in Mesos Backend Key: SPARK-32707 URL: https://issues.apache.org/jira/browse/SPARK-32707 Project: Spark

[jira] [Commented] (SPARK-32481) Support truncate table to move the data to trash

[ https://issues.apache.org/jira/browse/SPARK-32481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185220#comment-17185220 ] Apache Spark commented on SPARK-32481: -- User 'Udbhav30' has created a pull request for this issue:

[jira] [Commented] (SPARK-32481) Support truncate table to move the data to trash

[ https://issues.apache.org/jira/browse/SPARK-32481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185218#comment-17185218 ] Apache Spark commented on SPARK-32481: -- User 'Udbhav30' has created a pull request for this issue:

[jira] [Commented] (SPARK-31831) Flaky test: org.apache.spark.sql.hive.thriftserver.HiveSessionImplSuite.(It is not a test it is a sbt.testing.SuiteSelector)

[ https://issues.apache.org/jira/browse/SPARK-31831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185211#comment-17185211 ] Apache Spark commented on SPARK-31831: -- User 'alismess-db' has created a pull request for this

[jira] [Created] (SPARK-32706) Poor performance when casting invalid decimal string to decimal type

Yuming Wang created SPARK-32706: --- Summary: Poor performance when casting invalid decimal string to decimal type Key: SPARK-32706 URL: https://issues.apache.org/jira/browse/SPARK-32706 Project: Spark

[jira] [Assigned] (SPARK-32183) User Guide - PySpark Usage Guide for Pandas with Apache Arrow

[ https://issues.apache.org/jira/browse/SPARK-32183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32183: Assignee: Apache Spark > User Guide - PySpark Usage Guide for Pandas with Apache Arrow >

[jira] [Assigned] (SPARK-32183) User Guide - PySpark Usage Guide for Pandas with Apache Arrow

[ https://issues.apache.org/jira/browse/SPARK-32183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32183: Assignee: (was: Apache Spark) > User Guide - PySpark Usage Guide for Pandas with

[jira] [Commented] (SPARK-32183) User Guide - PySpark Usage Guide for Pandas with Apache Arrow

[ https://issues.apache.org/jira/browse/SPARK-32183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185076#comment-17185076 ] Apache Spark commented on SPARK-32183: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Commented] (SPARK-32705) EmptyHashedRelation$; no valid constructor

[ https://issues.apache.org/jira/browse/SPARK-32705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185000#comment-17185000 ] Apache Spark commented on SPARK-32705: -- User 'leanken' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32705) EmptyHashedRelation$; no valid constructor

[ https://issues.apache.org/jira/browse/SPARK-32705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32705: Assignee: Apache Spark > EmptyHashedRelation$; no valid constructor >

[jira] [Assigned] (SPARK-32705) EmptyHashedRelation$; no valid constructor

[ https://issues.apache.org/jira/browse/SPARK-32705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32705: Assignee: (was: Apache Spark) > EmptyHashedRelation$; no valid constructor >

[jira] [Created] (SPARK-32705) EmptyHashedRelation$; no valid constructor

Leanken.Lin created SPARK-32705: --- Summary: EmptyHashedRelation$; no valid constructor Key: SPARK-32705 URL: https://issues.apache.org/jira/browse/SPARK-32705 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-32361) Remove project if output is subset of child

[ https://issues.apache.org/jira/browse/SPARK-32361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184977#comment-17184977 ] L. C. Hsieh commented on SPARK-32361: - Isn't it already in physical plan phase? Removing such

[jira] [Resolved] (SPARK-32659) Fix the data issue of inserted DPP on non-atomic type

[ https://issues.apache.org/jira/browse/SPARK-32659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32659. - Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue resolved by pull

[jira] [Assigned] (SPARK-32659) Fix the data issue of inserted DPP on non-atomic type

[ https://issues.apache.org/jira/browse/SPARK-32659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32659: --- Assignee: Yuming Wang > Fix the data issue of inserted DPP on non-atomic type >

[jira] [Assigned] (SPARK-32481) Support truncate table to move the data to trash

[ https://issues.apache.org/jira/browse/SPARK-32481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32481: - Assignee: Udbhav Agrawal > Support truncate table to move the data to trash >

[jira] [Resolved] (SPARK-32481) Support truncate table to move the data to trash

[ https://issues.apache.org/jira/browse/SPARK-32481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32481. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29387