[jira] [Created] (SPARK-33114) Add metadata in MapStatus to support custom shuffle manager

2020-10-10 Thread BoYang (Jira)
BoYang created SPARK-33114: -- Summary: Add metadata in MapStatus to support custom shuffle manager Key: SPARK-33114 URL: https://issues.apache.org/jira/browse/SPARK-33114 Project: Spark Issue Type:

[jira] [Updated] (SPARK-21708) Migrate build to sbt 1.3.13

2020-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21708: -- Summary: Migrate build to sbt 1.3.13 (was: Migrate build to sbt .13) > Migrate build to

[jira] [Updated] (SPARK-21708) Migrate build to sbt 1.3.13

2020-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21708: -- Priority: Major (was: Minor) > Migrate build to sbt 1.3.13 > >

[jira] [Updated] (SPARK-21708) Migrate build to sbt .13

2020-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-21708: -- Summary: Migrate build to sbt .13 (was: use sbt 1.x) > Migrate build to sbt .13 >

[jira] [Commented] (SPARK-33104) Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in SparkHadoopUtil`

2020-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211809#comment-17211809 ] Dongjoon Hyun commented on SPARK-33104: --- Although this is a test case issue, I will raise this

[jira] [Updated] (SPARK-33104) Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in SparkHadoopUtil`

2020-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33104: -- Priority: Blocker (was: Major) > Fix `YarnClusterSuite.yarn-cluster should respect conf

[jira] [Updated] (SPARK-33104) Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in SparkHadoopUtil`

2020-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33104: -- Target Version/s: 3.1.0 > Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in

[jira] [Commented] (SPARK-33104) Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in SparkHadoopUtil`

2020-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211807#comment-17211807 ] Dongjoon Hyun commented on SPARK-33104: --- Please see

[jira] [Commented] (SPARK-33104) Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in SparkHadoopUtil`

2020-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211806#comment-17211806 ] Dongjoon Hyun commented on SPARK-33104: --- It turns out that Hadoop 2.7 / Hive 2.3 combination SBT

[jira] [Commented] (SPARK-33110) array_contains doesn't pick element in the array but just at the end

2020-10-10 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211803#comment-17211803 ] Takeshi Yamamuro commented on SPARK-33110: -- Could you please simplify it? Otherwise, it is

[jira] [Resolved] (SPARK-33107) Remove hive-2.3 workaround code

2020-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-33107. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29996

[jira] [Assigned] (SPARK-33107) Remove hive-2.3 workaround code

2020-10-10 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-33107: - Assignee: Yuming Wang > Remove hive-2.3 workaround code >

[jira] [Commented] (SPARK-32709) Write Hive ORC/Parquet bucketed table with hivehash (for Hive 1,2)

2020-10-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211799#comment-17211799 ] Apache Spark commented on SPARK-32709: -- User 'c21' has created a pull request for this issue:

[jira] [Commented] (SPARK-32709) Write Hive ORC/Parquet bucketed table with hivehash (for Hive 1,2)

2020-10-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211800#comment-17211800 ] Apache Spark commented on SPARK-32709: -- User 'c21' has created a pull request for this issue:

[jira] [Assigned] (SPARK-32709) Write Hive ORC/Parquet bucketed table with hivehash (for Hive 1,2)

2020-10-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32709: Assignee: (was: Apache Spark) > Write Hive ORC/Parquet bucketed table with hivehash

[jira] [Assigned] (SPARK-32709) Write Hive ORC/Parquet bucketed table with hivehash (for Hive 1,2)

2020-10-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32709: Assignee: Apache Spark > Write Hive ORC/Parquet bucketed table with hivehash (for Hive

[jira] [Commented] (SPARK-33086) Provide static annotatiions for pyspark.resource.* modules

2020-10-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211781#comment-17211781 ] Apache Spark commented on SPARK-33086: -- User 'zero323' has created a pull request for this issue:

[jira] [Commented] (SPARK-33086) Provide static annotatiions for pyspark.resource.* modules

2020-10-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211780#comment-17211780 ] Apache Spark commented on SPARK-33086: -- User 'zero323' has created a pull request for this issue:

[jira] [Updated] (SPARK-33110) array_contains doesn't pick element in the array but just at the end

2020-10-10 Thread Andrea Viano (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrea Viano updated SPARK-33110: - Attachment: testApriori3.csv > array_contains doesn't pick element in the array but just at the

[jira] [Commented] (SPARK-33110) array_contains doesn't pick element in the array but just at the end

2020-10-10 Thread Andrea Viano (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211769#comment-17211769 ] Andrea Viano commented on SPARK-33110: -- import findspark findspark.init() import pyspark from

[jira] [Updated] (SPARK-33113) [SparkR] gapply works with arrow disabled, fails with arrow enabled

2020-10-10 Thread Jacek Pliszka (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Pliszka updated SPARK-33113: -- Affects Version/s: 3.0.1 > [SparkR] gapply works with arrow disabled, fails with arrow

[jira] [Created] (SPARK-33113) [SparkR] gapply works with arrow disabled, fails with arrow enabled

2020-10-10 Thread Jacek Pliszka (Jira)
Jacek Pliszka created SPARK-33113: - Summary: [SparkR] gapply works with arrow disabled, fails with arrow enabled Key: SPARK-33113 URL: https://issues.apache.org/jira/browse/SPARK-33113 Project: Spark

[jira] [Commented] (SPARK-33110) array_contains doesn't pick element in the array but just at the end

2020-10-10 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211706#comment-17211706 ] Takeshi Yamamuro commented on SPARK-33110: -- Could you put complete code to reproduce this

[jira] [Updated] (SPARK-33110) array_contains doesn't pick element in the array but just at the end

2020-10-10 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-33110: - Priority: Major (was: Blocker) > array_contains doesn't pick element in the array but

[jira] [Updated] (SPARK-33110) array_contains doesn't pick element in the array but just at the end

2020-10-10 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-33110: - Component/s: (was: ML) SQL > array_contains doesn't pick element

[jira] [Commented] (SPARK-33112) Respect DatabaseMetaData.supportsBatchUpdates in JdbcUtils

2020-10-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211687#comment-17211687 ] Apache Spark commented on SPARK-33112: -- User 'ebyhr' has created a pull request for this issue:

[jira] [Assigned] (SPARK-33112) Respect DatabaseMetaData.supportsBatchUpdates in JdbcUtils

2020-10-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33112: Assignee: Apache Spark > Respect DatabaseMetaData.supportsBatchUpdates in JdbcUtils >

[jira] [Assigned] (SPARK-33112) Respect DatabaseMetaData.supportsBatchUpdates in JdbcUtils

2020-10-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33112: Assignee: (was: Apache Spark) > Respect DatabaseMetaData.supportsBatchUpdates in

[jira] [Commented] (SPARK-33112) Respect DatabaseMetaData.supportsBatchUpdates in JdbcUtils

2020-10-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211686#comment-17211686 ] Apache Spark commented on SPARK-33112: -- User 'ebyhr' has created a pull request for this issue:

[jira] [Updated] (SPARK-33112) Respect DatabaseMetaData.supportsBatchUpdates in JdbcUtils

2020-10-10 Thread Yuya Ebihara (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuya Ebihara updated SPARK-33112: - Description: Currently, JdbcUtils calls executeBatch regardless of 

[jira] [Created] (SPARK-33112) Respect DatabaseMetaData.supportsBatchUpdates in JdbcUtils

2020-10-10 Thread Yuya Ebihara (Jira)
Yuya Ebihara created SPARK-33112: Summary: Respect DatabaseMetaData.supportsBatchUpdates in JdbcUtils Key: SPARK-33112 URL: https://issues.apache.org/jira/browse/SPARK-33112 Project: Spark

[jira] [Assigned] (SPARK-33111) aft transform optimization

2020-10-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33111: Assignee: (was: Apache Spark) > aft transform optimization >

[jira] [Commented] (SPARK-33111) aft transform optimization

2020-10-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211661#comment-17211661 ] Apache Spark commented on SPARK-33111: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-33111) aft transform optimization

2020-10-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33111: Assignee: Apache Spark > aft transform optimization > -- > >

[jira] [Commented] (SPARK-33111) aft transform optimization

2020-10-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211660#comment-17211660 ] Apache Spark commented on SPARK-33111: -- User 'zhengruifeng' has created a pull request for this

[jira] [Created] (SPARK-33111) aft transform optimization

2020-10-10 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-33111: Summary: aft transform optimization Key: SPARK-33111 URL: https://issues.apache.org/jira/browse/SPARK-33111 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-33045) Implement built-in LIKE ANY and LIKE ALL UDF

2020-10-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211655#comment-17211655 ] Apache Spark commented on SPARK-33045: -- User 'beliefer' has created a pull request for this issue:

[jira] [Commented] (SPARK-33045) Implement built-in LIKE ANY and LIKE ALL UDF

2020-10-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211653#comment-17211653 ] Apache Spark commented on SPARK-33045: -- User 'beliefer' has created a pull request for this issue:

[jira] [Assigned] (SPARK-33045) Implement built-in LIKE ANY and LIKE ALL UDF

2020-10-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33045: Assignee: (was: Apache Spark) > Implement built-in LIKE ANY and LIKE ALL UDF >

[jira] [Assigned] (SPARK-33045) Implement built-in LIKE ANY and LIKE ALL UDF

2020-10-10 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33045: Assignee: Apache Spark > Implement built-in LIKE ANY and LIKE ALL UDF >

[jira] [Updated] (SPARK-33110) array_contains doesn't pick element in the array but just at the end

2020-10-10 Thread Andrea Viano (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrea Viano updated SPARK-33110: - Attachment: beer2.PNG beer1.PNG beer0.PNG > array_contains

[jira] [Created] (SPARK-33110) array_contains doesn't pick element in the array but just at the end

2020-10-10 Thread Andrea Viano (Jira)
Andrea Viano created SPARK-33110: Summary: array_contains doesn't pick element in the array but just at the end Key: SPARK-33110 URL: https://issues.apache.org/jira/browse/SPARK-33110 Project: Spark

[jira] [Commented] (SPARK-33109) Upgrade to SBT 1.4 and support `dependencyTree` back

2020-10-10 Thread Denis Pyshev (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211636#comment-17211636 ] Denis Pyshev commented on SPARK-33109: -- FYI SPARK-33080 has a build file fix to unblock upgrade to

[jira] [Commented] (SPARK-32929) StreamSuite failure on IBM Z: - SPARK-20432: union one stream with itself

2020-10-10 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211634#comment-17211634 ] Yang Jie commented on SPARK-32929: -- A little incomprehensible, Java should be able to shield the impact

[jira] [Commented] (SPARK-32978) Incorrect number of dynamic part metric

2020-10-10 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211630#comment-17211630 ] Yang Jie commented on SPARK-32978: -- It seems that there is no de-duplication mechanism > Incorrect