[jira] [Commented] (SPARK-26385) YARN - Spark Stateful Structured streaming HDFS_DELEGATION_TOKEN not found in cache

2020-04-07 Thread Zhou Jiashuai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077854#comment-17077854 ] Zhou Jiashuai commented on SPARK-26385: --- I see the same problem with spark structured streaming

[jira] [Comment Edited] (SPARK-31376) Non-global sort support for structured streaming

2020-04-07 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077821#comment-17077821 ] Jungtaek Lim edited comment on SPARK-31376 at 4/8/20, 4:46 AM: --- Btw it

[jira] [Commented] (SPARK-30839) Add version information for Spark configuration

2020-04-07 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077824#comment-17077824 ] Wenchen Fan commented on SPARK-30839: - Great work! Thanks [~beliefer] [~hyukjin.kwon]! > Add

[jira] [Commented] (SPARK-30839) Add version information for Spark configuration

2020-04-07 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077822#comment-17077822 ] jiaan.geng commented on SPARK-30839: I made the double check. [~hyukjin.kwon] Thanks for your hard

[jira] [Commented] (SPARK-31376) Non-global sort support for structured streaming

2020-04-07 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077821#comment-17077821 ] Jungtaek Lim commented on SPARK-31376: -- Btw it would be even better if you initiate the

[jira] [Commented] (SPARK-31376) Non-global sort support for structured streaming

2020-04-07 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077819#comment-17077819 ] Jungtaek Lim commented on SPARK-31376: -- I'm saying that sort is simply unavailable operation for

[jira] [Updated] (SPARK-31381) Upgrade built-in Hive 2.3.6 to 2.3.7

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31381: - Summary: Upgrade built-in Hive 2.3.6 to 2.3.7 (was: Upgrade built-in Hive 2.3 to 2.3.7) >

[jira] [Commented] (SPARK-31376) Non-global sort support for structured streaming

2020-04-07 Thread Adam Binford (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077803#comment-17077803 ] Adam Binford commented on SPARK-31376: -- You can't global sort because you can't sort an infinite

[jira] [Resolved] (SPARK-31351) Migration Guide Auditing for Spark 3.0 Release

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-31351. -- Fix Version/s: 3.1.0 3.0.0 Resolution: Fixed Fixed in

[jira] [Updated] (SPARK-31381) Upgrade built-in Hive 2.3 to 2.3.7

2020-04-07 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-31381: Description: Hive 2.3.7 fixed these issues: HIVE-21508: ClassCastException when initializing

[jira] [Created] (SPARK-31381) Upgrade built-in Hive 2.3 to 2.3.7

2020-04-07 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-31381: --- Summary: Upgrade built-in Hive 2.3 to 2.3.7 Key: SPARK-31381 URL: https://issues.apache.org/jira/browse/SPARK-31381 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-31376) Non-global sort support for structured streaming

2020-04-07 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077723#comment-17077723 ] Jungtaek Lim commented on SPARK-31376: -- I'll reflect the question; why do you think not allowing

[jira] [Commented] (SPARK-31370) AnalysisException when too many columns in join

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077705#comment-17077705 ] Dongjoon Hyun commented on SPARK-31370: --- I added some at `Affected Versions`. For 3.0.0 RC1, the

[jira] [Updated] (SPARK-31370) AnalysisException when too many columns in join

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31370: -- Affects Version/s: 2.1.3 > AnalysisException when too many columns in join >

[jira] [Updated] (SPARK-31370) AnalysisException when too many columns in join

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31370: -- Affects Version/s: 2.2.3 > AnalysisException when too many columns in join >

[jira] [Commented] (SPARK-31370) AnalysisException when too many columns in join

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077697#comment-17077697 ] Dongjoon Hyun commented on SPARK-31370: --- Thank you for reporting with the reproducible example,

[jira] [Resolved] (SPARK-31278) numOutputRows shows value from last micro batch when there is no new data

2020-04-07 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-31278. - Resolution: Fixed > numOutputRows shows value from last micro batch when there is no new data >

[jira] [Updated] (SPARK-31370) AnalysisException when too many columns in join

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31370: -- Affects Version/s: 3.0.0 > AnalysisException when too many columns in join >

[jira] [Assigned] (SPARK-31278) numOutputRows shows value from last micro batch when there is no new data

2020-04-07 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-31278: --- Assignee: Burak Yavuz > numOutputRows shows value from last micro batch when there is no

[jira] [Commented] (SPARK-31278) numOutputRows shows value from last micro batch when there is no new data

2020-04-07 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077695#comment-17077695 ] Burak Yavuz commented on SPARK-31278: - Resolved by [https://github.com/apache/spark/pull/28040] >

[jira] [Updated] (SPARK-31278) numOutputRows shows value from last micro batch when there is no new data

2020-04-07 Thread Burak Yavuz (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-31278: Fix Version/s: 3.0.0 > numOutputRows shows value from last micro batch when there is no new data

[jira] [Updated] (SPARK-31370) AnalysisException when too many columns in join

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31370: -- Affects Version/s: 2.3.4 > AnalysisException when too many columns in join >

[jira] [Updated] (SPARK-31370) AnalysisException when too many columns in join

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31370: -- Affects Version/s: 2.4.5 > AnalysisException when too many columns in join >

[jira] [Updated] (SPARK-31376) Non-global sort support for structured streaming

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31376: -- Affects Version/s: (was: 3.0.0) 3.1.0 > Non-global sort support

[jira] [Updated] (SPARK-31377) Add unit tests for "number of output rows" metric for joins in SQLMetricsSuite

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31377: -- Summary: Add unit tests for "number of output rows" metric for joins in SQLMetricsSuite

[jira] [Updated] (SPARK-31377) Add missing unit tests for "number of output rows" metric for joins in SQLMetricsSuite

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31377: -- Affects Version/s: (was: 3.0.0) 3.1.0 > Add missing unit tests for

[jira] [Updated] (SPARK-31377) Add missing unit tests for "number of output rows" metric for joins in SQLMetricsSuite

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31377: -- Priority: Minor (was: Major) > Add missing unit tests for "number of output rows" metric for

[jira] [Comment Edited] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

2020-04-07 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077674#comment-17077674 ] Venkata krishnan Sowrirajan edited comment on SPARK-22148 at 4/8/20, 12:04 AM:

[jira] [Comment Edited] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

2020-04-07 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077674#comment-17077674 ] Venkata krishnan Sowrirajan edited comment on SPARK-22148 at 4/8/20, 12:04 AM:

[jira] [Commented] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

2020-04-07 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077674#comment-17077674 ] Venkata krishnan Sowrirajan commented on SPARK-22148: - Thanks for responding

[jira] [Updated] (SPARK-25154) Support NOT IN sub-queries inside nested OR conditions.

2020-04-07 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-25154: - Affects Version/s: (was: 2.3.1) 3.1.0 > Support NOT IN

[jira] [Resolved] (SPARK-31008) Support json_array_length function

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31008. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 27759

[jira] [Assigned] (SPARK-31008) Support json_array_length function

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31008: - Assignee: Rakesh Raushan > Support json_array_length function >

[jira] [Commented] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

2020-04-07 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077614#comment-17077614 ] Thomas Graves commented on SPARK-22148: --- I'm not sure I follow what you are saying.  Are you just

[jira] [Updated] (SPARK-31377) Add missing unit tests for "number of output rows" metric for joins in SQLMetricsSuite

2020-04-07 Thread Srinivas Rishindra Pothireddi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srinivas Rishindra Pothireddi updated SPARK-31377: -- Description: For some combinations of join algorithm and join

[jira] [Commented] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

2020-04-07 Thread Venkata krishnan Sowrirajan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077603#comment-17077603 ] Venkata krishnan Sowrirajan commented on SPARK-22148: - [~irashid][~Dhruve Ashar]

[jira] [Resolved] (SPARK-31378) stage level scheduling dynamic allocation bug with initial num executors

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31378. --- Fix Version/s: 3.1.0 Assignee: Thomas Graves Resolution: Fixed This is

[jira] [Updated] (SPARK-31378) stage level scheduling dynamic allocation bug with initial num executors

2020-04-07 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-31378: -- Affects Version/s: (was: 3.0.0) 3.1.0 > stage level scheduling

[jira] [Updated] (SPARK-31380) Peak Execution Memory Quantile is not displayed in Spark History Server UI

2020-04-07 Thread Srinivas Rishindra Pothireddi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srinivas Rishindra Pothireddi updated SPARK-31380: -- Description: Peak Execution Memory Quantile is displayed in

[jira] [Updated] (SPARK-31380) Peak Execution Memory Quantile is not displayed in Spark History Server UI

2020-04-07 Thread Srinivas Rishindra Pothireddi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srinivas Rishindra Pothireddi updated SPARK-31380: -- Description: Peak Execution Memory Quantile is displayed in

[jira] [Updated] (SPARK-31380) Peak Execution Memory Quantile is not displayed in Spark History Server UI

2020-04-07 Thread Srinivas Rishindra Pothireddi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srinivas Rishindra Pothireddi updated SPARK-31380: -- Description: Peak Execution Memory Quantile is displayed in

[jira] [Updated] (SPARK-31380) Peak Execution Memory Quantile is not displayed in Spark History Server UI

2020-04-07 Thread Srinivas Rishindra Pothireddi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srinivas Rishindra Pothireddi updated SPARK-31380: -- Description: Peak Execution Memory Quantile is displayed in

[jira] [Updated] (SPARK-31380) Peak Execution Memory Quantile is not displayed in Spark History Server UI

2020-04-07 Thread Srinivas Rishindra Pothireddi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srinivas Rishindra Pothireddi updated SPARK-31380: -- Description: Peak Execution Memory Quantile is displayed in

[jira] [Created] (SPARK-31380) Peak Execution Memory Quantile is not displayed in Spark History Server UI

2020-04-07 Thread Srinivas Rishindra Pothireddi (Jira)
Srinivas Rishindra Pothireddi created SPARK-31380: - Summary: Peak Execution Memory Quantile is not displayed in Spark History Server UI Key: SPARK-31380 URL:

[jira] [Commented] (SPARK-30839) Add version information for Spark configuration

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077572#comment-17077572 ] Dongjoon Hyun commented on SPARK-30839: --- Great! Thank you so much, [~hyukjin.kwon]! > Add version

[jira] [Resolved] (SPARK-31208) Expose the ability for user to cleanup shuffle files

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31208. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28038

[jira] [Commented] (SPARK-25102) Write Spark version to ORC/Parquet file metadata

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077536#comment-17077536 ] Dongjoon Hyun commented on SPARK-25102: --- This is backported to `branch-2.4` via

[jira] [Updated] (SPARK-25102) Write Spark version to ORC/Parquet file metadata

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25102: -- Fix Version/s: 2.4.6 > Write Spark version to ORC/Parquet file metadata >

[jira] [Comment Edited] (SPARK-31347) Unable to run Spark Job on Federated Yarn Cluster, AMRMToken invalid

2020-04-07 Thread Babble Shack (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077533#comment-17077533 ] Babble Shack edited comment on SPARK-31347 at 4/7/20, 8:06 PM: --- I was able

[jira] [Comment Edited] (SPARK-31347) Unable to run Spark Job on Federated Yarn Cluster, AMRMToken invalid

2020-04-07 Thread Babble Shack (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077533#comment-17077533 ] Babble Shack edited comment on SPARK-31347 at 4/7/20, 8:02 PM: --- I was able

[jira] [Commented] (SPARK-31347) Unable to run Spark Job on Federated Yarn Cluster, AMRMToken invalid

2020-04-07 Thread Babble Shack (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077533#comment-17077533 ] Babble Shack commented on SPARK-31347: -- I was able to resolve this by adding ha configs used in

[jira] [Resolved] (SPARK-31113) Support DDL "SHOW VIEWS"

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31113. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27897

[jira] [Assigned] (SPARK-31113) Support DDL "SHOW VIEWS"

2020-04-07 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31113: - Assignee: Xin Wu > Support DDL "SHOW VIEWS" > > >

[jira] [Updated] (SPARK-31377) Add missing unit tests for "number of output rows" metric for joins in SQLMetricsSuite

2020-04-07 Thread Srinivas Rishindra Pothireddi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srinivas Rishindra Pothireddi updated SPARK-31377: -- Description: For some combinations of join algorithm and join

[jira] [Created] (SPARK-31379) Fix flaky test: o.a.s.scheduler.CoarseGrainedSchedulerBackendSuite.extra resources from executor

2020-04-07 Thread wuyi (Jira)
wuyi created SPARK-31379: Summary: Fix flaky test: o.a.s.scheduler.CoarseGrainedSchedulerBackendSuite.extra resources from executor Key: SPARK-31379 URL: https://issues.apache.org/jira/browse/SPARK-31379

[jira] [Created] (SPARK-31378) stage level scheduling dynamic allocation bug with initial num executors

2020-04-07 Thread Thomas Graves (Jira)
Thomas Graves created SPARK-31378: - Summary: stage level scheduling dynamic allocation bug with initial num executors Key: SPARK-31378 URL: https://issues.apache.org/jira/browse/SPARK-31378 Project:

[jira] [Resolved] (SPARK-30820) Add FMClassifier to SparkR

2020-04-07 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30820. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 27570

[jira] [Created] (SPARK-31377) Add missing unit tests for "number of output rows" metric for joins in SQLMetricsSuite

2020-04-07 Thread Srinivas Rishindra Pothireddi (Jira)
Srinivas Rishindra Pothireddi created SPARK-31377: - Summary: Add missing unit tests for "number of output rows" metric for joins in SQLMetricsSuite Key: SPARK-31377 URL:

[jira] [Assigned] (SPARK-30820) Add FMClassifier to SparkR

2020-04-07 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-30820: Assignee: Maciej Szymkiewicz > Add FMClassifier to SparkR > -- >

[jira] [Commented] (SPARK-31367) add octet_length to functions

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077259#comment-17077259 ] Hyukjin Kwon commented on SPARK-31367: -- For now, you can easily work around via

[jira] [Commented] (SPARK-31367) add octet_length to functions

2020-04-07 Thread Bill Schneider (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077228#comment-17077228 ] Bill Schneider commented on SPARK-31367: [~hyukjin.kwon] Why does DBMS support matter?  

[jira] [Created] (SPARK-31376) Non-global sort support for structured streaming

2020-04-07 Thread Adam Binford (Jira)
Adam Binford created SPARK-31376: Summary: Non-global sort support for structured streaming Key: SPARK-31376 URL: https://issues.apache.org/jira/browse/SPARK-31376 Project: Spark Issue Type:

[jira] [Updated] (SPARK-30839) Add version information for Spark configuration

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30839: - Fix Version/s: 3.1.0 3.0.0 > Add version information for Spark configuration

[jira] [Commented] (SPARK-30839) Add version information for Spark configuration

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077200#comment-17077200 ] Hyukjin Kwon commented on SPARK-30839: -- FYI [~cloud_fan], [~smilegator], [~dongjoon], I think we

[jira] [Updated] (SPARK-31295) Supplement version for configuration appear in doc

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31295: - Fix Version/s: 3.0.0 > Supplement version for configuration appear in doc >

[jira] [Updated] (SPARK-31282) Supplement version for configuration appear in security doc

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31282: - Fix Version/s: 3.0.0 > Supplement version for configuration appear in security doc >

[jira] [Updated] (SPARK-31269) Supplement version for configuration only appear in configuration doc

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31269: - Fix Version/s: 3.0.0 > Supplement version for configuration only appear in configuration doc >

[jira] [Commented] (SPARK-30839) Add version information for Spark configuration

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077199#comment-17077199 ] Hyukjin Kwon commented on SPARK-30839: -- [~beliefer], thanks for your efforts to port this back. I

[jira] [Updated] (SPARK-31279) Add version information to the configuration of Hive

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31279: - Fix Version/s: 3.0.0 > Add version information to the configuration of Hive >

[jira] [Updated] (SPARK-31002) Add version information to the configuration of Core

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31002: - Fix Version/s: 3.0.0 > Add version information to the configuration of Core >

[jira] [Updated] (SPARK-30841) Add version information to the configuration of SQL

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30841: - Fix Version/s: 3.0.0 > Add version information to the configuration of SQL >

[jira] [Updated] (SPARK-31215) Add version information to the static configuration of SQL

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31215: - Fix Version/s: 3.0.0 > Add version information to the static configuration of SQL >

[jira] [Updated] (SPARK-30913) Add version information to the configuration of Tests.scala

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30913: - Fix Version/s: 3.0.0 > Add version information to the configuration of Tests.scala >

[jira] [Updated] (SPARK-30911) Add version information to the configuration of Status

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30911: - Fix Version/s: 3.0.0 > Add version information to the configuration of Status >

[jira] [Updated] (SPARK-31228) Add version information to the configuration of Kafka

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31228: - Fix Version/s: 3.0.0 > Add version information to the configuration of Kafka >

[jira] [Updated] (SPARK-31141) Add version information to the configuration of Dstreams

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31141: - Fix Version/s: 3.0.0 > Add version information to the configuration of Dstreams >

[jira] [Updated] (SPARK-31002) Add version information to the configuration of Core

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31002: - Fix Version/s: (was: 3.0.0) > Add version information to the configuration of Core >

[jira] [Updated] (SPARK-30841) Add version information to the configuration of SQL

2020-04-07 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30841: - Fix Version/s: (was: 3.0.0) > Add version information to the configuration of SQL >

[jira] [Updated] (SPARK-31375) Overwriting into dynamic partitions is appending data in pyspark

2020-04-07 Thread Sai Krishna Chaitanya Chaganti (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Krishna Chaitanya Chaganti updated SPARK-31375: --- Description: While overwriting data in specific partitions

[jira] [Created] (SPARK-31375) Overwriting into dynamic partitions is appending data

2020-04-07 Thread Sai Krishna Chaitanya Chaganti (Jira)
Sai Krishna Chaitanya Chaganti created SPARK-31375: -- Summary: Overwriting into dynamic partitions is appending data Key: SPARK-31375 URL: https://issues.apache.org/jira/browse/SPARK-31375

[jira] [Updated] (SPARK-31375) Overwriting into dynamic partitions is appending data in pyspark

2020-04-07 Thread Sai Krishna Chaitanya Chaganti (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Krishna Chaitanya Chaganti updated SPARK-31375: --- Summary: Overwriting into dynamic partitions is appending

[jira] [Created] (SPARK-31374) Returning complex types in Pandas UDF

2020-04-07 Thread F. H. (Jira)
F. H. created SPARK-31374: - Summary: Returning complex types in Pandas UDF Key: SPARK-31374 URL: https://issues.apache.org/jira/browse/SPARK-31374 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-31373) Cluster tried to fetch blocks from blacklisted node of previous stage

2020-04-07 Thread Yuchen Feng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuchen Feng updated SPARK-31373: Description: We enabled blacklist on our Spark application but recently we saw some wierd issue.

[jira] [Updated] (SPARK-31373) Cluster tried to fetch blocks from blacklisted node of previous stage

2020-04-07 Thread Yuchen Feng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuchen Feng updated SPARK-31373: Environment: EMR cluster with r5.4xlarge and r5.8xlarge instances > Cluster tried to fetch blocks

[jira] [Updated] (SPARK-31373) Cluster tried to fetch blocks from blacklisted node of previous stage

2020-04-07 Thread Yuchen Feng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuchen Feng updated SPARK-31373: Description: We enabled blacklist on our Spark application but recently we saw some wierd issue.

[jira] [Updated] (SPARK-31373) Cluster tried to fetch blocks from blacklisted node of previous stage

2020-04-07 Thread Yuchen Feng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuchen Feng updated SPARK-31373: Issue Type: Question (was: Bug) > Cluster tried to fetch blocks from blacklisted node of

[jira] [Created] (SPARK-31373) Cluster tried to fetch blocks from blacklisted node of previous stage

2020-04-07 Thread Yuchen Feng (Jira)
Yuchen Feng created SPARK-31373: --- Summary: Cluster tried to fetch blocks from blacklisted node of previous stage Key: SPARK-31373 URL: https://issues.apache.org/jira/browse/SPARK-31373 Project: Spark

[jira] [Commented] (SPARK-31372) Check the validity of built-in function aliases

2020-04-07 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077070#comment-17077070 ] jiaan.geng commented on SPARK-31372: I'm working on. > Check the validity of built-in function

[jira] [Created] (SPARK-31372) Check the validity of built-in function aliases

2020-04-07 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-31372: -- Summary: Check the validity of built-in function aliases Key: SPARK-31372 URL: https://issues.apache.org/jira/browse/SPARK-31372 Project: Spark Issue Type:

[jira] [Commented] (SPARK-31371) FileStreamSource: Decide seen files on the checksum, instead of filename.

2020-04-07 Thread Prashant Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077048#comment-17077048 ] Prashant Sharma commented on SPARK-31371: - [~tdas] What do you think? > FileStreamSource:

[jira] [Created] (SPARK-31371) FileStreamSource: Decide seen files on the checksum, instead of filename.

2020-04-07 Thread Prashant Sharma (Jira)
Prashant Sharma created SPARK-31371: --- Summary: FileStreamSource: Decide seen files on the checksum, instead of filename. Key: SPARK-31371 URL: https://issues.apache.org/jira/browse/SPARK-31371

[jira] [Updated] (SPARK-31370) AnalysisException when too many columns in join

2020-04-07 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-31370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riccardo Delegà updated SPARK-31370: Description: Python version: '3.7.6 | packaged by conda-forge | (default, Jan 7 2020,

[jira] [Created] (SPARK-31370) AnalysisException when too many columns in join

2020-04-07 Thread Jira
Riccardo Delegà created SPARK-31370: --- Summary: AnalysisException when too many columns in join Key: SPARK-31370 URL: https://issues.apache.org/jira/browse/SPARK-31370 Project: Spark Issue