[jira] [Created] (SPARK-30806) Evaluate once per group in UnboundedWindowFunctionFrame

2020-02-12 Thread EdisonWang (Jira)
EdisonWang created SPARK-30806: -- Summary: Evaluate once per group in UnboundedWindowFunctionFrame Key: SPARK-30806 URL: https://issues.apache.org/jira/browse/SPARK-30806 Project: Spark Issue Typ

[jira] [Created] (SPARK-30805) Failed to get locally stored broadcast data: broadcast_30

2020-02-12 Thread Jiasi (Jira)
Jiasi created SPARK-30805: - Summary: Failed to get locally stored broadcast data: broadcast_30 Key: SPARK-30805 URL: https://issues.apache.org/jira/browse/SPARK-30805 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-30804) Measure and log elapsed time for "compact" operation in CompactibleFileStreamLog

2020-02-12 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-30804: Summary: Measure and log elapsed time for "compact" operation in CompactibleFileStreamLog Key: SPARK-30804 URL: https://issues.apache.org/jira/browse/SPARK-30804 Proj

[jira] [Resolved] (SPARK-30790) The datatype of map() should be map

2020-02-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30790. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27542 [https://gith

[jira] [Assigned] (SPARK-30790) The datatype of map() should be map

2020-02-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30790: --- Assignee: Rakesh Raushan > The datatype of map() should be map > --

[jira] [Updated] (SPARK-30791) Dataframe add sameResult and sementicHash method

2020-02-12 Thread Weichen Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-30791: --- Description: Sometimes, we want to check whether two dataframes are the same. There is already an i

[jira] [Updated] (SPARK-30791) Dataframe add sameResult and sementicHash method

2020-02-12 Thread Weichen Xu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-30791: --- Description: Sometimes, we want to check whether two dataframes are the same. There is already an i

[jira] [Updated] (SPARK-30803) Fix the home page link to Scala API document

2020-02-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30803: -- Priority: Blocker (was: Major) > Fix the home page link to Scala API document > -

[jira] [Created] (SPARK-30803) Fix the home page link to Scala API document

2020-02-12 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-30803: - Summary: Fix the home page link to Scala API document Key: SPARK-30803 URL: https://issues.apache.org/jira/browse/SPARK-30803 Project: Spark Issue Type: Bu

[jira] [Commented] (SPARK-16387) Reserved SQL words are not escaped by JDBC writer

2020-02-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-16387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035774#comment-17035774 ] Dongjoon Hyun commented on SPARK-16387: --- Hi, [~igreenfi]. What is the problem? For

[jira] [Resolved] (SPARK-29148) Modify dynamic allocation manager for stage level scheduling

2020-02-12 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-29148. --- Fix Version/s: 3.1.0 Resolution: Fixed > Modify dynamic allocation manager for stage

[jira] [Updated] (SPARK-30797) Set tradition user/group/other permission to ACL entries when setting up ACLs in truncate table

2020-02-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30797: -- Fix Version/s: 2.4.6 > Set tradition user/group/other permission to ACL entries when setting u

[jira] [Updated] (SPARK-30797) Set tradition user/group/other permission to ACL entries when setting up ACLs in truncate table

2020-02-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30797: -- Affects Version/s: 2.4.5 > Set tradition user/group/other permission to ACL entries when setti

[jira] [Resolved] (SPARK-30797) Set tradition user/group/other permission to ACL entries when setting up ACLs in truncate table

2020-02-12 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30797. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27548 [https://

[jira] [Updated] (SPARK-30801) Subqueries should not be AQE-ed if main query is not

2020-02-12 Thread Wei Xue (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Xue updated SPARK-30801: Description: Currently there are unsupported queries by AQE, e.g., queries that contain DPP filters. But i

[jira] [Created] (SPARK-30802) Use Summarizer instead of MultivariateOnlineSummarizer in Aggregator test suite

2020-02-12 Thread Huaxin Gao (Jira)
Huaxin Gao created SPARK-30802: -- Summary: Use Summarizer instead of MultivariateOnlineSummarizer in Aggregator test suite Key: SPARK-30802 URL: https://issues.apache.org/jira/browse/SPARK-30802 Project:

[jira] [Created] (SPARK-30801) Subqueries should not be AQE-ed if main query is not

2020-02-12 Thread Wei Xue (Jira)
Wei Xue created SPARK-30801: --- Summary: Subqueries should not be AQE-ed if main query is not Key: SPARK-30801 URL: https://issues.apache.org/jira/browse/SPARK-30801 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-30800) Refactor BarrierTaskContext barrier method to use allGather directly

2020-02-12 Thread Sarth Frey (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarth Frey updated SPARK-30800: --- Description: Issue is only present if this PR gets merged:  [https://github.com/apache/spark/pull/27

[jira] [Updated] (SPARK-30800) Refactor BarrierTaskContext barrier method to use allGather directly

2020-02-12 Thread Sarth Frey (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarth Frey updated SPARK-30800: --- Description: Issue is only present if this PR gets merged:  [https://github.com/apache/spark/pull/27

[jira] [Updated] (SPARK-30800) Refactor BarrierTaskContext barrier method to use allGather directly

2020-02-12 Thread Sarth Frey (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sarth Frey updated SPARK-30800: --- Description: Issue is only present if this PR gets merged:  [https://github.com/apache/spark/pull/27

[jira] [Created] (SPARK-30800) Refactor BarrierTaskContext barrier method to use allGather directly

2020-02-12 Thread Sarth Frey (Jira)
Sarth Frey created SPARK-30800: -- Summary: Refactor BarrierTaskContext barrier method to use allGather directly Key: SPARK-30800 URL: https://issues.apache.org/jira/browse/SPARK-30800 Project: Spark

[jira] [Resolved] (SPARK-30760) Port `millisToDays` and `daysToMillis` on Java 8 time API

2020-02-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30760. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27494 [https://gith

[jira] [Assigned] (SPARK-30760) Port `millisToDays` and `daysToMillis` on Java 8 time API

2020-02-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30760: --- Assignee: Maxim Gekk > Port `millisToDays` and `daysToMillis` on Java 8 time API >

[jira] [Assigned] (SPARK-30651) EXPLAIN EXTENDED does not show detail information for aggregate operators

2020-02-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30651: --- Assignee: Xin Wu > EXPLAIN EXTENDED does not show detail information for aggregate operator

[jira] [Resolved] (SPARK-30651) EXPLAIN EXTENDED does not show detail information for aggregate operators

2020-02-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30651. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27368 [https://gith

[jira] [Commented] (SPARK-30711) 64KB JVM bytecode limit - janino.InternalCompilerException

2020-02-12 Thread Kazuaki Ishizaki (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035464#comment-17035464 ] Kazuaki Ishizaki commented on SPARK-30711: -- The reason to show the exception as

[jira] [Updated] (SPARK-30528) Potential performance regression with DPP subquery duplication

2020-02-12 Thread Wei Xue (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Xue updated SPARK-30528: Summary: Potential performance regression with DPP subquery duplication (was: DPP issues) > Potential pe

[jira] [Created] (SPARK-30799) CatalogAndIdentifier shouldn't return wrong namespace

2020-02-12 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-30799: --- Summary: CatalogAndIdentifier shouldn't return wrong namespace Key: SPARK-30799 URL: https://issues.apache.org/jira/browse/SPARK-30799 Project: Spark Issue Typ

[jira] [Commented] (SPARK-30763) Fix java.lang.IndexOutOfBoundsException No group 1 for regexp_extract

2020-02-12 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035398#comment-17035398 ] jiaan.geng commented on SPARK-30763: Hive exception when idx > regex group count.

[jira] [Commented] (SPARK-16387) Reserved SQL words are not escaped by JDBC writer

2020-02-12 Thread Izek Greenfield (Jira)
[ https://issues.apache.org/jira/browse/SPARK-16387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035395#comment-17035395 ] Izek Greenfield commented on SPARK-16387: -   [~dongjoon] I think that PR fix o

[jira] [Commented] (SPARK-29474) CLI support for Spark-on-Docker-on-Yarn

2020-02-12 Thread Adam Antal (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035372#comment-17035372 ] Adam Antal commented on SPARK-29474: AFAIK YARN obtains the Docker image from the re

[jira] [Created] (SPARK-30798) Scope Session.active in QueryExecution

2020-02-12 Thread Ali Afroozeh (Jira)
Ali Afroozeh created SPARK-30798: Summary: Scope Session.active in QueryExecution Key: SPARK-30798 URL: https://issues.apache.org/jira/browse/SPARK-30798 Project: Spark Issue Type: Improvemen

[jira] [Resolved] (SPARK-30788) Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters

2020-02-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30788. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27524 [https://gith

[jira] [Assigned] (SPARK-30788) Support `SimpleDateFormat` and `FastDateFormat` as legacy date/timestamp formatters

2020-02-12 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30788: --- Assignee: Maxim Gekk > Support `SimpleDateFormat` and `FastDateFormat` as legacy date/times

[jira] [Commented] (SPARK-28067) Incorrect results in decimal aggregation with whole-stage code gen enabled

2020-02-12 Thread Javier Fuentes (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035286#comment-17035286 ] Javier Fuentes commented on SPARK-28067: [~dongjoon], [~msirek]  what should be

[jira] [Created] (SPARK-30797) Set tradition user/group/other permission to ACL entries when setting up ACLs in truncate table

2020-02-12 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-30797: --- Summary: Set tradition user/group/other permission to ACL entries when setting up ACLs in truncate table Key: SPARK-30797 URL: https://issues.apache.org/jira/browse/SPARK-30797

[jira] [Comment Edited] (SPARK-27710) ClassNotFoundException: $line196400984558.$read$ in OuterScopes

2020-02-12 Thread Jelmer Kuperus (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034532#comment-17034532 ] Jelmer Kuperus edited comment on SPARK-27710 at 2/12/20 8:11 AM: -

[jira] [Commented] (SPARK-27710) ClassNotFoundException: $line196400984558.$read$ in OuterScopes

2020-02-12 Thread Jelmer Kuperus (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035135#comment-17035135 ] Jelmer Kuperus commented on SPARK-27710: Turns out it was this https://issues.ap