[jira] [Commented] (SPARK-32169) Allow filter pushdown after a groupBy with collect_list

2020-07-03 Thread L. C. Hsieh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151196#comment-17151196 ] L. C. Hsieh commented on SPARK-32169: - collect_list is non-deterministic expression.

[jira] [Updated] (SPARK-32169) Allow filter pushdown after a groupBy with collect_list

2020-07-03 Thread Jorge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jorge updated SPARK-32169: -- Description: Consider the following statement: {code:python} import pyspark import pyspark.sql.functions as

[jira] [Updated] (SPARK-32169) Allow filter pushdown after a groupBy with collect_list

2020-07-03 Thread Jorge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jorge updated SPARK-32169: -- Description: Consider the following statement: {code:python} import pyspark import pyspark.sql.functions as

[jira] [Created] (SPARK-32169) Allow filter pushdown after a groupBy with collect_list

2020-07-03 Thread Jorge (Jira)
Jorge created SPARK-32169: - Summary: Allow filter pushdown after a groupBy with collect_list Key: SPARK-32169 URL: https://issues.apache.org/jira/browse/SPARK-32169 Project: Spark Issue Type: Improve

[jira] [Comment Edited] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

2020-07-03 Thread qingwu.fu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151133#comment-17151133 ] qingwu.fu edited comment on SPARK-30602 at 7/3/20, 10:56 PM: -

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

2020-07-03 Thread qingwu.fu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151133#comment-17151133 ] qingwu.fu commented on SPARK-30602: --- hi [~mshen], could I have the access auth to see

[jira] [Assigned] (SPARK-32168) DSv2 SQL overwrite incorrectly uses static plan with hidden partitions

2020-07-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32168: Assignee: Apache Spark > DSv2 SQL overwrite incorrectly uses static plan with hidden part

[jira] [Commented] (SPARK-32168) DSv2 SQL overwrite incorrectly uses static plan with hidden partitions

2020-07-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151118#comment-17151118 ] Apache Spark commented on SPARK-32168: -- User 'rdblue' has created a pull request fo

[jira] [Assigned] (SPARK-32168) DSv2 SQL overwrite incorrectly uses static plan with hidden partitions

2020-07-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32168: Assignee: (was: Apache Spark) > DSv2 SQL overwrite incorrectly uses static plan with

[jira] [Created] (SPARK-32168) DSv2 SQL overwrite incorrectly uses static plan with hidden partitions

2020-07-03 Thread Ryan Blue (Jira)
Ryan Blue created SPARK-32168: - Summary: DSv2 SQL overwrite incorrectly uses static plan with hidden partitions Key: SPARK-32168 URL: https://issues.apache.org/jira/browse/SPARK-32168 Project: Spark

[jira] [Commented] (SPARK-32167) nullability of GetArrayStructFields is incorrect

2020-07-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150834#comment-17150834 ] Apache Spark commented on SPARK-32167: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-32167) nullability of GetArrayStructFields is incorrect

2020-07-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32167: Assignee: Wenchen Fan (was: Apache Spark) > nullability of GetArrayStructFields is incor

[jira] [Assigned] (SPARK-32167) nullability of GetArrayStructFields is incorrect

2020-07-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32167: Assignee: Apache Spark (was: Wenchen Fan) > nullability of GetArrayStructFields is incor

[jira] [Created] (SPARK-32167) nullability of GetArrayStructFields is incorrect

2020-07-03 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-32167: --- Summary: nullability of GetArrayStructFields is incorrect Key: SPARK-32167 URL: https://issues.apache.org/jira/browse/SPARK-32167 Project: Spark Issue Type: Bu

[jira] [Updated] (SPARK-32167) nullability of GetArrayStructFields is incorrect

2020-07-03 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-32167: Labels: correctness (was: ) > nullability of GetArrayStructFields is incorrect >

[jira] [Commented] (SPARK-25433) Add support for PEX in PySpark

2020-07-03 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150831#comment-17150831 ] Fabian Höring commented on SPARK-25433: --- [~hyukjin.kwon] I actually noticed that i

[jira] [Comment Edited] (SPARK-25433) Add support for PEX in PySpark

2020-07-03 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150831#comment-17150831 ] Fabian Höring edited comment on SPARK-25433 at 7/3/20, 8:14 AM: --

[jira] [Comment Edited] (SPARK-25433) Add support for PEX in PySpark

2020-07-03 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150827#comment-17150827 ] Fabian Höring edited comment on SPARK-25433 at 7/3/20, 8:07 AM: --

[jira] [Commented] (SPARK-25433) Add support for PEX in PySpark

2020-07-03 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150827#comment-17150827 ] Fabian Höring commented on SPARK-25433: --- Yes, you can put me in the tickets. I can

[jira] [Commented] (SPARK-26533) Support query auto cancel on thriftserver

2020-07-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150763#comment-17150763 ] Apache Spark commented on SPARK-26533: -- User 'leoluan2009' has created a pull reque

[jira] [Assigned] (SPARK-26533) Support query auto cancel on thriftserver

2020-07-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26533: Assignee: Apache Spark > Support query auto cancel on thriftserver >

[jira] [Assigned] (SPARK-26533) Support query auto cancel on thriftserver

2020-07-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26533: Assignee: (was: Apache Spark) > Support query auto cancel on thriftserver > -

[jira] [Commented] (SPARK-26533) Support query auto cancel on thriftserver

2020-07-03 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150764#comment-17150764 ] Apache Spark commented on SPARK-26533: -- User 'leoluan2009' has created a pull reque

[jira] [Comment Edited] (SPARK-25433) Add support for PEX in PySpark

2020-07-03 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150831#comment-17150831 ] Fabian Höring edited comment on SPARK-25433 at 7/3/20, 9:04 AM: --

[jira] [Comment Edited] (SPARK-25433) Add support for PEX in PySpark

2020-07-03 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150827#comment-17150827 ] Fabian Höring edited comment on SPARK-25433 at 7/3/20, 8:53 AM: --

[jira] [Comment Edited] (SPARK-25433) Add support for PEX in PySpark

2020-07-03 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-25433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17150827#comment-17150827 ] Fabian Höring edited comment on SPARK-25433 at 7/3/20, 8:52 AM: --

[jira] [Reopened] (SPARK-31666) Cannot map hostPath volumes to container

2020-07-03 Thread Stephen Hopper (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Hopper reopened SPARK-31666: Hi [~dongjoon], This is still a bug. I should clarify the issue a bit more. In Spark 2.4, th