[jira] [Created] (SPARK-32739) support prune right for left semi join in DPP

2020-08-30 Thread Zhenhua Wang (Jira)
Zhenhua Wang created SPARK-32739: Summary: support prune right for left semi join in DPP Key: SPARK-32739 URL: https://issues.apache.org/jira/browse/SPARK-32739 Project: Spark Issue Type: Imp

[jira] [Assigned] (SPARK-32739) support prune right for left semi join in DPP

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32739: Assignee: Apache Spark > support prune right for left semi join in DPP >

[jira] [Assigned] (SPARK-32739) support prune right for left semi join in DPP

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32739: Assignee: (was: Apache Spark) > support prune right for left semi join in DPP > -

[jira] [Commented] (SPARK-32739) support prune right for left semi join in DPP

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187149#comment-17187149 ] Apache Spark commented on SPARK-32739: -- User 'wzhfy' has created a pull request for

[jira] [Created] (SPARK-32740) Refactor common partitioning/distribution logic to BaseAggregateExec

2020-08-30 Thread Cheng Su (Jira)
Cheng Su created SPARK-32740: Summary: Refactor common partitioning/distribution logic to BaseAggregateExec Key: SPARK-32740 URL: https://issues.apache.org/jira/browse/SPARK-32740 Project: Spark

[jira] [Commented] (SPARK-32740) Refactor common partitioning/distribution logic to BaseAggregateExec

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187150#comment-17187150 ] Apache Spark commented on SPARK-32740: -- User 'c21' has created a pull request for t

[jira] [Assigned] (SPARK-32740) Refactor common partitioning/distribution logic to BaseAggregateExec

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32740: Assignee: Apache Spark > Refactor common partitioning/distribution logic to BaseAggregate

[jira] [Assigned] (SPARK-32740) Refactor common partitioning/distribution logic to BaseAggregateExec

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32740: Assignee: (was: Apache Spark) > Refactor common partitioning/distribution logic to Ba

[jira] [Commented] (SPARK-32364) Use CaseInsensitiveMap for DataFrameReader/Writer options

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187152#comment-17187152 ] Apache Spark commented on SPARK-32364: -- User 'karolchmist' has created a pull reque

[jira] [Created] (SPARK-32741) Check if the same ExprId refers to the unique attribute in logical plans

2020-08-30 Thread Takeshi Yamamuro (Jira)
Takeshi Yamamuro created SPARK-32741: Summary: Check if the same ExprId refers to the unique attribute in logical plans Key: SPARK-32741 URL: https://issues.apache.org/jira/browse/SPARK-32741 Proj

[jira] [Assigned] (SPARK-32741) Check if the same ExprId refers to the unique attribute in logical plans

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32741: Assignee: Apache Spark > Check if the same ExprId refers to the unique attribute in logic

[jira] [Commented] (SPARK-32741) Check if the same ExprId refers to the unique attribute in logical plans

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187200#comment-17187200 ] Apache Spark commented on SPARK-32741: -- User 'maropu' has created a pull request fo

[jira] [Assigned] (SPARK-32741) Check if the same ExprId refers to the unique attribute in logical plans

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32741: Assignee: (was: Apache Spark) > Check if the same ExprId refers to the unique attribu

[jira] [Commented] (SPARK-32741) Check if the same ExprId refers to the unique attribute in logical plans

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187199#comment-17187199 ] Apache Spark commented on SPARK-32741: -- User 'maropu' has created a pull request fo

[jira] [Commented] (SPARK-32739) support prune right for left semi join in DPP

2020-08-30 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187202#comment-17187202 ] Takeshi Yamamuro commented on SPARK-32739: -- Could you fill the description, too

[jira] [Updated] (SPARK-32677) Load function resource before create

2020-08-30 Thread ulysses you (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ulysses you updated SPARK-32677: Summary: Load function resource before create (was: Cache function directly after create) > Load

[jira] [Created] (SPARK-32742) FileOutputCommitter warns "No Output found for attempt"

2020-08-30 Thread Ryan Luo (Jira)
Ryan Luo created SPARK-32742: Summary: FileOutputCommitter warns "No Output found for attempt" Key: SPARK-32742 URL: https://issues.apache.org/jira/browse/SPARK-32742 Project: Spark Issue Type: B

[jira] [Updated] (SPARK-32742) FileOutputCommitter warns "No Output found for attempt"

2020-08-30 Thread Ryan Luo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Luo updated SPARK-32742: - Priority: Blocker (was: Major) > FileOutputCommitter warns "No Output found for attempt" >

[jira] [Updated] (SPARK-32742) FileOutputCommitter warns "No Output found for attempt"

2020-08-30 Thread Ryan Luo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Luo updated SPARK-32742: - Component/s: (was: Spark Submit) Spark Core > FileOutputCommitter warns "No Output

[jira] [Updated] (SPARK-32742) FileOutputCommitter warns "No Output found for attempt"

2020-08-30 Thread Ryan Luo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Luo updated SPARK-32742: - Description: Hi team, This is my first time to report an issue here. We submitted and ran the spark jo

[jira] [Created] (SPARK-32743) Add distinct info at UnresolvedFunction toString

2020-08-30 Thread ulysses you (Jira)
ulysses you created SPARK-32743: --- Summary: Add distinct info at UnresolvedFunction toString Key: SPARK-32743 URL: https://issues.apache.org/jira/browse/SPARK-32743 Project: Spark Issue Type: Im

[jira] [Assigned] (SPARK-32481) Support truncate table to move the data to trash

2020-08-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32481: - Assignee: Udbhav Agrawal > Support truncate table to move the data to trash > -

[jira] [Resolved] (SPARK-32481) Support truncate table to move the data to trash

2020-08-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32481. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29552 [https://

[jira] [Updated] (SPARK-31511) Make BytesToBytesMap iterator() thread-safe

2020-08-30 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-31511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell updated SPARK-31511: -- Labels: correctness (was: ) > Make BytesToBytesMap iterator() thread-safe > -

[jira] [Updated] (SPARK-31511) Make BytesToBytesMap iterator() thread-safe

2020-08-30 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-31511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell updated SPARK-31511: -- Issue Type: Bug (was: Improvement) > Make BytesToBytesMap iterator() thread-safe > --

[jira] [Updated] (SPARK-31511) Make BytesToBytesMap iterator() thread-safe

2020-08-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31511: -- Fix Version/s: 3.0.2 > Make BytesToBytesMap iterator() thread-safe > -

[jira] [Commented] (SPARK-31511) Make BytesToBytesMap iterator() thread-safe

2020-08-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187334#comment-17187334 ] Dongjoon Hyun commented on SPARK-31511: --- This is backported to branch-3.0 via htt

[jira] [Commented] (SPARK-31511) Make BytesToBytesMap iterator() thread-safe

2020-08-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187337#comment-17187337 ] Dongjoon Hyun commented on SPARK-31511: --- Thank you for updating issue type and lab

[jira] [Updated] (SPARK-31511) Make BytesToBytesMap iterator() thread-safe

2020-08-30 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31511: -- Affects Version/s: 1.4.1 1.5.2 1.6.3

[jira] [Commented] (SPARK-32743) Add distinct info at UnresolvedFunction toString

2020-08-30 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187341#comment-17187341 ] Takeshi Yamamuro commented on SPARK-32743: -- Please describe more in the descrip

[jira] [Assigned] (SPARK-32743) Add distinct info at UnresolvedFunction toString

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32743: Assignee: (was: Apache Spark) > Add distinct info at UnresolvedFunction toString > --

[jira] [Assigned] (SPARK-32743) Add distinct info at UnresolvedFunction toString

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32743: Assignee: Apache Spark > Add distinct info at UnresolvedFunction toString > -

[jira] [Commented] (SPARK-32743) Add distinct info at UnresolvedFunction toString

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187346#comment-17187346 ] Apache Spark commented on SPARK-32743: -- User 'ulysses-you' has created a pull reque

[jira] [Commented] (SPARK-32743) Add distinct info at UnresolvedFunction toString

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187347#comment-17187347 ] Apache Spark commented on SPARK-32743: -- User 'ulysses-you' has created a pull reque

[jira] [Commented] (SPARK-32084) Replace dictionary-based function definitions to proper functions in functions.py

2020-08-30 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187364#comment-17187364 ] Nicholas Chammas commented on SPARK-32084: -- Can you share a couple of examples

[jira] [Commented] (SPARK-32084) Replace dictionary-based function definitions to proper functions in functions.py

2020-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187365#comment-17187365 ] Hyukjin Kwon commented on SPARK-32084: -- Hey [~nchammas], thanks for taking a look.

[jira] [Resolved] (SPARK-32733) Add extended information - arguments/examples/since/notes of expressions to the remarks field of GetFunctionsOperation

2020-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32733. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29577 [https://gi

[jira] [Assigned] (SPARK-32733) Add extended information - arguments/examples/since/notes of expressions to the remarks field of GetFunctionsOperation

2020-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32733: Assignee: Kent Yao > Add extended information - arguments/examples/since/notes of express

[jira] [Created] (SPARK-32744) request executor cores with decimal when spark on k8s

2020-08-30 Thread Yu Wang (Jira)
Yu Wang created SPARK-32744: --- Summary: request executor cores with decimal when spark on k8s Key: SPARK-32744 URL: https://issues.apache.org/jira/browse/SPARK-32744 Project: Spark Issue Type: Impro

[jira] [Updated] (SPARK-32744) request executor cores with decimal when spark on k8s

2020-08-30 Thread Yu Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Wang updated SPARK-32744: Attachment: screenshot-1.png > request executor cores with decimal when spark on k8s > ---

[jira] [Updated] (SPARK-32744) request executor cores with decimal when spark on k8s

2020-08-30 Thread Yu Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Wang updated SPARK-32744: Description: In current spark version which does not support to request executor cores with decimal whe

[jira] [Commented] (SPARK-32719) Add Flake8 check for missing imports

2020-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187381#comment-17187381 ] Hyukjin Kwon commented on SPARK-32719: -- Fixed in https://github.com/apache/spark/pu

[jira] [Resolved] (SPARK-32719) Add Flake8 check for missing imports

2020-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32719. -- Fix Version/s: 3.1.0 Resolution: Fixed > Add Flake8 check for missing imports > ---

[jira] [Assigned] (SPARK-32719) Add Flake8 check for missing imports

2020-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32719: Assignee: Fokko Driesprong > Add Flake8 check for missing imports > -

[jira] [Assigned] (SPARK-32376) Make unionByName null-filling behavior work with struct columns

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32376: Assignee: Apache Spark (was: L. C. Hsieh) > Make unionByName null-filling behavior work

[jira] [Assigned] (SPARK-32376) Make unionByName null-filling behavior work with struct columns

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32376: Assignee: L. C. Hsieh (was: Apache Spark) > Make unionByName null-filling behavior work

[jira] [Commented] (SPARK-32376) Make unionByName null-filling behavior work with struct columns

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187382#comment-17187382 ] Apache Spark commented on SPARK-32376: -- User 'viirya' has created a pull request fo

[jira] [Created] (SPARK-32745) Add an option to reject block migrations when under disk pressure

2020-08-30 Thread Holden Karau (Jira)
Holden Karau created SPARK-32745: Summary: Add an option to reject block migrations when under disk pressure Key: SPARK-32745 URL: https://issues.apache.org/jira/browse/SPARK-32745 Project: Spark

[jira] [Commented] (SPARK-32400) Test coverage of HiveScripTransformationExec

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187403#comment-17187403 ] Apache Spark commented on SPARK-32400: -- User 'AngersZh' has created a pull requ

[jira] [Created] (SPARK-32746) Not able to run Pandas UDF

2020-08-30 Thread Rahul Bhatia (Jira)
Rahul Bhatia created SPARK-32746: Summary: Not able to run Pandas UDF Key: SPARK-32746 URL: https://issues.apache.org/jira/browse/SPARK-32746 Project: Spark Issue Type: Bug Compone

[jira] [Updated] (SPARK-32746) Not able to run Pandas UDF

2020-08-30 Thread Rahul Bhatia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Bhatia updated SPARK-32746: - Description: Hi, I am facing issues in running Pandas UDF on a yarn cluster with multiple nodes

[jira] [Updated] (SPARK-32746) Not able to run Pandas UDF

2020-08-30 Thread Rahul Bhatia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Bhatia updated SPARK-32746: - Attachment: Screenshot 2020-08-31 at 9.04.07 AM.png > Not able to run Pandas UDF >

[jira] [Updated] (SPARK-32746) Not able to run Pandas UDF

2020-08-30 Thread Rahul Bhatia (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rahul Bhatia updated SPARK-32746: - Description: Hi, I am facing issues in running Pandas UDF on a yarn cluster with multiple nodes

[jira] [Commented] (SPARK-23812) DFS should be removed from unsupportedHiveNativeCommands in SqlBase.g4

2020-08-30 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187414#comment-17187414 ] Yuming Wang commented on SPARK-23812: - We don’t support {{dfs}}. {noformat} scala> s

[jira] [Created] (SPARK-32747) Deduplicate configuration set/unset in test_sparkSQL_arrow.R

2020-08-30 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-32747: Summary: Deduplicate configuration set/unset in test_sparkSQL_arrow.R Key: SPARK-32747 URL: https://issues.apache.org/jira/browse/SPARK-32747 Project: Spark

[jira] [Created] (SPARK-32748) Support local property propagation in SubqueryBroadcastExec

2020-08-30 Thread Zhenhua Wang (Jira)
Zhenhua Wang created SPARK-32748: Summary: Support local property propagation in SubqueryBroadcastExec Key: SPARK-32748 URL: https://issues.apache.org/jira/browse/SPARK-32748 Project: Spark

[jira] [Commented] (SPARK-32748) Support local property propagation in SubqueryBroadcastExec

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187436#comment-17187436 ] Apache Spark commented on SPARK-32748: -- User 'wzhfy' has created a pull request for

[jira] [Commented] (SPARK-32748) Support local property propagation in SubqueryBroadcastExec

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187435#comment-17187435 ] Apache Spark commented on SPARK-32748: -- User 'wzhfy' has created a pull request for

[jira] [Assigned] (SPARK-32748) Support local property propagation in SubqueryBroadcastExec

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32748: Assignee: Apache Spark > Support local property propagation in SubqueryBroadcastExec > --

[jira] [Assigned] (SPARK-32748) Support local property propagation in SubqueryBroadcastExec

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32748: Assignee: (was: Apache Spark) > Support local property propagation in SubqueryBroadca

[jira] [Updated] (SPARK-32749) Refactor DataFrameStatPerfSuite to use main method

2020-08-30 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-32749: Description: [https://github.com/apache/spark/blob/be2eca22e9888d2e2ea61b159a215c1f42f9261f/sql/co

[jira] [Created] (SPARK-32749) Refactor DataFrameStatPerfSuite to use main method

2020-08-30 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-32749: --- Summary: Refactor DataFrameStatPerfSuite to use main method Key: SPARK-32749 URL: https://issues.apache.org/jira/browse/SPARK-32749 Project: Spark Issue Type:

[jira] [Commented] (SPARK-32749) Refactor DataFrameStatPerfSuite to use main method

2020-08-30 Thread JiangYang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187438#comment-17187438 ] JiangYang commented on SPARK-32749: --- Im working on. > Refactor DataFrameStatPerfSuite

[jira] [Created] (SPARK-32750) Add code-gen for SortAggregateExec

2020-08-30 Thread Cheng Su (Jira)
Cheng Su created SPARK-32750: Summary: Add code-gen for SortAggregateExec Key: SPARK-32750 URL: https://issues.apache.org/jira/browse/SPARK-32750 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-32750) Add code-gen for SortAggregateExec

2020-08-30 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187441#comment-17187441 ] Cheng Su commented on SPARK-32750: -- Just FYI - will raise a PR in next couple of weeks.

[jira] [Commented] (SPARK-32746) Not able to run Pandas UDF

2020-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187443#comment-17187443 ] Hyukjin Kwon commented on SPARK-32746: -- [~rahulbhatia], please share your analysis

[jira] [Commented] (SPARK-32744) request executor cores with decimal when spark on k8s

2020-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187444#comment-17187444 ] Hyukjin Kwon commented on SPARK-32744: -- Is it realistic to have 2147483647+ cores?

[jira] [Updated] (SPARK-32742) FileOutputCommitter warns "No Output found for attempt"

2020-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32742: - Priority: Major (was: Blocker) > FileOutputCommitter warns "No Output found for attempt" >

[jira] [Commented] (SPARK-32742) FileOutputCommitter warns "No Output found for attempt"

2020-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187445#comment-17187445 ] Hyukjin Kwon commented on SPARK-32742: -- Please don't specify a blocker which is usu

[jira] [Created] (SPARK-32751) Add code-gen for ObjectHashAggregateExec

2020-08-30 Thread Cheng Su (Jira)
Cheng Su created SPARK-32751: Summary: Add code-gen for ObjectHashAggregateExec Key: SPARK-32751 URL: https://issues.apache.org/jira/browse/SPARK-32751 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-32751) Add code-gen for ObjectHashAggregateExec

2020-08-30 Thread Cheng Su (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187446#comment-17187446 ] Cheng Su commented on SPARK-32751: -- Just FYI - will raise a PR in next couple of weeks.

[jira] [Resolved] (SPARK-32737) readResolve() in BlockManagerId

2020-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32737. -- Resolution: Invalid Looks like a question. Please ask questions to mailing list, see also htt

[jira] [Updated] (SPARK-32732) Convert schema only once in OrcSerializer

2020-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32732: - Description: This is to track a TODO item in [Pull Request|https://github.com/apache/spark/pull/

[jira] [Resolved] (SPARK-32724) java.io.IOException: Stream is corrupted when tried to inner join 4 huge tables. Currently using pyspark version 2.4.0-cdh6.3.1

2020-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32724. -- Resolution: Invalid > java.io.IOException: Stream is corrupted when tried to inner join 4 huge

[jira] [Commented] (SPARK-32724) java.io.IOException: Stream is corrupted when tried to inner join 4 huge tables. Currently using pyspark version 2.4.0-cdh6.3.1

2020-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187449#comment-17187449 ] Hyukjin Kwon commented on SPARK-32724: -- Let's ask questions to the mailing list (h

[jira] [Resolved] (SPARK-32720) Spark 3 Fails to Cast DateType to StringType when comparing result of Max

2020-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32720. -- Resolution: Cannot Reproduce > Spark 3 Fails to Cast DateType to StringType when comparing res

[jira] [Resolved] (SPARK-32495) Update jackson-databind versions to fix various vulnerabilities.

2020-08-30 Thread Prashant Sharma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma resolved SPARK-32495. - Resolution: Won't Fix Resolving it as won't fix for now, as most of us feel the behaviou

[jira] [Updated] (SPARK-32744) request executor cores with decimal when spark on k8s

2020-08-30 Thread Yu Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Wang updated SPARK-32744: Attachment: screenshot-2.png > request executor cores with decimal when spark on k8s > ---

[jira] [Updated] (SPARK-32744) request executor cores with decimal when spark on k8s

2020-08-30 Thread Yu Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Wang updated SPARK-32744: Attachment: screenshot-3.png > request executor cores with decimal when spark on k8s > ---

[jira] [Commented] (SPARK-32744) request executor cores with decimal when spark on k8s

2020-08-30 Thread Yu Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187457#comment-17187457 ] Yu Wang commented on SPARK-32744: - !screenshot-2.png! [~hyukjin.kwon] Thank you for yo

[jira] [Commented] (SPARK-32138) Drop Python 2, 3.4 and 3.5 in codes and documentation

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187458#comment-17187458 ] Apache Spark commented on SPARK-32138: -- User 'zero323' has created a pull request f

[jira] [Assigned] (SPARK-32681) PySpark type hints support

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32681: Assignee: Apache Spark (was: Maciej Szymkiewicz) > PySpark type hints support >

[jira] [Commented] (SPARK-32681) PySpark type hints support

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187464#comment-17187464 ] Apache Spark commented on SPARK-32681: -- User 'zero323' has created a pull request f

[jira] [Assigned] (SPARK-32681) PySpark type hints support

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32681: Assignee: Maciej Szymkiewicz (was: Apache Spark) > PySpark type hints support >

[jira] [Created] (SPARK-32752) Alias breaks for interval typed literals

2020-08-30 Thread Kent Yao (Jira)
Kent Yao created SPARK-32752: Summary: Alias breaks for interval typed literals Key: SPARK-32752 URL: https://issues.apache.org/jira/browse/SPARK-32752 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-32747) Deduplicate configuration set/unset in test_sparkSQL_arrow.R

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32747: Assignee: Apache Spark > Deduplicate configuration set/unset in test_sparkSQL_arrow.R > -

[jira] [Commented] (SPARK-32747) Deduplicate configuration set/unset in test_sparkSQL_arrow.R

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187473#comment-17187473 ] Apache Spark commented on SPARK-32747: -- User 'HyukjinKwon' has created a pull reque

[jira] [Assigned] (SPARK-32747) Deduplicate configuration set/unset in test_sparkSQL_arrow.R

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32747: Assignee: (was: Apache Spark) > Deduplicate configuration set/unset in test_sparkSQL_

[jira] [Commented] (SPARK-32747) Deduplicate configuration set/unset in test_sparkSQL_arrow.R

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187475#comment-17187475 ] Apache Spark commented on SPARK-32747: -- User 'HyukjinKwon' has created a pull reque

[jira] [Commented] (SPARK-32691) Test org.apache.spark.DistributedSuite failed on arm64 jenkins

2020-08-30 Thread huangtianhua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187478#comment-17187478 ] huangtianhua commented on SPARK-32691: -- [~dongjoon] Seems it doesn't with 'with rep

[jira] [Issue Comment Deleted] (SPARK-32681) PySpark type hints support

2020-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-32681: - Comment: was deleted (was: User 'zero323' has created a pull request for this issue: https://git

[jira] [Commented] (SPARK-32714) Port pyspark-stubs

2020-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187485#comment-17187485 ] Hyukjin Kwon commented on SPARK-32714: -- https://github.com/apache/spark/pull/29591

[jira] [Assigned] (SPARK-32714) Port pyspark-stubs

2020-08-30 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-32714: Assignee: Maciej Szymkiewicz > Port pyspark-stubs > -- > >

[jira] [Commented] (SPARK-32714) Port pyspark-stubs

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187486#comment-17187486 ] Apache Spark commented on SPARK-32714: -- User 'zero323' has created a pull request f

[jira] [Assigned] (SPARK-32714) Port pyspark-stubs

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32714: Assignee: Maciej Szymkiewicz (was: Apache Spark) > Port pyspark-stubs >

[jira] [Assigned] (SPARK-32714) Port pyspark-stubs

2020-08-30 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32714: Assignee: Apache Spark (was: Maciej Szymkiewicz) > Port pyspark-stubs >

[jira] [Resolved] (SPARK-32740) Refactor common partitioning/distribution logic to BaseAggregateExec

2020-08-30 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-32740. -- Fix Version/s: 3.1.0 Assignee: Cheng Su Resolution: Fixed Resolved by 

[jira] [Created] (SPARK-32753) Deduplicating and repartitioning the same column create duplicate rows with AQE

2020-08-30 Thread Manu Zhang (Jira)
Manu Zhang created SPARK-32753: -- Summary: Deduplicating and repartitioning the same column create duplicate rows with AQE Key: SPARK-32753 URL: https://issues.apache.org/jira/browse/SPARK-32753 Project:

[jira] [Updated] (SPARK-32753) Deduplicating and repartitioning the same column create duplicate rows with AQE

2020-08-30 Thread Manu Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manu Zhang updated SPARK-32753: --- Description: To reproduce: {code:java} spark.range(10).union(spark.range(10)).createOrReplaceTempVie

[jira] [Commented] (SPARK-32750) Add code-gen for SortAggregateExec

2020-08-30 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17187495#comment-17187495 ] Takeshi Yamamuro commented on SPARK-32750: -- Just FYI: [https://github.com/apach

  1   2   >