[jira] [Resolved] (SPARK-29991) Support `hive-1.2/2.3` in PR Builder

2019-11-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-29991. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26695

[jira] [Updated] (SPARK-29991) Support `test-hive1.2` in PR Builder

2019-11-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29991: -- Summary: Support `test-hive1.2` in PR Builder (was: Support `hive-1.2/2.3` in PR Builder) >

[jira] [Updated] (SPARK-30022) Supporting Parsing of Simple Hive Virtual View created from Presto

2019-11-27 Thread Arun Ravi M V (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Ravi M V updated SPARK-30022: -- Description:   We have an environment were we use Apache Spark and Presto (both backed by

[jira] [Updated] (SPARK-30022) Supporting Parsing of Simple Hive Virtual View created from Presto

2019-11-27 Thread Arun Ravi M V (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Ravi M V updated SPARK-30022: -- Description:   We have an environment were we use Apache Spark and Presto (both backed by

[jira] [Updated] (SPARK-30022) Supporting Parsing of Simple Hive Virtual View created from Presto

2019-11-27 Thread Arun Ravi M V (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Ravi M V updated SPARK-30022: -- Description:   We have an environment were we use Apache Spark and Presto (both backed by

[jira] [Updated] (SPARK-30022) Supporting Parsing of Simple Hive Virtual View created from Presto

2019-11-27 Thread Arun Ravi M V (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Ravi M V updated SPARK-30022: -- Description:   We have an environment were we use Apache Spark and Presto (both backed by

[jira] [Updated] (SPARK-30022) Supporting Parsing of Simple Hive Virtual View created from Presto

2019-11-27 Thread Arun Ravi M V (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Ravi M V updated SPARK-30022: -- Summary: Supporting Parsing of Simple Hive Virtual View created from Presto (was: Supporting

[jira] [Updated] (SPARK-30022) Supporting reading of Hive Virtual View created from Presto

2019-11-27 Thread Arun Ravi M V (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Ravi M V updated SPARK-30022: -- Summary: Supporting reading of Hive Virtual View created from Presto (was: Supporting

[jira] [Created] (SPARK-30066) Columnar execution support for interval types

2019-11-27 Thread Kent Yao (Jira)
Kent Yao created SPARK-30066: Summary: Columnar execution support for interval types Key: SPARK-30066 URL: https://issues.apache.org/jira/browse/SPARK-30066 Project: Spark Issue Type:

[jira] [Commented] (SPARK-29836) PostgreSQL dialect: cast

2019-11-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984098#comment-16984098 ] Dongjoon Hyun commented on SPARK-29836: --- According to the dev mailing list, it seems that we are

[jira] [Assigned] (SPARK-30034) Use Apache Hive 2.3 dependency by default

2019-11-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30034: - Assignee: Dongjoon Hyun > Use Apache Hive 2.3 dependency by default >

[jira] [Created] (SPARK-30065) Unable to drop na with duplicate columns

2019-11-27 Thread Terry Kim (Jira)
Terry Kim created SPARK-30065: - Summary: Unable to drop na with duplicate columns Key: SPARK-30065 URL: https://issues.apache.org/jira/browse/SPARK-30065 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-30065) Unable to drop na with duplicate columns

2019-11-27 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984044#comment-16984044 ] Terry Kim commented on SPARK-30065: --- I am working on this. > Unable to drop na with duplicate columns

[jira] [Resolved] (SPARK-29989) Update release-script for `hive-1.2/2.3` combination

2019-11-27 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-29989. - Fix Version/s: 3.0.0 Resolution: Fixed > Update release-script for `hive-1.2/2.3` combination >

[jira] [Commented] (SPARK-30064) Incomplete Graphical SQL Plan in Spark application UI's SQL tab for SaveIntoDataSourceCommand

2019-11-27 Thread Reza Safi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984030#comment-16984030 ] Reza Safi commented on SPARK-30064: --- I am working on a fix for this and hopefully will send a PR

[jira] [Commented] (SPARK-30064) Incomplete Graphical SQL Plan in Spark application UI's SQL tab for SaveIntoDataSourceCommand

2019-11-27 Thread Reza Safi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16984016#comment-16984016 ] Reza Safi commented on SPARK-30064: --- I also attached the incorrect query plan. > Incomplete

[jira] [Updated] (SPARK-30064) Incomplete Graphical SQL Plan in Spark application UI's SQL tab for SaveIntoDataSourceCommand

2019-11-27 Thread Reza Safi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reza Safi updated SPARK-30064: -- Attachment: Screen Shot 2019-11-21 at 3.42.16 PM.png > Incomplete Graphical SQL Plan in Spark

[jira] [Created] (SPARK-30064) Incomplete Graphical SQL Plan in Spark application UI's SQL tab for SaveIntoDataSourceCommand

2019-11-27 Thread Reza Safi (Jira)
Reza Safi created SPARK-30064: - Summary: Incomplete Graphical SQL Plan in Spark application UI's SQL tab for SaveIntoDataSourceCommand Key: SPARK-30064 URL: https://issues.apache.org/jira/browse/SPARK-30064

[jira] [Updated] (SPARK-30063) Failure when returning a value from multiple Pandas UDFs

2019-11-27 Thread Tim Kellogg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Kellogg updated SPARK-30063: Description: I have 20 Pandas UDFs that I'm trying to evaluate all at the same time. *

[jira] [Updated] (SPARK-30063) Failure when returning a value from multiple Pandas UDFs

2019-11-27 Thread Tim Kellogg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Kellogg updated SPARK-30063: Description: I have 20 Pandas UDFs that I'm trying to evaluate all at the same time. *

[jira] [Updated] (SPARK-30063) Failure when returning a value from multiple Pandas UDFs

2019-11-27 Thread Tim Kellogg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Kellogg updated SPARK-30063: Description: I have 20 Pandas UDFs that I'm trying to evaluate all at the same time. *

[jira] [Assigned] (SPARK-29991) Support `hive-1.2/2.3` in PR Builder

2019-11-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-29991: - Assignee: Dongjoon Hyun > Support `hive-1.2/2.3` in PR Builder >

[jira] [Updated] (SPARK-30063) Failure when returning a value from multiple Pandas UDFs

2019-11-27 Thread Tim Kellogg (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Kellogg updated SPARK-30063: Attachment: spark-debug.txt > Failure when returning a value from multiple Pandas UDFs >

[jira] [Created] (SPARK-30063) Failure when returning a value from multiple Pandas UDFs

2019-11-27 Thread Tim Kellogg (Jira)
Tim Kellogg created SPARK-30063: --- Summary: Failure when returning a value from multiple Pandas UDFs Key: SPARK-30063 URL: https://issues.apache.org/jira/browse/SPARK-30063 Project: Spark Issue

[jira] [Reopened] (SPARK-28461) Pad Decimal numbers with trailing zeros to the scale of the column

2019-11-27 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-28461: --- > Pad Decimal numbers with trailing zeros to the scale of the column >

[jira] [Commented] (SPARK-29903) Add documentation for recursiveFileLookup

2019-11-27 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983833#comment-16983833 ] Xiao Li commented on SPARK-29903: - We also need to update the corresponding parts in PySpark  > Add

[jira] [Issue Comment Deleted] (SPARK-29367) pandas udf not working with latest pyarrow release (0.15.0)

2019-11-27 Thread Luis Blanche (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luis Blanche updated SPARK-29367: - Comment: was deleted (was: Hello,  I have tried the fix proposed by [~bryanc] and the issue

[jira] [Commented] (SPARK-27025) Speed up toLocalIterator

2019-11-27 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983646#comment-16983646 ] Sean R. Owen commented on SPARK-27025: -- It's a duplicate JIRA, so I closed it. It's always possible

[jira] [Resolved] (SPARK-29852) Implement parallel preemptive RDD.toLocalIterator and Dataset.toLocalIterator

2019-11-27 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29852. -- Resolution: Duplicate > Implement parallel preemptive RDD.toLocalIterator and

[jira] [Updated] (SPARK-30057) Add a statement of platforms that Spark runs on

2019-11-27 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-30057: - Priority: Trivial (was: Major) This should not be marked "Major" > Add a statement of

[jira] [Commented] (SPARK-27025) Speed up toLocalIterator

2019-11-27 Thread Peng Cheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983635#comment-16983635 ] Peng Cheng commented on SPARK-27025: Am I too late for this issue? I submitted SPARK-29852. Do you

[jira] [Commented] (SPARK-14203) TPC-DS Query 86 - When there is a rollup , the parser keels over on grouping_id() on a single column

2019-11-27 Thread sdhalex (Jira)
[ https://issues.apache.org/jira/browse/SPARK-14203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983596#comment-16983596 ] sdhalex commented on SPARK-14203: - Do this issue only happen in spark 1.6? I mean if you don`t take any

[jira] [Commented] (SPARK-29836) PostgreSQL dialect: cast

2019-11-27 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983583#comment-16983583 ] Sean R. Owen commented on SPARK-29836: -- I don't know enough about this to comment, other than to

[jira] [Commented] (SPARK-29367) pandas udf not working with latest pyarrow release (0.15.0)

2019-11-27 Thread Luis Blanche (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983578#comment-16983578 ] Luis Blanche commented on SPARK-29367: -- Hello,  I have tried the fix proposed by [~bryanc] and the

[jira] [Commented] (SPARK-27780) Shuffle server & client should be versioned to enable smoother upgrade

2019-11-27 Thread deshanxiao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983536#comment-16983536 ] deshanxiao commented on SPARK-27780: I can't argee it more. Add shuffle service version is very

[jira] [Created] (SPARK-30062) bug with DB2Driver using mode("overwrite") option("truncate",True)

2019-11-27 Thread Guy Huinen (Jira)
Guy Huinen created SPARK-30062: -- Summary: bug with DB2Driver using mode("overwrite") option("truncate",True) Key: SPARK-30062 URL: https://issues.apache.org/jira/browse/SPARK-30062 Project: Spark

[jira] [Created] (SPARK-30061) BufferedWriter should be closed

2019-11-27 Thread xzh_dz (Jira)
xzh_dz created SPARK-30061: -- Summary: BufferedWriter should be closed Key: SPARK-30061 URL: https://issues.apache.org/jira/browse/SPARK-30061 Project: Spark Issue Type: Wish Components:

[jira] [Created] (SPARK-30060) Uniform naming for Spark Metrics configuration parameters

2019-11-27 Thread Luca Canali (Jira)
Luca Canali created SPARK-30060: --- Summary: Uniform naming for Spark Metrics configuration parameters Key: SPARK-30060 URL: https://issues.apache.org/jira/browse/SPARK-30060 Project: Spark

[jira] [Updated] (SPARK-30060) Uniform naming for Spark Metrics configuration parameters

2019-11-27 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-30060: Description: Currently Spark has a few parameters to enable/disable metrics reporting. Their

[jira] [Commented] (SPARK-29793) Display plan evolve history in AQE

2019-11-27 Thread Manu Zhang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983339#comment-16983339 ] Manu Zhang commented on SPARK-29793: [~maryannxue], Is the {{AdaptiveSparkPlan(isFinal=true)}}

[jira] [Resolved] (SPARK-30056) Skip building test artifacts in `dev/make-distribution.sh`

2019-11-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30056. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26689

[jira] [Resolved] (SPARK-28461) Pad Decimal numbers with trailing zeros to the scale of the column

2019-11-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28461. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25214

[jira] [Assigned] (SPARK-28461) Pad Decimal numbers with trailing zeros to the scale of the column

2019-11-27 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-28461: Assignee: Yuming Wang > Pad Decimal numbers with trailing zeros to the scale of the

[jira] [Resolved] (SPARK-29768) nondeterministic expression fails column pruning

2019-11-27 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29768. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26629