[jira] [Created] (SPARK-30200) Add ExplainMode for Dataset.explain

2019-12-09 Thread Takeshi Yamamuro (Jira)
Takeshi Yamamuro created SPARK-30200: Summary: Add ExplainMode for Dataset.explain Key: SPARK-30200 URL: https://issues.apache.org/jira/browse/SPARK-30200 Project: Spark Issue Type:

[jira] [Updated] (SPARK-30199) Recover spark.ui.port and spark.blockManager.port from checkpoint

2019-12-09 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30199: -- Issue Type: Improvement (was: Bug) > Recover spark.ui.port and spark.blockManager.port from

[jira] [Created] (SPARK-30199) Recover spark.ui.port and spark.blockManager.port from checkpoint

2019-12-09 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-30199: - Summary: Recover spark.ui.port and spark.blockManager.port from checkpoint Key: SPARK-30199 URL: https://issues.apache.org/jira/browse/SPARK-30199 Project: Spark

[jira] [Created] (SPARK-30198) BytesToBytesMap does not grow internal long array as expected

2019-12-09 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-30198: --- Summary: BytesToBytesMap does not grow internal long array as expected Key: SPARK-30198 URL: https://issues.apache.org/jira/browse/SPARK-30198 Project: Spark

[jira] [Commented] (SPARK-30130) Hardcoded numeric values in common table expressions which utilize GROUP BY are interpreted as ordinal positions

2019-12-09 Thread Ankit Raj Boudh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992185#comment-16992185 ] Ankit Raj Boudh commented on SPARK-30130: - hi Matt boegner, can you please me to reproduce this

[jira] [Comment Edited] (SPARK-30130) Hardcoded numeric values in common table expressions which utilize GROUP BY are interpreted as ordinal positions

2019-12-09 Thread Ankit Raj Boudh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992185#comment-16992185 ] Ankit Raj Boudh edited comment on SPARK-30130 at 12/10/19 4:41 AM: --- hi

[jira] [Assigned] (SPARK-30196) Bump lz4-java version to 1.7.0

2019-12-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30196: Assignee: Takeshi Yamamuro > Bump lz4-java version to 1.7.0 >

[jira] [Resolved] (SPARK-30196) Bump lz4-java version to 1.7.0

2019-12-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30196. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26823

[jira] [Resolved] (SPARK-30193) Wrong 2nd argument type. Found: 'org.apache.spark.sql.Column', required: 'scala.collection.Seq'

2019-12-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30193. -- Resolution: Not A Problem > Wrong 2nd argument type. Found: 'org.apache.spark.sql.Column',

[jira] [Resolved] (SPARK-30179) Improve test in SingleSessionSuite

2019-12-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30179. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26812

[jira] [Assigned] (SPARK-30179) Improve test in SingleSessionSuite

2019-12-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30179: Assignee: Yuming Wang > Improve test in SingleSessionSuite >

[jira] [Created] (SPARK-30197) Add `requirements.txt` file to `python` directory

2019-12-09 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-30197: - Summary: Add `requirements.txt` file to `python` directory Key: SPARK-30197 URL: https://issues.apache.org/jira/browse/SPARK-30197 Project: Spark Issue

[jira] [Created] (SPARK-30196) Bump lz4-java version to 1.7.0

2019-12-09 Thread Takeshi Yamamuro (Jira)
Takeshi Yamamuro created SPARK-30196: Summary: Bump lz4-java version to 1.7.0 Key: SPARK-30196 URL: https://issues.apache.org/jira/browse/SPARK-30196 Project: Spark Issue Type:

[jira] [Created] (SPARK-30195) Some imports, function need more explicit resolution in 2.13

2019-12-09 Thread Sean R. Owen (Jira)
Sean R. Owen created SPARK-30195: Summary: Some imports, function need more explicit resolution in 2.13 Key: SPARK-30195 URL: https://issues.apache.org/jira/browse/SPARK-30195 Project: Spark

[jira] [Resolved] (SPARK-30158) Resolve Array + reference type compile problems in 2.13, with sc.parallelize

2019-12-09 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30158. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26787

[jira] [Assigned] (SPARK-30146) add setWeightCol to GBTs in PySpark

2019-12-09 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-30146: Assignee: Huaxin Gao > add setWeightCol to GBTs in PySpark >

[jira] [Resolved] (SPARK-30146) add setWeightCol to GBTs in PySpark

2019-12-09 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30146. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26774

[jira] [Resolved] (SPARK-30180) listJars() and listFiles() function display issue when file path contains spaces

2019-12-09 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30180. -- Resolution: Not A Problem > listJars() and listFiles() function display issue when file path

[jira] [Updated] (SPARK-30193) Wrong 2nd argument type. Found: 'org.apache.spark.sql.Column', required: 'scala.collection.Seq'

2019-12-09 Thread Ankit Raj Boudh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Raj Boudh updated SPARK-30193: Description:  

[jira] [Updated] (SPARK-30193) Wrong 2nd argument type. Found: 'org.apache.spark.sql.Column', required: 'scala.collection.Seq'

2019-12-09 Thread Ankit Raj Boudh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Raj Boudh updated SPARK-30193: Description:  

[jira] [Updated] (SPARK-30193) Wrong 2nd argument type. Found: 'org.apache.spark.sql.Column', required: 'scala.collection.Seq'

2019-12-09 Thread Ankit Raj Boudh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Raj Boudh updated SPARK-30193: Summary: Wrong 2nd argument type. Found: 'org.apache.spark.sql.Column', required:

[jira] [Updated] (SPARK-30108) Add robust accumulator for observable metrics

2019-12-09 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-30108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell updated SPARK-30108: -- Description: Spark accumulators reflect the work that has been done, and not the data

[jira] [Created] (SPARK-30194) Re-enable checkstyle for Java

2019-12-09 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-30194: - Summary: Re-enable checkstyle for Java Key: SPARK-30194 URL: https://issues.apache.org/jira/browse/SPARK-30194 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-30193) remove unused import file

2019-12-09 Thread Ankit Raj Boudh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991756#comment-16991756 ] Ankit Raj Boudh commented on SPARK-30193: - i am working in this jira > remove unused import

[jira] [Updated] (SPARK-30193) remove unused import file

2019-12-09 Thread Ankit Raj Boudh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Raj Boudh updated SPARK-30193: Description: unused import file.  

[jira] [Created] (SPARK-30193) remove unused import file

2019-12-09 Thread Ankit Raj Boudh (Jira)
Ankit Raj Boudh created SPARK-30193: --- Summary: remove unused import file Key: SPARK-30193 URL: https://issues.apache.org/jira/browse/SPARK-30193 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-22876) spark.yarn.am.attemptFailuresValidityInterval does not work correctly

2019-12-09 Thread Andrew White (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991752#comment-16991752 ] Andrew White commented on SPARK-22876: -- This issue is most certainly unresolved, and the

[jira] [Resolved] (SPARK-27189) Add Executor metrics and memory usage instrumentation to the metrics system

2019-12-09 Thread Imran Rashid (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-27189. -- Fix Version/s: 3.0.0 Resolution: Fixed Fixed by

[jira] [Assigned] (SPARK-27189) Add Executor metrics and memory usage instrumentation to the metrics system

2019-12-09 Thread Imran Rashid (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-27189: Assignee: Luca Canali > Add Executor metrics and memory usage instrumentation to the

[jira] [Created] (SPARK-30192) support column position in DS v2

2019-12-09 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-30192: --- Summary: support column position in DS v2 Key: SPARK-30192 URL: https://issues.apache.org/jira/browse/SPARK-30192 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-30191) AM should update pending resource request faster when driver lost executor

2019-12-09 Thread Max Xie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Xie updated SPARK-30191: - Description: I run spark on yarn.  I found that when driver lost its executors because of machine

[jira] [Created] (SPARK-30191) AM should update pending resource request faster when driver lost executor

2019-12-09 Thread Max Xie (Jira)
Max Xie created SPARK-30191: Summary: AM should update pending resource request faster when driver lost executor Key: SPARK-30191 URL: https://issues.apache.org/jira/browse/SPARK-30191 Project: Spark

[jira] [Created] (SPARK-30190) HistoryServerDiskManager will fail on appStoreDir in s3

2019-12-09 Thread thierry accart (Jira)
thierry accart created SPARK-30190: -- Summary: HistoryServerDiskManager will fail on appStoreDir in s3 Key: SPARK-30190 URL: https://issues.apache.org/jira/browse/SPARK-30190 Project: Spark

[jira] [Resolved] (SPARK-30159) Fix the method calls of QueryTest.checkAnswer

2019-12-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30159. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26788

[jira] [Commented] (SPARK-30162) Filter is not being pushed down for Parquet files

2019-12-09 Thread Nasir Ali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991573#comment-16991573 ] Nasir Ali commented on SPARK-30162: --- [~aman_omer] added in my question > Filter is not being pushed

[jira] [Updated] (SPARK-30162) Filter is not being pushed down for Parquet files

2019-12-09 Thread Nasir Ali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nasir Ali updated SPARK-30162: -- Description: Filters are not pushed down in Spark 3.0 preview. Also the output of "explain" method

[jira] [Resolved] (SPARK-30145) sparkContext.addJar fails when file path contains spaces

2019-12-09 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30145. -- Resolution: Duplicate > sparkContext.addJar fails when file path contains spaces >

[jira] [Commented] (SPARK-30072) Create dedicated planner for subqueries

2019-12-09 Thread Xiaoju Wu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991554#comment-16991554 ] Xiaoju Wu commented on SPARK-30072: --- [~afroozeh] I think the change from checking if queryExecution

[jira] [Created] (SPARK-30189) Interval from year-month/date-time string handling whitespaces

2019-12-09 Thread Kent Yao (Jira)
Kent Yao created SPARK-30189: Summary: Interval from year-month/date-time string handling whitespaces Key: SPARK-30189 URL: https://issues.apache.org/jira/browse/SPARK-30189 Project: Spark

[jira] [Created] (SPARK-30188) Enable Adaptive Query Execution default

2019-12-09 Thread Ke Jia (Jira)
Ke Jia created SPARK-30188: -- Summary: Enable Adaptive Query Execution default Key: SPARK-30188 URL: https://issues.apache.org/jira/browse/SPARK-30188 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-30187) NULL handling in PySpark-PandasUDF

2019-12-09 Thread Abdeali Kothari (Jira)
Abdeali Kothari created SPARK-30187: --- Summary: NULL handling in PySpark-PandasUDF Key: SPARK-30187 URL: https://issues.apache.org/jira/browse/SPARK-30187 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-30186) support Dynamic Partition Pruning in Adaptive Execution

2019-12-09 Thread Xiaoju Wu (Jira)
Xiaoju Wu created SPARK-30186: - Summary: support Dynamic Partition Pruning in Adaptive Execution Key: SPARK-30186 URL: https://issues.apache.org/jira/browse/SPARK-30186 Project: Spark Issue

[jira] [Updated] (SPARK-30180) listJars() and listFiles() function display issue when file path contains spaces

2019-12-09 Thread Ankit Raj Boudh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Raj Boudh updated SPARK-30180: Summary: listJars() and listFiles() function display issue when file path contains spaces

[jira] [Commented] (SPARK-26433) Tail method for spark DataFrame

2019-12-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991520#comment-16991520 ] Hyukjin Kwon commented on SPARK-26433: -- Looking back this JIRA, I realised that I underestimated it

[jira] [Reopened] (SPARK-26433) Tail method for spark DataFrame

2019-12-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-26433: -- > Tail method for spark DataFrame > --- > > Key:

[jira] [Resolved] (SPARK-26433) Tail method for spark DataFrame

2019-12-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26433. -- Resolution: Duplicate > Tail method for spark DataFrame > --- > >

[jira] [Resolved] (SPARK-30138) Separate configuration key of max iterations for analyzer and optimizer

2019-12-09 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-30138. -- Fix Version/s: 3.0.0 Assignee: Hu Fuwang Resolution: Fixed Resolved

[jira] [Updated] (SPARK-30185) Implement Dataset.tail API

2019-12-09 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30185: - Description: I would like to propose an API called DataFrame.tail. *Background & Motivation*

[jira] [Created] (SPARK-30185) Implement Dataset.tail API

2019-12-09 Thread Hyukjin Kwon (Jira)
Hyukjin Kwon created SPARK-30185: Summary: Implement Dataset.tail API Key: SPARK-30185 URL: https://issues.apache.org/jira/browse/SPARK-30185 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-30184) Implement a helper method for aliasing functions

2019-12-09 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Omer updated SPARK-30184: -- Summary: Implement a helper method for aliasing functions (was: Implement a helper method for

[jira] [Created] (SPARK-30184) Implement a helper method for aliasing

2019-12-09 Thread Aman Omer (Jira)
Aman Omer created SPARK-30184: - Summary: Implement a helper method for aliasing Key: SPARK-30184 URL: https://issues.apache.org/jira/browse/SPARK-30184 Project: Spark Issue Type: Improvement

[jira] [Issue Comment Deleted] (SPARK-30176) Eliminate warnings: part 6

2019-12-09 Thread Rakesh Raushan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh Raushan updated SPARK-30176: --- Comment: was deleted (was: i will work on this.) > Eliminate warnings: part 6 >

[jira] [Issue Comment Deleted] (SPARK-30175) Eliminate warnings: part 5

2019-12-09 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Katta updated SPARK-30175: -- Comment: was deleted (was: thanks for raising, will raise PR soon) > Eliminate warnings:

[jira] [Commented] (SPARK-30108) Add robust accumulator for observable metrics

2019-12-09 Thread Jira
[ https://issues.apache.org/jira/browse/SPARK-30108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991381#comment-16991381 ] Herman van Hövell commented on SPARK-30108: --- [~Ankitraj] Great to hear! Here are some

[jira] [Created] (SPARK-30183) Disallow to specify reserved properties in CREATE NAMESPACE syntax

2019-12-09 Thread Kent Yao (Jira)
Kent Yao created SPARK-30183: Summary: Disallow to specify reserved properties in CREATE NAMESPACE syntax Key: SPARK-30183 URL: https://issues.apache.org/jira/browse/SPARK-30183 Project: Spark

[jira] [Created] (SPARK-30182) Support nested aggregates

2019-12-09 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-30182: -- Summary: Support nested aggregates Key: SPARK-30182 URL: https://issues.apache.org/jira/browse/SPARK-30182 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-30181) Throws runtime exception when filter metastore partition key that's not string type or integral types

2019-12-09 Thread Yu-Jhe Li (Jira)
Yu-Jhe Li created SPARK-30181: - Summary: Throws runtime exception when filter metastore partition key that's not string type or integral types Key: SPARK-30181 URL: https://issues.apache.org/jira/browse/SPARK-30181