[jira] [Created] (SPARK-30200) Add ExplainMode for Dataset.explain

Takeshi Yamamuro created SPARK-30200: Summary: Add ExplainMode for Dataset.explain Key: SPARK-30200 URL: https://issues.apache.org/jira/browse/SPARK-30200 Project: Spark Issue Type: Impro

[jira] [Updated] (SPARK-30199) Recover spark.ui.port and spark.blockManager.port from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-30199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30199: -- Issue Type: Improvement (was: Bug) > Recover spark.ui.port and spark.blockManager.port from c

[jira] [Created] (SPARK-30199) Recover spark.ui.port and spark.blockManager.port from checkpoint

Dongjoon Hyun created SPARK-30199: - Summary: Recover spark.ui.port and spark.blockManager.port from checkpoint Key: SPARK-30199 URL: https://issues.apache.org/jira/browse/SPARK-30199 Project: Spark

[jira] [Created] (SPARK-30198) BytesToBytesMap does not grow internal long array as expected

L. C. Hsieh created SPARK-30198: --- Summary: BytesToBytesMap does not grow internal long array as expected Key: SPARK-30198 URL: https://issues.apache.org/jira/browse/SPARK-30198 Project: Spark

[jira] [Commented] (SPARK-30130) Hardcoded numeric values in common table expressions which utilize GROUP BY are interpreted as ordinal positions

[ https://issues.apache.org/jira/browse/SPARK-30130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992185#comment-16992185 ] Ankit Raj Boudh commented on SPARK-30130: - hi Matt boegner, can you please me to

[jira] [Comment Edited] (SPARK-30130) Hardcoded numeric values in common table expressions which utilize GROUP BY are interpreted as ordinal positions

[ https://issues.apache.org/jira/browse/SPARK-30130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16992185#comment-16992185 ] Ankit Raj Boudh edited comment on SPARK-30130 at 12/10/19 4:41 AM: ---

[jira] [Assigned] (SPARK-30196) Bump lz4-java version to 1.7.0

[ https://issues.apache.org/jira/browse/SPARK-30196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30196: Assignee: Takeshi Yamamuro > Bump lz4-java version to 1.7.0 > ---

[jira] [Resolved] (SPARK-30196) Bump lz4-java version to 1.7.0

[ https://issues.apache.org/jira/browse/SPARK-30196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30196. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26823 [https://gi

[jira] [Resolved] (SPARK-30193) Wrong 2nd argument type. Found: 'org.apache.spark.sql.Column', required: 'scala.collection.Seq'

[ https://issues.apache.org/jira/browse/SPARK-30193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30193. -- Resolution: Not A Problem > Wrong 2nd argument type. Found: 'org.apache.spark.sql.Column', req

[jira] [Resolved] (SPARK-30179) Improve test in SingleSessionSuite

[ https://issues.apache.org/jira/browse/SPARK-30179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30179. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26812 [https://gi

[jira] [Assigned] (SPARK-30179) Improve test in SingleSessionSuite

[ https://issues.apache.org/jira/browse/SPARK-30179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30179: Assignee: Yuming Wang > Improve test in SingleSessionSuite >

[jira] [Created] (SPARK-30197) Add `requirements.txt` file to `python` directory

Dongjoon Hyun created SPARK-30197: - Summary: Add `requirements.txt` file to `python` directory Key: SPARK-30197 URL: https://issues.apache.org/jira/browse/SPARK-30197 Project: Spark Issue Typ

[jira] [Created] (SPARK-30196) Bump lz4-java version to 1.7.0

Takeshi Yamamuro created SPARK-30196: Summary: Bump lz4-java version to 1.7.0 Key: SPARK-30196 URL: https://issues.apache.org/jira/browse/SPARK-30196 Project: Spark Issue Type: Improvemen

[jira] [Created] (SPARK-30195) Some imports, function need more explicit resolution in 2.13

Sean R. Owen created SPARK-30195: Summary: Some imports, function need more explicit resolution in 2.13 Key: SPARK-30195 URL: https://issues.apache.org/jira/browse/SPARK-30195 Project: Spark

[jira] [Resolved] (SPARK-30158) Resolve Array + reference type compile problems in 2.13, with sc.parallelize

[ https://issues.apache.org/jira/browse/SPARK-30158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30158. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26787 [https://gi

[jira] [Assigned] (SPARK-30146) add setWeightCol to GBTs in PySpark

[ https://issues.apache.org/jira/browse/SPARK-30146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-30146: Assignee: Huaxin Gao > add setWeightCol to GBTs in PySpark >

[jira] [Resolved] (SPARK-30146) add setWeightCol to GBTs in PySpark

[ https://issues.apache.org/jira/browse/SPARK-30146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30146. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26774 [https://gi

[jira] [Resolved] (SPARK-30180) listJars() and listFiles() function display issue when file path contains spaces

[ https://issues.apache.org/jira/browse/SPARK-30180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30180. -- Resolution: Not A Problem > listJars() and listFiles() function display issue when file path c

[jira] [Updated] (SPARK-30193) Wrong 2nd argument type. Found: 'org.apache.spark.sql.Column', required: 'scala.collection.Seq'

[ https://issues.apache.org/jira/browse/SPARK-30193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Raj Boudh updated SPARK-30193: Description:   [https://github.com/apache/spark/pull/26773/checks?check_run_id=340387020]

[jira] [Updated] (SPARK-30193) Wrong 2nd argument type. Found: 'org.apache.spark.sql.Column', required: 'scala.collection.Seq'

[ https://issues.apache.org/jira/browse/SPARK-30193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Raj Boudh updated SPARK-30193: Description:   [https://github.com/apache/spark/pull/26773/checks?check_run_id=340387020]

[jira] [Updated] (SPARK-30193) Wrong 2nd argument type. Found: 'org.apache.spark.sql.Column', required: 'scala.collection.Seq'

[ https://issues.apache.org/jira/browse/SPARK-30193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Raj Boudh updated SPARK-30193: Summary: Wrong 2nd argument type. Found: 'org.apache.spark.sql.Column', required: 'scala.c

[jira] [Updated] (SPARK-30108) Add robust accumulator for observable metrics

[ https://issues.apache.org/jira/browse/SPARK-30108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hövell updated SPARK-30108: -- Description: Spark accumulators reflect the work that has been done, and not the data

[jira] [Created] (SPARK-30194) Re-enable checkstyle for Java

Dongjoon Hyun created SPARK-30194: - Summary: Re-enable checkstyle for Java Key: SPARK-30194 URL: https://issues.apache.org/jira/browse/SPARK-30194 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-30193) remove unused import file

[ https://issues.apache.org/jira/browse/SPARK-30193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991756#comment-16991756 ] Ankit Raj Boudh commented on SPARK-30193: - i am working in this jira > remove u

[jira] [Updated] (SPARK-30193) remove unused import file

[ https://issues.apache.org/jira/browse/SPARK-30193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Raj Boudh updated SPARK-30193: Description: unused import file.   [https://github.com/apache/spark/pull/26773/checks?ch

[jira] [Created] (SPARK-30193) remove unused import file

Ankit Raj Boudh created SPARK-30193: --- Summary: remove unused import file Key: SPARK-30193 URL: https://issues.apache.org/jira/browse/SPARK-30193 Project: Spark Issue Type: Bug Com

[jira] [Commented] (SPARK-22876) spark.yarn.am.attemptFailuresValidityInterval does not work correctly

[ https://issues.apache.org/jira/browse/SPARK-22876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991752#comment-16991752 ] Andrew White commented on SPARK-22876: -- This issue is most certainly unresolved, an

[jira] [Resolved] (SPARK-27189) Add Executor metrics and memory usage instrumentation to the metrics system

[ https://issues.apache.org/jira/browse/SPARK-27189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-27189. -- Fix Version/s: 3.0.0 Resolution: Fixed Fixed by https://github.com/apache/spark/pull/24

[jira] [Assigned] (SPARK-27189) Add Executor metrics and memory usage instrumentation to the metrics system

[ https://issues.apache.org/jira/browse/SPARK-27189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-27189: Assignee: Luca Canali > Add Executor metrics and memory usage instrumentation to the metr

[jira] [Created] (SPARK-30192) support column position in DS v2

Wenchen Fan created SPARK-30192: --- Summary: support column position in DS v2 Key: SPARK-30192 URL: https://issues.apache.org/jira/browse/SPARK-30192 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-30191) AM should update pending resource request faster when driver lost executor

[ https://issues.apache.org/jira/browse/SPARK-30191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Max Xie updated SPARK-30191: - Description: I run spark on yarn.  I found that when driver lost its executors because of machine hardw

[jira] [Created] (SPARK-30191) AM should update pending resource request faster when driver lost executor

Max Xie created SPARK-30191: Summary: AM should update pending resource request faster when driver lost executor Key: SPARK-30191 URL: https://issues.apache.org/jira/browse/SPARK-30191 Project: Spark

[jira] [Created] (SPARK-30190) HistoryServerDiskManager will fail on appStoreDir in s3

thierry accart created SPARK-30190: -- Summary: HistoryServerDiskManager will fail on appStoreDir in s3 Key: SPARK-30190 URL: https://issues.apache.org/jira/browse/SPARK-30190 Project: Spark I

[jira] [Resolved] (SPARK-30159) Fix the method calls of QueryTest.checkAnswer

[ https://issues.apache.org/jira/browse/SPARK-30159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30159. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26788 [https://gi

[jira] [Commented] (SPARK-30162) Filter is not being pushed down for Parquet files

[ https://issues.apache.org/jira/browse/SPARK-30162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991573#comment-16991573 ] Nasir Ali commented on SPARK-30162: --- [~aman_omer] added in my question > Filter is no

[jira] [Updated] (SPARK-30162) Filter is not being pushed down for Parquet files

[ https://issues.apache.org/jira/browse/SPARK-30162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nasir Ali updated SPARK-30162: -- Description: Filters are not pushed down in Spark 3.0 preview. Also the output of "explain" method is

[jira] [Resolved] (SPARK-30145) sparkContext.addJar fails when file path contains spaces

[ https://issues.apache.org/jira/browse/SPARK-30145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30145. -- Resolution: Duplicate > sparkContext.addJar fails when file path contains spaces > ---

[jira] [Commented] (SPARK-30072) Create dedicated planner for subqueries

[ https://issues.apache.org/jira/browse/SPARK-30072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991554#comment-16991554 ] Xiaoju Wu commented on SPARK-30072: --- [~afroozeh] I think the change from checking if q

[jira] [Created] (SPARK-30189) Interval from year-month/date-time string handling whitespaces

Kent Yao created SPARK-30189: Summary: Interval from year-month/date-time string handling whitespaces Key: SPARK-30189 URL: https://issues.apache.org/jira/browse/SPARK-30189 Project: Spark Issue

[jira] [Created] (SPARK-30188) Enable Adaptive Query Execution default

Ke Jia created SPARK-30188: -- Summary: Enable Adaptive Query Execution default Key: SPARK-30188 URL: https://issues.apache.org/jira/browse/SPARK-30188 Project: Spark Issue Type: New Feature

[jira] [Created] (SPARK-30187) NULL handling in PySpark-PandasUDF

Abdeali Kothari created SPARK-30187: --- Summary: NULL handling in PySpark-PandasUDF Key: SPARK-30187 URL: https://issues.apache.org/jira/browse/SPARK-30187 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-30186) support Dynamic Partition Pruning in Adaptive Execution

Xiaoju Wu created SPARK-30186: - Summary: support Dynamic Partition Pruning in Adaptive Execution Key: SPARK-30186 URL: https://issues.apache.org/jira/browse/SPARK-30186 Project: Spark Issue Type:

[jira] [Updated] (SPARK-30180) listJars() and listFiles() function display issue when file path contains spaces

[ https://issues.apache.org/jira/browse/SPARK-30180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankit Raj Boudh updated SPARK-30180: Summary: listJars() and listFiles() function display issue when file path contains spaces

[jira] [Commented] (SPARK-26433) Tail method for spark DataFrame

[ https://issues.apache.org/jira/browse/SPARK-26433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991520#comment-16991520 ] Hyukjin Kwon commented on SPARK-26433: -- Looking back this JIRA, I realised that I u

[jira] [Reopened] (SPARK-26433) Tail method for spark DataFrame

[ https://issues.apache.org/jira/browse/SPARK-26433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-26433: -- > Tail method for spark DataFrame > --- > > Key: SPARK

[jira] [Resolved] (SPARK-26433) Tail method for spark DataFrame

[ https://issues.apache.org/jira/browse/SPARK-26433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26433. -- Resolution: Duplicate > Tail method for spark DataFrame > --- > >

[jira] [Resolved] (SPARK-30138) Separate configuration key of max iterations for analyzer and optimizer

[ https://issues.apache.org/jira/browse/SPARK-30138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-30138. -- Fix Version/s: 3.0.0 Assignee: Hu Fuwang Resolution: Fixed Resolved by

[jira] [Updated] (SPARK-30185) Implement Dataset.tail API

[ https://issues.apache.org/jira/browse/SPARK-30185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30185: - Description: I would like to propose an API called DataFrame.tail. *Background & Motivation* M

[jira] [Created] (SPARK-30185) Implement Dataset.tail API

Hyukjin Kwon created SPARK-30185: Summary: Implement Dataset.tail API Key: SPARK-30185 URL: https://issues.apache.org/jira/browse/SPARK-30185 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-30184) Implement a helper method for aliasing functions

[ https://issues.apache.org/jira/browse/SPARK-30184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Omer updated SPARK-30184: -- Summary: Implement a helper method for aliasing functions (was: Implement a helper method for aliasin

[jira] [Created] (SPARK-30184) Implement a helper method for aliasing

Aman Omer created SPARK-30184: - Summary: Implement a helper method for aliasing Key: SPARK-30184 URL: https://issues.apache.org/jira/browse/SPARK-30184 Project: Spark Issue Type: Improvement

[jira] [Issue Comment Deleted] (SPARK-30176) Eliminate warnings: part 6

[ https://issues.apache.org/jira/browse/SPARK-30176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh Raushan updated SPARK-30176: --- Comment: was deleted (was: i will work on this.) > Eliminate warnings: part 6 > ---

[jira] [Issue Comment Deleted] (SPARK-30175) Eliminate warnings: part 5

[ https://issues.apache.org/jira/browse/SPARK-30175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Katta updated SPARK-30175: -- Comment: was deleted (was: thanks for raising, will raise PR soon) > Eliminate warnings: part

[jira] [Commented] (SPARK-30108) Add robust accumulator for observable metrics

[ https://issues.apache.org/jira/browse/SPARK-30108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991381#comment-16991381 ] Herman van Hövell commented on SPARK-30108: --- [~Ankitraj] Great to hear! Here a

[jira] [Created] (SPARK-30183) Disallow to specify reserved properties in CREATE NAMESPACE syntax

Kent Yao created SPARK-30183: Summary: Disallow to specify reserved properties in CREATE NAMESPACE syntax Key: SPARK-30183 URL: https://issues.apache.org/jira/browse/SPARK-30183 Project: Spark

[jira] [Created] (SPARK-30182) Support nested aggregates

jiaan.geng created SPARK-30182: -- Summary: Support nested aggregates Key: SPARK-30182 URL: https://issues.apache.org/jira/browse/SPARK-30182 Project: Spark Issue Type: Sub-task Componen

[jira] [Created] (SPARK-30181) Throws runtime exception when filter metastore partition key that's not string type or integral types

Yu-Jhe Li created SPARK-30181: - Summary: Throws runtime exception when filter metastore partition key that's not string type or integral types Key: SPARK-30181 URL: https://issues.apache.org/jira/browse/SPARK-30181