[jira] [Resolved] (SPARK-30202) impl QuantileTransform

2020-04-19 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-30202. -- Resolution: Not A Problem > impl QuantileTransform > -- > >

[jira] [Resolved] (SPARK-31256) Dropna doesn't work for struct columns

2020-04-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31256. - Fix Version/s: 2.4.6 3.0.0 Assignee: Terry Kim Resolution:

[jira] [Created] (SPARK-31494) flatten the result dataframe of ANOVATest

2020-04-19 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-31494: Summary: flatten the result dataframe of ANOVATest Key: SPARK-31494 URL: https://issues.apache.org/jira/browse/SPARK-31494 Project: Spark Issue Type:

[jira] [Created] (SPARK-31493) Optimize InSet to In according partition size at InSubqueryExec

2020-04-19 Thread ulysses you (Jira)
ulysses you created SPARK-31493: --- Summary: Optimize InSet to In according partition size at InSubqueryExec Key: SPARK-31493 URL: https://issues.apache.org/jira/browse/SPARK-31493 Project: Spark

[jira] [Created] (SPARK-31492) flatten the result dataframe of FValueTest

2020-04-19 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-31492: Summary: flatten the result dataframe of FValueTest Key: SPARK-31492 URL: https://issues.apache.org/jira/browse/SPARK-31492 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-31433) Summarizer supports string arguments

2020-04-19 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-31433. -- Resolution: Not A Problem > Summarizer supports string arguments >

[jira] [Issue Comment Deleted] (SPARK-31256) Dropna doesn't work for struct columns

2020-04-19 Thread JinxinTang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JinxinTang updated SPARK-31256: --- Comment: was deleted (was: This feature may not be supported since spark 2.4.5,due to former bug

[jira] [Commented] (SPARK-26385) YARN - Spark Stateful Structured streaming HDFS_DELEGATION_TOKEN not found in cache

2020-04-19 Thread Zhou Jiashuai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087326#comment-17087326 ] Zhou Jiashuai commented on SPARK-26385: --- My spark version is spark-2.4.4-bin-hadoop2.7. > YARN -

[jira] [Commented] (SPARK-26385) YARN - Spark Stateful Structured streaming HDFS_DELEGATION_TOKEN not found in cache

2020-04-19 Thread Zhou Jiashuai (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087325#comment-17087325 ] Zhou Jiashuai commented on SPARK-26385: --- The log of AMCredentialRenewer is as follows: {code:java}

[jira] [Commented] (SPARK-31256) Dropna doesn't work for struct columns

2020-04-19 Thread JinxinTang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087306#comment-17087306 ] JinxinTang commented on SPARK-31256: This feature may not be supported since spark 2.4.5,due to

[jira] [Commented] (SPARK-31365) Enable nested predicate pushdown per data sources

2020-04-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087277#comment-17087277 ] Hyukjin Kwon commented on SPARK-31365: -- Hi [~kabhwan], have you made some progresses on this? >

[jira] [Updated] (SPARK-31365) Enable nested predicate pushdown per data sources

2020-04-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31365: - Priority: Blocker (was: Major) > Enable nested predicate pushdown per data sources >

[jira] [Updated] (SPARK-31365) Enable nested predicate pushdown per data sources

2020-04-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31365: - Target Version/s: 3.0.0 > Enable nested predicate pushdown per data sources >

[jira] [Updated] (SPARK-31234) ResetCommand should not wipe out all configs

2020-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31234: -- Fix Version/s: 2.4.6 > ResetCommand should not wipe out all configs >

[jira] [Commented] (SPARK-28367) Kafka connector infinite wait because metadata never updated

2020-04-19 Thread Ismael Juma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087242#comment-17087242 ] Ismael Juma commented on SPARK-28367: - This should be unblocked since Spark has upgraded to Kafka

[jira] [Commented] (SPARK-31464) Upgrade Kafka to 2.5.0

2020-04-19 Thread Ismael Juma (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087240#comment-17087240 ] Ismael Juma commented on SPARK-31464: - Thanks for upgrading so quickly. :) > Upgrade Kafka to 2.5.0

[jira] [Commented] (SPARK-31256) Dropna doesn't work for struct columns

2020-04-19 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087204#comment-17087204 ] Terry Kim commented on SPARK-31256: --- I created [https://github.com/apache/spark/pull/28266] to address

[jira] [Commented] (SPARK-31482) spark.kubernetes.driver.podTemplateFile Configuration not used by the job

2020-04-19 Thread Ed Mitchell (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087195#comment-17087195 ] Ed Mitchell commented on SPARK-31482: - Yeah. It's actually there but there's some unescaped HTML

[jira] [Commented] (SPARK-31482) spark.kubernetes.driver.podTemplateFile Configuration not used by the job

2020-04-19 Thread Pradeep Misra (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087167#comment-17087167 ] Pradeep Misra commented on SPARK-31482: --- Have created another bug for documentation - 

[jira] [Created] (SPARK-31491) Re-arrange Data Types page to document Floating Point Special Values

2020-04-19 Thread Huaxin Gao (Jira)
Huaxin Gao created SPARK-31491: -- Summary: Re-arrange Data Types page to document Floating Point Special Values Key: SPARK-31491 URL: https://issues.apache.org/jira/browse/SPARK-31491 Project: Spark

[jira] [Commented] (SPARK-31482) spark.kubernetes.driver.podTemplateFile Configuration not used by the job

2020-04-19 Thread Ed Mitchell (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087150#comment-17087150 ] Ed Mitchell commented on SPARK-31482: - I'm not sure where the bug is here TBH. I don't see anything

[jira] [Created] (SPARK-31490) Benchmark conversions to/from Java 8 date-time types

2020-04-19 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31490: -- Summary: Benchmark conversions to/from Java 8 date-time types Key: SPARK-31490 URL: https://issues.apache.org/jira/browse/SPARK-31490 Project: Spark Issue Type:

[jira] [Updated] (SPARK-31234) ResetCommand should not wipe out all configs

2020-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31234: -- Affects Version/s: 2.0.2 > ResetCommand should not wipe out all configs >

[jira] [Updated] (SPARK-31234) ResetCommand should not wipe out all configs

2020-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31234: -- Affects Version/s: 2.1.3 > ResetCommand should not wipe out all configs >

[jira] [Updated] (SPARK-31234) ResetCommand should not wipe out all configs

2020-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31234: -- Affects Version/s: 2.2.3 > ResetCommand should not wipe out all configs >

[jira] [Updated] (SPARK-31234) ResetCommand should not wipe out all configs

2020-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31234: -- Affects Version/s: 2.3.4 > ResetCommand should not wipe out all configs >

[jira] [Updated] (SPARK-31234) ResetCommand should not wipe out all configs

2020-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31234: -- Affects Version/s: 2.4.5 > ResetCommand should not wipe out all configs >

[jira] [Commented] (SPARK-31234) ResetCommand should not wipe out all configs

2020-04-19 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17087142#comment-17087142 ] Xiao Li commented on SPARK-31234: - I reopened the ticket and target it to Spark 2.4.6. > ResetCommand

[jira] [Updated] (SPARK-31234) ResetCommand should not wipe out all configs

2020-04-19 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-31234: Target Version/s: 2.4.6 > ResetCommand should not wipe out all configs >

[jira] [Updated] (SPARK-31234) ResetCommand should not wipe out all configs

2020-04-19 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-31234: Priority: Blocker (was: Major) > ResetCommand should not wipe out all configs >

[jira] [Reopened] (SPARK-31234) ResetCommand should not wipe out all configs

2020-04-19 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reopened SPARK-31234: - > ResetCommand should not wipe out all configs > > >

[jira] [Assigned] (SPARK-31464) Upgrade Kafka to 2.5.0

2020-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31464: - Assignee: Dongjoon Hyun > Upgrade Kafka to 2.5.0 > -- > >

[jira] [Resolved] (SPARK-31464) Upgrade Kafka to 2.5.0

2020-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31464. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28235

[jira] [Created] (SPARK-31489) Failure on pushing down filters with java.time.LocalDate values in ORC

2020-04-19 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31489: -- Summary: Failure on pushing down filters with java.time.LocalDate values in ORC Key: SPARK-31489 URL: https://issues.apache.org/jira/browse/SPARK-31489 Project: Spark

[jira] [Resolved] (SPARK-31484) Avoid file already existing exception when checkpointing

2020-04-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31484. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28255

[jira] [Updated] (SPARK-31488) Support `java.time.LocalDate` in Parquet filter pushdown

2020-04-19 Thread Maxim Gekk (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxim Gekk updated SPARK-31488: --- Description: Currently, ParquetFilters supports only java.sql.Date values of DateType, and

[jira] [Created] (SPARK-31488) Support `java.time.LocalDate` in Parquet filter pushdown

2020-04-19 Thread Maxim Gekk (Jira)
Maxim Gekk created SPARK-31488: -- Summary: Support `java.time.LocalDate` in Parquet filter pushdown Key: SPARK-31488 URL: https://issues.apache.org/jira/browse/SPARK-31488 Project: Spark Issue

[jira] [Created] (SPARK-31487) Move slots check of barrier job from DAGScheduler to TaskSchedulerImpl

2020-04-19 Thread wuyi (Jira)
wuyi created SPARK-31487: Summary: Move slots check of barrier job from DAGScheduler to TaskSchedulerImpl Key: SPARK-31487 URL: https://issues.apache.org/jira/browse/SPARK-31487 Project: Spark

[jira] [Created] (SPARK-31486) spark.submit.waitForCompletion flag to control spark-submit exit in Standalone Cluster Mode

2020-04-19 Thread Akshat Bordia (Jira)
Akshat Bordia created SPARK-31486: - Summary: spark.submit.waitForCompletion flag to control spark-submit exit in Standalone Cluster Mode Key: SPARK-31486 URL: https://issues.apache.org/jira/browse/SPARK-31486

[jira] [Created] (SPARK-31485) Barrier stage can hang if only partial tasks launched

2020-04-19 Thread wuyi (Jira)
wuyi created SPARK-31485: Summary: Barrier stage can hang if only partial tasks launched Key: SPARK-31485 URL: https://issues.apache.org/jira/browse/SPARK-31485 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-31375) Overwriting into dynamic partitions is appending data in pyspark

2020-04-19 Thread Sai Krishna Chaitanya Chaganti (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086877#comment-17086877 ] Sai Krishna Chaitanya Chaganti edited comment on SPARK-31375 at 4/19/20, 9:35 AM:

[jira] [Commented] (SPARK-31375) Overwriting into dynamic partitions is appending data in pyspark

2020-04-19 Thread Sai Krishna Chaitanya Chaganti (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086877#comment-17086877 ] Sai Krishna Chaitanya Chaganti commented on SPARK-31375: [~hyukjin.kwon] Hope