[jira] [Resolved] (SPARK-31223) Update py code to generate data in testsuites

[ https://issues.apache.org/jira/browse/SPARK-31223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-31223. -- Resolution: Fixed > Update py code to generate data in testsuites >

[jira] [Assigned] (SPARK-31223) Update py code to generate data in testsuites

[ https://issues.apache.org/jira/browse/SPARK-31223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-31223: Assignee: Huaxin Gao (was: zhengruifeng) > Update py code to generate data in

[jira] [Assigned] (SPARK-31223) Update py code to generate data in testsuites

[ https://issues.apache.org/jira/browse/SPARK-31223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-31223: Assignee: zhengruifeng > Update py code to generate data in testsuites >

[jira] [Assigned] (SPARK-30623) Spark external shuffle allow disable of separate event loop group

[ https://issues.apache.org/jira/browse/SPARK-30623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30623: - Assignee: Yuanjian Li > Spark external shuffle allow disable of separate event loop

[jira] [Resolved] (SPARK-30623) Spark external shuffle allow disable of separate event loop group

[ https://issues.apache.org/jira/browse/SPARK-30623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30623. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27665

[jira] [Created] (SPARK-31264) Repartition by dynamic partition columns before insert table

Yuming Wang created SPARK-31264: --- Summary: Repartition by dynamic partition columns before insert table Key: SPARK-31264 URL: https://issues.apache.org/jira/browse/SPARK-31264 Project: Spark

[jira] [Updated] (SPARK-30623) Spark external shuffle allow disable of separate event loop group

[ https://issues.apache.org/jira/browse/SPARK-30623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30623: -- Affects Version/s: (was: 2.4.4) > Spark external shuffle allow disable of separate event

[jira] [Commented] (SPARK-30623) Spark external shuffle allow disable of separate event loop group

[ https://issues.apache.org/jira/browse/SPARK-30623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067359#comment-17067359 ] Dongjoon Hyun commented on SPARK-30623: --- Hi, [~tgraves]. Since SPARK-24355 is fixed at 3.0.0, I

[jira] [Created] (SPARK-31263) Enable yarn shuffle service close the idle connections

feiwang created SPARK-31263: --- Summary: Enable yarn shuffle service close the idle connections Key: SPARK-31263 URL: https://issues.apache.org/jira/browse/SPARK-31263 Project: Spark Issue Type:

[jira] [Created] (SPARK-31262) Test case import another test case contains bracketed comments, can't display bracketed comments in golden files well.

jiaan.geng created SPARK-31262: -- Summary: Test case import another test case contains bracketed comments, can't display bracketed comments in golden files well. Key: SPARK-31262 URL:

[jira] [Resolved] (SPARK-30923) Spark MLlib, GraphX 3.0 QA umbrella

[ https://issues.apache.org/jira/browse/SPARK-30923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-30923. -- Resolution: Fixed > Spark MLlib, GraphX 3.0 QA umbrella > ---

[jira] [Updated] (SPARK-31260) How to speed up WholeStageCodegen in Spark SQL Query?

[ https://issues.apache.org/jira/browse/SPARK-31260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] HongJin updated SPARK-31260: Description: It's took about 2mins for one 248 MB file. 2 files ~ 5 mins How can I tune or maximize the

[jira] [Created] (SPARK-31261) Avoid npe when reading bad csv input with `columnNameCorruptRecord` specified

Zhenhua Wang created SPARK-31261: Summary: Avoid npe when reading bad csv input with `columnNameCorruptRecord` specified Key: SPARK-31261 URL: https://issues.apache.org/jira/browse/SPARK-31261

[jira] [Created] (SPARK-31260) How to speed up WholeStageCodegen in Spark SQL Query?

HongJin created SPARK-31260: --- Summary: How to speed up WholeStageCodegen in Spark SQL Query? Key: SPARK-31260 URL: https://issues.apache.org/jira/browse/SPARK-31260 Project: Spark Issue Type:

[jira] [Commented] (SPARK-30466) remove dependency on jackson-mapper-asl-1.9.13 and jackson-core-asl-1.9.13

[ https://issues.apache.org/jira/browse/SPARK-30466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067300#comment-17067300 ] Nicholas Marion commented on SPARK-30466: - It is worth noting that the following dependencies

[jira] [Created] (SPARK-31259) Fix log error of curRequestSize in ShuffleBlockFetcherIterator

wuyi created SPARK-31259: Summary: Fix log error of curRequestSize in ShuffleBlockFetcherIterator Key: SPARK-31259 URL: https://issues.apache.org/jira/browse/SPARK-31259 Project: Spark Issue Type:

[jira] [Updated] (SPARK-31258) sbt unidoc fail to resolve arvo dependency

[ https://issues.apache.org/jira/browse/SPARK-31258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-31258: - Summary: sbt unidoc fail to resolve arvo dependency (was: sbt unidoc fail to resolving arvo

[jira] [Created] (SPARK-31258) sbt unidoc fail to resolving arvo dependency

Kent Yao created SPARK-31258: Summary: sbt unidoc fail to resolving arvo dependency Key: SPARK-31258 URL: https://issues.apache.org/jira/browse/SPARK-31258 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-30095) create function syntax has to be enhance in Doc for multiple dependent jars

[ https://issues.apache.org/jira/browse/SPARK-30095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067221#comment-17067221 ] Huaxin Gao commented on SPARK-30095: [~abhishek.akg] Any update on this? > create function syntax

[jira] [Created] (SPARK-31257) Fix ambiguous two different CREATE TABLE syntaxes

Jungtaek Lim created SPARK-31257: Summary: Fix ambiguous two different CREATE TABLE syntaxes Key: SPARK-31257 URL: https://issues.apache.org/jira/browse/SPARK-31257 Project: Spark Issue

[jira] [Created] (SPARK-31256) Dropna doesn't work for struct columns

Michael Souder created SPARK-31256: -- Summary: Dropna doesn't work for struct columns Key: SPARK-31256 URL: https://issues.apache.org/jira/browse/SPARK-31256 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-31255) DataSourceV2: Add metadata columns

[ https://issues.apache.org/jira/browse/SPARK-31255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated SPARK-31255: -- Issue Type: New Feature (was: Bug) > DataSourceV2: Add metadata columns >

[jira] [Created] (SPARK-31255) DataSourceV2: Add metadata columns

Ryan Blue created SPARK-31255: - Summary: DataSourceV2: Add metadata columns Key: SPARK-31255 URL: https://issues.apache.org/jira/browse/SPARK-31255 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-31254) `HiveResult.toHiveString` does not use the current session time zone

Maxim Gekk created SPARK-31254: -- Summary: `HiveResult.toHiveString` does not use the current session time zone Key: SPARK-31254 URL: https://issues.apache.org/jira/browse/SPARK-31254 Project: Spark

[jira] [Assigned] (SPARK-31244) Use Minio instead of Ceph in K8S DepsTestsSuite

[ https://issues.apache.org/jira/browse/SPARK-31244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-31244: - Assignee: Dongjoon Hyun > Use Minio instead of Ceph in K8S DepsTestsSuite >

[jira] [Resolved] (SPARK-31244) Use Minio instead of Ceph in K8S DepsTestsSuite

[ https://issues.apache.org/jira/browse/SPARK-31244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31244. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 28015

[jira] [Commented] (SPARK-31249) Flaky Test: CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied

[ https://issues.apache.org/jira/browse/SPARK-31249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066939#comment-17066939 ] Xingbo Jiang commented on SPARK-31249: -- I can't reproduce this failure, maybe the ExecutorAdded

[jira] [Updated] (SPARK-31246) GracefulShutdown does not work when application is terminated from RestSubmissionClient or YarnClient

[ https://issues.apache.org/jira/browse/SPARK-31246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-31246: Priority: Major (was: Blocker) > GracefulShutdown does not work when application is terminated from >

[jira] [Resolved] (SPARK-31133) fix sql ref doc for DML

[ https://issues.apache.org/jira/browse/SPARK-31133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31133. --- Fix Version/s: 3.0.0 Resolution: Fixed This is resolved via

[jira] [Created] (SPARK-31253) add metrics to shuffle reader

Wenchen Fan created SPARK-31253: --- Summary: add metrics to shuffle reader Key: SPARK-31253 URL: https://issues.apache.org/jira/browse/SPARK-31253 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-31181) Remove the default value assumption on CREATE TABLE test cases

[ https://issues.apache.org/jira/browse/SPARK-31181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31181: -- Parent: SPARK-31085 Issue Type: Sub-task (was: Improvement) > Remove the default

[jira] [Closed] (SPARK-31136) Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax

[ https://issues.apache.org/jira/browse/SPARK-31136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-31136. - > Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax >

[jira] [Updated] (SPARK-31147) forbid CHAR type in non-Hive-Serde tables

[ https://issues.apache.org/jira/browse/SPARK-31147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31147: -- Parent: SPARK-31085 Issue Type: Sub-task (was: Bug) > forbid CHAR type in

[jira] [Resolved] (SPARK-31147) forbid CHAR type in non-Hive-Serde tables

[ https://issues.apache.org/jira/browse/SPARK-31147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-31147. --- Fix Version/s: 3.1.0 Resolution: Fixed This is resolved via

[jira] [Created] (SPARK-31252) Flaky test: ElementTrackingStoreSuite.asynchronous tracking single-fire

Gabor Somogyi created SPARK-31252: - Summary: Flaky test: ElementTrackingStoreSuite.asynchronous tracking single-fire Key: SPARK-31252 URL: https://issues.apache.org/jira/browse/SPARK-31252 Project:

[jira] [Resolved] (SPARK-31196) Server-side processing of History UI list of applications

[ https://issues.apache.org/jira/browse/SPARK-31196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pavol Vidlička resolved SPARK-31196. Resolution: Won't Fix After looking into what would need to be changed to implement

[jira] [Created] (SPARK-31251) Flaky Test: StreamingContextSuite.stop gracefully

Hyukjin Kwon created SPARK-31251: Summary: Flaky Test: StreamingContextSuite.stop gracefully Key: SPARK-31251 URL: https://issues.apache.org/jira/browse/SPARK-31251 Project: Spark Issue

[jira] [Updated] (SPARK-31248) Flaky Test: ExecutorAllocationManagerSuite.interleaving add and remove

[ https://issues.apache.org/jira/browse/SPARK-31248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31248: - Description:

[jira] [Updated] (SPARK-31248) Flaky Test: ExecutorAllocationManagerSuite.interleaving add and remove

[ https://issues.apache.org/jira/browse/SPARK-31248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31248: - Summary: Flaky Test: ExecutorAllocationManagerSuite.interleaving add and remove (was: Flaky

[jira] [Created] (SPARK-31250) Flaky Test: KafkaDelegationTokenSuite.(It is not a test it is a sbt.testing.SuiteSelector)

Hyukjin Kwon created SPARK-31250: Summary: Flaky Test: KafkaDelegationTokenSuite.(It is not a test it is a sbt.testing.SuiteSelector) Key: SPARK-31250 URL: https://issues.apache.org/jira/browse/SPARK-31250

[jira] [Created] (SPARK-31249) Flaky Test: CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied

Hyukjin Kwon created SPARK-31249: Summary: Flaky Test: CoarseGrainedSchedulerBackendSuite.custom log url for Spark UI is applied Key: SPARK-31249 URL: https://issues.apache.org/jira/browse/SPARK-31249

[jira] [Created] (SPARK-31248) Flaky Test: org.apache.spark.ExecutorAllocationManagerSuite.interleaving add and remove

Hyukjin Kwon created SPARK-31248: Summary: Flaky Test: org.apache.spark.ExecutorAllocationManagerSuite.interleaving add and remove Key: SPARK-31248 URL: https://issues.apache.org/jira/browse/SPARK-31248

[jira] [Created] (SPARK-31247) Flaky test: KafkaContinuousSourceSuite.assign from latest offsets (failOnDataLoss: false)

Gabor Somogyi created SPARK-31247: - Summary: Flaky test: KafkaContinuousSourceSuite.assign from latest offsets (failOnDataLoss: false) Key: SPARK-31247 URL: https://issues.apache.org/jira/browse/SPARK-31247

[jira] [Updated] (SPARK-31228) Add version information to the configuration of Kafka

[ https://issues.apache.org/jira/browse/SPARK-31228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-31228: -- Description:

[jira] [Commented] (SPARK-26341) Expose executor memory metrics at the stage level, in the Stages tab

[ https://issues.apache.org/jira/browse/SPARK-26341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066572#comment-17066572 ] angerszhu commented on SPARK-26341: --- I have do this in our own version, will raise a pr soon these

[jira] [Created] (SPARK-31246) GracefulShutdown does not work when application is terminated from RestSubmissionClient or YarnClient

Gajanan Hebbar created SPARK-31246: -- Summary: GracefulShutdown does not work when application is terminated from RestSubmissionClient or YarnClient Key: SPARK-31246 URL:

[jira] [Resolved] (SPARK-31232) Specify formats of `spark.sql.session.timeZone`

[ https://issues.apache.org/jira/browse/SPARK-31232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-31232. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27999

[jira] [Assigned] (SPARK-31232) Specify formats of `spark.sql.session.timeZone`

[ https://issues.apache.org/jira/browse/SPARK-31232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-31232: --- Assignee: Maxim Gekk > Specify formats of `spark.sql.session.timeZone` >

[jira] [Comment Edited] (SPARK-31218) counts in BinaryClassificationMetrics should be cached

[ https://issues.apache.org/jira/browse/SPARK-31218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066474#comment-17066474 ] CacheCheck edited comment on SPARK-31218 at 3/25/20, 8:40 AM: -- I mean rdd

[jira] [Comment Edited] (SPARK-31218) counts in BinaryClassificationMetrics should be cached

[ https://issues.apache.org/jira/browse/SPARK-31218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066474#comment-17066474 ] CacheCheck edited comment on SPARK-31218 at 3/25/20, 8:37 AM: -- I mean rdd

[jira] [Commented] (SPARK-31218) counts in BinaryClassificationMetrics should be cached

[ https://issues.apache.org/jira/browse/SPARK-31218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066474#comment-17066474 ] CacheCheck commented on SPARK-31218: I mean rdd {{counts}} belong in the lazy val block below

[jira] [Assigned] (SPARK-30822) Pyspark queries fail if terminated with a semicolon

[ https://issues.apache.org/jira/browse/SPARK-30822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30822: --- Assignee: Samuel Setegne > Pyspark queries fail if terminated with a semicolon >

[jira] [Updated] (SPARK-31184) Support getTablesByType API of Hive Client

[ https://issues.apache.org/jira/browse/SPARK-31184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31184: -- Fix Version/s: (was: 3.1.0) 3.0.0 > Support getTablesByType API of

[jira] [Updated] (SPARK-31184) Support getTablesByType API of Hive Client

[ https://issues.apache.org/jira/browse/SPARK-31184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-31184: -- Affects Version/s: (was: 3.1.0) 3.0.0 > Support getTablesByType

[jira] [Resolved] (SPARK-30822) Pyspark queries fail if terminated with a semicolon

[ https://issues.apache.org/jira/browse/SPARK-30822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30822. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27567

[jira] [Commented] (SPARK-31238) Incompatible ORC dates with Spark 2.4

[ https://issues.apache.org/jira/browse/SPARK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066442#comment-17066442 ] Dongjoon Hyun commented on SPARK-31238: --- Thank you so much, [~maxgekk]! > Incompatible ORC dates

[jira] [Created] (SPARK-31245) Document the backward incompatibility issue for Encoders.kryo and Encoders.javaSerialization

Shixiong Zhu created SPARK-31245: Summary: Document the backward incompatibility issue for Encoders.kryo and Encoders.javaSerialization Key: SPARK-31245 URL: https://issues.apache.org/jira/browse/SPARK-31245

[jira] [Commented] (SPARK-31238) Incompatible ORC dates with Spark 2.4

[ https://issues.apache.org/jira/browse/SPARK-31238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066427#comment-17066427 ] Maxim Gekk commented on SPARK-31238: I am working on the issue. > Incompatible ORC dates with Spark

[jira] [Created] (SPARK-31244) Use Minio instead of Ceph in K8S DepsTestsSuite

Dongjoon Hyun created SPARK-31244: - Summary: Use Minio instead of Ceph in K8S DepsTestsSuite Key: SPARK-31244 URL: https://issues.apache.org/jira/browse/SPARK-31244 Project: Spark Issue

[jira] [Commented] (SPARK-31218) counts in BinaryClassificationMetrics should be cached

[ https://issues.apache.org/jira/browse/SPARK-31218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066407#comment-17066407 ] L. C. Hsieh commented on SPARK-31218: - Can you tell more about it? The only one place calling

[jira] [Updated] (SPARK-31243) add ANOVATest and FValueTest to PySpark

[ https://issues.apache.org/jira/browse/SPARK-31243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Huaxin Gao updated SPARK-31243: --- Parent: SPARK-26111 Issue Type: Sub-task (was: New Feature) > add ANOVATest and FValueTest

[jira] [Created] (SPARK-31243) add ANOVATest and FValueTest to PySpark

Huaxin Gao created SPARK-31243: -- Summary: add ANOVATest and FValueTest to PySpark Key: SPARK-31243 URL: https://issues.apache.org/jira/browse/SPARK-31243 Project: Spark Issue Type: New Feature