[jira] [Assigned] (SPARK-27250) Scala 2.11 maven compile should target Java 1.8

[ https://issues.apache.org/jira/browse/SPARK-27250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-27250: - Assignee: John Zhuge > Scala 2.11 maven compile should target Java 1.8 > --

[jira] [Resolved] (SPARK-27241) Add map_keys and map_values support to SelectedField in nested schema pruning

[ https://issues.apache.org/jira/browse/SPARK-27241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27241. --- Resolution: Fixed Fix Version/s: 3.0.0 This is resolved via https://github.com/apache

[jira] [Assigned] (SPARK-27241) Add map_keys and map_values support to SelectedField in nested schema pruning

[ https://issues.apache.org/jira/browse/SPARK-27241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-27241: - Assignee: Liang-Chi Hsieh > Add map_keys and map_values support to SelectedField in nes

[jira] [Resolved] (SPARK-27251) @volatile var cannot be defined in case class in Scala 2.11

[ https://issues.apache.org/jira/browse/SPARK-27251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27251. --- Resolution: Fixed Fix Version/s: 3.0.0 This is resolved via https://github.com/apache

[jira] [Updated] (SPARK-27251) @volatile var cannot be defined in case class in Scala 2.11

[ https://issues.apache.org/jira/browse/SPARK-27251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27251: -- Component/s: Build > @volatile var cannot be defined in case class in Scala 2.11 > ---

[jira] [Assigned] (SPARK-27251) @volatile var cannot be defined in case class in Scala 2.11

[ https://issues.apache.org/jira/browse/SPARK-27251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-27251: - Assignee: Takeshi Yamamuro > @volatile var cannot be defined in case class in Scala 2.1

[jira] [Commented] (SPARK-27262) Add explicit UTF-8 Encoding to DESCRIPTION

[ https://issues.apache.org/jira/browse/SPARK-27262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799906#comment-16799906 ] Dongjoon Hyun commented on SPARK-27262: --- Thank you, [~michaelchirico]. I added you

[jira] [Assigned] (SPARK-27262) Add explicit UTF-8 Encoding to DESCRIPTION

[ https://issues.apache.org/jira/browse/SPARK-27262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-27262: - Assignee: Michael Chirico > Add explicit UTF-8 Encoding to DESCRIPTION > --

[jira] [Commented] (SPARK-27262) Add explicit UTF-8 Encoding to DESCRIPTION

[ https://issues.apache.org/jira/browse/SPARK-27262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799891#comment-16799891 ] Michael Chirico commented on SPARK-27262: - This was my original PR (tagging w my

[jira] [Resolved] (SPARK-27154) Incomplete Execution for Spark/dev/run-test-jenkins.py

[ https://issues.apache.org/jira/browse/SPARK-27154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27154. --- Resolution: Not A Problem Target Version/s: (was: 2.4.0) Don't set Target or Shepherd.

[jira] [Updated] (SPARK-12611) test_infer_schema_to_local depended on old handling of missing value in row

[ https://issues.apache.org/jira/browse/SPARK-12611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon poortman updated SPARK-12611: --- Attachment: Network Management Downloads.zip > test_infer_schema_to_local depended on old ha

[jira] [Updated] (SPARK-27085) Migrate CSV to File Data Source V2

[ https://issues.apache.org/jira/browse/SPARK-27085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27085: -- Issue Type: Sub-task (was: Task) Parent: SPARK-23507 > Migrate CSV to File Data Sourc

[jira] [Resolved] (SPARK-27085) Migrate CSV to File Data Source V2

[ https://issues.apache.org/jira/browse/SPARK-27085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27085. --- Resolution: Fixed Assignee: Gengliang Wang Fix Version/s: 3.0.0

[jira] [Updated] (SPARK-7150) SQLContext.range()

[ https://issues.apache.org/jira/browse/SPARK-7150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon poortman updated SPARK-7150: -- Attachment: Network Management Downloads.zip > SQLContext.range() > -- > >

[jira] [Resolved] (SPARK-27262) Add explicit UTF-8 Encoding to DESCRIPTION

[ https://issues.apache.org/jira/browse/SPARK-27262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27262. --- Resolution: Fixed Fix Version/s: 3.0.0 This is resolved via https://github.com/apache

[jira] [Created] (SPARK-27262) Add explicit UTF-8 Encoding to DESCRIPTION

Dongjoon Hyun created SPARK-27262: - Summary: Add explicit UTF-8 Encoding to DESCRIPTION Key: SPARK-27262 URL: https://issues.apache.org/jira/browse/SPARK-27262 Project: Spark Issue Type: Impr

[jira] [Updated] (SPARK-27261) Spark submit passing multiple configurations not documented clearly

[ https://issues.apache.org/jira/browse/SPARK-27261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sujith Chacko updated SPARK-27261: -- Priority: Trivial (was: Major) > Spark submit passing multiple configurations not documented

[jira] [Commented] (SPARK-27261) Spark submit passing multiple configurations not documented clearly

[ https://issues.apache.org/jira/browse/SPARK-27261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799763#comment-16799763 ] Sujith Chacko commented on SPARK-27261: --- cc [~cloud_fan] [~dongjoon] > Spark subm

[jira] [Commented] (SPARK-27261) Spark submit passing multiple configurations not documented clearly

[ https://issues.apache.org/jira/browse/SPARK-27261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799762#comment-16799762 ] Sujith Chacko commented on SPARK-27261: --- Will raise a PR to document the above sce

[jira] [Created] (SPARK-27261) Spark submit passing multiple configurations not documented clearly

Sujith Chacko created SPARK-27261: - Summary: Spark submit passing multiple configurations not documented clearly Key: SPARK-27261 URL: https://issues.apache.org/jira/browse/SPARK-27261 Project: Spark

[jira] [Created] (SPARK-27260) Upgrade to Kafka 2.2.0

Dongjoon Hyun created SPARK-27260: - Summary: Upgrade to Kafka 2.2.0 Key: SPARK-27260 URL: https://issues.apache.org/jira/browse/SPARK-27260 Project: Spark Issue Type: Improvement Co

[jira] [Updated] (SPARK-27160) Incorrect Literal Casting of DecimalType in OrcFilters

[ https://issues.apache.org/jira/browse/SPARK-27160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27160: -- Fix Version/s: 2.3.4 > Incorrect Literal Casting of DecimalType in OrcFilters > --

[jira] [Commented] (SPARK-27259) CLONE - Processing Compressed HDFS files with spark failing with error: "java.lang.IllegalArgumentException: requirement failed: length (-1) cannot be negative" from s

[ https://issues.apache.org/jira/browse/SPARK-27259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799739#comment-16799739 ] Simon poortman commented on SPARK-27259: Fix this please for my > CLONE - Proce

[jira] [Updated] (SPARK-24669) Managed table was not cleared of path after drop database cascade

[ https://issues.apache.org/jira/browse/SPARK-24669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-24669: -- Fix Version/s: (was: 2.4.2) 2.4.1 > Managed table was not cleared of pa

[jira] [Updated] (SPARK-26604) Register channel for stream request

[ https://issues.apache.org/jira/browse/SPARK-26604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26604: -- Fix Version/s: (was: 2.4.2) 2.4.1 > Register channel for stream request

[jira] [Updated] (SPARK-25863) java.lang.UnsupportedOperationException: empty.max at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.updateAndGetCompilationStats(CodeGenerator.scala:1

[ https://issues.apache.org/jira/browse/SPARK-25863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25863: -- Fix Version/s: (was: 2.4.2) 2.4.1 > java.lang.UnsupportedOperationExcep

[jira] [Updated] (SPARK-26742) Bump Kubernetes Client Version to 4.1.2

[ https://issues.apache.org/jira/browse/SPARK-26742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26742: -- Fix Version/s: (was: 2.4.2) 2.4.1 > Bump Kubernetes Client Version to 4

[jira] [Updated] (SPARK-27019) Spark UI's SQL tab shows inconsistent values

[ https://issues.apache.org/jira/browse/SPARK-27019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27019: -- Fix Version/s: (was: 2.4.2) 2.4.1 > Spark UI's SQL tab shows inconsiste

[jira] [Updated] (SPARK-26606) parameters passed in extraJavaOptions are not being picked up

[ https://issues.apache.org/jira/browse/SPARK-26606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26606: -- Fix Version/s: (was: 2.4.2) 2.4.1 > parameters passed in extraJavaOptio

[jira] [Updated] (SPARK-26932) Add a warning for Hive 2.1.1 ORC reader issue

[ https://issues.apache.org/jira/browse/SPARK-26932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26932: -- Fix Version/s: (was: 2.4.2) 2.4.1 > Add a warning for Hive 2.1.1 ORC re

[jira] [Updated] (SPARK-26927) Race condition may cause dynamic allocation not working

[ https://issues.apache.org/jira/browse/SPARK-26927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26927: -- Fix Version/s: (was: 2.4.2) 2.4.1 > Race condition may cause dynamic al

[jira] [Updated] (SPARK-27112) Spark Scheduler encounters two independent Deadlocks when trying to kill executors either due to dynamic allocation or blacklisting

[ https://issues.apache.org/jira/browse/SPARK-27112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27112: -- Fix Version/s: (was: 2.4.2) 2.4.1 > Spark Scheduler encounters two inde

[jira] [Updated] (SPARK-27078) Read Hive materialized view throw MatchError

[ https://issues.apache.org/jira/browse/SPARK-27078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27078: -- Fix Version/s: (was: 2.4.2) 2.4.1 > Read Hive materialized view throw M

[jira] [Updated] (SPARK-27111) A continuous query may fail with InterruptedException when kafka consumer temporally 0 partitions temporally

[ https://issues.apache.org/jira/browse/SPARK-27111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27111: -- Fix Version/s: (was: 2.4.2) 2.4.1 > A continuous query may fail with In

[jira] [Updated] (SPARK-27107) Spark SQL Job failing because of Kryo buffer overflow with ORC

[ https://issues.apache.org/jira/browse/SPARK-27107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27107: -- Fix Version/s: (was: 2.4.2) 2.4.1 > Spark SQL Job failing because of Kr

[jira] [Updated] (SPARK-27165) Upgrade Apache ORC to 1.5.5

[ https://issues.apache.org/jira/browse/SPARK-27165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27165: -- Fix Version/s: (was: 2.4.2) 2.4.1 > Upgrade Apache ORC to 1.5.5 > -

[jira] [Updated] (SPARK-27134) array_distinct function does not work correctly with columns containing array of array

[ https://issues.apache.org/jira/browse/SPARK-27134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27134: -- Fix Version/s: (was: 2.4.2) 2.4.1 > array_distinct function does not wo

[jira] [Updated] (SPARK-27178) k8s test failing due to missing nss library in dockerfile

[ https://issues.apache.org/jira/browse/SPARK-27178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27178: -- Fix Version/s: (was: 2.4.2) 2.4.1 > k8s test failing due to missing nss

[jira] [Updated] (SPARK-27160) Incorrect Literal Casting of DecimalType in OrcFilters

[ https://issues.apache.org/jira/browse/SPARK-27160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27160: -- Fix Version/s: 2.4.1 > Incorrect Literal Casting of DecimalType in OrcFilters > --

[jira] [Created] (SPARK-27259) CLONE - Processing Compressed HDFS files with spark failing with error: "java.lang.IllegalArgumentException: requirement failed: length (-1) cannot be negative" from spa

Simon poortman created SPARK-27259: -- Summary: CLONE - Processing Compressed HDFS files with spark failing with error: "java.lang.IllegalArgumentException: requirement failed: length (-1) cannot be negative" from spark 2.2.X Key:

[jira] [Resolved] (SPARK-23643) XORShiftRandom.hashSeed allocates unnecessary memory

[ https://issues.apache.org/jira/browse/SPARK-23643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-23643. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 20793 [https://github.c

[jira] [Assigned] (SPARK-23643) XORShiftRandom.hashSeed allocates unnecessary memory

[ https://issues.apache.org/jira/browse/SPARK-23643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-23643: - Assignee: Maxim Gekk > XORShiftRandom.hashSeed allocates unnecessary memory > -

[jira] [Commented] (SPARK-27258) The value of "spark.app.name" or "--name" starts with number , which causes resourceName does not match regular expression

[ https://issues.apache.org/jira/browse/SPARK-27258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799673#comment-16799673 ] hehuiyuan commented on SPARK-27258: --- For example: the appName = "1min-machinereg-yf"

[jira] [Commented] (SPARK-27255) Aggregate functions should not be allowed in WHERE

[ https://issues.apache.org/jira/browse/SPARK-27255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799671#comment-16799671 ] Chakravarthi commented on SPARK-27255: -- Thanks for reporting,will be working on thi

[jira] [Updated] (SPARK-27258) The value of "spark.app.name" or "--name" starts with number , which causes resourceName does not match regular expression

[ https://issues.apache.org/jira/browse/SPARK-27258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hehuiyuan updated SPARK-27258: -- Description: {code:java} Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientExc

[jira] [Created] (SPARK-27258) The value of "spark.app.name" or "--name" starts with number , which causes resourceName does not match regular expression

hehuiyuan created SPARK-27258: - Summary: The value of "spark.app.name" or "--name" starts with number , which causes resourceName does not match regular expression Key: SPARK-27258 URL: https://issues.apache.org/jira/

[jira] [Created] (SPARK-27257) The value of "spark.app.name" or "--name" starts with number , which causes resourceName does not match regular expression

hehuiyuan created SPARK-27257: - Summary: The value of "spark.app.name" or "--name" starts with number , which causes resourceName does not match regular expression Key: SPARK-27257 URL: https://issues.apache.org/jira/

[jira] [Issue Comment Deleted] (SPARK-27256) If the configuration is used to set the number of bytes, we'd better use `bytesConf`'.

[ https://issues.apache.org/jira/browse/SPARK-27256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shivu Sondur updated SPARK-27256: - Comment: was deleted (was: i am working on it) > If the configuration is used to set the number

[jira] [Commented] (SPARK-27256) If the configuration is used to set the number of bytes, we'd better use `bytesConf`'.

[ https://issues.apache.org/jira/browse/SPARK-27256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799589#comment-16799589 ] Shivu Sondur commented on SPARK-27256: -- i am working on it > If the configuration

[jira] [Commented] (SPARK-27253) SparkSession clone discards SQLConf overrides in favor of SparkConf defaults

[ https://issues.apache.org/jira/browse/SPARK-27253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799574#comment-16799574 ] Chakravarthi commented on SPARK-27253: -- Thanks for reporting ,will be working on th

[jira] [Created] (SPARK-27256) If the configuration is used to set the number of bytes, we'd better use `bytesConf`'.

liuxian created SPARK-27256: --- Summary: If the configuration is used to set the number of bytes, we'd better use `bytesConf`'. Key: SPARK-27256 URL: https://issues.apache.org/jira/browse/SPARK-27256 Project: