[jira] [Created] (SPARK-48565) Fix thread dump display in UI
Cheng Pan created SPARK-48565: - Summary: Fix thread dump display in UI Key: SPARK-48565 URL: https://issues.apache.org/jira/browse/SPARK-48565 Project: Spark Issue Type: Bug Components: UI Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-48238) Spark fail to start due to class o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
[
https://issues.apache.org/jira/browse/SPARK-48238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17846913#comment-17846913
]
Cheng Pan commented on SPARK-48238:
---
[~dongjoon] [~HF] I opened [https://github.com/apache/spark/pull/46611] to
address the YARN incompatible issue by re-implementing a functionally
equivalent Filter, please let me know what you think about this approach.
> Spark fail to start due to class
> o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
> ---
>
> Key: SPARK-48238
> URL: https://issues.apache.org/jira/browse/SPARK-48238
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 4.0.0
>Reporter: Cheng Pan
>Priority: Blocker
> Labels: pull-request-available
>
> I tested the latest master branch, it failed to start on YARN mode
> {code:java}
> dev/make-distribution.sh --tgz -Phive,hive-thriftserver,yarn{code}
>
> {code:java}
> $ bin/spark-sql --master yarn
> WARNING: Using incubator modules: jdk.incubator.vector
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
> setLogLevel(newLevel).
> 2024-05-10 17:58:17 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 2024-05-10 17:58:18 WARN Client: Neither spark.yarn.jars nor
> spark.yarn.archive} is set, falling back to uploading libraries under
> SPARK_HOME.
> 2024-05-10 17:58:25 ERROR SparkContext: Error initializing SparkContext.
> org.sparkproject.jetty.util.MultiException: Multiple exceptions
> at
> org.sparkproject.jetty.util.MultiException.ifExceptionThrow(MultiException.java:117)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:751)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:392)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:902)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:306)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:514)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2$adapted(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
> ~[scala-library-2.13.13.jar:?]
> at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
> ~[scala-library-2.13.13.jar:?]
> at scala.collection.AbstractIterable.foreach(Iterable.scala:935)
> ~[scala-library-2.13.13.jar:?]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1$adapted(SparkUI.scala:79)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?]
> at org.apache.spark.ui.SparkUI.attachAllHandlers(SparkUI.scala:79)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.SparkContext.$anonfun$new$31(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.SparkContext.$anonfun$new$31$adapted(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?]
> at org.apache.spark.SparkContext.(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2963)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1118)
> ~[spark-sql_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.getOrElse(Option.scala:201) [scala-library-2.13.13.jar:?]
> at
> org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1112)
>
[jira] [Created] (SPARK-48257) Polish POM for Hive dependencies
Cheng Pan created SPARK-48257: - Summary: Polish POM for Hive dependencies Key: SPARK-48257 URL: https://issues.apache.org/jira/browse/SPARK-48257 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48255) Guava should not respect hadoop.deps.scope
Cheng Pan created SPARK-48255: - Summary: Guava should not respect hadoop.deps.scope Key: SPARK-48255 URL: https://issues.apache.org/jira/browse/SPARK-48255 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48254) Enhance Guava version extraction rule in dev/test-dependencies.sh
Cheng Pan created SPARK-48254: - Summary: Enhance Guava version extraction rule in dev/test-dependencies.sh Key: SPARK-48254 URL: https://issues.apache.org/jira/browse/SPARK-48254 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-48238) Spark fail to start due to class o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
[
https://issues.apache.org/jira/browse/SPARK-48238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17845781#comment-17845781
]
Cheng Pan commented on SPARK-48238:
---
[~HF] javax => jakarta migration would be complex on the Hadoop side, as Jetty
version couples with Jersey, there is a long standing issue of Jersey upgrading
tracked in HADOOP-15984.
For reverting, I would suggest reverting to 9.4 instead of 10, see
https://github.com/apache/spark/pull/43765#issuecomment-2106600268
> Spark fail to start due to class
> o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
> ---
>
> Key: SPARK-48238
> URL: https://issues.apache.org/jira/browse/SPARK-48238
> Project: Spark
> Issue Type: Sub-task
> Components: Build
>Affects Versions: 4.0.0
>Reporter: Cheng Pan
>Priority: Blocker
>
> I tested the latest master branch, it failed to start on YARN mode
> {code:java}
> dev/make-distribution.sh --tgz -Phive,hive-thriftserver,yarn{code}
>
> {code:java}
> $ bin/spark-sql --master yarn
> WARNING: Using incubator modules: jdk.incubator.vector
> Setting default log level to "WARN".
> To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
> setLogLevel(newLevel).
> 2024-05-10 17:58:17 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 2024-05-10 17:58:18 WARN Client: Neither spark.yarn.jars nor
> spark.yarn.archive} is set, falling back to uploading libraries under
> SPARK_HOME.
> 2024-05-10 17:58:25 ERROR SparkContext: Error initializing SparkContext.
> org.sparkproject.jetty.util.MultiException: Multiple exceptions
> at
> org.sparkproject.jetty.util.MultiException.ifExceptionThrow(MultiException.java:117)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:751)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:392)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:902)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:306)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:514)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2$adapted(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
> ~[scala-library-2.13.13.jar:?]
> at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
> ~[scala-library-2.13.13.jar:?]
> at scala.collection.AbstractIterable.foreach(Iterable.scala:935)
> ~[scala-library-2.13.13.jar:?]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1(SparkUI.scala:81)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1$adapted(SparkUI.scala:79)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?]
> at org.apache.spark.ui.SparkUI.attachAllHandlers(SparkUI.scala:79)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.SparkContext.$anonfun$new$31(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.SparkContext.$anonfun$new$31$adapted(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?]
> at org.apache.spark.SparkContext.(SparkContext.scala:690)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2963)
> ~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at
> org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1118)
> ~[spark-sql_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
> at scala.Option.getOrElse(Option.scala:201) [scala-library-2.13.13.jar:?]
> at
>
[jira] [Created] (SPARK-48238) Spark fail to start due to class o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
Cheng Pan created SPARK-48238:
-
Summary: Spark fail to start due to class
o.a.h.yarn.server.webproxy.amfilter.AmIpFilter is not a jakarta.servlet.Filter
Key: SPARK-48238
URL: https://issues.apache.org/jira/browse/SPARK-48238
Project: Spark
Issue Type: Bug
Components: Build
Affects Versions: 4.0.0
Reporter: Cheng Pan
I tested the latest master branch, it failed to start on YARN mode
{code:java}
dev/make-distribution.sh --tgz -Phive,hive-thriftserver,yarn{code}
{code:java}
$ bin/spark-sql --master yarn
WARNING: Using incubator modules: jdk.incubator.vector
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
2024-05-10 17:58:17 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
2024-05-10 17:58:18 WARN Client: Neither spark.yarn.jars nor
spark.yarn.archive} is set, falling back to uploading libraries under
SPARK_HOME.
2024-05-10 17:58:25 ERROR SparkContext: Error initializing SparkContext.
org.sparkproject.jetty.util.MultiException: Multiple exceptions
at
org.sparkproject.jetty.util.MultiException.ifExceptionThrow(MultiException.java:117)
~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.sparkproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:751)
~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.sparkproject.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:392)
~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.sparkproject.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:902)
~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.sparkproject.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:306)
~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:514)
~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2(SparkUI.scala:81)
~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$2$adapted(SparkUI.scala:81)
~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
~[scala-library-2.13.13.jar:?]
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
~[scala-library-2.13.13.jar:?]
at scala.collection.AbstractIterable.foreach(Iterable.scala:935)
~[scala-library-2.13.13.jar:?]
at
org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1(SparkUI.scala:81)
~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.spark.ui.SparkUI.$anonfun$attachAllHandlers$1$adapted(SparkUI.scala:79)
~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?]
at org.apache.spark.ui.SparkUI.attachAllHandlers(SparkUI.scala:79)
~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.spark.SparkContext.$anonfun$new$31(SparkContext.scala:690)
~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.spark.SparkContext.$anonfun$new$31$adapted(SparkContext.scala:690)
~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.13.jar:?]
at org.apache.spark.SparkContext.(SparkContext.scala:690)
~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2963)
~[spark-core_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:1118)
~[spark-sql_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at scala.Option.getOrElse(Option.scala:201) [scala-library-2.13.13.jar:?]
at
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1112)
[spark-sql_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:64)
[spark-hive-thriftserver_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.(SparkSQLCLIDriver.scala:405)
[spark-hive-thriftserver_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:162)
[spark-hive-thriftserver_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
[spark-hive-thriftserver_2.13-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at
[jira] [Created] (SPARK-48231) Remove unused CodeHaus Jackson dependencies
Cheng Pan created SPARK-48231: - Summary: Remove unused CodeHaus Jackson dependencies Key: SPARK-48231 URL: https://issues.apache.org/jira/browse/SPARK-48231 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48230) Remove unused jodd-core
Cheng Pan created SPARK-48230: - Summary: Remove unused jodd-core Key: SPARK-48230 URL: https://issues.apache.org/jira/browse/SPARK-48230 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-48038) Promote driverServiceName to KubernetesDriverConf
[ https://issues.apache.org/jira/browse/SPARK-48038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-48038: -- Summary: Promote driverServiceName to KubernetesDriverConf (was: Promote driverServiceName to DriverServiceFeatureStep) > Promote driverServiceName to KubernetesDriverConf > - > > Key: SPARK-48038 > URL: https://issues.apache.org/jira/browse/SPARK-48038 > Project: Spark > Issue Type: Improvement > Components: Kubernetes >Affects Versions: 4.0.0 >Reporter: Cheng Pan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-48038) Promote driverServiceName to DriverServiceFeatureStep
Cheng Pan created SPARK-48038: - Summary: Promote driverServiceName to DriverServiceFeatureStep Key: SPARK-48038 URL: https://issues.apache.org/jira/browse/SPARK-48038 Project: Spark Issue Type: Improvement Components: Kubernetes Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47974) Remove install_scala from build/mvn
Cheng Pan created SPARK-47974: - Summary: Remove install_scala from build/mvn Key: SPARK-47974 URL: https://issues.apache.org/jira/browse/SPARK-47974 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47954) Support creating ingress entry for external UI access
Cheng Pan created SPARK-47954: - Summary: Support creating ingress entry for external UI access Key: SPARK-47954 URL: https://issues.apache.org/jira/browse/SPARK-47954 Project: Spark Issue Type: Test Components: Kubernetes Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-47869) Upgrade built in hive to Hive-4.0
[ https://issues.apache.org/jira/browse/SPARK-47869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839669#comment-17839669 ] Cheng Pan commented on SPARK-47869: --- cross link SPARK-44114 > Upgrade built in hive to Hive-4.0 > - > > Key: SPARK-47869 > URL: https://issues.apache.org/jira/browse/SPARK-47869 > Project: Spark > Issue Type: Task > Components: Spark Core >Affects Versions: 3.5.1 >Reporter: Simhadri Govindappa >Priority: Major > > Hive 4.0 has been released. It brings in a lot of new features, bug fixes and > performance improvements. > We would like to update the version of hive used in spark to hive-4.0 > [https://lists.apache.org/thread/2jqpvsx8n801zb5pmlhb8f4zloq27p82] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-47928) Speed up test "Add jar support Ivy URI in SQL"
[ https://issues.apache.org/jira/browse/SPARK-47928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-47928: -- Affects Version/s: 3.2.0 (was: 4.0.0) > Speed up test "Add jar support Ivy URI in SQL" > -- > > Key: SPARK-47928 > URL: https://issues.apache.org/jira/browse/SPARK-47928 > Project: Spark > Issue Type: Test > Components: SQL >Affects Versions: 3.2.0 >Reporter: Cheng Pan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47928) Speed up test "Add jar support Ivy URI in SQL"
Cheng Pan created SPARK-47928: - Summary: Speed up test "Add jar support Ivy URI in SQL" Key: SPARK-47928 URL: https://issues.apache.org/jira/browse/SPARK-47928 Project: Spark Issue Type: Test Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47850) Support converting insert for unpartitioned Hive table
Cheng Pan created SPARK-47850: - Summary: Support converting insert for unpartitioned Hive table Key: SPARK-47850 URL: https://issues.apache.org/jira/browse/SPARK-47850 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47847) Deprecate spark.network.remoteReadNioBufferConversion
Cheng Pan created SPARK-47847: - Summary: Deprecate spark.network.remoteReadNioBufferConversion Key: SPARK-47847 URL: https://issues.apache.org/jira/browse/SPARK-47847 Project: Spark Issue Type: Improvement Components: Shuffle, Spark Core Affects Versions: 3.5.2 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47835) Remove switch for remoteReadNioBufferConversion
Cheng Pan created SPARK-47835: - Summary: Remove switch for remoteReadNioBufferConversion Key: SPARK-47835 URL: https://issues.apache.org/jira/browse/SPARK-47835 Project: Spark Issue Type: Improvement Components: Shuffle Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47833) Supply caller stackstrace for checkAndGlobPathIfNecessary AnalysisException
Cheng Pan created SPARK-47833: - Summary: Supply caller stackstrace for checkAndGlobPathIfNecessary AnalysisException Key: SPARK-47833 URL: https://issues.apache.org/jira/browse/SPARK-47833 Project: Spark Issue Type: Improvement Components: Spark Core, SQL Affects Versions: 3.1.3 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47778) Promote `--wait` option to all services
Cheng Pan created SPARK-47778: - Summary: Promote `--wait` option to all services Key: SPARK-47778 URL: https://issues.apache.org/jira/browse/SPARK-47778 Project: Spark Issue Type: New Feature Components: Deploy Affects Versions: 4.0.0 Reporter: Cheng Pan SPARK-47040 add `--wait` support to `start-connect-server.sh` ./sbin/start-connect-server.sh [--wait] [options] In [https://github.com/apache/spark/pull/45852,] we discussed and reached a consensus to promote `–wait` to all service for consistency -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47610) Always set io.netty.tryReflectionSetAccessible=true
Cheng Pan created SPARK-47610: - Summary: Always set io.netty.tryReflectionSetAccessible=true Key: SPARK-47610 URL: https://issues.apache.org/jira/browse/SPARK-47610 Project: Spark Issue Type: Improvement Components: Build, Spark Core Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-47103) Make the default storage level of intermediate datasets for MLlib configurable
Cheng Pan created SPARK-47103: - Summary: Make the default storage level of intermediate datasets for MLlib configurable Key: SPARK-47103 URL: https://issues.apache.org/jira/browse/SPARK-47103 Project: Spark Issue Type: Improvement Components: MLlib Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-46977) A failed request to obtain a token from one NameNode should not block subsequent token requests
Cheng Pan created SPARK-46977: - Summary: A failed request to obtain a token from one NameNode should not block subsequent token requests Key: SPARK-46977 URL: https://issues.apache.org/jira/browse/SPARK-46977 Project: Spark Issue Type: Improvement Components: Security, Spark Core Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-46918) Replace self-defined variables with Hadoop ContainerExitStatus
Cheng Pan created SPARK-46918: - Summary: Replace self-defined variables with Hadoop ContainerExitStatus Key: SPARK-46918 URL: https://issues.apache.org/jira/browse/SPARK-46918 Project: Spark Issue Type: Improvement Components: YARN Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-46695) Always setting hive.execution.engine to mr
Cheng Pan created SPARK-46695: - Summary: Always setting hive.execution.engine to mr Key: SPARK-46695 URL: https://issues.apache.org/jira/browse/SPARK-46695 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-46675) Remove unused inferTimestampNTZ in ParquetReadSupport
Cheng Pan created SPARK-46675: - Summary: Remove unused inferTimestampNTZ in ParquetReadSupport Key: SPARK-46675 URL: https://issues.apache.org/jira/browse/SPARK-46675 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-46669) Bump Kubernetes Client 6.10.0
[ https://issues.apache.org/jira/browse/SPARK-46669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan resolved SPARK-46669. --- Resolution: Duplicate > Bump Kubernetes Client 6.10.0 > - > > Key: SPARK-46669 > URL: https://issues.apache.org/jira/browse/SPARK-46669 > Project: Spark > Issue Type: Dependency upgrade > Components: k8s >Affects Versions: 4.0.0 >Reporter: Cheng Pan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-46669) Bump Kubernetes Client 6.10.0
Cheng Pan created SPARK-46669: - Summary: Bump Kubernetes Client 6.10.0 Key: SPARK-46669 URL: https://issues.apache.org/jira/browse/SPARK-46669 Project: Spark Issue Type: Dependency upgrade Components: k8s Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-46499) Bump sbt-eclipse 6.2.0
Cheng Pan created SPARK-46499: - Summary: Bump sbt-eclipse 6.2.0 Key: SPARK-46499 URL: https://issues.apache.org/jira/browse/SPARK-46499 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-46420) Remove unused transport from SparkSQLCLIDriver
Cheng Pan created SPARK-46420: - Summary: Remove unused transport from SparkSQLCLIDriver Key: SPARK-46420 URL: https://issues.apache.org/jira/browse/SPARK-46420 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-45937) Fix documentation of spark.executor.maxNumFailures
[
https://issues.apache.org/jira/browse/SPARK-45937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17787926#comment-17787926
]
Cheng Pan commented on SPARK-45937:
---
[~tgraves] sorry, I missed this ticket and opened a new one SPARK-45969, PR is
ready for review https://github.com/apache/spark/pull/43863
> Fix documentation of spark.executor.maxNumFailures
> --
>
> Key: SPARK-45937
> URL: https://issues.apache.org/jira/browse/SPARK-45937
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 3.5.0
>Reporter: Thomas Graves
>Priority: Critical
>
> https://issues.apache.org/jira/browse/SPARK-41210 added support for
> spark.executor.maxNumFailures on Kubernetes, it made this config generic and
> deprecated the yarn version. This config isn't documented and defaults are
> not documented.
>
> [https://github.com/apache/spark/commit/40872e9a094f8459b0b6f626937ced48a8d98efb]
> \
> It also added {color:#0a3069}spark.executor.failuresValidityInterval.{color}
>
> {color:#0a3069}Both need to have default values specified for yarn and k8s,
> it also needs to remove the yarn documentation for equivalent configs
> spark.yarn.max.executor.failures configuration{color}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45969) Document configuration change of executor failure tracker
Cheng Pan created SPARK-45969: - Summary: Document configuration change of executor failure tracker Key: SPARK-45969 URL: https://issues.apache.org/jira/browse/SPARK-45969 Project: Spark Issue Type: Documentation Components: Documentation Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45906) Fix error message extraction from ResourceNotFoundException
Cheng Pan created SPARK-45906: - Summary: Fix error message extraction from ResourceNotFoundException Key: SPARK-45906 URL: https://issues.apache.org/jira/browse/SPARK-45906 Project: Spark Issue Type: Bug Components: YARN Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45670) SparkSubmit does not support --total-executor-cores when deploying on K8s
Cheng Pan created SPARK-45670: - Summary: SparkSubmit does not support --total-executor-cores when deploying on K8s Key: SPARK-45670 URL: https://issues.apache.org/jira/browse/SPARK-45670 Project: Spark Issue Type: Bug Components: Spark Submit Affects Versions: 3.5.0, 3.4.1, 3.3.3 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45646) Remove hardcoding time variables prior to Hive 2.0
Cheng Pan created SPARK-45646: - Summary: Remove hardcoding time variables prior to Hive 2.0 Key: SPARK-45646 URL: https://issues.apache.org/jira/browse/SPARK-45646 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45389) Correct MetaException matching rule on getting partition metadata
Cheng Pan created SPARK-45389: - Summary: Correct MetaException matching rule on getting partition metadata Key: SPARK-45389 URL: https://issues.apache.org/jira/browse/SPARK-45389 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.3.3 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45388) Eliminate unnecessary reflection invocation in Hive shim classes
Cheng Pan created SPARK-45388: - Summary: Eliminate unnecessary reflection invocation in Hive shim classes Key: SPARK-45388 URL: https://issues.apache.org/jira/browse/SPARK-45388 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45358) Remove shim classes for Hive prior 2.0.0
Cheng Pan created SPARK-45358: - Summary: Remove shim classes for Hive prior 2.0.0 Key: SPARK-45358 URL: https://issues.apache.org/jira/browse/SPARK-45358 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-45004) Adding extension for Spark SQL authorization with Ranger-Hive policies
[ https://issues.apache.org/jira/browse/SPARK-45004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17768581#comment-17768581 ] Cheng Pan commented on SPARK-45004: --- [~Ahmadkhani] The Apache Kyuubi community maintains such an extension https://github.com/apache/kyuubi/tree/master/extensions/spark/kyuubi-spark-authz > Adding extension for Spark SQL authorization with Ranger-Hive policies > -- > > Key: SPARK-45004 > URL: https://issues.apache.org/jira/browse/SPARK-45004 > Project: Spark > Issue Type: New Feature > Components: Spark Core, SQL >Affects Versions: 3.4.2 >Reporter: Hasan >Priority: Major > Original Estimate: 24h > Remaining Estimate: 24h > > Apache Ranger have not plugin for Spark SQL authorization and due to this > limitation it's required some extra work and using extensions to manage Spark > authorizations. > > Spark HWC have some performance issues and Cloudera (CDP) solution is complex > and required extra copy of data (big data!). > > Seems natively adding plugins like "Submarine Spark Security Plugin" in Spark > will help to implementation of standard and high performance solution. > Reference: > [https://submarine.apache.org/zh-cn/docs/0.6.0/userDocs/submarine-security/spark-security/] > > > This plugin works fine with table/column/row level and masking options. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45292) Remove Guava from shared classes from IsolatedClientLoader
Cheng Pan created SPARK-45292: - Summary: Remove Guava from shared classes from IsolatedClientLoader Key: SPARK-45292 URL: https://issues.apache.org/jira/browse/SPARK-45292 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-45089) Remove obsolete repo of DB2 JDBC driver
Cheng Pan created SPARK-45089: - Summary: Remove obsolete repo of DB2 JDBC driver Key: SPARK-45089 URL: https://issues.apache.org/jira/browse/SPARK-45089 Project: Spark Issue Type: Test Components: Build, Tests Affects Versions: 4.0.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44938) Change default value of spark.sql.maxSinglePartitionBytes to 128m
Cheng Pan created SPARK-44938: - Summary: Change default value of spark.sql.maxSinglePartitionBytes to 128m Key: SPARK-44938 URL: https://issues.apache.org/jira/browse/SPARK-44938 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44811) Upgrade Guava to 32+
Cheng Pan created SPARK-44811: - Summary: Upgrade Guava to 32+ Key: SPARK-44811 URL: https://issues.apache.org/jira/browse/SPARK-44811 Project: Spark Issue Type: Dependency upgrade Components: Build Affects Versions: 4.0.0 Reporter: Cheng Pan It's a long-standing issue, and was reported many times, e.g. SPARK-38262 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44727) Improve the error message for dynamic allocation conditions
Cheng Pan created SPARK-44727: - Summary: Improve the error message for dynamic allocation conditions Key: SPARK-44727 URL: https://issues.apache.org/jira/browse/SPARK-44727 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44677) Drop legacy Hive-based ORC file format
Cheng Pan created SPARK-44677:
-
Summary: Drop legacy Hive-based ORC file format
Key: SPARK-44677
URL: https://issues.apache.org/jira/browse/SPARK-44677
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.0.0
Reporter: Cheng Pan
Currently, Spark allows to use spark.sql.orc.impl=native/hive to switch the ORC
FileFormat implementation.
SPARK-23456(2.4) switched the default value of spark.sql.orc.impl from "hive"
to "native". and prepared to drop the "hive" implementation in the future.
> ... eventually, Apache Spark will drop old Hive-based ORC code.
The native implementation works well during the whole Spark 3.x period, so it's
a good time to consider dropping the "hive" one in Spark 4.0.
Also, we should take care about the backward-compatibility during change.
> BTW, IIRC, there was a different at Hive ORC CHAR implementation before. So,
> we couldn't remove it for backward-compatibility issues. Since Spark
> implements many CHAR features, we need to re-verify that {{native}}
> implementation has all legacy Hive-based ORC features
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44669) Parquet/ORC files written using Hive Serde should has file extension
Cheng Pan created SPARK-44669: - Summary: Parquet/ORC files written using Hive Serde should has file extension Key: SPARK-44669 URL: https://issues.apache.org/jira/browse/SPARK-44669 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.3.2 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44525) Improve error message when Invoke method is not found
Cheng Pan created SPARK-44525: - Summary: Improve error message when Invoke method is not found Key: SPARK-44525 URL: https://issues.apache.org/jira/browse/SPARK-44525 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.3.2 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-44489) Make InsertIntoDataSourceDirCommand extends DataWritingCommand
[ https://issues.apache.org/jira/browse/SPARK-44489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan resolved SPARK-44489. --- Resolution: Not A Problem > Make InsertIntoDataSourceDirCommand extends DataWritingCommand > -- > > Key: SPARK-44489 > URL: https://issues.apache.org/jira/browse/SPARK-44489 > Project: Spark > Issue Type: Improvement > Components: SQL >Affects Versions: 3.5.0 >Reporter: Cheng Pan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44489) Make InsertIntoDataSourceDirCommand extends DataWritingCommand
Cheng Pan created SPARK-44489: - Summary: Make InsertIntoDataSourceDirCommand extends DataWritingCommand Key: SPARK-44489 URL: https://issues.apache.org/jira/browse/SPARK-44489 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44204) Add missing recordHiveCall for getPartitionNames
Cheng Pan created SPARK-44204: - Summary: Add missing recordHiveCall for getPartitionNames Key: SPARK-44204 URL: https://issues.apache.org/jira/browse/SPARK-44204 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.3.2 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44180) DistributionAndOrderingUtils should apply ResolveTimeZone
Cheng Pan created SPARK-44180: - Summary: DistributionAndOrderingUtils should apply ResolveTimeZone Key: SPARK-44180 URL: https://issues.apache.org/jira/browse/SPARK-44180 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.4.1 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-44070) Bump snappy-java 1.1.10.1
Cheng Pan created SPARK-44070: - Summary: Bump snappy-java 1.1.10.1 Key: SPARK-44070 URL: https://issues.apache.org/jira/browse/SPARK-44070 Project: Spark Issue Type: Dependency upgrade Components: Build Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43880) Organize hadoop-cloud in standard maven project structure
[ https://issues.apache.org/jira/browse/SPARK-43880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-43880: -- Parent: SPARK-43197 Issue Type: Sub-task (was: Improvement) > Organize hadoop-cloud in standard maven project structure > - > > Key: SPARK-43880 > URL: https://issues.apache.org/jira/browse/SPARK-43880 > Project: Spark > Issue Type: Sub-task > Components: Build >Affects Versions: 3.5.0 >Reporter: Cheng Pan >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43880) Organize hadoop-cloud in standard maven project structure
Cheng Pan created SPARK-43880: - Summary: Organize hadoop-cloud in standard maven project structure Key: SPARK-43880 URL: https://issues.apache.org/jira/browse/SPARK-43880 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43751) Document for unbase64 behavior change
Cheng Pan created SPARK-43751: - Summary: Document for unbase64 behavior change Key: SPARK-43751 URL: https://issues.apache.org/jira/browse/SPARK-43751 Project: Spark Issue Type: Documentation Components: Documentation, SQL Affects Versions: 3.3.3, 3.4.1, 3.5.0 Reporter: Cheng Pan Document behavior change caused by https://issues.apache.org/jira/browse/SPARK-37820 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43575) Exclude duplicated classes from kafka assembly jar
Cheng Pan created SPARK-43575: - Summary: Exclude duplicated classes from kafka assembly jar Key: SPARK-43575 URL: https://issues.apache.org/jira/browse/SPARK-43575 Project: Spark Issue Type: Improvement Components: Build, Structured Streaming Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43519) Bump Parquet 1.13.1
Cheng Pan created SPARK-43519: - Summary: Bump Parquet 1.13.1 Key: SPARK-43519 URL: https://issues.apache.org/jira/browse/SPARK-43519 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43489) Remove protobuf 2.5.0
Cheng Pan created SPARK-43489: - Summary: Remove protobuf 2.5.0 Key: SPARK-43489 URL: https://issues.apache.org/jira/browse/SPARK-43489 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43484) Kafka/Kinesis Assembly should not package hadoop-client-runtime
[ https://issues.apache.org/jira/browse/SPARK-43484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-43484: -- Summary: Kafka/Kinesis Assembly should not package hadoop-client-runtime (was: Kafka Assembly should not package hadoop-client-runtime) > Kafka/Kinesis Assembly should not package hadoop-client-runtime > --- > > Key: SPARK-43484 > URL: https://issues.apache.org/jira/browse/SPARK-43484 > Project: Spark > Issue Type: Bug > Components: Build, Structured Streaming >Affects Versions: 3.2.4, 3.3.2, 3.4.0, 3.5.0 >Reporter: Cheng Pan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43484) Kafka Assembly should not package hadoop-client-runtime
Cheng Pan created SPARK-43484: - Summary: Kafka Assembly should not package hadoop-client-runtime Key: SPARK-43484 URL: https://issues.apache.org/jira/browse/SPARK-43484 Project: Spark Issue Type: Bug Components: Build, Structured Streaming Affects Versions: 3.4.0, 3.3.2, 3.2.4, 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43455) Bump kubernetes-client 6.6.1
Cheng Pan created SPARK-43455: - Summary: Bump kubernetes-client 6.6.1 Key: SPARK-43455 URL: https://issues.apache.org/jira/browse/SPARK-43455 Project: Spark Issue Type: Improvement Components: Build, Kubernetes Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43448) Remove dummy hadoop-openstack
Cheng Pan created SPARK-43448: - Summary: Remove dummy hadoop-openstack Key: SPARK-43448 URL: https://issues.apache.org/jira/browse/SPARK-43448 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43395) Excluding macOS tar extended metadata in make-distribution.sh
Cheng Pan created SPARK-43395: - Summary: Excluding macOS tar extended metadata in make-distribution.sh Key: SPARK-43395 URL: https://issues.apache.org/jira/browse/SPARK-43395 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 3.4.0, 3.3.2, 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43395) Exclude macOS tar extended metadata in make-distribution.sh
[ https://issues.apache.org/jira/browse/SPARK-43395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-43395: -- Summary: Exclude macOS tar extended metadata in make-distribution.sh (was: Excluding macOS tar extended metadata in make-distribution.sh) > Exclude macOS tar extended metadata in make-distribution.sh > --- > > Key: SPARK-43395 > URL: https://issues.apache.org/jira/browse/SPARK-43395 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.3.2, 3.4.0, 3.5.0 >Reporter: Cheng Pan >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43394) Upgrade maven to 3.8.8
[ https://issues.apache.org/jira/browse/SPARK-43394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Pan updated SPARK-43394: -- Priority: Trivial (was: Major) > Upgrade maven to 3.8.8 > -- > > Key: SPARK-43394 > URL: https://issues.apache.org/jira/browse/SPARK-43394 > Project: Spark > Issue Type: Improvement > Components: Build >Affects Versions: 3.5.0 >Reporter: Cheng Pan >Priority: Trivial > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43394) Upgrade maven to 3.8.8
Cheng Pan created SPARK-43394: - Summary: Upgrade maven to 3.8.8 Key: SPARK-43394 URL: https://issues.apache.org/jira/browse/SPARK-43394 Project: Spark Issue Type: Improvement Components: Build Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43390) DSv2 allows CTAS/RTAS to reserve schema nullability
Cheng Pan created SPARK-43390: - Summary: DSv2 allows CTAS/RTAS to reserve schema nullability Key: SPARK-43390 URL: https://issues.apache.org/jira/browse/SPARK-43390 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43356) Migrate deprecated createOrReplace to serverSideApply
[
https://issues.apache.org/jira/browse/SPARK-43356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheng Pan updated SPARK-43356:
--
Description:
{{public interface CreateOrReplaceable extends Replaceable {}}
{{ /**}}
{{ * Creates a provided resource in a Kubernetes Cluster. If creation}}
{{ * fails with a HTTP_CONFLICT, it tries to replace resource.}}
{{ *}}
{{ * @return created item returned in kubernetes api response}}
{{ *}}
{{ * @deprecated please use \{@link ServerSideApplicable#serverSideApply()}
or attempt a create and edit/patch operation.}}
{{ */}}
{{ @Deprecated}}
{{ T createOrReplace();}}
{{ /**}}
{{ * Creates an item}}
{{ *}}
{{ * @return the item from the api server}}
{{ */}}
{{ T create();}}
{{}}}
was:
{{public interface CreateOrReplaceable extends Replaceable {}}{{ /**}}
{{ * Creates a provided resource in a Kubernetes Cluster. If creation}}
{{ * fails with a HTTP_CONFLICT, it tries to replace resource.}}
{{ *}}
{{ * @return created item returned in kubernetes api response}}
{{ *}}
{{ * @deprecated please use \{@link ServerSideApplicable#serverSideApply()}
or attempt a create and edit/patch operation.}}
{{ */}}
{{ @Deprecated}}
{{ T createOrReplace();}}
{{ /**}}
{{ * Creates an item}}
{{ *}}
{{ * @return the item from the api server}}
{{ */}}
{{ T create();}}
{{}}}
> Migrate deprecated createOrReplace to serverSideApply
> -
>
> Key: SPARK-43356
> URL: https://issues.apache.org/jira/browse/SPARK-43356
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
>Affects Versions: 3.5.0
>Reporter: Cheng Pan
>Priority: Major
>
>
>
> {{public interface CreateOrReplaceable extends Replaceable {}}
> {{ /**}}
> {{ * Creates a provided resource in a Kubernetes Cluster. If creation}}
> {{ * fails with a HTTP_CONFLICT, it tries to replace resource.}}
> {{ *}}
> {{ * @return created item returned in kubernetes api response}}
> {{ *}}
> {{ * @deprecated please use \{@link ServerSideApplicable#serverSideApply()}
> or attempt a create and edit/patch operation.}}
> {{ */}}
> {{ @Deprecated}}
> {{ T createOrReplace();}}
>
> {{ /**}}
> {{ * Creates an item}}
> {{ *}}
> {{ * @return the item from the api server}}
> {{ */}}
> {{ T create();}}
> {{}}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-43356) Migrate deprecated createOrReplace to serverSideApply
[
https://issues.apache.org/jira/browse/SPARK-43356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheng Pan updated SPARK-43356:
--
Description:
{{public interface CreateOrReplaceable extends Replaceable {}}{{ /**}}
{{ * Creates a provided resource in a Kubernetes Cluster. If creation}}
{{ * fails with a HTTP_CONFLICT, it tries to replace resource.}}
{{ *}}
{{ * @return created item returned in kubernetes api response}}
{{ *}}
{{ * @deprecated please use \{@link ServerSideApplicable#serverSideApply()}
or attempt a create and edit/patch operation.}}
{{ */}}
{{ @Deprecated}}
{{ T createOrReplace();}}
{{ /**}}
{{ * Creates an item}}
{{ *}}
{{ * @return the item from the api server}}
{{ */}}
{{ T create();}}
{{}}}
was:
{{public interface CreateOrReplaceable extends Replaceable {}}{{ /**}}
{{ * Creates a provided resource in a Kubernetes Cluster. If creation}}
{{ * fails with a HTTP_CONFLICT, it tries to replace resource.}}
{{ *}}
{{ * @return created item returned in kubernetes api response}}
{{ *}}
{{ * @deprecated please use \{@link ServerSideApplicable#serverSideApply()}
or attempt a create and edit/patch operation.}}
{{ */}}
{{ @Deprecated}}
{{ T createOrReplace();}}{{ /**}}
{{ * Creates an item}}
{{ *}}
{{ * @return the item from the api server}}
{{ */}}
{{ T create();}}
{{}}}
> Migrate deprecated createOrReplace to serverSideApply
> -
>
> Key: SPARK-43356
> URL: https://issues.apache.org/jira/browse/SPARK-43356
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
>Affects Versions: 3.5.0
>Reporter: Cheng Pan
>Priority: Major
>
>
>
> {{public interface CreateOrReplaceable extends Replaceable {}}{{ /**}}
> {{ * Creates a provided resource in a Kubernetes Cluster. If creation}}
> {{ * fails with a HTTP_CONFLICT, it tries to replace resource.}}
> {{ *}}
> {{ * @return created item returned in kubernetes api response}}
> {{ *}}
> {{ * @deprecated please use \{@link ServerSideApplicable#serverSideApply()}
> or attempt a create and edit/patch operation.}}
> {{ */}}
> {{ @Deprecated}}
> {{ T createOrReplace();}}
>
> {{ /**}}
> {{ * Creates an item}}
> {{ *}}
> {{ * @return the item from the api server}}
> {{ */}}
> {{ T create();}}
> {{}}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43356) Migrate deprecated createOrReplace to serverSideApply
Cheng Pan created SPARK-43356:
-
Summary: Migrate deprecated createOrReplace to serverSideApply
Key: SPARK-43356
URL: https://issues.apache.org/jira/browse/SPARK-43356
Project: Spark
Issue Type: Improvement
Components: Kubernetes
Affects Versions: 3.5.0
Reporter: Cheng Pan
{{public interface CreateOrReplaceable extends Replaceable {}}{{ /**}}
{{ * Creates a provided resource in a Kubernetes Cluster. If creation}}
{{ * fails with a HTTP_CONFLICT, it tries to replace resource.}}
{{ *}}
{{ * @return created item returned in kubernetes api response}}
{{ *}}
{{ * @deprecated please use \{@link ServerSideApplicable#serverSideApply()}
or attempt a create and edit/patch operation.}}
{{ */}}
{{ @Deprecated}}
{{ T createOrReplace();}}{{ /**}}
{{ * Creates an item}}
{{ *}}
{{ * @return the item from the api server}}
{{ */}}
{{ T create();}}
{{}}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43355) Upgrade fabric8io/kubernetes-client 6.6.0
Cheng Pan created SPARK-43355: - Summary: Upgrade fabric8io/kubernetes-client 6.6.0 Key: SPARK-43355 URL: https://issues.apache.org/jira/browse/SPARK-43355 Project: Spark Issue Type: Dependency upgrade Components: Build, Kubernetes Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43352) `setPythonSparkConfProperties` should use conf instead of sparkAppConf
Cheng Pan created SPARK-43352: - Summary: `setPythonSparkConfProperties` should use conf instead of sparkAppConf Key: SPARK-43352 URL: https://issues.apache.org/jira/browse/SPARK-43352 Project: Spark Issue Type: Test Components: Kubernetes, Tests Affects Versions: 3.2.4 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43320) Directly call Hive 2.3.9 API
Cheng Pan created SPARK-43320: - Summary: Directly call Hive 2.3.9 API Key: SPARK-43320 URL: https://issues.apache.org/jira/browse/SPARK-43320 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43319) Remove usage of deprecated DefaultKubernetesClient
Cheng Pan created SPARK-43319: - Summary: Remove usage of deprecated DefaultKubernetesClient Key: SPARK-43319 URL: https://issues.apache.org/jira/browse/SPARK-43319 Project: Spark Issue Type: Test Components: Kubernetes, Tests Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43248) Unnecessary serialize/deserialize of Path on parallel gather partition stats
Cheng Pan created SPARK-43248: - Summary: Unnecessary serialize/deserialize of Path on parallel gather partition stats Key: SPARK-43248 URL: https://issues.apache.org/jira/browse/SPARK-43248 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43211) Remove Hadoop2 support in IsolatedClientLoader
Cheng Pan created SPARK-43211: - Summary: Remove Hadoop2 support in IsolatedClientLoader Key: SPARK-43211 URL: https://issues.apache.org/jira/browse/SPARK-43211 Project: Spark Issue Type: Sub-task Components: SQL Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43208) IsolatedClassLoader should close barrier class InputStream after reading
Cheng Pan created SPARK-43208: - Summary: IsolatedClassLoader should close barrier class InputStream after reading Key: SPARK-43208 URL: https://issues.apache.org/jira/browse/SPARK-43208 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 3.3.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43202) Replace reflection w/ direct calling for YARN Resource API
Cheng Pan created SPARK-43202: - Summary: Replace reflection w/ direct calling for YARN Resource API Key: SPARK-43202 URL: https://issues.apache.org/jira/browse/SPARK-43202 Project: Spark Issue Type: Sub-task Components: YARN Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43200) Remove Hadoop 2 reference in docs
Cheng Pan created SPARK-43200: - Summary: Remove Hadoop 2 reference in docs Key: SPARK-43200 URL: https://issues.apache.org/jira/browse/SPARK-43200 Project: Spark Issue Type: Sub-task Components: Documentation Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43195) Remove unnecessary serializable wrapper in HadoopFSUtils
Cheng Pan created SPARK-43195: - Summary: Remove unnecessary serializable wrapper in HadoopFSUtils Key: SPARK-43195 URL: https://issues.apache.org/jira/browse/SPARK-43195 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43193) Remove workaround for HADOOP-12074
Cheng Pan created SPARK-43193: - Summary: Remove workaround for HADOOP-12074 Key: SPARK-43193 URL: https://issues.apache.org/jira/browse/SPARK-43193 Project: Spark Issue Type: Improvement Components: Structured Streaming Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43191) Replace reflection w/ direct calling for Hadoop CallerContext
Cheng Pan created SPARK-43191: - Summary: Replace reflection w/ direct calling for Hadoop CallerContext Key: SPARK-43191 URL: https://issues.apache.org/jira/browse/SPARK-43191 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43187) Remove workaround for MiniKdc's BindException
Cheng Pan created SPARK-43187: - Summary: Remove workaround for MiniKdc's BindException Key: SPARK-43187 URL: https://issues.apache.org/jira/browse/SPARK-43187 Project: Spark Issue Type: Test Components: Tests Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43186) Remove workaround for FileSinkDesc
Cheng Pan created SPARK-43186: - Summary: Remove workaround for FileSinkDesc Key: SPARK-43186 URL: https://issues.apache.org/jira/browse/SPARK-43186 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43171) Support dynamic changing unix user in Pod
Cheng Pan created SPARK-43171: - Summary: Support dynamic changing unix user in Pod Key: SPARK-43171 URL: https://issues.apache.org/jira/browse/SPARK-43171 Project: Spark Issue Type: Improvement Components: Kubernetes Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-43150) Remove workaround for PARQUET-2160
Cheng Pan created SPARK-43150: - Summary: Remove workaround for PARQUET-2160 Key: SPARK-43150 URL: https://issues.apache.org/jira/browse/SPARK-43150 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-42949) Simplify code for NAAJ
Cheng Pan created SPARK-42949: - Summary: Simplify code for NAAJ Key: SPARK-42949 URL: https://issues.apache.org/jira/browse/SPARK-42949 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 3.5.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-42906) Resource name prefix should start with an alphabetic character
Cheng Pan created SPARK-42906: - Summary: Resource name prefix should start with an alphabetic character Key: SPARK-42906 URL: https://issues.apache.org/jira/browse/SPARK-42906 Project: Spark Issue Type: Bug Components: Kubernetes Affects Versions: 3.2.3 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-42813) Print application info when waitAppCompletion is false
Cheng Pan created SPARK-42813: - Summary: Print application info when waitAppCompletion is false Key: SPARK-42813 URL: https://issues.apache.org/jira/browse/SPARK-42813 Project: Spark Issue Type: Bug Components: Kubernetes Affects Versions: 3.3.2 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-41952) Upgrade Parquet to fix off-heap memory leaks in Zstd codec
[ https://issues.apache.org/jira/browse/SPARK-41952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17691012#comment-17691012 ] Cheng Pan commented on SPARK-41952: --- Fix on Spark side is feasible, I'm working on this. > Upgrade Parquet to fix off-heap memory leaks in Zstd codec > -- > > Key: SPARK-41952 > URL: https://issues.apache.org/jira/browse/SPARK-41952 > Project: Spark > Issue Type: Bug > Components: Input/Output >Affects Versions: 3.1.3, 3.3.1, 3.2.3 >Reporter: Alexey Kudinkin >Priority: Critical > > Recently, native memory leak have been discovered in Parquet in conjunction > of it using Zstd decompressor from luben/zstd-jni library (PARQUET-2160). > This is very problematic to a point where we can't use Parquet w/ Zstd due to > pervasive OOMs taking down our executors and disrupting our jobs. > Luckily fix addressing this had already landed in Parquet: > [https://github.com/apache/parquet-mr/pull/982] > > Now, we just need to > # Updated version of Parquet is released in a timely manner > # Spark is upgraded onto this new version in the upcoming release > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-41788) Move InsertIntoStatement to basicLogicalOperators
Cheng Pan created SPARK-41788: - Summary: Move InsertIntoStatement to basicLogicalOperators Key: SPARK-41788 URL: https://issues.apache.org/jira/browse/SPARK-41788 Project: Spark Issue Type: Task Components: SQL Affects Versions: 3.4.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-41667) Expose env var SPARK_DRIVER_POD_NAME in Driver Pod
Cheng Pan created SPARK-41667: - Summary: Expose env var SPARK_DRIVER_POD_NAME in Driver Pod Key: SPARK-41667 URL: https://issues.apache.org/jira/browse/SPARK-41667 Project: Spark Issue Type: Improvement Components: Kubernetes Affects Versions: 3.4.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-41458) Correctly transform the SPI services for Yarn Shuffle Service
Cheng Pan created SPARK-41458: - Summary: Correctly transform the SPI services for Yarn Shuffle Service Key: SPARK-41458 URL: https://issues.apache.org/jira/browse/SPARK-41458 Project: Spark Issue Type: Bug Components: Shuffle, YARN Affects Versions: 3.3.0, 3.2.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-41451) Avoid using empty abbrevMarker in StringUtils.abbreviate
Cheng Pan created SPARK-41451: - Summary: Avoid using empty abbrevMarker in StringUtils.abbreviate Key: SPARK-41451 URL: https://issues.apache.org/jira/browse/SPARK-41451 Project: Spark Issue Type: Improvement Components: Kubernetes Affects Versions: 3.3.1, 3.3.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-41376) Executor netty direct memory check should respect spark.shuffle.io.preferDirectBufs
Cheng Pan created SPARK-41376: - Summary: Executor netty direct memory check should respect spark.shuffle.io.preferDirectBufs Key: SPARK-41376 URL: https://issues.apache.org/jira/browse/SPARK-41376 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 3.2.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-41210) Window based executor failure tracking mechanism
Cheng Pan created SPARK-41210: - Summary: Window based executor failure tracking mechanism Key: SPARK-41210 URL: https://issues.apache.org/jira/browse/SPARK-41210 Project: Spark Issue Type: New Feature Components: Kubernetes Affects Versions: 3.4.0 Reporter: Cheng Pan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-39601) AllocationFailure should not be treated as exitCausedByApp when driver is shutting down
[
https://issues.apache.org/jira/browse/SPARK-39601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheng Pan updated SPARK-39601:
--
Description:
I observed some Spark Applications successfully completed all jobs but failed
during the shutting down phase w/ reason: Max number of executor failures (16)
reached, the timeline is
Driver - Job success, Spark starts shutting down procedure.
{code:java}
2022-06-23 19:50:55 CST AbstractConnector INFO - Stopped
Spark@74e9431b{HTTP/1.1, (http/1.1)}
{0.0.0.0:0}
2022-06-23 19:50:55 CST SparkUI INFO - Stopped Spark web UI at
http://hadoop2627.xxx.org:28446
2022-06-23 19:50:55 CST YarnClusterSchedulerBackend INFO - Shutting down all
executors
{code}
Driver - A container allocate successful during shutting down phase.
{code:java}
2022-06-23 19:52:21 CST YarnAllocator INFO - Launching container
container_e94_1649986670278_7743380_02_25 on host hadoop4388.xxx.org for
executor with ID 24 for ResourceProfile Id 0{code}
Executor - The executor can not connect to driver endpoint because driver
already stopped the endpoint.
{code:java}
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1911)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:393)
at
org.apache.spark.executor.YarnCoarseGrainedExecutorBackend$.main(YarnCoarseGrainedExecutorBackend.scala:81)
at
org.apache.spark.executor.YarnCoarseGrainedExecutorBackend.main(YarnCoarseGrainedExecutorBackend.scala)
Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$9(CoarseGrainedExecutorBackend.scala:413)
at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
at
scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
at scala.collection.immutable.Range.foreach(Range.scala:158)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$7(CoarseGrainedExecutorBackend.scala:411)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
... 4 more
Caused by: org.apache.spark.rpc.RpcEndpointNotFoundException: Cannot find
endpoint: spark://coarsegrainedschedu...@hadoop2627.xxx.org:21956
at
org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$asyncSetupEndpointRefByURI$1(NettyRpcEnv.scala:148)
at
org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$asyncSetupEndpointRefByURI$1$adapted(NettyRpcEnv.scala:144)
at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at org.apache.spark.util.ThreadUtils$$anon$1.execute(ThreadUtils.scala:99)
at
scala.concurrent.impl.ExecutionContextImpl$$anon$4.execute(ExecutionContextImpl.scala:138)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72)
at
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288)
at
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288)
at
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288){code}
Driver - YarnAllocator received container launch error message and treat it as
`exitCausedByApp`
{code:java}
2022-06-23 19:52:27 CST YarnAllocator INFO - Completed container
container_e94_1649986670278_7743380_02_25 on host: hadoop4388.xxx.org
(state: COMPLETE, exit status: 1)
2022-06-23 19:52:27 CST YarnAllocator WARN - Container from a bad node:
container_e94_1649986670278_7743380_02_25 on host: hadoop4388.xxx.org. Exit
status: 1. Diagnostics: [2022-06-23 19:52:24.932]Exception from
container-launch.
Container id: container_e94_1649986670278_7743380_02_25
Exit code: 1
Shell output: main : command provided 1
main : run as user is bdms_pm
main : requested yarn user is bdms_pm
Getting exit code file...
Creating script paths...
Writing pid file...
Writing to tmp file
[jira] [Updated] (SPARK-39601) AllocationFailure should not be treated as exitCausedByApp when driver is shutting down
[
https://issues.apache.org/jira/browse/SPARK-39601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheng Pan updated SPARK-39601:
--
Description:
I observed some Spark Applications successfully completed all jobs but failed
during the shutting down phase w/ reason: Max number of executor failures (16)
reached, the timeline is
Driver - Job success, Spark starts shutting down procedure.
{code:java}
2022-06-23 19:50:55 CST AbstractConnector INFO - Stopped
Spark@74e9431b{HTTP/1.1, (http/1.1)}
{0.0.0.0:0}
2022-06-23 19:50:55 CST SparkUI INFO - Stopped Spark web UI at
http://hadoop2627.xxx.org:28446
2022-06-23 19:50:55 CST YarnClusterSchedulerBackend INFO - Shutting down all
executors
{code}
Driver - A container allocate successful during shutting down phase.
{code:java}
2022-06-23 19:52:21 CST YarnAllocator INFO - Launching container
container_e94_1649986670278_7743380_02_25 on host hadoop4388.xxx.org for
executor with ID 24 for ResourceProfile Id 0{code}
Executor - The executor can not connect to driver endpoint because driver
already stopped the endpoint.
{code:java}
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1911)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:393)
at
org.apache.spark.executor.YarnCoarseGrainedExecutorBackend$.main(YarnCoarseGrainedExecutorBackend.scala:81)
at
org.apache.spark.executor.YarnCoarseGrainedExecutorBackend.main(YarnCoarseGrainedExecutorBackend.scala)
Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$9(CoarseGrainedExecutorBackend.scala:413)
at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
at
scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
at scala.collection.immutable.Range.foreach(Range.scala:158)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$7(CoarseGrainedExecutorBackend.scala:411)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
... 4 more
Caused by: org.apache.spark.rpc.RpcEndpointNotFoundException: Cannot find
endpoint: spark://coarsegrainedschedu...@hadoop2627.xxx.org:21956
at
org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$asyncSetupEndpointRefByURI$1(NettyRpcEnv.scala:148)
at
org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$asyncSetupEndpointRefByURI$1$adapted(NettyRpcEnv.scala:144)
at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at org.apache.spark.util.ThreadUtils$$anon$1.execute(ThreadUtils.scala:99)
at
scala.concurrent.impl.ExecutionContextImpl$$anon$4.execute(ExecutionContextImpl.scala:138)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72)
at
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288)
at
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288)
at
scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288){code}
Driver - YarnAllocator received container launch error message and treat it as
`exitCausedByApp`
{code:java}
2022-06-23 19:52:27 CST YarnAllocator INFO - Completed container
container_e94_1649986670278_7743380_02_25 on host: hadoop4388.xxx.org
(state: COMPLETE, exit status: 1)
2022-06-23 19:52:27 CST YarnAllocator WARN - Container from a bad node:
container_e94_1649986670278_7743380_02_25 on host: hadoop4388.xxx.org. Exit
status: 1. Diagnostics: [2022-06-23 19:52:24.932]Exception from
container-launch.
Container id: container_e94_1649986670278_7743380_02_25
Exit code: 1
Shell output: main : command provided 1
main : run as user is bdms_pm
main : requested yarn user is bdms_pm
Getting exit code file...
Creating script paths...
Writing pid file...
Writing to tmp file
[jira] [Updated] (SPARK-39601) AllocationFailure should not be treated as exitCausedByApp when driver is shutting down
[
https://issues.apache.org/jira/browse/SPARK-39601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheng Pan updated SPARK-39601:
--
Description:
I observed some Spark Applications successfully completed all jobs but failed
during the shutting down phase w/ reason: Max number of executor failures (16)
reached, the timeline is
Driver - Job success, Spark starts shutting down procedure.
```
2022-06-23 19:50:55 CST AbstractConnector INFO - Stopped
Spark@74e9431b\{HTTP/1.1, (http/1.1)}{0.0.0.0:0}
2022-06-23 19:50:55 CST SparkUI INFO - Stopped Spark web UI at
http://hadoop2627.xxx.org:28446
2022-06-23 19:50:55 CST YarnClusterSchedulerBackend INFO - Shutting down all
executors
```
Driver - A container allocate successful during shutting down phase.
```
2022-06-23 19:52:21 CST YarnAllocator INFO - Launching container
container_e94_1649986670278_7743380_02_25 on host hadoop4388.xxx.org for
executor with ID 24 for ResourceProfile Id 0
```
Executor - The executor can not connect to driver endpoint because driver
already stopped the endpoint.
```
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1911)
at
org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:393)
at
org.apache.spark.executor.YarnCoarseGrainedExecutorBackend$.main(YarnCoarseGrainedExecutorBackend.scala:81)
at
org.apache.spark.executor.YarnCoarseGrainedExecutorBackend.main(YarnCoarseGrainedExecutorBackend.scala)
Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:301)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$9(CoarseGrainedExecutorBackend.scala:413)
at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
at
scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:877)
at scala.collection.immutable.Range.foreach(Range.scala:158)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:876)
at
org.apache.spark.executor.CoarseGrainedExecutorBackend$.$anonfun$run$7(CoarseGrainedExecutorBackend.scala:411)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62)
at
org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
... 4 more
Caused by: org.apache.spark.rpc.RpcEndpointNotFoundException: Cannot find
endpoint: spark://coarsegrainedschedu...@hadoop2627.xxx.org:21956
at
org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$asyncSetupEndpointRefByURI$1(NettyRpcEnv.scala:148)
at
org.apache.spark.rpc.netty.NettyRpcEnv.$anonfun$asyncSetupEndpointRefByURI$1$adapted(NettyRpcEnv.scala:144)
at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at org.apache.spark.util.ThreadUtils$$anon$1.execute(ThreadUtils.scala:99)
at
scala.concurrent.impl.ExecutionContextImpl$$anon$4.execute(ExecutionContextImpl.scala:138)
at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:72)
at
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1(Promise.scala:288)
at
scala.concurrent.impl.Promise$DefaultPromise.$anonfun$tryComplete$1$adapted(Promise.scala:288)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:288)
```
Driver - YarnAllocator received container launch error message and treat it as
`exitCausedByApp`
```
2022-06-23 19:52:27 CST YarnAllocator INFO - Completed container
container_e94_1649986670278_7743380_02_25 on host: hadoop4388.xxx.org
(state: COMPLETE, exit status: 1)
2022-06-23 19:52:27 CST YarnAllocator WARN - Container from a bad node:
container_e94_1649986670278_7743380_02_25 on host: hadoop4388.xxx.org. Exit
status: 1. Diagnostics: [2022-06-23 19:52:24.932]Exception from
container-launch.
Container id: container_e94_1649986670278_7743380_02_25
Exit code: 1
Shell output: main : command provided 1
main : run as user is bdms_pm
main : requested yarn user is bdms_pm
Getting exit code file...
Creating script paths...
Writing pid file...
Writing to tmp file
