[jira] [Commented] (SPARK-33104) Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in SparkHadoopUtil`
[ https://issues.apache.org/jira/browse/SPARK-33104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238988#comment-17238988 ] Apache Spark commented on SPARK-33104: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/30508 > Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in > SparkHadoopUtil` > > > Key: SPARK-33104 > URL: https://issues.apache.org/jira/browse/SPARK-33104 > Project: Spark > Issue Type: Bug > Components: Tests >Affects Versions: 3.1.0 >Reporter: Dongjoon Hyun >Assignee: Hyukjin Kwon >Priority: Blocker > Fix For: 3.1.0 > > > - > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/1377/testReport/org.apache.spark.deploy.yarn/YarnClusterSuite/yarn_cluster_should_respect_conf_overrides_in_SparkHadoopUtil__SPARK_16414__SPARK_23630_/ > {code} > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN DefaultContainerExecutor: > Exit code from container container_1602245728426_0006_02_01 is : 15 > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN DefaultContainerExecutor: > Exception from container-launch with container ID: > container_1602245728426_0006_02_01 and exit code: 15 > ExitCodeException exitCode=15: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:585) > at org.apache.hadoop.util.Shell.run(Shell.java:482) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN ContainerLaunch: Container > exited with a non-zero exit code 15 > 20/10/09 05:18:13.237 AsyncDispatcher event handler WARN NMAuditLogger: > USER=jenkins OPERATION=Container Finished - Failed TARGET=ContainerImpl > RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE > APPID=application_1602245728426_0006 > CONTAINERID=container_1602245728426_0006_02_01 > 20/10/09 05:18:13.244 Socket Reader #1 for port 37112 INFO Server: Auth > successful for appattempt_1602245728426_0006_02 (auth:SIMPLE) > 20/10/09 05:18:13.326 IPC Parameter Sending Thread #0 DEBUG Client: IPC > Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins sending #37 > 20/10/09 05:18:13.327 IPC Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins DEBUG Client: IPC > Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins got value #37 > 20/10/09 05:18:13.328 main DEBUG ProtobufRpcEngine: Call: > getApplicationReport took 2ms > 20/10/09 05:18:13.328 main INFO Client: Application report for > application_1602245728426_0006 (state: FINISHED) > 20/10/09 05:18:13.328 main DEBUG Client: >client token: N/A >diagnostics: User class threw exception: > org.scalatest.exceptions.TestFailedException: null was not equal to > "testvalue" > at > org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:344) > at > org.scalatest.matchers.should.Matchers$ShouldMethodHelperClass.shouldMatcher(Matchers.scala:6778) > at > org.scalatest.matchers.should.Matchers$AnyShouldWrapper.should(Matchers.scala:6822) > at > org.apache.spark.deploy.yarn.YarnClusterDriverUseSparkHadoopUtilConf$.$anonfun$main$2(YarnClusterSuite.scala:383) > at > scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) > at > scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) > at > org.apache.spark.deploy.yarn.YarnClusterDriverUseSparkHadoopUtilConf$.main(YarnClusterSuite.scala:382) > at > org.apache.spark.deploy.yarn.YarnClusterDriverUseSparkHadoopUtilConf.main(YarnClusterSuite.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >
[jira] [Commented] (SPARK-33104) Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in SparkHadoopUtil`
[ https://issues.apache.org/jira/browse/SPARK-33104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219083#comment-17219083 ] Apache Spark commented on SPARK-33104: -- User 'HyukjinKwon' has created a pull request for this issue: https://github.com/apache/spark/pull/30133 > Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in > SparkHadoopUtil` > > > Key: SPARK-33104 > URL: https://issues.apache.org/jira/browse/SPARK-33104 > Project: Spark > Issue Type: Bug > Components: Tests >Affects Versions: 3.1.0 >Reporter: Dongjoon Hyun >Priority: Blocker > > - > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/1377/testReport/org.apache.spark.deploy.yarn/YarnClusterSuite/yarn_cluster_should_respect_conf_overrides_in_SparkHadoopUtil__SPARK_16414__SPARK_23630_/ > {code} > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN DefaultContainerExecutor: > Exit code from container container_1602245728426_0006_02_01 is : 15 > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN DefaultContainerExecutor: > Exception from container-launch with container ID: > container_1602245728426_0006_02_01 and exit code: 15 > ExitCodeException exitCode=15: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:585) > at org.apache.hadoop.util.Shell.run(Shell.java:482) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN ContainerLaunch: Container > exited with a non-zero exit code 15 > 20/10/09 05:18:13.237 AsyncDispatcher event handler WARN NMAuditLogger: > USER=jenkins OPERATION=Container Finished - Failed TARGET=ContainerImpl > RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE > APPID=application_1602245728426_0006 > CONTAINERID=container_1602245728426_0006_02_01 > 20/10/09 05:18:13.244 Socket Reader #1 for port 37112 INFO Server: Auth > successful for appattempt_1602245728426_0006_02 (auth:SIMPLE) > 20/10/09 05:18:13.326 IPC Parameter Sending Thread #0 DEBUG Client: IPC > Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins sending #37 > 20/10/09 05:18:13.327 IPC Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins DEBUG Client: IPC > Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins got value #37 > 20/10/09 05:18:13.328 main DEBUG ProtobufRpcEngine: Call: > getApplicationReport took 2ms > 20/10/09 05:18:13.328 main INFO Client: Application report for > application_1602245728426_0006 (state: FINISHED) > 20/10/09 05:18:13.328 main DEBUG Client: >client token: N/A >diagnostics: User class threw exception: > org.scalatest.exceptions.TestFailedException: null was not equal to > "testvalue" > at > org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:344) > at > org.scalatest.matchers.should.Matchers$ShouldMethodHelperClass.shouldMatcher(Matchers.scala:6778) > at > org.scalatest.matchers.should.Matchers$AnyShouldWrapper.should(Matchers.scala:6822) > at > org.apache.spark.deploy.yarn.YarnClusterDriverUseSparkHadoopUtilConf$.$anonfun$main$2(YarnClusterSuite.scala:383) > at > scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) > at > scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) > at > org.apache.spark.deploy.yarn.YarnClusterDriverUseSparkHadoopUtilConf$.main(YarnClusterSuite.scala:382) > at > org.apache.spark.deploy.yarn.YarnClusterDriverUseSparkHadoopUtilConf.main(YarnClusterSuite.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at >
[jira] [Commented] (SPARK-33104) Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in SparkHadoopUtil`
[ https://issues.apache.org/jira/browse/SPARK-33104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218119#comment-17218119 ] Hyukjin Kwon commented on SPARK-33104: -- cc [~Ngone51] FYI since you're taking a look for this. > Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in > SparkHadoopUtil` > > > Key: SPARK-33104 > URL: https://issues.apache.org/jira/browse/SPARK-33104 > Project: Spark > Issue Type: Bug > Components: Tests >Affects Versions: 3.1.0 >Reporter: Dongjoon Hyun >Priority: Blocker > > - > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/1377/testReport/org.apache.spark.deploy.yarn/YarnClusterSuite/yarn_cluster_should_respect_conf_overrides_in_SparkHadoopUtil__SPARK_16414__SPARK_23630_/ > {code} > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN DefaultContainerExecutor: > Exit code from container container_1602245728426_0006_02_01 is : 15 > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN DefaultContainerExecutor: > Exception from container-launch with container ID: > container_1602245728426_0006_02_01 and exit code: 15 > ExitCodeException exitCode=15: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:585) > at org.apache.hadoop.util.Shell.run(Shell.java:482) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN ContainerLaunch: Container > exited with a non-zero exit code 15 > 20/10/09 05:18:13.237 AsyncDispatcher event handler WARN NMAuditLogger: > USER=jenkins OPERATION=Container Finished - Failed TARGET=ContainerImpl > RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE > APPID=application_1602245728426_0006 > CONTAINERID=container_1602245728426_0006_02_01 > 20/10/09 05:18:13.244 Socket Reader #1 for port 37112 INFO Server: Auth > successful for appattempt_1602245728426_0006_02 (auth:SIMPLE) > 20/10/09 05:18:13.326 IPC Parameter Sending Thread #0 DEBUG Client: IPC > Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins sending #37 > 20/10/09 05:18:13.327 IPC Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins DEBUG Client: IPC > Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins got value #37 > 20/10/09 05:18:13.328 main DEBUG ProtobufRpcEngine: Call: > getApplicationReport took 2ms > 20/10/09 05:18:13.328 main INFO Client: Application report for > application_1602245728426_0006 (state: FINISHED) > 20/10/09 05:18:13.328 main DEBUG Client: >client token: N/A >diagnostics: User class threw exception: > org.scalatest.exceptions.TestFailedException: null was not equal to > "testvalue" > at > org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:344) > at > org.scalatest.matchers.should.Matchers$ShouldMethodHelperClass.shouldMatcher(Matchers.scala:6778) > at > org.scalatest.matchers.should.Matchers$AnyShouldWrapper.should(Matchers.scala:6822) > at > org.apache.spark.deploy.yarn.YarnClusterDriverUseSparkHadoopUtilConf$.$anonfun$main$2(YarnClusterSuite.scala:383) > at > scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) > at > scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) > at > org.apache.spark.deploy.yarn.YarnClusterDriverUseSparkHadoopUtilConf$.main(YarnClusterSuite.scala:382) > at > org.apache.spark.deploy.yarn.YarnClusterDriverUseSparkHadoopUtilConf.main(YarnClusterSuite.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at
[jira] [Commented] (SPARK-33104) Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in SparkHadoopUtil`
[ https://issues.apache.org/jira/browse/SPARK-33104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211809#comment-17211809 ] Dongjoon Hyun commented on SPARK-33104: --- Although this is a test case issue, I will raise this issue as a blocker because this is an evidence of SBT 1.3 regression . We need to fix this before Apache Spark 3.1.0. Otherwise, this can hide more issues. > Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in > SparkHadoopUtil` > > > Key: SPARK-33104 > URL: https://issues.apache.org/jira/browse/SPARK-33104 > Project: Spark > Issue Type: Bug > Components: Tests >Affects Versions: 3.1.0 >Reporter: Dongjoon Hyun >Priority: Major > > - > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/1377/testReport/org.apache.spark.deploy.yarn/YarnClusterSuite/yarn_cluster_should_respect_conf_overrides_in_SparkHadoopUtil__SPARK_16414__SPARK_23630_/ > {code} > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN DefaultContainerExecutor: > Exit code from container container_1602245728426_0006_02_01 is : 15 > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN DefaultContainerExecutor: > Exception from container-launch with container ID: > container_1602245728426_0006_02_01 and exit code: 15 > ExitCodeException exitCode=15: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:585) > at org.apache.hadoop.util.Shell.run(Shell.java:482) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN ContainerLaunch: Container > exited with a non-zero exit code 15 > 20/10/09 05:18:13.237 AsyncDispatcher event handler WARN NMAuditLogger: > USER=jenkins OPERATION=Container Finished - Failed TARGET=ContainerImpl > RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE > APPID=application_1602245728426_0006 > CONTAINERID=container_1602245728426_0006_02_01 > 20/10/09 05:18:13.244 Socket Reader #1 for port 37112 INFO Server: Auth > successful for appattempt_1602245728426_0006_02 (auth:SIMPLE) > 20/10/09 05:18:13.326 IPC Parameter Sending Thread #0 DEBUG Client: IPC > Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins sending #37 > 20/10/09 05:18:13.327 IPC Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins DEBUG Client: IPC > Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins got value #37 > 20/10/09 05:18:13.328 main DEBUG ProtobufRpcEngine: Call: > getApplicationReport took 2ms > 20/10/09 05:18:13.328 main INFO Client: Application report for > application_1602245728426_0006 (state: FINISHED) > 20/10/09 05:18:13.328 main DEBUG Client: >client token: N/A >diagnostics: User class threw exception: > org.scalatest.exceptions.TestFailedException: null was not equal to > "testvalue" > at > org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:344) > at > org.scalatest.matchers.should.Matchers$ShouldMethodHelperClass.shouldMatcher(Matchers.scala:6778) > at > org.scalatest.matchers.should.Matchers$AnyShouldWrapper.should(Matchers.scala:6822) > at > org.apache.spark.deploy.yarn.YarnClusterDriverUseSparkHadoopUtilConf$.$anonfun$main$2(YarnClusterSuite.scala:383) > at > scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) > at > scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) > at > org.apache.spark.deploy.yarn.YarnClusterDriverUseSparkHadoopUtilConf$.main(YarnClusterSuite.scala:382) > at > org.apache.spark.deploy.yarn.YarnClusterDriverUseSparkHadoopUtilConf.main(YarnClusterSuite.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at >
[jira] [Commented] (SPARK-33104) Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in SparkHadoopUtil`
[ https://issues.apache.org/jira/browse/SPARK-33104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211807#comment-17211807 ] Dongjoon Hyun commented on SPARK-33104: --- Please see https://github.com/apache/spark/pull/29286#issuecomment-706630303 > Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in > SparkHadoopUtil` > > > Key: SPARK-33104 > URL: https://issues.apache.org/jira/browse/SPARK-33104 > Project: Spark > Issue Type: Bug > Components: Tests >Affects Versions: 3.1.0 >Reporter: Dongjoon Hyun >Priority: Major > > - > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/1377/testReport/org.apache.spark.deploy.yarn/YarnClusterSuite/yarn_cluster_should_respect_conf_overrides_in_SparkHadoopUtil__SPARK_16414__SPARK_23630_/ > {code} > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN DefaultContainerExecutor: > Exit code from container container_1602245728426_0006_02_01 is : 15 > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN DefaultContainerExecutor: > Exception from container-launch with container ID: > container_1602245728426_0006_02_01 and exit code: 15 > ExitCodeException exitCode=15: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:585) > at org.apache.hadoop.util.Shell.run(Shell.java:482) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN ContainerLaunch: Container > exited with a non-zero exit code 15 > 20/10/09 05:18:13.237 AsyncDispatcher event handler WARN NMAuditLogger: > USER=jenkins OPERATION=Container Finished - Failed TARGET=ContainerImpl > RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE > APPID=application_1602245728426_0006 > CONTAINERID=container_1602245728426_0006_02_01 > 20/10/09 05:18:13.244 Socket Reader #1 for port 37112 INFO Server: Auth > successful for appattempt_1602245728426_0006_02 (auth:SIMPLE) > 20/10/09 05:18:13.326 IPC Parameter Sending Thread #0 DEBUG Client: IPC > Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins sending #37 > 20/10/09 05:18:13.327 IPC Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins DEBUG Client: IPC > Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins got value #37 > 20/10/09 05:18:13.328 main DEBUG ProtobufRpcEngine: Call: > getApplicationReport took 2ms > 20/10/09 05:18:13.328 main INFO Client: Application report for > application_1602245728426_0006 (state: FINISHED) > 20/10/09 05:18:13.328 main DEBUG Client: >client token: N/A >diagnostics: User class threw exception: > org.scalatest.exceptions.TestFailedException: null was not equal to > "testvalue" > at > org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:344) > at > org.scalatest.matchers.should.Matchers$ShouldMethodHelperClass.shouldMatcher(Matchers.scala:6778) > at > org.scalatest.matchers.should.Matchers$AnyShouldWrapper.should(Matchers.scala:6822) > at > org.apache.spark.deploy.yarn.YarnClusterDriverUseSparkHadoopUtilConf$.$anonfun$main$2(YarnClusterSuite.scala:383) > at > scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) > at > scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) > at > org.apache.spark.deploy.yarn.YarnClusterDriverUseSparkHadoopUtilConf$.main(YarnClusterSuite.scala:382) > at > org.apache.spark.deploy.yarn.YarnClusterDriverUseSparkHadoopUtilConf.main(YarnClusterSuite.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[jira] [Commented] (SPARK-33104) Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in SparkHadoopUtil`
[ https://issues.apache.org/jira/browse/SPARK-33104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211806#comment-17211806 ] Dongjoon Hyun commented on SPARK-33104: --- It turns out that Hadoop 2.7 / Hive 2.3 combination SBT build is broken by SPARK-21708 . - https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/ *BEFORE THIS COMMIT* {code} $ build/sbt "yarn/testOnly *.YarnClusterSuite -- -z SparkHadoopUtil" -Pyarn -Phadoop-2.7 -Phive -Phive-2.3 ... [info] YarnClusterSuite: [info] - yarn-cluster should respect conf overrides in SparkHadoopUtil (SPARK-16414, SPARK-23630) (12 seconds, 193 milliseconds) [info] ScalaTest [info] Run completed in 29 seconds, 774 milliseconds. [info] Total number of tests run: 1 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. [info] Passed: Total 1, Failed 0, Errors 0, Passed 1 [success] Total time: 122 s, completed Oct 10, 2020 5:21:55 PM ~s:23afc930ae $ git log --oneline -n1 23afc930ae (HEAD) [SPARK-26499][SQL][FOLLOWUP] Print the loading provider exception starting from the INFO level {code} *AFTER THIS COMMIT* {code} $ build/sbt "yarn/testOnly *.YarnClusterSuite -- -z SparkHadoopUtil" -Pyarn -Phadoop-2.7 -Phive -Phive-2.3 ... [info] *** 1 TEST FAILED *** [error] Failed: Total 1, Failed 1, Errors 0, Passed 0 [error] Failed tests: [error] org.apache.spark.deploy.yarn.YarnClusterSuite [error] (yarn / Test / testOnly) sbt.TestsFailedException: Tests unsuccessful [error] Total time: 132 s (02:12), completed Oct 10, 2020 5:28:29 PM {code} > Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in > SparkHadoopUtil` > > > Key: SPARK-33104 > URL: https://issues.apache.org/jira/browse/SPARK-33104 > Project: Spark > Issue Type: Bug > Components: Tests >Affects Versions: 3.1.0 >Reporter: Dongjoon Hyun >Priority: Major > > - > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/1377/testReport/org.apache.spark.deploy.yarn/YarnClusterSuite/yarn_cluster_should_respect_conf_overrides_in_SparkHadoopUtil__SPARK_16414__SPARK_23630_/ > {code} > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN DefaultContainerExecutor: > Exit code from container container_1602245728426_0006_02_01 is : 15 > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN DefaultContainerExecutor: > Exception from container-launch with container ID: > container_1602245728426_0006_02_01 and exit code: 15 > ExitCodeException exitCode=15: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:585) > at org.apache.hadoop.util.Shell.run(Shell.java:482) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN ContainerLaunch: Container > exited with a non-zero exit code 15 > 20/10/09 05:18:13.237 AsyncDispatcher event handler WARN NMAuditLogger: > USER=jenkins OPERATION=Container Finished - Failed TARGET=ContainerImpl > RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE > APPID=application_1602245728426_0006 > CONTAINERID=container_1602245728426_0006_02_01 > 20/10/09 05:18:13.244 Socket Reader #1 for port 37112 INFO Server: Auth > successful for appattempt_1602245728426_0006_02 (auth:SIMPLE) > 20/10/09 05:18:13.326 IPC Parameter Sending Thread #0 DEBUG Client: IPC > Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins sending #37 > 20/10/09 05:18:13.327 IPC Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins DEBUG Client: IPC > Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins got value #37 > 20/10/09 05:18:13.328 main DEBUG ProtobufRpcEngine: Call: > getApplicationReport took 2ms > 20/10/09 05:18:13.328 main INFO Client: Application report for >
[jira] [Commented] (SPARK-33104) Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in SparkHadoopUtil`
[ https://issues.apache.org/jira/browse/SPARK-33104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17211563#comment-17211563 ] Yang Jie commented on SPARK-33104: -- Is this an inevitable problem? `mvn test` can pass, may need to add some log to determine the file loading path of `core-site.xml` > Fix `YarnClusterSuite.yarn-cluster should respect conf overrides in > SparkHadoopUtil` > > > Key: SPARK-33104 > URL: https://issues.apache.org/jira/browse/SPARK-33104 > Project: Spark > Issue Type: Bug > Components: Tests >Affects Versions: 3.1.0 >Reporter: Dongjoon Hyun >Priority: Major > > - > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7-hive-2.3/1377/testReport/org.apache.spark.deploy.yarn/YarnClusterSuite/yarn_cluster_should_respect_conf_overrides_in_SparkHadoopUtil__SPARK_16414__SPARK_23630_/ > {code} > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN DefaultContainerExecutor: > Exit code from container container_1602245728426_0006_02_01 is : 15 > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN DefaultContainerExecutor: > Exception from container-launch with container ID: > container_1602245728426_0006_02_01 and exit code: 15 > ExitCodeException exitCode=15: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:585) > at org.apache.hadoop.util.Shell.run(Shell.java:482) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 20/10/09 05:18:13.211 ContainersLauncher #0 WARN ContainerLaunch: Container > exited with a non-zero exit code 15 > 20/10/09 05:18:13.237 AsyncDispatcher event handler WARN NMAuditLogger: > USER=jenkins OPERATION=Container Finished - Failed TARGET=ContainerImpl > RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE > APPID=application_1602245728426_0006 > CONTAINERID=container_1602245728426_0006_02_01 > 20/10/09 05:18:13.244 Socket Reader #1 for port 37112 INFO Server: Auth > successful for appattempt_1602245728426_0006_02 (auth:SIMPLE) > 20/10/09 05:18:13.326 IPC Parameter Sending Thread #0 DEBUG Client: IPC > Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins sending #37 > 20/10/09 05:18:13.327 IPC Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins DEBUG Client: IPC > Client (1123559518) connection to > amp-jenkins-worker-04.amp/192.168.10.24:43090 from jenkins got value #37 > 20/10/09 05:18:13.328 main DEBUG ProtobufRpcEngine: Call: > getApplicationReport took 2ms > 20/10/09 05:18:13.328 main INFO Client: Application report for > application_1602245728426_0006 (state: FINISHED) > 20/10/09 05:18:13.328 main DEBUG Client: >client token: N/A >diagnostics: User class threw exception: > org.scalatest.exceptions.TestFailedException: null was not equal to > "testvalue" > at > org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:344) > at > org.scalatest.matchers.should.Matchers$ShouldMethodHelperClass.shouldMatcher(Matchers.scala:6778) > at > org.scalatest.matchers.should.Matchers$AnyShouldWrapper.should(Matchers.scala:6822) > at > org.apache.spark.deploy.yarn.YarnClusterDriverUseSparkHadoopUtilConf$.$anonfun$main$2(YarnClusterSuite.scala:383) > at > scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) > at > scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) > at > org.apache.spark.deploy.yarn.YarnClusterDriverUseSparkHadoopUtilConf$.main(YarnClusterSuite.scala:382) > at > org.apache.spark.deploy.yarn.YarnClusterDriverUseSparkHadoopUtilConf.main(YarnClusterSuite.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at >