[
https://issues.apache.org/jira/browse/HIVE-29015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Butao Zhang updated HIVE-29015:
-------------------------------
Description:
I'm not sure if this is a problem. So creating this issue will facilitate our
discussion.
{*}This is how I conducted the test{*}:
* Hive was deployed on JDK 17, while Hadoop was deployed on JDK 8.
However, when I submitted the Tez task, the Tez am would fail to start with
exception: {color:#de350b}Unrecognized option:
--add-opens=java.base/java.net=ALL-UNNAMED{color}
{code:java}
INFO : Tez session hasn't been created yet. Opening session
ERROR : Failed to execute tez graph.
org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown.
Application application_1749800465429_0005 failed 2 times due to AM Container
for appattempt_1749800465429_0005_000002 exited with exitCode: 1
Failing this attempt.Diagnostics: [2025-06-13 15:46:17.468]Exception from
container-launch.
Container id: container_1749800465429_0005_02_000001
Exit code: 1[2025-06-13 15:46:17.471]Container exited with a non-zero exit code
1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Unrecognized option: --add-opens=java.base/java.net=ALL-UNNAMED
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
[2025-06-13 15:46:17.472]Container exited with a non-zero exit code 1. Error
file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Unrecognized option: --add-opens=java.base/java.net=ALL-UNNAMED
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
For more detailed output, check the application tracking page:
http://127.0.0.1:8188/applicationhistory/app/application_1749800465429_0005
Then click on links to logs of each attempt.
. Failing the application.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:995)
~[tez-api-0.10.5.jar:0.10.5]
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:964)
~[tez-api-0.10.5.jar:0.10.5]
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:484)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:384)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.openInternal(TezSessionPoolSession.java:124)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:250)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.tez.TezTask.ensureSessionHasResources(TezTask.java:481)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:232)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:354)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:327)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:244)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:105)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:345)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:189)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:142)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:137)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:190)
~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:234)
~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:334)
~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at
java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
~[?:?]
at java.base/javax.security.auth.Subject.doAs(Subject.java:439) ~[?:?]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
~[hadoop-common-3.4.1.jar:?]
at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:354)
~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
~[?:?]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
~[?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
~[?:?]
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
~[?:?]
at java.base/java.lang.Thread.run(Thread.java:833) [?:?]
ERROR : FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.tez.TezTask. TezSession has already shutdown.
Application application_1749800465429_0005 failed 2 times due to AM Container
for appattempt_1749800465429_0005_000002 exited with exitCode: 1
{code}
* Hive was deployed on JDK 17, and Hadoop 3.4.1 was also deployed on JDK 17.
{color:#de350b}But If I switch the JDK8 used by Hadoop to JDK17, then the
Tez tasks can run normally.{color}
Due to the actual production environment, users may not readily change the JDK
of Hadoop. Users may prefer to have Hadoop continue to use JDK 8, while Hive
uses JDK 17.
{color:#de350b}So my question is: If Hive is deployed on JDK 17, can we submit
Tez tasks in the Hadoop cluster (with JDK 8)? {color}
> Hive(jdk17) + Hadoop(jdk8) can not submit Tez job
> -------------------------------------------------
>
> Key: HIVE-29015
> URL: https://issues.apache.org/jira/browse/HIVE-29015
> Project: Hive
> Issue Type: Test
> Reporter: Butao Zhang
> Priority: Major
>
> I'm not sure if this is a problem. So creating this issue will facilitate our
> discussion.
>
> {*}This is how I conducted the test{*}:
> * Hive was deployed on JDK 17, while Hadoop was deployed on JDK 8.
> However, when I submitted the Tez task, the Tez am would fail to start with
> exception: {color:#de350b}Unrecognized option:
> --add-opens=java.base/java.net=ALL-UNNAMED{color}
>
> {code:java}
> INFO : Tez session hasn't been created yet. Opening session
> ERROR : Failed to execute tez graph.
> org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown.
> Application application_1749800465429_0005 failed 2 times due to AM Container
> for appattempt_1749800465429_0005_000002 exited with exitCode: 1
> Failing this attempt.Diagnostics: [2025-06-13 15:46:17.468]Exception from
> container-launch.
> Container id: container_1749800465429_0005_02_000001
> Exit code: 1[2025-06-13 15:46:17.471]Container exited with a non-zero exit
> code 1. Error file: prelaunch.err.
> Last 4096 bytes of prelaunch.err :
> Last 4096 bytes of stderr :
> Unrecognized option: --add-opens=java.base/java.net=ALL-UNNAMED
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> [2025-06-13 15:46:17.472]Container exited with a non-zero exit code 1. Error
> file: prelaunch.err.
> Last 4096 bytes of prelaunch.err :
> Last 4096 bytes of stderr :
> Unrecognized option: --add-opens=java.base/java.net=ALL-UNNAMED
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> For more detailed output, check the application tracking page:
> http://127.0.0.1:8188/applicationhistory/app/application_1749800465429_0005
> Then click on links to logs of each attempt.
> . Failing the application.
> at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:995)
> ~[tez-api-0.10.5.jar:0.10.5]
> at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:964)
> ~[tez-api-0.10.5.jar:0.10.5]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:484)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:384)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.openInternal(TezSessionPoolSession.java:124)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:250)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.ensureSessionHasResources(TezTask.java:481)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:232)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:354)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:327)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:244)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:105)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:345)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:189)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:142)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:137)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:190)
> ~[hive-exec-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:234)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:334)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
> ~[?:?]
> at java.base/javax.security.auth.Subject.doAs(Subject.java:439) ~[?:?]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
> ~[hadoop-common-3.4.1.jar:?]
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:354)
> ~[hive-service-4.1.0-SNAPSHOT.jar:4.1.0-SNAPSHOT]
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
> ~[?:?]
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> ~[?:?]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> ~[?:?]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> ~[?:?]
> at java.base/java.lang.Thread.run(Thread.java:833) [?:?]
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.tez.TezTask. TezSession has already shutdown.
> Application application_1749800465429_0005 failed 2 times due to AM Container
> for appattempt_1749800465429_0005_000002 exited with exitCode: 1
> {code}
>
>
> * Hive was deployed on JDK 17, and Hadoop 3.4.1 was also deployed on JDK 17.
> {color:#de350b}But If I switch the JDK8 used by Hadoop to JDK17, then
> the Tez tasks can run normally.{color}
>
>
> Due to the actual production environment, users may not readily change the
> JDK of Hadoop. Users may prefer to have Hadoop continue to use JDK 8, while
> Hive uses JDK 17.
> {color:#de350b}So my question is: If Hive is deployed on JDK 17, can we
> submit Tez tasks in the Hadoop cluster (with JDK 8)? {color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
