alexdongli0829 commented on PR #422:
URL: https://github.com/apache/tez/pull/422#issuecomment-3043334805
Integration test on cluster
Prepare the DAG which is big enough (adjust the IPC limit
"ipc.maximum.data.length" if necessary)
Before the fix
```
2025-06-30T10:18:18,760 INFO [ce4666f9-a278-4f15-be97-ae59b727e14b
main([])]: client.TezClient (:()) - Send dag plan using YARN local resources
since it's too large, dag plan size=385547, max dag plan size through
IPC=128974848, max IPC message size= 134217728
2025-06-30T10:18:18,809 INFO [ce4666f9-a278-4f15-be97-ae59b727e14b
main([])]: exec.Task (:()) - Dag submit failed due to DestHost:destPort
ip-172-31-93-189.ec2.internal:8020 , LocalHost:localPort
ip-172-31-93-68.ec2.internal/172.31.93.68:0. Failed on local exception:
java.io.IOException: org.apache.hadoop.security.AccessControlException: Client
cannot authenticate via:[TOKEN, KERBEROS]
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at
java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:964)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:939)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1679)
at org.apache.hadoop.ipc.Client.call(Client.java:1620)
at org.apache.hadoop.ipc.Client.call(Client.java:1517)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:258)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:139)
```
After the fix
```
2025-06-30T10:34:52,975 INFO [8f4c8a93-f6a9-4d6d-a813-cb946649815a
main([])]: client.TezClient (:()) - Send dag plan using YARN local resources
since it's too large, dag plan size=389516, max dag plan size through
IPC=128974848, max IPC message size= 134217728
2025-06-30T10:34:53,171 INFO [8f4c8a93-f6a9-4d6d-a813-cb946649815a
main([])]: client.FrameworkClient (:()) - Submitted dag to TezSession,
sessionName=HIVE-8f4c8a93-f6a9-4d6d-a813-cb946649815a,
applicationId=application_1751278111719_0008, dagId=dag_1751278111719_0008_1,
dagName=select count(*) from drone_orders where ...0 (Stage-1)
2025-06-30T10:34:53,490 INFO [8f4c8a93-f6a9-4d6d-a813-cb946649815a
main([])]: SessionState (:()) - Status: Running (Executing on YARN cluster with
App id application_1751278111719_0008)
2025-06-30T10:34:53,505 INFO [8f4c8a93-f6a9-4d6d-a813-cb946649815a
main([])]: SessionState (:()) - Map 1: -/- Reducer 2: 0/1
2025-06-30T10:34:56,532 INFO [8f4c8a93-f6a9-4d6d-a813-cb946649815a
main([])]: SessionState (:()) - Map 1: -/- Reducer 2: 0/1
2025-06-30T10:34:57,541 INFO [8f4c8a93-f6a9-4d6d-a813-cb946649815a
main([])]: SessionState (:()) - Map 1: 0/54 Reducer 2: 0/1
2025-06-30T10:35:00,565 INFO [8f4c8a93-f6a9-4d6d-a813-cb946649815a
main([])]: SessionState (:()) - Map 1: 0/54 Reducer 2: 0/1
2025-06-30T10:35:01,070 INFO [8f4c8a93-f6a9-4d6d-a813-cb946649815a
main([])]: SessionState (:()) - Map 1: 0(+3)/54 Reducer 2: 0/1
2025-06-30T10:35:02,084 INFO [8f4c8a93-f6a9-4d6d-a813-cb946649815a
main([])]: SessionState (:()) - Map 1: 0(+5)/54 Reducer 2: 0/1
2025-06-30T10:35:02,589 INFO [8f4c8a93-f6a9-4d6d-a813-cb946649815a
main([])]: SessionState (:()) - Map 1: 0(+7)/54 Reducer 2: 0/1
2025-06-30T10:35:03,598 INFO [8f4c8a93-f6a9-4d6d-a813-cb946649815a
main([])]: SessionState (:()) - Map 1: 0(+9)/54 Reducer 2: 0/1
2025-06-30T10:35:04,103 INFO [8f4c8a93-f6a9-4d6d-a813-cb946649815a
main([])]: SessionState (:()) - Map 1: 0(+11)/54 Reducer 2: 0/1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
