Re: [PR] TEZ-4682: [Cloud] Tez AM docker image [tez]

Mon, 23 Mar 2026 05:18:47 -0700 


abstractdog commented on PR #456:
URL: https://github.com/apache/tez/pull/456#issuecomment-4110201982

   > 1. I have used official hadoop docker image 3.4.2-lean
   > 2. namenode and datanode will be separate service otherwise we need to use 
a custom entrypoint to start in a single container as happening in 
minimal-hadoop docker image. I think it is better to have separate. I have 
added a wait also to ensure namenode is not in safemode before datanode and 
tez-am is up.
   > 3. Handled the review comments regarding docs and file naming conventions.
   > 4. Added a sample program to run in tez-docker am and here it get tricky 
⚠️. It is throwing same error as mentioned in TEZ-4686 with standalone program.
   >    Steps:
   > 
   > ```
   > 1. cd tez-dist/src/docker/tez-am/
   > 2. docker compose up -d --build
   > ```
   > 
   > <img alt="Screenshot 2026-03-20 at 12 26 15 AM" width="1231" height="278" 
src="https://private-user-images.githubusercontent.com/49618756/566467592-4019dec9-6ee0-48c8-ab44-d42677281ad6.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzQyNjc0OTcsIm5iZiI6MTc3NDI2NzE5NywicGF0aCI6Ii80OTYxODc1Ni81NjY0Njc1OTItNDAxOWRlYzktNmVlMC00OGM4LWFiNDQtZDQyNjc3MjgxYWQ2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAzMjMlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMzIzVDExNTk1N1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWZlNjhkNDFhMDQwN2EyMzAwNDI4NWI1MWIxOTljM2Q1ZjIwYzIxMWIwMWZkZTQxNzFiMzJjN2Y5ZDBhMDdkMzgmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.gpSz1iPbTtcPYu2ytLP3MuWuLuBZDuaX41P_0tpbspE";>
   > **Everything should be running at this point**
   > 
   > ```
   > 3. docker exec -it tez-am bash
   > 4. echo "Hello world Hello" > /tmp/input.txt
   > 5. java -cp ./*:./lib/*:tez-examples-1.0.0-SNAPSHOT.jar 
org.apache.tez.examples.ExternalAmWordCount /tmp/input.txt /tmp/output
   > ```
   > 
   > **With TEZ-4686 cherrypick:**
   > 
   > ```
   > ❯ docker exec -it tez-am bash
   > bash-5.1$ echo "Hello world Hello" > /tmp/input.txt
   > java -cp ./*:./lib/*:tez-examples-1.0.0-SNAPSHOT.jar 
org.apache.tez.examples.ExternalAmWordCount /tmp/input.txt /tmp/output
   > log4j:WARN No appenders could be found for logger 
(org.apache.hadoop.util.Shell).
   > log4j:WARN Please initialize the log4j system properly.
   > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for 
more info.
   > bash-5.1$ ls /tmp/output
   > attempt_1773945902335_0000_1_00_000000_0_10003     
attempt_1773945902335_0000_1_00_000000_0_10003_0  part-v001-o000-r-00000  
_SUCCESS
   > bash-5.1$ cat /tmp/output/part-v001-o000-r-00000
   > Hello      2
   > world      1
   > bash-5.1$
   > ```
   > 
   > **Without TEZ-4686 cherrypick:**
   > 
   > ```
   > ❯ docker exec -it tez-am bash
   > bash-5.1$ echo "Hello world Hello" > /tmp/input.txt
   > java -cp ./*:./lib/*:tez-examples-1.0.0-SNAPSHOT.jar 
org.apache.tez.examples.ExternalAmWordCount /tmp/input.txt /tmp/output
   > log4j:WARN No appenders could be found for logger 
(org.apache.hadoop.util.Shell).
   > log4j:WARN Please initialize the log4j system properly.
   > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for 
more info.
   > Exception in thread "main" java.lang.NullPointerException: Cannot invoke 
"org.apache.tez.client.registry.AMRecord.getApplicationId()" because 
"this.amRecord" is null
   >    at 
org.apache.tez.client.registry.zookeeper.ZkFrameworkClient.createApplication(ZkFrameworkClient.java:114)
   >    at 
org.apache.tez.client.TezClient.createApplication(TezClient.java:1103)
   >    at org.apache.tez.client.TezClient.start(TezClient.java:399)
   >    at 
org.apache.tez.examples.ExternalAmWordCount.main(ExternalAmWordCount.java:74)
   > ```
   
   looks good @Aggarwal-Raghav : the NPE is because the example program uses 
`TezClient.start`, which eventually leads to a `createApplication` call instead 
of discovering one from the registry, this is what's described on 
[HIVE-29477](https://issues.apache.org/jira/browse/HIVE-29477) and workarounded 
like this:
   
https://github.com/apache/hive/pull/6343/changes#diff-acfe9151d94581378224e8c415c3eb8eedbfa2c5118c1ac3a1945a2146c62802R104-R118
   
   in the long run we can make `TezClient.start` to magically handle this, but 
in the meantime, to unblock this PR, the example program can utilize TezClient 
the same way the above Hive class does: if you can make this work, this PR is 
ready
   
   other than that, left minor comments regarding the example program


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to