abstractdog commented on PR #6343: URL: https://github.com/apache/hive/pull/6343#issuecomment-4023522660
> hive.server2.tez.external.sessions.namespace > Hi @abstractdog , i tested this patch on my local setup (hadoop, hive, TEZ-4682 branch and Zookeeper - installed via brew) and **its working**🚀. Ran a basic insert command > > Attaching the HS2 logs and tez_am docker logs > > [tez-am-working.log](https://github.com/user-attachments/files/25818516/tez-am-working.log) [working_hs2.log](https://github.com/user-attachments/files/25818518/working_hs2.log) > > **Few things to note:** > > 1. Add the following in `/etc/hosts/` file. this is necesary for docker to communicate with hdfs running on localhost > `127.0.0.1 host.docker.internal` > 2. tez-am docker image needs hive-exec jar i.e resorce localization. For now I added it using the plugin directory way i.e. `-v "/Users/raghav/Desktop/plugin:/opt/tez/plugins" \` and plugin directory contains only hive-exec jar otherwise . > > ``` > Vertex vertex_1769280834537_0000_1_01 [Reducer 2] killed/failed due to: INIT_FAILURE] .... > vertex=vertex_1769280834537_0000_1_00 [Map 1], org.apache.tez.dag.api .TezUncheckedException: java. lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.ha > doop.hive.ql.io. CombineHiveInputFormat not found > ``` > > 5. `core-site.xml` > > ``` > <property> > <name>fs.defaultFS</name> > <value>hdfs://host.docker.internal:9000</value> > </property> > ``` > > 6. `hdfs-site.xml` Connecting docker AM to datanode was really problematic, took lot of time 😅 > > ``` > <property> > <name>dfs.datanode.use.datanode.hostname</name> > <value>true</value> > </property> > > <property> > <name>dfs.datanode.address</name> > <value>host.docker.internal:9866</value> > </property> > > <property> > <name>dfs.datanode.hostname</name> > <value>host.docker.internal</value> > </property> > ``` > > 7. In `docker tez-am tez-site.xml ` (Will update the PR for this as well in TEZ-4682) > > ``` > <property> > <name>dfs.client.use.datanode.hostname</name> > <value>true</value> > </property> > ``` > > 8. Some `--add-opens` in tez-entrypoint.sh needs to be updated. Basically hive project --add-opens are also required. Will update the PR for TEZ-4682. > 9. `hive-site.xml` > > ``` > <property> > <name>hive.server2.use.external.sessions</name> > <value>true</value> > </property> > > <property> > <name>hive.server2.tez.external.sessions.namespace</name> > <value>/tez-external-sessions</value> > </property> > > <property> > <name>hive.server2.tez.external.sessions.registry.class</name> > <value>org.apache.hadoop.hive.ql.exec.tez.ZookeeperExternalSessionsRegistryClient</value> > </property> > > <property> > <name>hive.zookeeper.quorum</name> > <value>localhost:2181</value> > </property> > ``` > > **NOTE/IMPORTANT**: There was a flaky behaviour observed with `tez-conf.pb` in tez-staging directory in hdfs. It was throwing error for `No file found`. Not sure why it was occuring but it was fixed after some time automatically and not facing it again. Will post here if I face it again. > > <img alt="Screenshot 2026-03-08 at 2 38 12 AM" width="762" height="387" src="https://private-user-images.githubusercontent.com/49618756/559835670-92ee985e-d1d2-4691-b5ca-53013bcbbd77.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzMwNTMyODYsIm5iZiI6MTc3MzA1Mjk4NiwicGF0aCI6Ii80OTYxODc1Ni81NTk4MzU2NzAtOTJlZTk4NWUtZDFkMi00NjkxLWI1Y2EtNTMwMTNiY2JiZDc3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAzMDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMzA5VDEwNDMwNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQzODk0OTZiYjNhNmQ1MWU5NzM5ZTA2NTQzZGM1OTlkM2I1NTliMGZlOTUwZTcxYjhhNmQxMTljMjJjMGY2MGQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.UPL300XjZ8GR9-0otEjLrUteLmUs_SEIpSiNE63910w";> @Aggarwal-Raghav : this is awesome, thanks for testing! let me share the action items I can think of here to make this and TEZ-4682 happen: 1. /etc/hosts workaround: external setup steps, I wish we could get rid of them, but maybe it's crucial for the first iteration: I believe hive docker page can iterate about this: https://hive.apache.org/docs/latest/admin/setting-up-hive-with-docker/ 2. tez-plugins folder: that's awesome, properly documented on TEZ-4682 side (if it's not enough, we can fully make it work out of the box in the scope of [HIVE-29419](https://issues.apache.org/jira/browse/HIVE-29419) 5/6: core-site.xml + hdfs-site.changes: these are must-have items on the hive site, however it would be better if I can have the hdfs setup and these xml configs at the same place, otherwise, the values might seem a bit vulnerable in a sense that no one knows under what circumstances `host.docker.internal:9866` and `hdfs://host.docker.internal:9000` are valid and working, this is what might be addressed by [HIVE-29493](https://issues.apache.org/jira/browse/HIVE-29493) 8. hive-related add-opens to tez config: in general, tez should not contain hive related stuff given that tez doesn't depend on hive, however, in case of add-opens args, I feel it might be beneficial to have them there as long as those don't refer to anything "hive", only java packages 9. hive-site.xml config: definitely something to be included here in this PR or a related one -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
![https://private-user-images.githubusercontent.com/49618756/559835670-92ee985e-d1d2-4691-b5ca-53013bcbbd77.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzMwNTMyODYsIm5iZiI6MTc3MzA1Mjk4NiwicGF0aCI6Ii80OTYxODc1Ni81NTk4MzU2NzAtOTJlZTk4NWUtZDFkMi00NjkxLWI1Y2EtNTMwMTNiY2JiZDc3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAzMDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMzA5VDEwNDMwNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQzODk0OTZiYjNhNmQ1MWU5NzM5ZTA2NTQzZGM1OTlkM2I1NTliMGZlOTUwZTcxYjhhNmQxMTljMjJjMGY2MGQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.UPL300XjZ8GR9-0otEjLrUteLmUs_SEIpSiNE63910w"](https://private-user-images.githubusercontent.com/49618756/559835670-92ee985e-d1d2-4691-b5ca-53013bcbbd77.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzMwNTMyODYsIm5iZiI6MTc3MzA1Mjk4NiwicGF0aCI6Ii80OTYxODc1Ni81NTk4MzU2NzAtOTJlZTk4NWUtZDFkMi00NjkxLWI1Y2EtNTMwMTNiY2JiZDc3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAzMDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMzA5VDEwNDMwNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQzODk0OTZiYjNhNmQ1MWU5NzM5ZTA2NTQzZGM1OTlkM2I1NTliMGZlOTUwZTcxYjhhNmQxMTljMjJjMGY2MGQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.UPL300XjZ8GR9-0otEjLrUteLmUs_SEIpSiNE63910w"){kind=link}