danny0405 commented on code in PR #19812:
URL: https://github.com/apache/hudi/pull/19812#discussion_r3938674169
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java:
##########
@@ -77,27 +77,28 @@ public HiveQueryDDLExecutor(HiveSyncConfig config,
IMetaStoreClient metaStoreCli
this.metaStoreClient = metaStoreClient;
this.driverPool = driverPool;
this.metaStoreClientPool = metaStoreClientPool;
+ // SessionState.start() binds the session it starts to this thread,
displacing the caller's.
+ // Statements and the teardown bind ours themselves, so the thread is
handed back once the
+ // Driver is built -- its constructor is what reads SessionState.get().
+ SessionState previousSession = SessionState.get();
+ ClassLoader previousLoader =
Thread.currentThread().getContextClassLoader();
try {
- this.sessionState = new SessionState(config.getHiveConf(),
+ // Its own conf copy, as HiveDriverPool's workers get: a session stamps
hive.session.id and a
+ // UDFClassLoader onto the conf it is handed, and close() deletes the
directories that id
+ // names and closes that loader. config's HiveConf outlives us and both
pools copy it.
+ HiveConf sessionConf = new HiveConf(config.getHiveConf());
+ // An inherited ID would make close() delete the caller's session
directories.
+ sessionConf.setVar(HiveConf.ConfVars.HIVESESSIONID, "");
+ this.sessionState = new SessionState(sessionConf,
UserGroupInformation.getCurrentUser().getShortUserName());
Review Comment:
[P2] Isolate the downloaded-resource directory as well
The fresh session ID isolates the scratch paths, but
`hive.downloaded.resources.dir` is also copied unchanged. When that setting is
a fixed path (for example `/tmp/app-hive-resources`, without
`${hive.session.id}`), the executor and caller still share it. Hive 2.3.10's
`SessionState.close()` recursively deletes the configured directory itself; it
does not append the session ID. The new teardown therefore removes the live
caller's downloaded JARs/files, which can break subsequent queries that use
those resource paths.
Reproduced at this head with two real sessions: create a resource file in
the caller's configured directory, construct the executor from that caller's
conf, and close it. The caller is restored but its file is deleted. A control
using a `${hive.session.id}_resources` path passes.
Please give the executor a private downloaded-resource directory in its
copied conf before creating the session, and extend the ownership regression
test to assert that caller resources survive as well as the local/HDFS scratch
files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
