difin commented on code in PR #5123:
URL: https://github.com/apache/hive/pull/5123#discussion_r1568989334
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/compaction/IcebergMajorQueryCompactor.java:
##########
@@ -44,22 +51,69 @@ public boolean run(CompactorContext context) throws
IOException, HiveException,
Map<String, String> tblProperties = context.getTable().getParameters();
LOG.debug("Initiating compaction for the {} table", compactTableName);
- String compactionQuery = String.format("insert overwrite table %s select *
from %<s",
- compactTableName);
+ HiveConf conf = new HiveConf(context.getConf());
+ String partSpec = context.getCompactionInfo().partName;
+ String compactionQuery;
+
+ if (partSpec == null) {
+ HiveConf.setVar(conf, ConfVars.REWRITE_POLICY,
RewritePolicy.ALL_PARTITIONS.name());
+ compactionQuery = String.format("insert overwrite table %s select * from
%<s", compactTableName);
+ } else {
+ Table table = IcebergTableUtil.getTable(conf, context.getTable());
+ PartitionData partitionData = DataFiles.data(table.spec(), partSpec);
+ compactionQuery = String.format("insert overwrite table %1$s
partition(%2$s) select * from %1$s where %3$s",
+ compactTableName, partDataToSQL(partitionData, partSpec, ","),
+ partDataToSQL(partitionData, partSpec, " and "));
+ }
+
+ SessionState sessionState = setupQueryCompactionSession(conf,
context.getCompactionInfo(), tblProperties);
+ sessionState.setConf(conf);
Review Comment:
When the cloned conf is passed to the session, and the session is not null,
the conf is not set in the session and any changes defined in it get lost. I
found it during debugging when trying to understand why in a q-test that has 2
compaction calls the second compaction command is not working.
```
public static SessionState setUpSessionState(HiveConf conf, String user,
boolean doStart) {
SessionState sessionState = SessionState.get();
if (sessionState == null) {
// Note: we assume that workers run on the same threads repeatedly, so
we can set up
// the session here and it will be reused without explicitly
storing in the worker.
sessionState = new SessionState(conf, user);
if (doStart) {
// TODO: Required due to SessionState.getHDFSSessionPath. Why wasn't
it required before?
sessionState.setIsHiveServerQuery(true);
SessionState.start(sessionState);
}
SessionState.setCurrentSessionState(sessionState);
}
return sessionState;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
