marton-bod commented on a change in pull request #2325:
URL: https://github.com/apache/iceberg/pull/2325#discussion_r593312779
##########
File path: hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java
##########
@@ -547,4 +541,42 @@ public void setConf(Configuration conf) {
public Configuration getConf() {
return conf;
}
+
+ @VisibleForTesting
+ HiveClientPool clientPool() {
+ synchronized (CLIENT_POOL_CACHE) {
+ String metastoreUri = conf.get(HiveConf.ConfVars.METASTOREURIS.varname,
"");
+ Pair<HiveClientPool, Long> cacheEntry =
CLIENT_POOL_CACHE.getIfPresent(metastoreUri);
+ HiveClientPool clientPool = cacheEntry == null ? new
HiveClientPool(clientPoolSize, conf) : cacheEntry.first();
+ CLIENT_POOL_CACHE.put(metastoreUri, Pair.of(clientPool,
System.currentTimeMillis() + evictionInterval));
+ return clientPool;
+ }
+ }
+
+ private void scheduleCacheCleaner() {
+ if (cleaner == null) {
+ synchronized (HiveCatalog.class) {
+ if (cleaner == null) {
+ cleaner = Executors.newSingleThreadScheduledExecutor(
+ new ThreadFactoryBuilder()
+ .setDaemon(true)
+
.setNameFormat("iceberg-client-pool-cache-cleaner-%d")
+ .build());
+ }
+ long cleanerInterval = conf.getLong(CACHE_CLEANER_INTERVAL,
CACHE_CLEANER_INTERVAL_DEFAULT);
+ ScheduledFuture<?> futures = cleaner.scheduleWithFixedDelay(() -> {
+ synchronized (CLIENT_POOL_CACHE) {
+ long currentTime = System.currentTimeMillis();
+ CLIENT_POOL_CACHE.asMap().entrySet().stream()
+ .filter(e -> e.getValue().second() <= currentTime)
+ .forEach(e -> {
+ HiveClientPool pool = e.getValue().first();
+ CLIENT_POOL_CACHE.invalidate(e.getKey());
+ pool.close();
Review comment:
If we decide to close the pool here regardless of its usage by any
clients, it's probably worth pointing out in documentation that the pool has an
expiration date and users should not cache it in a variable for too long. For
example:
```
pool = getPool()
// some long running other operation
pool.getTable("table"); // -> FAIL
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
