devmadhuu commented on code in PR #10133:
URL: https://github.com/apache/ozone/pull/10133#discussion_r3151790350
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/TarExtractor.java:
##########
@@ -163,8 +162,7 @@ private void writeFile(Path outputDir, String fileName,
byte[] fileData) {
public void start() {
if (executorServiceStarted.compareAndSet(false, true)) {
- this.executor =
- new ThreadPoolExecutor(0, threadPoolSize, 60L, TimeUnit.SECONDS, new
LinkedBlockingQueue<>(), threadFactory);
+ this.executor = Executors.newFixedThreadPool(threadPoolSize,
threadFactory);
Review Comment:
Just a thought: The pool size is `Math.max(64,
Runtime.getRuntime().availableProcessors())` at the call site in
`OzoneManagerServiceProviderImpl`. On high-CPU machines this could be large.
With the old code it barely mattered; now it will actually create that many
threads. On a 256-core host that's 256 threads all blocking on I/O at the same
time. Consider documenting or probably IMO we should be capping this.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/DataNodeMetricsService.java:
##########
@@ -97,13 +96,10 @@ public DataNodeMetricsService(
this.metricsServiceProviderFactory = metricsServiceProviderFactory;
this.lastCollectionEndTime.set(-minimumApiDelayMs);
int corePoolSize = Runtime.getRuntime().availableProcessors() * 2;
- this.executorService = new ThreadPoolExecutor(
- corePoolSize, MAX_POOL_SIZE,
- KEEP_ALIVE_TIME, TimeUnit.SECONDS,
- new LinkedBlockingQueue<>(),
- new ThreadFactoryBuilder()
- .setNameFormat("DataNodeMetricsCollector-%d")
- .build());
+ ThreadFactory threadFactory = new ThreadFactoryBuilder()
Review Comment:
The original pool was intended (if misconfigured) to scale up to 500 threads
under load (e.g., Recon polling thousands of DataNodes simultaneously). The fix
correctly reflects actual behavior today, but it also avoiding efficient future
scaling. If the intent was truly to allow bursting to a larger pool, can we
think of using a bounded queue rather than simply removing it. At minimum, a
comment explaining the corePoolSize choice and impact on removing the
MAX_POOL_SIZE = 500.
Bu wait, this could make it slower in large cluster having hundreds of
datanodes where it makes concurrent DataNode queries. The original
MAX_POOL_SIZE = 500 captured the right intent — it was just broken by the
unbounded LinkedBlockingQueue. A better fix would be to make the pool size
configurable with a sensible default:
```
// e.g., add a config key OZONE_RECON_DN_METRICS_THREAD_COUNT, default 128
or min(nodeCount, 500)
int poolSize = config.getInt(OZONE_RECON_DN_METRICS_THREAD_COUNT, 128);
this.executorService = Executors.newFixedThreadPool(poolSize, threadFactory);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]