devmadhuu commented on code in PR #10133:
URL: https://github.com/apache/ozone/pull/10133#discussion_r3151790350


##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/TarExtractor.java:
##########
@@ -163,8 +162,7 @@ private void writeFile(Path outputDir, String fileName, 
byte[] fileData) {
 
   public void start() {
     if (executorServiceStarted.compareAndSet(false, true)) {
-      this.executor =
-          new ThreadPoolExecutor(0, threadPoolSize, 60L, TimeUnit.SECONDS, new 
LinkedBlockingQueue<>(), threadFactory);
+      this.executor = Executors.newFixedThreadPool(threadPoolSize, 
threadFactory);

Review Comment:
   Just a thought: The pool size is `Math.max(64, 
Runtime.getRuntime().availableProcessors())` at the call site in 
`OzoneManagerServiceProviderImpl`. On high-CPU machines this could be large. 
With the old  code it barely mattered; now it will actually create that many 
threads. On a 256-core host that's 256 threads all blocking on I/O at the same 
time. Consider documenting or probably IMO we should be capping this.



##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/api/DataNodeMetricsService.java:
##########
@@ -97,13 +96,10 @@ public DataNodeMetricsService(
     this.metricsServiceProviderFactory = metricsServiceProviderFactory;
     this.lastCollectionEndTime.set(-minimumApiDelayMs);
     int corePoolSize = Runtime.getRuntime().availableProcessors() * 2;
-    this.executorService = new ThreadPoolExecutor(
-        corePoolSize, MAX_POOL_SIZE,
-        KEEP_ALIVE_TIME, TimeUnit.SECONDS,
-        new LinkedBlockingQueue<>(),
-        new ThreadFactoryBuilder()
-            .setNameFormat("DataNodeMetricsCollector-%d")
-            .build());
+    ThreadFactory threadFactory = new ThreadFactoryBuilder()

Review Comment:
   The original pool was intended (if misconfigured) to scale up to 500 threads 
under load (e.g., Recon polling thousands of DataNodes simultaneously). The fix 
correctly reflects actual behavior today, but it also avoiding efficient future 
scaling. If the intent was truly to allow bursting to a larger pool, can we 
think of using a bounded queue rather than simply removing it. At minimum, a 
comment explaining the corePoolSize choice and impact on removing the 
MAX_POOL_SIZE = 500.
   
   Bu wait,  this could make it slower in large cluster having hundreds of 
datanodes where it makes concurrent DataNode queries. The original 
MAX_POOL_SIZE = 500 captured the right intent — it was just broken by the 
unbounded LinkedBlockingQueue. A better fix would be to make the pool size 
configurable with a sensible default:
   
   ```
   // e.g., add a config key OZONE_RECON_DN_METRICS_THREAD_COUNT, default 128 
or min(nodeCount, 500)
   int poolSize = config.getInt(OZONE_RECON_DN_METRICS_THREAD_COUNT, 128);
   this.executorService = Executors.newFixedThreadPool(poolSize, threadFactory);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to