FMX commented on code in PR #3485:
URL: https://github.com/apache/celeborn/pull/3485#discussion_r2370829131
##########
client/src/main/java/org/apache/celeborn/client/read/DfsPartitionReader.java:
##########
@@ -172,10 +172,13 @@ public DfsPartitionReader(
this.startChunkIndex,
this.endChunkIndex,
chunkOffsets);
+
if (this.numChunks > 0) {
- fetchThread =
- ThreadUtils.newDaemonSingleThreadExecutor(
- "celeborn-client-dfs-partition-fetcher" +
location.getStorageInfo().getFilePath());
+ this.minFetchInFlight = Math.min(conf.clientFetchMaxReqsInFlight(),
numChunks);
+ this.fetchExecutor =
+ ThreadUtils.newDaemonFixedThreadPool(
Review Comment:
This is a partition reader which means that there will be many instances of
this class and this is executed in the executor process. I think you can add a
new config for this thread pool otherwise this may consume too many threads
especially when the prefetch is enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
