saurabhd336 commented on code in PR #3132: URL: https://github.com/apache/celeborn/pull/3132#discussion_r1986678373
########## common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala: ########## @@ -982,6 +982,8 @@ class CelebornConf(loadDefaults: Boolean) extends Cloneable with Logging with Se def clientFetchTimeoutMs: Long = get(CLIENT_FETCH_TIMEOUT) def clientFetchBufferSize: Int = get(CLIENT_FETCH_BUFFER_SIZE).toInt def clientFetchMaxReqsInFlight: Int = get(CLIENT_FETCH_MAX_REQS_IN_FLIGHT) + def isWorkerPartitionReaderCheckpointEnabled: Boolean = + get(WORKER_PARTITION_READER_CHECKPOINT_ENABLE) Review Comment: @zaynt4606 It is possible to fallback to reading from the same server even when replication is enabled. Eg: when `celeborn.client.adaptive.optimizeSkewedPartitionRead.enabled` is set to true and partitions are being split. I'd let `CelebornInputStream` itself decide whether or not to restore checkpoint when creating a reader -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
