zuston commented on issue #2569: URL: https://github.com/apache/uniffle/issues/2569#issuecomment-3173475614
> I saw how TaskShuffleWriteTimes works, correct me if I wrong, it sends writes statistics to Coordinator server with gRPC once its has finished and this statistic are sent fromRssShuffleWriter. Similarly, TaskShuffleReadMetric in RssShuffleReader. But RssShuffleDataIterator it's another phase in the reading process that iteratively gets blocks processed from Uniffle Server. But here, we are not tracking fecthWaitTime and more important metrics to Coordinator, so proposal here is maybe enable another grpc method, to send this metric the same way as the other two? Not precise, the shuffle write times will be collected in the spark executor shuffleWriter side, and then will be sent to the spark driver's UniffleShuffleManager, and then all the metrics will be aggregated in the `UniffleListener`, after that, the metrics will be shown in the spark Uniffle UI tab, like the following screenshot. <img width="1914" height="618" alt="Image" src="https://github.com/user-attachments/assets/7cc31de9-2ea1-408e-85d7-16bb7e4cada1"; /> And so, for the shuffle read times, I hope the following phase could be collected 1. Uncompression times (this may be as the majority part of the whole progress) 2. Deser time 3. Fetch wait time 4. buffer copy times 5. ... The shuffle read times metrics aggregation logic is like the write times, so you can follow the same style. BTW, the grpc transfer could be reused by `ShuffleManagerGrpcService.reportShuffleReadMetric` , the thing you should do is that to extend the existing protobuf definition . Feel free to reach me if you have any questions @SalvadorRomo -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
