xumanbu commented on issue #791: URL: https://github.com/apache/incubator-uniffle/issues/791#issuecomment-1495274306
> Why shuffleId with too many partitions will make namenode qps high? Could you help describe more? @xumanbu such as shuffle server have 100 shuffleId, earch shuffleId have 2048 partitions, when shuffle server Memory usage reaches the watermark, and new shuffle partition data write into continuely, so 100 * 2048 * 2(index file and data file) = 409600 files will flush to hdfs in a short time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
