zuston commented on issue #791: URL: https://github.com/apache/incubator-uniffle/issues/791#issuecomment-1495297662
> > Why shuffleId with too many partitions will make namenode qps high? Could you help describe more? @xumanbu > > such as shuffle server have 100 shuffleId, earch shuffleId have 2048 partitions, when shuffle server Memory usage reaches the watermark, and new shuffle partition data write into continuely, so 100 * 2048 * 2(index file and data file) = 409600 files will flush to hdfs in a short time. This will only occur in `MEMORY_HDFS`, right? So I think the best deploy mode is `MEMORY_LOCALFILE_HDFS`. And the HDFS is only as the HUGE PARTITIONs' house instead of normal partitions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
