jerqi commented on issue #186: URL: https://github.com/apache/incubator-uniffle/issues/186#issuecomment-1225306182
> Thank you for your detailed explanation. I have no doubt about the first two points, but the third point is that I want to know what you mean by the retry of HDFS cluster. And the fourth point is that we really have no way to know the amount of shuffle data, so the current idea is that if there are no files at the beginning, we can only compare the remaining capacity of the namespace. If there are already shuffle files, Then we can compare the ratio of the size of all the shuffle files and the remaining capacity of the namespace under different HDFS paths. Third point, sorry... I should give more explanation. If shuffle server fail to write data to HDFS because HDFS high load, shuffle will retry. If there are many retries about HDFS, it means that HDFS have bad status, we should avoid using it. Fourth, your solution is not a perfect solution, it must depend data in the production environment to prove the effectiveness. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
