YuanbenWang commented on PR #8598: URL: https://github.com/apache/hadoop/pull/8598#issuecomment-5909626799
@Hexiaoqiao Thanks for raising this concern. I conducted two complementary benchmark experiments to evaluate both the benefit of parallel XML processing and the cost of merging temporary files. The first experiment used small fsimages containing 2K, 20K, and 200K files. Each configuration was run five times, and the median wall-clock time was recorded: | Files | -m 1 | -m 2 | -m 4 | -m 8 | Best speedup | |------:|-----:|-----:|-----:|-----:|-------------:| | 2K | 0.67s | 0.63s | 0.61s | 0.60s | 1.12x | | 20K | 1.59s | 1.53s | 1.42s | 1.42s | 1.12x | | 200K | 9.14s | 7.21s | 6.58s | 6.21s | 1.47x | This experiment shows that parallelism provides little benefit for very small fsimages, where JVM startup, thread scheduling, and temporary-file overhead represent a relatively large portion of the total runtime. However, the benefit becomes visible as the image grows. For the 200K-file image, `-m 8` was 1.47x faster than serial processing. I then conducted a second experiment using synthetic Hadoop-native fsimages containing 1M, 10M, and 100M files. These tests wrote the XML output to real local files, so the reported wall-clock time includes fsimage parsing, XML generation, temporary-file I/O, and the final merge: | Files | -m 1 | -m 2 | -m 4 | -m 8 | Best speedup | |------:|-----:|-----:|-----:|-----:|-------------:| | 1M | 82.28s | 40.41s | 33.05s | 35.51s | 2.49x (`-m 4`) | | 10M | 697.98s | 381.96s | 314.43s | 314.99s | 2.22x (`-m 4`) | | 100M | 6353.90s | 4277.06s | 3985.24s | 3294.85s | 1.93x (`-m 8`) | The measured merge times were: - 1M files: 249–1,185 ms - 10M files: 5.9–6.3 s - 100M files: 115–146 s The merge cost therefore increases with the image size and is no longer negligible for the 100M-file image. However, it does not eliminate the end-to-end benefit of parallel processing. In this benchmark, `-m 8` reduced the 100M-file runtime by approximately 51 minutes compared with serial processing. All parallelism settings produced byte-for-byte identical XML output for each image. The generated fsimages were validated by OIV and `FSImageUtil.loadSummary()`. Since the large images were synthetic benchmark images and were not loaded into a running NameNode, these results should be interpreted as OIV parsing/output benchmarks rather than NameNode restore benchmarks. Overall, these two experiments indicate that parallel XML processing is not particularly useful for very small fsimages, but provides substantial benefits for sufficiently large fsimages. The merge introduces a real and measurable trade-off at very large scale, but it remains smaller than the time saved by parallel processing in these experiments. The optimal parallelism depends on the fsimage size and the CPU/storage environment. I will attach the detailed benchmark report, including the test environment, image-generation method, raw measurements, merge timings, and correctness checks, to the HDFS-16512 JIRA issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
