FrankYang0529 commented on PR #76279: URL: https://github.com/apache/airflow/pull/76279#issuecomment-6622732443
@Dev-iL Thanks for the review. The problem is in two layer - build and load. - Build: before this change, the stash step compressed the 2.30 GiB tar with zlib level 6 on one CPU core. Compressing and uploading the 681 MB result took 58 s. With this change, `zstd -6 -T0` uses all CPU cores and takes 10 to 13 s, and uploading the 563 MB file takes 4 s. So the build was CPU-bound. - Each job that loads the image: restoring the stash went from 28.2 s to 11.8 s. With this change, the restore time is mostly download time, because the stash no longer compresses the file. `docker image load` took 36.9 s with the plain tar and 37.9 s with the zstd file. The plain tar needs no decompression, but loading it still took 36.9 s. So Docker spends most of the load time unpacking the image layers and writing them to disk. Loading the zstd file, which Docker has to decompress, took only 1 s longer than loading the plain tar. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
