================
@@ -906,6 +906,16 @@ CreateFileHandler(MemoryBuffer &FirstInput,
}
OffloadBundlerConfig::OffloadBundlerConfig() {
+ if (llvm::compression::zstd::isAvailable()) {
+ CompressionFormat = llvm::compression::Format::Zstd;
+ // Use a high zstd compress level by default for better size reduction.
----------------
yxsamliu wrote:
> Also, I've just discovered that zstd already has `--long` option:
> https://github.com/facebook/zstd/blob/b293d2ebc3a5d29309390a70b3e7861b6f5133ec/lib/zstd.h#L394
>
> ```
> ZSTD_c_enableLongDistanceMatching=160, /* Enable long distance matching.
> * This parameter is designed to improve
> compression ratio
> * for large inputs, by finding large
> matches at long distance.
> * It increases memory usage and window
> size.
> * Note: enabling this parameter
> increases default ZSTD_c_windowLog to 128 MB
> * except when expressly set to a
> different value.
> * Note: will be enabled by default if
> ZSTD_c_windowLog >= 128 MB and
> * compression strategy >= ZSTD_btopt (==
> compression level 16+) */
> ```
>
> This sounds like something we could use here.
Thanks this option is promising. Here is some benchmark result of a fat binary
containing 13 code objects each of which is about 2.7MB.
The following data is without `--long`. The numbers are compression level,
original size -> compressed size (compression rate), compression speed,
decompression speed.
```
$ zstd -b1 -e22 -f --ultra tmp.o
1#tmp.o : 34864866 -> 9169246 (3.802), 657.0 MB/s ,1691.0 MB/s
2#tmp.o : 34864866 -> 7352667 (4.742), 626.3 MB/s ,1903.8 MB/s
3#tmp.o : 34864866 -> 6885718 (5.063), 488.1 MB/s ,1900.2 MB/s
4#tmp.o : 34864866 -> 6700508 (5.203), 416.7 MB/s ,1897.2 MB/s
5#tmp.o : 34864866 -> 6405252 (5.443), 236.4 MB/s ,1918.8 MB/s
6#tmp.o : 34864866 -> 6336706 (5.502), 211.8 MB/s ,1941.4 MB/s
7#tmp.o : 34864866 -> 6170409 (5.650), 153.5 MB/s ,2032.5 MB/s
8#tmp.o : 34864866 -> 6121226 (5.696), 131.1 MB/s ,2071.5 MB/s
9#tmp.o : 34864866 -> 6098948 (5.717), 124.9 MB/s ,2080.4 MB/s
10#tmp.o : 34864866 -> 2555599 (13.64), 179.4 MB/s ,3504.2 MB/s
11#tmp.o : 34864866 -> 2545375 (13.70), 119.4 MB/s ,3516.8 MB/s
12#tmp.o : 34864866 -> 2542711 (13.71), 107.2 MB/s ,3518.4 MB/s
13#tmp.o : 34864866 -> 2601619 (13.40), 58.4 MB/s ,3507.6 MB/s
14#tmp.o : 34864866 -> 2590656 (13.46), 46.2 MB/s ,3520.4 MB/s
15#tmp.o : 34864866 -> 2518599 (13.84), 28.4 MB/s ,3557.4 MB/s
16#tmp.o : 34864866 -> 2527122 (13.80), 20.8 MB/s ,3348.5 MB/s
17#tmp.o : 34864866 -> 2277125 (15.31), 19.0 MB/s ,3370.6 MB/s
18#tmp.o : 34864866 -> 2138918 (16.30), 15.0 MB/s ,3182.2 MB/s
19#tmp.o : 34864866 -> 2118238 (16.46), 8.82 MB/s ,3194.5 MB/s
20#tmp.o : 34864866 -> 2041007 (17.08), 8.31 MB/s ,3178.4 MB/s
21#tmp.o : 34864866 -> 2039075 (17.10), 5.21 MB/s ,3170.6 MB/s
22#tmp.o : 34864866 -> 2038568 (17.10), 3.60 MB/s ,3171.5 MB/s
```
The following data are with `--long`:
```
$ zstd --long -b1 -e22 -f --ultra tmp.o
1#tmp.o : 34864866 -> 3281430 (10.62), 375.0 MB/s ,3531.9 MB/s
2#tmp.o : 34864866 -> 2854143 (12.22), 360.6 MB/s ,3536.7 MB/s
3#tmp.o : 34864866 -> 2648807 (13.16), 325.4 MB/s ,3462.7 MB/s
4#tmp.o : 34864866 -> 2548618 (13.68), 309.6 MB/s ,3345.9 MB/s
5#tmp.o : 34864866 -> 2540406 (13.72), 265.8 MB/s ,3297.8 MB/s
6#tmp.o : 34864866 -> 2518788 (13.84), 251.9 MB/s ,3296.0 MB/s
7#tmp.o : 34864866 -> 2451360 (14.22), 206.5 MB/s ,3446.9 MB/s
8#tmp.o : 34864866 -> 2421083 (14.40), 186.5 MB/s ,3522.7 MB/s
9#tmp.o : 34864866 -> 2406717 (14.49), 172.0 MB/s ,3472.2 MB/s
10#tmp.o : 34864866 -> 2392819 (14.57), 139.6 MB/s ,3439.4 MB/s
11#tmp.o : 34864866 -> 2386599 (14.61), 113.0 MB/s ,3415.2 MB/s
12#tmp.o : 34864866 -> 2385088 (14.62), 104.5 MB/s ,3430.0 MB/s
13#tmp.o : 34864866 -> 2389264 (14.59), 69.5 MB/s ,3422.9 MB/s
14#tmp.o : 34864866 -> 2382705 (14.63), 61.2 MB/s ,3428.6 MB/s
15#tmp.o : 34864866 -> 2372640 (14.69), 51.2 MB/s ,3446.7 MB/s
16#tmp.o : 34864866 -> 2209022 (15.78), 20.5 MB/s ,3483.3 MB/s
17#tmp.o : 34864866 -> 2168474 (16.08), 18.2 MB/s ,3381.5 MB/s
18#tmp.o : 34864866 -> 2065724 (16.88), 14.2 MB/s ,3187.8 MB/s
19#tmp.o : 34864866 -> 2042810 (17.07), 8.50 MB/s ,3195.7 MB/s
20#tmp.o : 34864866 -> 2040443 (17.09), 8.13 MB/s ,3173.0 MB/s
21#tmp.o : 34864866 -> 2038794 (17.10), 5.11 MB/s ,3174.3 MB/s
22#tmp.o : 34864866 -> 2038375 (17.10), 3.54 MB/s ,3177.8 MB/s
```
>From the data we can see, with `--long`, even with compression level 3, the
>compressed file size is below one code object, whereas the compression speed
>is very fast. Higher compression level only improves compression rate slightly
>but with a much slower speed. Therefore compression level 3 seems to be more
>favorable.
https://github.com/llvm/llvm-project/pull/83605
_______________________________________________
cfe-commits mailing list
cfe-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
