Hi Manu, Thanks for the careful review and the second round of timings. You are right about the commit message, and I can now say so from my own numbers.
I built v9 on a bare-metal box (c6id.metal, 128 vCPU, local NVMe, THP off, performance governor) and ran the A/B you'd expect: v9 against the same tree with the decompression reverted to the real v7 path, that is a fresh ZSTD_createDCtx()/free per value, not the reset removed. The reset-removed build is not a fair speed baseline, because it corrupts a plain slice scan; I used it only to reproduce that corruption, which it does. Median of 30 paired rounds, alternating order, on 80k x 46kB docs plus 160k x 4kB plus 200 x 1MB, all external, warm: query v9 ms v7 ms ratio v9 faster wins 32kB prefix(100B) 2441.7 3975.8 0.614 39% 30/30 32kB slice@20k 8876.7 9316.3 0.953 5% 30/30 32kB full read 9774.2 10222.8 0.956 4% 30/30 4kB prefix(100B) 501.4 517.7 0.969 3% 30/30 1MB slice@500k 647.8 647.0 1.001 0% 10/30 So the big number only shows up for a small prefix or slice of a large value, where the allocation dwarfs the decode. For whole-value reads and for 4kB values it's a few percent, which is your point. I'll change the 0003 commit message to something like: Reusing the context saves the per-value allocation. For whole-value reads and small (4kB) values this is a few percent; it rises to about 40% for small slices and prefixes of large (32kB) values, where the allocation dominates the decode. Your repro passes here too: v9 gives exactly the three expected "compressed zstd data is corrupt" errors and no ASan reports, and with the reset removed B2 returns wrong data with no error. A bare count(length(substr(v,1,100))) over 80k external zstd values is enough to show it: v9 and true v7 both return 80000, the reset-removed build raises the corrupt error. Two more things from the same box, since they bear on the series. Dictionaries. Compression ratio (raw/relation size), 150k rows per shape: shape pglz lz4 zstd zstd+dict logs 1.57 1.57 2.75 4.32 text 16.7 19.7 26.9 63.8 random 0.92 0.92 1.37 (no dict: ZDICT declined to train) The dictionary is where the structured-but-small values win: +57% over plain zstd on logs, more on templated text. Per-value CLI numbers on ~1kB docs line up: lz4 2.0x -> 3.6x with a dictionary, zstd 2.9x -> 5.2x. The proportional gain is largest for small values and fades as values grow, for both codecs, which fits the dictionary being a cold-start aid. Lifecycle under load. I ran a 65-minute torture test on a cassert build: 48 pgbench clients reading slices, prefixes and whole values, inserting and updating dict-compressed rows, while a background loop retrained the dictionary, ran VACUUM FULL (which retires superseded dictionaries), added and dropped a second zstd_dict column, vacuumed, and ran pg_amcheck --heapallindexed. A control column holds md5(v) as written, checked after every step. Across the whole run: zero bad signatures, zero amcheck errors, zero failed transactions, no assertion failure or crash, and the dictionary count cycled as designed (retrain adds one, the next rewrite retires the old ones). An earlier run I killed with -m immediate at 5.6M rows came back through crash recovery with every row still matching its signature. I'll fold the commit-message change into the next version along with anything else that comes out of review, rather than post a v10 that differs only in that one sentence. If you want to compare setups, send your scripts and per-run numbers and I'll diff them against mine. best. -greg
