HippoBaro commented on PR #11278: URL: https://github.com/apache/arrow-rs/pull/11278#issuecomment-5976249333
> ```string_dictionary/bloom_filter 1.00 101.5±1.26ms 2.5 GB/sec 1.01 102.5±1.49ms 2.5 > GB/sec > string_dictionary/cdc 1.19 55.2±0.78ms 4.7 GB/sec 1.00 46.3±0.39ms 5.6 GB/sec > string_dictionary/default 1.00 57.2±1.24ms 4.5 GB/sec 1.00 57.4±1.02ms 4.5 >GB/sec > string_dictionary/number_distinct_values 1.00 115.8±0.67ms 2.2 GB/sec 1.03 119.2±1.42ms 2.2 GB/sec > string_dictionary/parquet_2 1.00 48.6±3.96ms 5.3 GB/sec 1.12 54.6±5.43ms 4.7 GB/sec > string_dictionary/zstd 1.00 215.5±0.64ms 1225.8 MB/sec 1.02 219.0±1.02ms 1206.0 MB/sec > string_dictionary/zstd_parquet_2 1.00 193.0±4.40ms 1368.8 MB/sec 1.02 196.5±4.96ms 1343.9 MB/sec > ``` These results are strange. I can’t find an algorithmic explanation for the opposite movements in `/cdc` and `/parquet_2`. All the changes here are in shared schema/type-checking setup, so I’d expect any direct overhead to affect the variants similarly rather than move them in opposite directions. I reran the benchmark multiple times on my Graviton4 benchmark box ([configured like this](https://github.com/apache/arrow-rs/pull/11267#issuecomment-5876792873)). It uses the same Neoverse V2 cores, and I couldn’t reproduce either effect: ```text group base changed ----- ---- ------- string_dictionary/bloom_filter 1.00 95.9±0.16ms 2.7 GB/sec 1.00 96.1±0.41ms 2.7 GB/sec string_dictionary/cdc 1.00 58.1±0.15ms 4.4 GB/sec 1.01 58.6±1.36ms 4.4 GB/sec string_dictionary/default 1.00 58.7±0.45ms 4.4 GB/sec 1.01 59.1±0.61ms 4.4 GB/sec string_dictionary/number_distinct_values 1.00 119.4±2.07ms 2.2 GB/sec 1.01 120.7±0.21ms 2.1 GB/sec string_dictionary/parquet_2 1.00 59.2±5.42ms 4.4 GB/sec 1.00 59.3±5.19ms 4.4 GB/sec string_dictionary/zstd 1.00 222.8±0.67ms 1185.3 MB/sec 1.00 223.7±0.58ms 1180.7 MB/sec string_dictionary/zstd_parquet_2 1.01 210.0±4.14ms 1258.0 MB/sec 1.00 208.8±4.45ms 1265.2 MB/sec string_dictionary_low_cardinality_100/bloom_filter 1.00 23.0±0.05ms 174.2 MB/sec 1.00 23.0±0.05ms 173.9 MB/sec string_dictionary_low_cardinality_100/cdc 1.00 25.2±0.06ms 158.7 MB/sec 1.01 25.5±0.37ms 156.9 MB/sec string_dictionary_low_cardinality_100/default 1.00 22.9±0.04ms 175.0 MB/sec 1.00 23.0±0.06ms 174.6 MB/sec string_dictionary_low_cardinality_100/number_distinct_values 1.00 46.8±0.07ms 85.5 MB/sec 1.00 47.1±0.09ms 85.1 MB/sec string_dictionary_low_cardinality_100/parquet_2 1.00 22.8±0.05ms 176.0 MB/sec 1.01 22.9±0.06ms 175.0 MB/sec string_dictionary_low_cardinality_100/zstd 1.00 23.2±0.05ms 172.4 MB/sec 1.00 23.3±0.06ms 171.7 MB/sec string_dictionary_low_cardinality_100/zstd_parquet_2 1.00 23.1±0.05ms 173.3 MB/sec 1.01 23.3±0.07ms 172.2 MB/sec string_dictionary_low_cardinality_20/bloom_filter 1.00 23.3±0.05ms 171.6 MB/sec 1.00 23.4±0.05ms 171.3 MB/sec string_dictionary_low_cardinality_20/cdc 1.00 25.6±0.07ms 156.4 MB/sec 1.02 26.1±0.73ms 153.6 MB/sec string_dictionary_low_cardinality_20/default 1.00 23.3±0.06ms 172.3 MB/sec 1.00 23.3±0.05ms 171.9 MB/sec string_dictionary_low_cardinality_20/number_distinct_values 1.00 48.0±0.07ms 83.5 MB/sec 1.00 47.8±0.07ms 83.8 MB/sec string_dictionary_low_cardinality_20/parquet_2 1.00 23.2±0.05ms 172.4 MB/sec 1.00 23.2±0.05ms 172.5 MB/sec string_dictionary_low_cardinality_20/zstd 1.00 23.4±0.06ms 171.3 MB/sec 1.00 23.4±0.05ms 170.9 MB/sec string_dictionary_low_cardinality_20/zstd_parquet_2 1.00 23.3±0.05ms 171.8 MB/sec 1.01 23.4±0.05ms 170.9 MB/sec string_dictionary_low_cardinality_400/bloom_filter 1.01 23.8±0.07ms 168.9 MB/sec 1.00 23.6±0.07ms 169.8 MB/sec string_dictionary_low_cardinality_400/cdc 1.00 25.9±0.07ms 154.9 MB/sec 1.00 26.0±0.41ms 154.2 MB/sec string_dictionary_low_cardinality_400/default 1.01 23.7±0.07ms 169.6 MB/sec 1.00 23.5±0.06ms 170.6 MB/sec string_dictionary_low_cardinality_400/number_distinct_values 1.00 47.9±0.09ms 83.7 MB/sec 1.00 47.9±0.11ms 83.8 MB/sec string_dictionary_low_cardinality_400/parquet_2 1.01 23.7±0.06ms 169.1 MB/sec 1.00 23.6±0.08ms 170.1 MB/sec string_dictionary_low_cardinality_400/zstd 1.00 24.0±0.08ms 167.0 MB/sec 1.00 24.1±0.06ms 166.8 MB/sec string_dictionary_low_cardinality_400/zstd_parquet_2 1.00 24.1±0.05ms 166.2 MB/sec 1.00 24.1±0.07ms 166.3 MB/sec ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
