felipecrv commented on PR #35:
URL: https://github.com/apache/arrow-experiments/pull/35#issuecomment-2347414612
Now including dictionary encoding (always on for these examples) and IPC
buffer compression.
```
# uncompressed
803M out.arrows
# HTTP response compression # IPC buffer compression
208M out.arrows.zstd 220M out.arrows+zstd
247M out.arrows.gz
38M out.arrows.br
404M out.arrows+lz4
```
<details>
<summary>Timings</summary>
```
[identity]: Requesting data from http://127.0.0.1:8008 with `identity`
compression strategy.
[identity]: Schema received in 0.007 seconds. schema=(ticker, price, volume).
[identity]: First batch received and processed in 0.007 seconds
[identity]: Processing of all batches completed in 0.194 seconds.
[identity]: ReadStats(num_messages=6838, num_record_batches=6836,
num_dictionary_batches=1, num_dictionary_deltas=0, num_replaced_dictionaries=0)
[zstd]: Requesting data from http://127.0.0.1:8008 with `zstd`
compression strategy.
[zstd]: Schema received in 0.006 seconds. schema=(ticker, price, volume).
[zstd]: First batch received and processed in 0.006 seconds
[zstd]: Processing of all batches completed in 1.934 seconds.
[zstd]: ReadStats(num_messages=6838, num_record_batches=6836,
num_dictionary_batches=1, num_dictionary_deltas=0, num_replaced_dictionaries=0)
[br]: Requesting data from http://127.0.0.1:8008 with `br` compression
strategy.
[br]: Schema received in 0.111 seconds. schema=(ticker, price, volume).
[br]: First batch received and processed in 0.111 seconds
[br]: Processing of all batches completed in 7.824 seconds.
[br]: ReadStats(num_messages=6838, num_record_batches=6836,
num_dictionary_batches=1, num_dictionary_deltas=0, num_replaced_dictionaries=0)
[gzip]: Requesting data from http://127.0.0.1:8008 with `gzip`
compression strategy.
[gzip]: Schema received in 0.026 seconds. schema=(ticker, price, volume).
[gzip]: First batch received and processed in 0.026 seconds
[gzip]: Processing of all batches completed in 41.153 seconds.
[gzip]: ReadStats(num_messages=6838, num_record_batches=6836,
num_dictionary_batches=1, num_dictionary_deltas=0, num_replaced_dictionaries=0)
[identity+zstd]: Requesting data from http://127.0.0.1:8008 with
`identity+zstd` compression strategy.
[identity+zstd]: Schema received in 0.001 seconds. schema=(ticker, price,
volume).
[identity+zstd]: First batch received and processed in 0.001 seconds
[identity+zstd]: Processing of all batches completed in 0.180 seconds.
[identity+zstd]: ReadStats(num_messages=6838, num_record_batches=6836,
num_dictionary_batches=1, num_dictionary_deltas=0, num_replaced_dictionaries=0)
[identity+lz4]: Requesting data from http://127.0.0.1:8008 with
`identity+lz4` compression strategy.
[identity+lz4]: Schema received in 0.001 seconds. schema=(ticker, price,
volume).
[identity+lz4]: First batch received and processed in 0.001 seconds
[identity+lz4]: Processing of all batches completed in 0.184 seconds.
[identity+lz4]: ReadStats(num_messages=6838, num_record_batches=6836,
num_dictionary_batches=1, num_dictionary_deltas=0, num_replaced_dictionaries=0)
```
<details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
