taepper commented on PR #51396:
URL: https://github.com/apache/arrow/pull/51396#issuecomment-6518696895
This advantage still holds for larger datasets:
| Dataset | RapidJSON (main) | simdjson, column-wise (a5d38cd523) |
simdjson, row-wise
(https://github.com/taepper/arrow/commit/5247184017656960f6615a4f3befca1ca0f6fe9b)
|
|---|---:|---:|---:|
| **JSON -> Arrow** (parse + convert) | | | |
|
[twitter.json](https://github.com/simdjson/simdjson/blob/master/jsonexamples/twitter.json),
100 statuses x 100 = 10,000 rows, 47 MB | 401 ms (wrong result) | 104 ms |
**49 ms** |
|
[canada.json](https://github.com/boostorg/json/blob/develop/bench/data/canada.json),
1 feature x 100 = 100 rows, 209 MB | segfault | 1,115 ms | **325 ms** |
|
[canada.json](https://github.com/boostorg/json/blob/develop/bench/data/canada.json),
1 feature x 1,000 = 1,000 rows, 2.1 GB | segfault | 9,550 ms | **3,224 ms** |
| example records, 3 rows x ~33k = 100,000 rows, 10.8 MB | 319 ms | 132 ms
| **22 ms** |
| **Arrow -> JSON** (to strings) | | | |
| twitter.json, 10,000 rows | 642 ms | 66 ms | 68 ms |
| canada.json, 100 rows | segfault | 4,147 ms | 4,169 ms |
| canada.json, 1,000 rows | segfault | 41,642 ms | 41,896 ms |
| example records, 100,000 rows | 21,074 ms | 34.5 ms | 34.3 ms |
Also note peak memory reduced from 26 GB to 19 GB with the row-wise version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
