alexandre-normand opened a new pull request, #1654: URL: https://github.com/apache/iceberg-go/pull/1654
We've got a production service that's streaming records into iceberg at _some_ scale and found that one of the lowest hanging fruits in terms of CPU usage (which turned out to be a bottleneck) is the json decoding of iceberg REST catalog responses. Granted, this is exacerbated on our side because we have a long snapshot history and that inflates responses but I thought we might not be the only ones that could benefit from this performance improvement. The actual change is to swap out `encoding/json` for the drop-in replacement that is [github.com/goccy/go-json](https://github.com/goccy/go-json). We'd probably want to migrate back to `encoding/json` v2 which brings performance improvements but given it's still experimental, `goccy/go-json` is a fair compromise. You can see the profiling comparison below before/after swapping `encoding/json` for `github.com/goccy/go-json`: <img width="1715" height="964" alt="performance improvement" src="https://github.com/user-attachments/assets/0724093a-2573-4ed4-848a-2d14eeb8614d"; /> With the same resources, we went from 75% to 100% CPU utilization down to 34% with this change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
