mbasmanova commented on PR #48468: URL: https://github.com/apache/arrow/pull/48468#issuecomment-4957770389
Hi @wecharyu @wgtmac @pitrou — we work on [Velox](https://github.com/facebookincubator/velox), a C++ query engine used by Prestissimo (Presto on Velox) and by Gluten (Spark on Velox). Velox writes Parquet through Arrow's C++ writer, so we've been following this issue. For what it's worth, we agree with keeping the writer's contract exact. Row-count is a precise, deterministic contract; byte-limiting is inherently an estimate (the encoded size isn't known until the data is written), so it's the kind of imprecise decision an application is better placed to own — it can judge how good the estimate needs to be for its data and its tolerance — rather than something the library bakes in for everyone. That points to a small seam that would let applications do exactly that while the writer stays exact: allow the row-count target to be **adjusted between batches** (today it's fixed at construction). The application translates its byte budget into a row count from stats it already has (`currentRowGroupTotalBytes` + row count) and sets the next target; the writer still only ever cuts at an exact row. It also reuses the split-by-row-count loop the writer already has, rather than each caller reimplementing it. We can do this above the writer too — slice batches ourselves and finish row groups at the chosen point — so we're not blocked (that's roughly what we're doing today: [facebookincubator/velox#18104](https://github.com/facebookincubator/velox/pull/18104)). An adjustable target would just let us drive the writer's existing split instead of duplicating it outside. One side question, @yonipeleg33: the same contract concern would seem to apply to arrow-rs, which recently added byte-limiting *in the writer* with per-batch estimation ([apache/arrow-rs#9357](https://github.com/apache/arrow-rs/pull/9357)). We're curious how that was reconciled there — not to argue either way, just trying to understand the thinking across the implementations. Thanks for engaging on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
