scovich commented on PR #7535: URL: https://github.com/apache/arrow-rs/pull/7535#issuecomment-2945462274
> > @alamb About the builder APIs — one question I’ve been thinking about: in the Go version, the single-buffer builder is impressively efficient and avoids most overhead. In contrast, the nested builder I’ve been working on in Rust offers more ergonomic APIs, but it’s harder to optimize due to intermediate buffers and lifetime management. Do you have any advice on how I can start? > > Impressively effiicient and simpler without overhead sounds pretty great to me > > One suggestion is to basically sketch out what the two APIs might look like (I know you have one already) so we can try and evaluate the code > > Another suggestion is to make a PR with a basic builder / framework to start (e.g. for primitives). If you can break it up into smaller PRs I think it might be easier to get you specific feedback on potential alternative approaches Intuitively, it seems like would want the "nested" builders to all be working with "views" of a single buffer? Ultimately, a builder would either be appending to the end of the buffer, or filling in bytes that were previously allocated. And from the playing around I did previously, I _think_ all the filling in is a local operation (e.g. an array needs to pre-allocate space for the N offsets, but can't populate the offset values until the value bytes have actually been written). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
