ennuite commented on PR #1036: URL: https://github.com/apache/arrow-java/pull/1036#issuecomment-5439594968
@rustyconover Hi! I want to continue with this PR eventually, I'm finishing some other work before I continue with this. Maybe closing this is a lack of understanding of Github etiquette: I don't have time to finish it before 20.0.0, I tried removing the milestone but I couldn't so I closed it to let maintainers know I couldn't finish it on time for the release. I'm targeting this for the next release (21.0.0). In terms of the state of this PR: I think the low level plumbing is mostly done, but to have feature parity with [your Rust PR](https://github.com/apache/arrow-rs/pull/9445) there is still quite a bit of work: - figure out how to represent per-batch metadata in Java’s higher-level APIs so that slicing and other transformations preserve it; - expose metadata through ArrowStreamReader and ArrowFileReader; - allow ArrowStreamWriter and ArrowFileWriter to write it; - propagate it through Flight without conflating it with Flight application metadata; - add public-API, multi-batch, and (potentially) PyArrow interoperability tests. Right now, metadata round-trips through the low-level ArrowRecordBatch/MessageSerializer APIs, but it is discarded when a batch is loaded into a VectorSchemaRoot, and the standard writer path has no way to supply it. Do you agree with this summary? I don't have the bandwidth to tackle all of this right now and I don't think what I have implemented is useful as its own PR. As I'm not very familiar with Arrow IPC, it might take me some time to go through all the things even once I do grab this :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
