codewarnab commented on issue #5981: URL: https://github.com/apache/arrow-rs/issues/5981#issuecomment-6759164667
Here is my approach. I split it into 3 PRs so nothing breaks right away. and also keeping each pr diff minimal and reviewable ### PR 1 (not breaking, draft coming): - Add `IpcSchema` to `arrow-ipc`. It holds the `Schema` and a tree of dictionary IDs with the same shape as the schema. A tree handles nested types and projection. - The reader fills it from the IPC schema message. So the IDs always come from the file, never from field order. This keeps C++ (parent-first IDs) and Java (shared IDs) files working. - `FileReader`, `StreamReader` and `StreamDecoder` use it internally. No API change there. - Add new variants that take `IpcSchema` for `read_record_batch`, `read_dictionary`, `FileDecoder::new` and `flight_data_to_arrow_batch`. The old ones keep working and still use `Field::dict_id`. ### PR 2: deprecate the old variants. ### PR 3 (next major release): remove `Field::dict_id` and the old variants. This closes the issue. @alamb about side effects: PR 1 does not change what we read or write. `Field::dict_id` is still filled in. The writer already ignores it. I will add tests for nested and shared IDs and run the integration tests. @Jeffrey about ergonomics: users who pass their own `Schema` get an `IpcSchema` from the same schema message that they already parse. I am open to better names. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
