RanaPriyansh commented on code in PR #11293:
URL: https://github.com/apache/arrow-rs/pull/11293#discussion_r4270973248
##########
arrow-flight/src/encode.rs:
##########
@@ -269,6 +274,8 @@ impl FlightDataEncoderBuilder {
pub struct FlightDataEncoder {
/// Input stream
inner: BoxStream<'static, Result<RecordBatch>>,
+ /// Schema of input batches before dictionary handling
+ input_schema: Option<SchemaRef>,
Review Comment:
`schema` contains the encoded schema. In Hydrate mode, [dictionary
preparation](https://github.com/apache/arrow-rs/blob/f84141eb9b7860b3d928f240a58aa84ba77c169f/arrow-flight/src/encode.rs#L550)
replaces dictionary types with their value types. For example, an input
`Dictionary(Int32, Utf8)` field becomes `Utf8`.
`input_schema` preserves the original schema so each batch can be checked
before that conversion. Reusing `schema` for this comparison would reject valid
dictionary batches. Comparing only converted schemas would also miss
differences in the original dictionary types.
The existing
[`test_encoder_accepts_changed_dictionary_values_with_same_schema`](https://github.com/apache/arrow-rs/blob/f84141eb9b7860b3d928f240a58aa84ba77c169f/arrow-flight/tests/encode_decode.rs#L461)
covers matching input schemas in both dictionary modes. Keeping both fields
also preserves the encoded schema returned by `known_schema()`.
Codex used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
