codewarnab commented on code in PR #12147:
URL: https://github.com/apache/arrow-rs/pull/12147#discussion_r4565777756
##########
arrow-ipc/src/ipc_schema.rs:
##########
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF)
under one or more contributor license agreements
+
+//! [`IpcSchema`]: an Arrow [`Schema`](arrow_schema::Schema) together with the
IPC dictionary IDs of its fields
+
+use std::sync::Arc;
+
+use arrow_schema::{ArrowError, DataType, Field, Fields, SchemaRef};
+
+/// An Arrow [`Schema`](arrow_schema::Schema) together with the dictionary IDs
assigned to its
+/// dictionary-encoded fields by an IPC schema message.
+///
+/// In the Arrow IPC format, every dictionary-encoded field carries a
+/// dictionary ID, and dictionary batches refer to their field by that ID.
+/// The IDs are a property of the IPC encoding, not of the logical schema, so
+/// they are kept here instead of on [`Field`].
+///
+/// The IDs are stored in a tree with the same shape as the schema, so nested
+/// fields (for example a dictionary inside a struct, or inside the values of
+/// another dictionary) get their own ID, and several fields can share one ID.
+///
+/// Create one from an IPC schema message with [`IpcSchema::try_from_fb`]. The
+/// readers in this crate do this for you; use it directly with the low-level
+/// APIs such as [`read_record_batch_with_ipc_schema`] and
+/// [`FileDecoder::new_with_ipc_schema`].
+///
+/// [`read_record_batch_with_ipc_schema`]:
crate::reader::read_record_batch_with_ipc_schema
+/// [`FileDecoder::new_with_ipc_schema`]:
crate::reader::FileDecoder::new_with_ipc_schema
+#[derive(Debug, Clone, PartialEq)]
+pub struct IpcSchema {
Review Comment:
**Start here.** `IpcSchema` = the logical `Schema` plus a tree of dictionary
ids with the same shape. Fields are `pub(crate)` so the tree layout stays an
internal detail; only `try_from_fb` and `schema()` are public.
##########
arrow-ipc/src/ipc_schema.rs:
##########
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF)
under one or more contributor license agreements
+
+//! [`IpcSchema`]: an Arrow [`Schema`](arrow_schema::Schema) together with the
IPC dictionary IDs of its fields
+
+use std::sync::Arc;
+
+use arrow_schema::{ArrowError, DataType, Field, Fields, SchemaRef};
+
+/// An Arrow [`Schema`](arrow_schema::Schema) together with the dictionary IDs
assigned to its
+/// dictionary-encoded fields by an IPC schema message.
+///
+/// In the Arrow IPC format, every dictionary-encoded field carries a
+/// dictionary ID, and dictionary batches refer to their field by that ID.
+/// The IDs are a property of the IPC encoding, not of the logical schema, so
+/// they are kept here instead of on [`Field`].
+///
+/// The IDs are stored in a tree with the same shape as the schema, so nested
+/// fields (for example a dictionary inside a struct, or inside the values of
+/// another dictionary) get their own ID, and several fields can share one ID.
+///
+/// Create one from an IPC schema message with [`IpcSchema::try_from_fb`]. The
+/// readers in this crate do this for you; use it directly with the low-level
+/// APIs such as [`read_record_batch_with_ipc_schema`] and
+/// [`FileDecoder::new_with_ipc_schema`].
+///
+/// [`read_record_batch_with_ipc_schema`]:
crate::reader::read_record_batch_with_ipc_schema
+/// [`FileDecoder::new_with_ipc_schema`]:
crate::reader::FileDecoder::new_with_ipc_schema
+#[derive(Debug, Clone, PartialEq)]
+pub struct IpcSchema {
+ pub(crate) schema: SchemaRef,
+ pub(crate) dict_ids: Arc<[DictIdNode]>,
+}
+
+impl IpcSchema {
+ /// Reads the schema and its dictionary IDs from an IPC schema message.
+ ///
+ /// The IDs always come from the message, so files that number
+ /// dictionaries in any order, or share one dictionary between fields,
+ /// are read correctly.
+ pub fn try_from_fb(fb: crate::Schema) -> Result<Self, ArrowError> {
+ let schema = crate::convert::try_fb_to_schema(fb)?;
+ let dict_ids = match fb.fields() {
+ Some(fields) => fields.iter().map(DictIdNode::from_fb).collect(),
+ None => Arc::from([]),
+ };
+ Ok(Self {
+ schema: Arc::new(schema),
+ dict_ids,
+ })
+ }
+
+ /// Builds an [`IpcSchema`] from the deprecated [`Field::dict_id`] values
+ /// stored on `schema`.
+ ///
+ /// This is how the APIs that only take a [`Schema`](arrow_schema::Schema)
find dictionary IDs.
+ pub(crate) fn from_schema_dict_ids(schema: SchemaRef) -> Self {
+ let dict_ids = DictIdNode::from_fields(schema.fields());
+ Self { schema, dict_ids }
+ }
+
+ /// The Arrow schema
+ pub fn schema(&self) -> &SchemaRef {
+ &self.schema
+ }
+
+ /// Dictionary ID trees, one per top-level field
+ pub(crate) fn dict_ids(&self) -> &Arc<[DictIdNode]> {
+ &self.dict_ids
+ }
+
+ /// Returns the first field, in depth-first order, that uses dictionary
+ /// `id`, together with its node in the ID tree.
+ pub(crate) fn field_with_dict_id(&self, id: i64) -> Option<(&Field,
&DictIdNode)> {
+ find_in_fields(self.schema.fields(), &self.dict_ids, id)
+ }
+}
+
+/// The dictionary ID of one field and the IDs of its children.
+///
+/// `children` follow [`child_fields`]: for a dictionary field they are the
+/// children of its value type, matching the IPC flatbuffer layout.
+#[derive(Debug, Clone, Default, PartialEq)]
+pub(crate) struct DictIdNode {
+ pub(crate) dict_id: Option<i64>,
+ pub(crate) children: Vec<DictIdNode>,
+}
+
+static EMPTY_NODE: DictIdNode = DictIdNode {
Review Comment:
Missing nodes resolve to this empty node instead of erroring. A dictionary
field that hits it still fails later with the existing "does not have dict id"
error, so malformed input is not silently accepted.
##########
arrow-ipc/src/reader.rs:
##########
@@ -160,8 +166,7 @@ impl RecordBatchDecoder<'_> {
let index_node = self.next_node(field)?;
let index_buffers = [self.next_buffer()?, self.next_buffer()?];
- #[expect(deprecated)]
- let dict_id = field.dict_id().ok_or_else(|| {
+ let dict_id = dict_node.dict_id.ok_or_else(|| {
Review Comment:
The core behaviour change: the id now comes from the tree node passed down
through `create_array`, not from `Field::dict_id`. Every recursive call above
passes `dict_node.child(i)` alongside the child field.
##########
arrow-ipc/src/ipc_schema.rs:
##########
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF)
under one or more contributor license agreements
+
+//! [`IpcSchema`]: an Arrow [`Schema`](arrow_schema::Schema) together with the
IPC dictionary IDs of its fields
+
+use std::sync::Arc;
+
+use arrow_schema::{ArrowError, DataType, Field, Fields, SchemaRef};
+
+/// An Arrow [`Schema`](arrow_schema::Schema) together with the dictionary IDs
assigned to its
+/// dictionary-encoded fields by an IPC schema message.
+///
+/// In the Arrow IPC format, every dictionary-encoded field carries a
+/// dictionary ID, and dictionary batches refer to their field by that ID.
+/// The IDs are a property of the IPC encoding, not of the logical schema, so
+/// they are kept here instead of on [`Field`].
+///
+/// The IDs are stored in a tree with the same shape as the schema, so nested
+/// fields (for example a dictionary inside a struct, or inside the values of
+/// another dictionary) get their own ID, and several fields can share one ID.
+///
+/// Create one from an IPC schema message with [`IpcSchema::try_from_fb`]. The
+/// readers in this crate do this for you; use it directly with the low-level
+/// APIs such as [`read_record_batch_with_ipc_schema`] and
+/// [`FileDecoder::new_with_ipc_schema`].
+///
+/// [`read_record_batch_with_ipc_schema`]:
crate::reader::read_record_batch_with_ipc_schema
+/// [`FileDecoder::new_with_ipc_schema`]:
crate::reader::FileDecoder::new_with_ipc_schema
+#[derive(Debug, Clone, PartialEq)]
+pub struct IpcSchema {
+ pub(crate) schema: SchemaRef,
+ pub(crate) dict_ids: Arc<[DictIdNode]>,
+}
+
+impl IpcSchema {
+ /// Reads the schema and its dictionary IDs from an IPC schema message.
+ ///
+ /// The IDs always come from the message, so files that number
+ /// dictionaries in any order, or share one dictionary between fields,
+ /// are read correctly.
+ pub fn try_from_fb(fb: crate::Schema) -> Result<Self, ArrowError> {
+ let schema = crate::convert::try_fb_to_schema(fb)?;
+ let dict_ids = match fb.fields() {
+ Some(fields) => fields.iter().map(DictIdNode::from_fb).collect(),
+ None => Arc::from([]),
+ };
+ Ok(Self {
+ schema: Arc::new(schema),
+ dict_ids,
+ })
+ }
+
+ /// Builds an [`IpcSchema`] from the deprecated [`Field::dict_id`] values
+ /// stored on `schema`.
+ ///
+ /// This is how the APIs that only take a [`Schema`](arrow_schema::Schema)
find dictionary IDs.
+ pub(crate) fn from_schema_dict_ids(schema: SchemaRef) -> Self {
+ let dict_ids = DictIdNode::from_fields(schema.fields());
+ Self { schema, dict_ids }
+ }
+
+ /// The Arrow schema
+ pub fn schema(&self) -> &SchemaRef {
+ &self.schema
+ }
+
+ /// Dictionary ID trees, one per top-level field
+ pub(crate) fn dict_ids(&self) -> &Arc<[DictIdNode]> {
+ &self.dict_ids
+ }
+
+ /// Returns the first field, in depth-first order, that uses dictionary
+ /// `id`, together with its node in the ID tree.
+ pub(crate) fn field_with_dict_id(&self, id: i64) -> Option<(&Field,
&DictIdNode)> {
+ find_in_fields(self.schema.fields(), &self.dict_ids, id)
+ }
+}
+
+/// The dictionary ID of one field and the IDs of its children.
+///
+/// `children` follow [`child_fields`]: for a dictionary field they are the
+/// children of its value type, matching the IPC flatbuffer layout.
+#[derive(Debug, Clone, Default, PartialEq)]
+pub(crate) struct DictIdNode {
+ pub(crate) dict_id: Option<i64>,
+ pub(crate) children: Vec<DictIdNode>,
+}
+
+static EMPTY_NODE: DictIdNode = DictIdNode {
+ dict_id: None,
+ children: Vec::new(),
+};
+
+impl DictIdNode {
+ /// The node at index `i` of `nodes`, or an empty node when there is none
+ pub(crate) fn nth(nodes: &[DictIdNode], i: usize) -> &DictIdNode {
+ nodes.get(i).unwrap_or(&EMPTY_NODE)
+ }
+
+ /// The node for child `i`, or an empty node when there is none
+ pub(crate) fn child(&self, i: usize) -> &DictIdNode {
+ self.children.get(i).unwrap_or(&EMPTY_NODE)
+ }
+
+ fn from_fb(field: crate::Field) -> Self {
+ Self {
+ dict_id: field.dictionary().map(|d| d.id()),
+ children: field
+ .children()
+ .map(|c| c.iter().map(Self::from_fb).collect())
+ .unwrap_or_default(),
+ }
+ }
+
+ fn from_fields(fields: &Fields) -> Arc<[Self]> {
+ fields.iter().map(|f| Self::from_field(f)).collect()
+ }
+
+ fn from_field(field: &Field) -> Self {
+ let dict_id = match field.data_type() {
+ #[expect(deprecated)]
+ DataType::Dictionary(_, _) => field.dict_id(),
+ _ => None,
+ };
+ Self {
+ dict_id,
+ children: child_fields(field.data_type())
+ .into_iter()
+ .map(Self::from_field)
+ .collect(),
+ }
+ }
+}
+
+/// The child fields of `data_type`, in IPC order.
+///
+/// For a dictionary these are the children of its value type, because the
+/// IPC format describes a dictionary field by its value type.
+pub(crate) fn child_fields(data_type: &DataType) -> Vec<&Field> {
+ match data_type {
+ DataType::Struct(fields) => fields.iter().map(|f|
f.as_ref()).collect(),
+ DataType::Union(fields, _) => fields.iter().map(|(_, f)|
f.as_ref()).collect(),
+ DataType::List(f)
+ | DataType::LargeList(f)
+ | DataType::ListView(f)
+ | DataType::LargeListView(f)
+ | DataType::FixedSizeList(f, _)
+ | DataType::Map(f, _) => vec![f.as_ref()],
+ DataType::RunEndEncoded(run_ends, values) => vec![run_ends.as_ref(),
values.as_ref()],
+ DataType::Dictionary(_, value_type) => child_fields(value_type),
+ _ => vec![],
+ }
+}
+
+fn find_in_fields<'a>(
+ fields: &'a Fields,
+ nodes: &'a [DictIdNode],
+ id: i64,
+) -> Option<(&'a Field, &'a DictIdNode)> {
+ fields
+ .iter()
+ .enumerate()
+ .find_map(|(i, f)| find_in_field(f,
nodes.get(i).unwrap_or(&EMPTY_NODE), id))
+}
+
+fn find_in_field<'a>(
+ field: &'a Field,
+ node: &'a DictIdNode,
+ id: i64,
+) -> Option<(&'a Field, &'a DictIdNode)> {
+ if node.dict_id == Some(id) && matches!(field.data_type(),
DataType::Dictionary(_, _)) {
Review Comment:
Lookup is depth-first and returns the first match, same order as the removed
`first_field_with_dict_id`. The test
`test_ipc_schema_field_with_dict_id_matches_fields_with_dict_id` checks it
against `Schema::fields_with_dict_id`.
##########
arrow-ipc/src/reader.rs:
##########
@@ -450,6 +455,9 @@ pub struct RecordBatchDecoder<'a> {
schema: SchemaRef,
/// Decoded dictionaries indexed by dictionary id
dictionaries_by_id: &'a HashMap<i64, ArrayRef>,
+ /// Dictionary ids of the schema fields, one tree per top-level field.
+ /// When `None`, they are taken from the deprecated `Field::dict_id`.
+ dict_ids: Option<Arc<[DictIdNode]>>,
Review Comment:
`None` keeps the old behaviour for the public `RecordBatchDecoder`
constructors. `with_dict_ids` is `pub(crate)` on purpose so this PR adds no new
public surface on the decoder.
##########
arrow-flight/src/utils.rs:
##########
@@ -26,13 +24,13 @@ pub fn flight_data_to_batches(flight_data: &[FlightData])
-> Result<Vec<RecordBa
let ipc_schema: arrow_ipc::Schema = message
.header_as_schema()
.ok_or_else(|| ArrowError::CastError("Cannot get header as
Schema".to_string()))?;
- let schema = try_fb_to_schema(ipc_schema)?;
- let schema = Arc::new(schema);
+ let ipc_schema = IpcSchema::try_from_fb(ipc_schema)?;
Review Comment:
Only `flight_data_to_batches` moves to the new path. `FlightDataDecoder` and
the Flight SQL client are left for a follow-up (see PR description).
##########
arrow-ipc/src/reader.rs:
##########
@@ -975,6 +1037,11 @@ fn get_dictionary_values(
dictionaries_by_id,
metadata,
)?
+ // The value type's children carry the ids of nested dictionaries
Review Comment:
Nested dictionaries inside dictionary values: the values are decoded as a
one-column batch, so we wrap the field's child nodes in a synthetic root node.
Probably the subtlest spot in the PR.
##########
arrow-ipc/src/reader/stream.rs:
##########
@@ -21,7 +22,7 @@ use crate::{CONTINUATION_MARKER, MessageHeader};
#[derive(Debug, Default)]
pub struct StreamDecoder {
/// The schema of this decoder, if read
- schema: Option<SchemaRef>,
+ schema: Option<IpcSchema>,
Review Comment:
`StreamDecoder` stores `IpcSchema` internally; the public `schema()` still
returns `SchemaRef`, so no API change.
##########
arrow-ipc/src/ipc_schema.rs:
##########
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF)
under one or more contributor license agreements
+
+//! [`IpcSchema`]: an Arrow [`Schema`](arrow_schema::Schema) together with the
IPC dictionary IDs of its fields
+
+use std::sync::Arc;
+
+use arrow_schema::{ArrowError, DataType, Field, Fields, SchemaRef};
+
+/// An Arrow [`Schema`](arrow_schema::Schema) together with the dictionary IDs
assigned to its
+/// dictionary-encoded fields by an IPC schema message.
+///
+/// In the Arrow IPC format, every dictionary-encoded field carries a
+/// dictionary ID, and dictionary batches refer to their field by that ID.
+/// The IDs are a property of the IPC encoding, not of the logical schema, so
+/// they are kept here instead of on [`Field`].
+///
+/// The IDs are stored in a tree with the same shape as the schema, so nested
+/// fields (for example a dictionary inside a struct, or inside the values of
+/// another dictionary) get their own ID, and several fields can share one ID.
+///
+/// Create one from an IPC schema message with [`IpcSchema::try_from_fb`]. The
+/// readers in this crate do this for you; use it directly with the low-level
+/// APIs such as [`read_record_batch_with_ipc_schema`] and
+/// [`FileDecoder::new_with_ipc_schema`].
+///
+/// [`read_record_batch_with_ipc_schema`]:
crate::reader::read_record_batch_with_ipc_schema
+/// [`FileDecoder::new_with_ipc_schema`]:
crate::reader::FileDecoder::new_with_ipc_schema
+#[derive(Debug, Clone, PartialEq)]
+pub struct IpcSchema {
+ pub(crate) schema: SchemaRef,
+ pub(crate) dict_ids: Arc<[DictIdNode]>,
+}
+
+impl IpcSchema {
+ /// Reads the schema and its dictionary IDs from an IPC schema message.
+ ///
+ /// The IDs always come from the message, so files that number
+ /// dictionaries in any order, or share one dictionary between fields,
+ /// are read correctly.
+ pub fn try_from_fb(fb: crate::Schema) -> Result<Self, ArrowError> {
+ let schema = crate::convert::try_fb_to_schema(fb)?;
+ let dict_ids = match fb.fields() {
+ Some(fields) => fields.iter().map(DictIdNode::from_fb).collect(),
+ None => Arc::from([]),
+ };
+ Ok(Self {
+ schema: Arc::new(schema),
+ dict_ids,
+ })
+ }
+
+ /// Builds an [`IpcSchema`] from the deprecated [`Field::dict_id`] values
+ /// stored on `schema`.
+ ///
+ /// This is how the APIs that only take a [`Schema`](arrow_schema::Schema)
find dictionary IDs.
+ pub(crate) fn from_schema_dict_ids(schema: SchemaRef) -> Self {
Review Comment:
Back-compat bridge: the old APIs that only get a `Schema`
(`read_record_batch`, `read_dictionary`, `FileDecoder::new`) build the tree
from the deprecated `Field::dict_id` here, so their behaviour is unchanged.
##########
arrow-ipc/src/ipc_schema.rs:
##########
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: Apache-2.0
+// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for
additional information regarding copyright ownership
+// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF)
under one or more contributor license agreements
+
+//! [`IpcSchema`]: an Arrow [`Schema`](arrow_schema::Schema) together with the
IPC dictionary IDs of its fields
+
+use std::sync::Arc;
+
+use arrow_schema::{ArrowError, DataType, Field, Fields, SchemaRef};
+
+/// An Arrow [`Schema`](arrow_schema::Schema) together with the dictionary IDs
assigned to its
+/// dictionary-encoded fields by an IPC schema message.
+///
+/// In the Arrow IPC format, every dictionary-encoded field carries a
+/// dictionary ID, and dictionary batches refer to their field by that ID.
+/// The IDs are a property of the IPC encoding, not of the logical schema, so
+/// they are kept here instead of on [`Field`].
+///
+/// The IDs are stored in a tree with the same shape as the schema, so nested
+/// fields (for example a dictionary inside a struct, or inside the values of
+/// another dictionary) get their own ID, and several fields can share one ID.
+///
+/// Create one from an IPC schema message with [`IpcSchema::try_from_fb`]. The
+/// readers in this crate do this for you; use it directly with the low-level
+/// APIs such as [`read_record_batch_with_ipc_schema`] and
+/// [`FileDecoder::new_with_ipc_schema`].
+///
+/// [`read_record_batch_with_ipc_schema`]:
crate::reader::read_record_batch_with_ipc_schema
+/// [`FileDecoder::new_with_ipc_schema`]:
crate::reader::FileDecoder::new_with_ipc_schema
+#[derive(Debug, Clone, PartialEq)]
+pub struct IpcSchema {
+ pub(crate) schema: SchemaRef,
+ pub(crate) dict_ids: Arc<[DictIdNode]>,
+}
+
+impl IpcSchema {
+ /// Reads the schema and its dictionary IDs from an IPC schema message.
+ ///
+ /// The IDs always come from the message, so files that number
+ /// dictionaries in any order, or share one dictionary between fields,
+ /// are read correctly.
+ pub fn try_from_fb(fb: crate::Schema) -> Result<Self, ArrowError> {
+ let schema = crate::convert::try_fb_to_schema(fb)?;
+ let dict_ids = match fb.fields() {
+ Some(fields) => fields.iter().map(DictIdNode::from_fb).collect(),
+ None => Arc::from([]),
+ };
+ Ok(Self {
+ schema: Arc::new(schema),
+ dict_ids,
+ })
+ }
+
+ /// Builds an [`IpcSchema`] from the deprecated [`Field::dict_id`] values
+ /// stored on `schema`.
+ ///
+ /// This is how the APIs that only take a [`Schema`](arrow_schema::Schema)
find dictionary IDs.
+ pub(crate) fn from_schema_dict_ids(schema: SchemaRef) -> Self {
+ let dict_ids = DictIdNode::from_fields(schema.fields());
+ Self { schema, dict_ids }
+ }
+
+ /// The Arrow schema
+ pub fn schema(&self) -> &SchemaRef {
+ &self.schema
+ }
+
+ /// Dictionary ID trees, one per top-level field
+ pub(crate) fn dict_ids(&self) -> &Arc<[DictIdNode]> {
+ &self.dict_ids
+ }
+
+ /// Returns the first field, in depth-first order, that uses dictionary
+ /// `id`, together with its node in the ID tree.
+ pub(crate) fn field_with_dict_id(&self, id: i64) -> Option<(&Field,
&DictIdNode)> {
+ find_in_fields(self.schema.fields(), &self.dict_ids, id)
+ }
+}
+
+/// The dictionary ID of one field and the IDs of its children.
+///
+/// `children` follow [`child_fields`]: for a dictionary field they are the
+/// children of its value type, matching the IPC flatbuffer layout.
+#[derive(Debug, Clone, Default, PartialEq)]
+pub(crate) struct DictIdNode {
+ pub(crate) dict_id: Option<i64>,
+ pub(crate) children: Vec<DictIdNode>,
+}
+
+static EMPTY_NODE: DictIdNode = DictIdNode {
+ dict_id: None,
+ children: Vec::new(),
+};
+
+impl DictIdNode {
+ /// The node at index `i` of `nodes`, or an empty node when there is none
+ pub(crate) fn nth(nodes: &[DictIdNode], i: usize) -> &DictIdNode {
+ nodes.get(i).unwrap_or(&EMPTY_NODE)
+ }
+
+ /// The node for child `i`, or an empty node when there is none
+ pub(crate) fn child(&self, i: usize) -> &DictIdNode {
+ self.children.get(i).unwrap_or(&EMPTY_NODE)
+ }
+
+ fn from_fb(field: crate::Field) -> Self {
+ Self {
+ dict_id: field.dictionary().map(|d| d.id()),
+ children: field
+ .children()
+ .map(|c| c.iter().map(Self::from_fb).collect())
+ .unwrap_or_default(),
+ }
+ }
+
+ fn from_fields(fields: &Fields) -> Arc<[Self]> {
+ fields.iter().map(|f| Self::from_field(f)).collect()
+ }
+
+ fn from_field(field: &Field) -> Self {
+ let dict_id = match field.data_type() {
+ #[expect(deprecated)]
+ DataType::Dictionary(_, _) => field.dict_id(),
+ _ => None,
+ };
+ Self {
+ dict_id,
+ children: child_fields(field.data_type())
+ .into_iter()
+ .map(Self::from_field)
+ .collect(),
+ }
+ }
+}
+
+/// The child fields of `data_type`, in IPC order.
+///
+/// For a dictionary these are the children of its value type, because the
+/// IPC format describes a dictionary field by its value type.
+pub(crate) fn child_fields(data_type: &DataType) -> Vec<&Field> {
Review Comment:
Key invariant: this child order must match the flatbuffer `children()` order
used in `from_fb`, and the order `create_array` walks children. For a
dictionary we descend into the value type, because IPC describes a dictionary
field by its value type. Worth checking each arm against the decoder.
##########
arrow-ipc/src/reader.rs:
##########
@@ -3902,9 +3983,131 @@ mod tests {
);
}
+ /// Writes `batch` to an IPC file and returns the buffer and footer offsets
+ fn write_file_for_ipc_schema_test(batch: &RecordBatch) -> Buffer {
+ let mut out = vec![];
+ let mut writer = crate::writer::FileWriter::try_new(&mut out,
&batch.schema()).unwrap();
+ writer.write(batch).unwrap();
+ writer.finish().unwrap();
+ drop(writer);
+ Buffer::from_vec(out)
+ }
+
+ fn nested_dict_batch() -> RecordBatch {
+ let dict = |values: &[&str]| -> ArrayRef {
+ Arc::new(
+ values
+ .iter()
+ .copied()
+ .collect::<DictionaryArray<Int32Type>>(),
+ )
+ };
+ let struct_array = StructArray::from(vec![(
+ Arc::new(Field::new(
+ "s_dict",
+ DataType::Dictionary(Box::new(DataType::Int32),
Box::new(DataType::Utf8)),
+ true,
+ )),
+ dict(&["x", "y", "x"]),
+ )]);
+ let list_values = dict(&["p", "q", "q", "p"]);
+ let list_field =
Arc::new(Field::new_list_field(list_values.data_type().clone(), true));
+ let list_array = ListArray::new(
+ list_field,
+ OffsetBuffer::from_lengths([1, 2, 1]),
+ list_values,
+ None,
+ );
+ RecordBatch::try_from_iter(vec![
+ ("top", dict(&["a", "b", "a"])),
+ ("st", Arc::new(struct_array) as ArrayRef),
+ ("lst", Arc::new(list_array) as ArrayRef),
+ ])
+ .unwrap()
+ }
+
+ /// Strips the deprecated `Field::dict_id`, so a decoder can only find the
+ /// dictionary ids through an [`IpcSchema`].
+ fn schema_without_dict_ids(schema: &Schema) -> Schema {
+ // Round trip through `Field::new`, which sets every dict_id to 0
+ fn strip(field: &Field) -> Field {
+ let data_type = match field.data_type() {
+ DataType::Struct(fields) => {
+ DataType::Struct(fields.iter().map(|f| strip(f)).collect())
+ }
+ DataType::List(f) => DataType::List(Arc::new(strip(f))),
+ other => other.clone(),
+ };
+ Field::new(field.name(), data_type, field.is_nullable())
+ }
+ Schema::new(schema.fields().iter().map(|f|
strip(f)).collect::<Vec<_>>())
+ }
+
+ #[test]
+ fn test_ipc_schema_reads_dict_ids_from_message() {
Review Comment:
Main test: strips `Field::dict_id` from the schema, then shows the new path
still decodes (ids from the tree) while `FileDecoder::new` cannot. Also covers
projection.
##########
arrow-ipc/src/reader.rs:
##########
@@ -564,6 +581,10 @@ impl<'a> RecordBatchDecoder<'a> {
let options =
RecordBatchOptions::new().with_row_count(Some(self.batch.length() as usize));
let schema = Arc::clone(&self.schema);
+ let dict_ids = match &self.dict_ids {
+ Some(dict_ids) => Arc::clone(dict_ids),
+ None =>
Arc::clone(IpcSchema::from_schema_dict_ids(Arc::clone(&schema)).dict_ids()),
Review Comment:
Fallback path: builds the tree from `Field::dict_id` once per batch. With
projection, the full tree is indexed by the original column index (`idx`), not
the projected position.
##########
arrow-ipc/src/reader.rs:
##########
@@ -847,13 +901,56 @@ pub fn read_dictionary_impl(
metadata: &MetadataVersion,
require_alignment: bool,
skip_validation: UnsafeFlag,
+) -> Result<(), ArrowError> {
+ let ipc_schema = IpcSchema::from_schema_dict_ids(Arc::new(schema.clone()));
Review Comment:
`read_dictionary_impl` is now a thin wrapper; all logic lives in
`read_dictionary_with_ipc_schema_impl`. One extra `Schema` clone per dictionary
batch on the legacy path only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]