linliu-code opened a new issue, #11958:
URL: https://github.com/apache/arrow-rs/issues/11958
**Describe the bug**
When a reader schema is supplied, a writer union with more than two branches
whose branches reference named types by name (for example `["null", "Bw",
"Sw"]`, where `Bw` and `Sw` are records defined earlier in the schema) is
resolved incorrectly: every by-name branch is mapped to reader branch 0. If
branch 0 is `null`, a non-null value of that union is decoded as null, its
bytes are not consumed, and the rest of the record (and every following record)
is decoded from the wrong offset. Decoding then fails with an error that
depends on the bytes that follow, e.g. `Parser error: bad varint` or `Parser
error: Union branch index 61 not resolvable by reader schema`.
This happens even when the reader schema is identical to the writer schema.
Decoding the same bytes without a reader schema is correct.
We found this while reading Apache Hudi metadata-table files, whose
column-statistics record has `minValue` defined as a union of inline wrapper
records and `maxValue` as a union referencing the same records by name.
**To Reproduce**
Writer schema (also used as the reader schema):
```json
{"type": "record", "name": "Root", "fields": [
{"name": "minValue", "type": ["null",
{"type": "record", "name": "Bw", "fields": [{"name": "value", "type":
"bytes"}]},
{"type": "record", "name": "Sw", "fields": [{"name": "value", "type":
"string"}]}]},
{"name": "maxValue", "type": ["null", "Bw", "Sw"]},
{"name": "tail", "type": "long"}
]}
```
Dependencies: `arrow-avro = "60.0.0"`, `apache-avro = "0.22"`.
```rust
use std::io::Cursor;
use apache_avro::{types::Value, Schema as ApacheSchema, Writer};
use arrow_avro::reader::ReaderBuilder;
use arrow_avro::schema::AvroSchema;
fn main() {
let schema_json = r#"{"type": "record", "name": "Root", "fields": [
{"name": "minValue", "type": ["null",
{"type": "record", "name": "Bw", "fields": [{"name": "value",
"type": "bytes"}]},
{"type": "record", "name": "Sw", "fields": [{"name": "value",
"type": "string"}]}]},
{"name": "maxValue", "type": ["null", "Bw", "Sw"]},
{"name": "tail", "type": "long"}]}"#;
let schema = ApacheSchema::parse_str(schema_json).unwrap();
let bw = |v: &[u8]| Value::Union(1,
Box::new(Value::Record(vec![("value".into(), Value::Bytes(v.to_vec()))])));
let mut bytes = Vec::new();
{
let mut w = Writer::new(&schema, &mut bytes).unwrap();
for tail in [1i64, 2] {
w.append_value_ref(&Value::Record(vec![
("minValue".into(), bw(b"a")),
("maxValue".into(), bw(b"zz")),
("tail".into(), Value::Long(tail)),
])).unwrap();
}
w.flush().unwrap();
}
// Writer schema only: decodes correctly.
let mut r =
ReaderBuilder::new().build(Cursor::new(bytes.clone())).unwrap();
println!("{:?}", r.next().unwrap().map(|b| b.num_rows()));
// Reader schema == writer schema: fails.
let mut r = ReaderBuilder::new()
.with_reader_schema(AvroSchema::new(schema_json.to_string()))
.build(Cursor::new(bytes))
.unwrap();
println!("{:?}", r.next().unwrap().map(|b| b.num_rows()));
}
```
**Expected behavior**
Both reads return the two rows, with `maxValue` decoded as the `Bw` branch
holding `"zz"` and `tail` = 1, 2.
**Actual behavior**
```text
Ok(2)
Err(AvroError("Parser error: Union branch index 61 not resolvable by reader
schema"))
```
The writer-only read succeeds. The read with the reader schema fails:
`maxValue`'s `Bw` branch is mapped to the `null` branch, so the 3 bytes
encoding `"zz"` (length prefix 0x04, then `z`, `z`) are not consumed as
`maxValue`; `tail` is read from the length prefix (0x04 decodes to 2), and the
next record starts at the first `z` (`Union branch index 61` is the zig-zag
decoding of `z`, 0x7a). With other data the error is `Parser error: bad varint`.
Reproduced on `main` (5db01b1fa) and on the 58.4.1 and 60.0.0 releases.
**Root cause**
`Maker::resolve_type` in `arrow-avro/src/codec.rs` has two arms that accept
a named-type reference against any schema on the other side, without comparing
names:
https://github.com/apache/arrow-rs/blob/0b94195639ccbae0d615596cdb11445cde1441b6/arrow-avro/src/codec.rs#L1965-L1966
```rust
(Schema::TypeName(TypeName::Ref(_)), _) => self.parse_type(reader_schema,
namespace),
(_, Schema::TypeName(TypeName::Ref(_))) => self.parse_type(reader_schema,
namespace),
```
For a union that is not a two-branch nullable union, `resolve_unions` calls
`find_best_union_match` for each writer branch, which returns the first reader
branch whose resolution is an exact match:
https://github.com/apache/arrow-rs/blob/0b94195639ccbae0d615596cdb11445cde1441b6/arrow-avro/src/codec.rs#L1973
For a by-name writer branch, the first reader branch tried (`null`) hits the
first arm, parses successfully with no resolution info, and is taken as an
exact match. Branches defined inline are not affected because they go through
`resolve_records` / `resolve_enums` / `resolve_fixed`, which check names.
A reference should only resolve against a reference or a named definition
with the same full name (or a matching alias).
**Additional context**
A related case that the name check alone does not address: for a two-branch
nullable union `["null", "T"]`, `resolve_type` resolves the non-null branches
directly, and a writer-side reference resolves to the *reader's* definition of
`T` via `parse_type`, without resolving it against the writer's definition. If
the writer and reader define the named type `T` differently (schema evolution
of `T`), values read through the reference are decoded with the reader's layout
rather than the writer's, which can mis-decode. Fixing that requires access to
the writer's definition of the referenced type at resolution time, which the
resolver currently stores only as a parsed `AvroDataType`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
