linliu-code opened a new issue, #11960:
URL: https://github.com/apache/arrow-rs/issues/11960
**Describe the bug**
When a reader schema is supplied, `arrow-avro` compares the names of writer
and reader records (and enums and fixed types) using only each type's own
`namespace` attribute. A nested named type with no `namespace` attribute
inherits the namespace of the enclosing named type, per the spec, but the
comparison treats it as having no namespace. Two schemas that differ only in
whether the inherited namespace is written out explicitly are therefore treated
as different names.
This affects files written by `apache-avro` (Rust), which writes the
inherited namespace explicitly on every nested named type in the container-file
header. Reading such a file with the schema the user wrote (the same schema,
with the namespace left inherited) fails:
- if the nested record is a union branch, the branch is left unmapped and
decoding fails with `Parser error: Union branch index 1 not resolvable by
reader schema`;
- if the nested record is a plain field type, building the reader fails with
`Record name mismatch writer=Bw, reader=Bw`.
Reading without a reader schema, or with a reader schema that spells out the
inherited namespace, works.
The Avro specification, section "Names":
> A simple name only is specified (a name that contains no dots). In this
case the namespace is taken from the most tightly enclosing named schema or
protocol, and the fullname is constructed from that namespace and the name.
So in the schema below both the writer's and the reader's nested records
have the full name `ns.Bw`.
**To Reproduce**
Dependencies: `arrow-avro = "60.0.0"`, `apache-avro = "0.22"`.
```rust
use std::io::Cursor;
use apache_avro::{types::Value, Schema as ApacheSchema, Writer};
use arrow_avro::reader::ReaderBuilder;
use arrow_avro::schema::AvroSchema;
fn read(bytes: &[u8], reader_schema: Option<&str>) -> Result<usize, String> {
let mut b = ReaderBuilder::new();
if let Some(s) = reader_schema {
b = b.with_reader_schema(AvroSchema::new(s.to_string()));
}
let mut r = b.build(Cursor::new(bytes.to_vec())).map_err(|e|
format!("{e:?}"))?;
r.next().unwrap().map(|b| b.num_rows()).map_err(|e| format!("{e:?}"))
}
fn main() {
// Bw and Sw have no namespace attribute, so their full names are ns.Bw
and ns.Sw.
let schema_json = r#"{"type": "record", "name": "Root", "namespace":
"ns", "fields": [
{"name": "v", "type": ["null",
{"type": "record", "name": "Bw", "fields": [{"name": "value",
"type": "bytes"}]},
{"type": "record", "name": "Sw", "fields": [{"name": "value",
"type": "string"}]}]},
{"name": "tail", "type": "long"}]}"#;
let schema = ApacheSchema::parse_str(schema_json).unwrap();
let mut bytes = Vec::new();
{
let mut w = Writer::new(&schema, &mut bytes).unwrap();
for tail in [1i64, 2] {
let bw = Value::Record(vec![("value".into(),
Value::Bytes(b"a".to_vec()))]);
w.append_value_ref(&Value::Record(vec![
("v".into(), Value::Union(1, Box::new(bw))),
("tail".into(), Value::Long(tail)),
])).unwrap();
}
w.flush().unwrap();
}
// The same schema with the inherited namespace written out on Bw and Sw,
// as apache_avro stores it in the file header.
let explicit_json =
r#"{"type":"record","namespace":"ns","name":"Root","fields":[{"name":"v","type":["null",{"type":"record","namespace":"ns","name":"Bw","fields":[{"name":"value","type":"bytes"}]},{"type":"record","namespace":"ns","name":"Sw","fields":[{"name":"value","type":"string"}]}]},{"name":"tail","type":"long"}]}"#;
println!("no reader schema: {:?}", read(&bytes, None));
println!("reader schema = schema_json: {:?}", read(&bytes,
Some(schema_json)));
println!("reader schema = explicit_json: {:?}", read(&bytes,
Some(explicit_json)));
// Not specific to unions: a plain nested record field fails at build
time.
let rec_json = r#"{"type": "record", "name": "Root", "namespace": "ns",
"fields": [
{"name": "r", "type": {"type": "record", "name": "Bw", "fields":
[{"name": "value", "type": "bytes"}]}}]}"#;
let rec_schema = ApacheSchema::parse_str(rec_json).unwrap();
let mut rec_bytes = Vec::new();
{
let mut w = Writer::new(&rec_schema, &mut rec_bytes).unwrap();
let bw = Value::Record(vec![("value".into(),
Value::Bytes(b"a".to_vec()))]);
w.append_value_ref(&Value::Record(vec![("r".into(), bw)])).unwrap();
w.flush().unwrap();
}
println!("nested record, reader = rec_json: {:?}", read(&rec_bytes,
Some(rec_json)));
}
```
The header schema `apache-avro` writes for `schema_json` (the `avro.schema`
metadata value, verbatim) is the `explicit_json` string above:
```json
{"type":"record","namespace":"ns","name":"Root","fields":[{"name":"v","type":["null",{"type":"record","namespace":"ns","name":"Bw","fields":[{"name":"value","type":"bytes"}]},{"type":"record","namespace":"ns","name":"Sw","fields":[{"name":"value","type":"string"}]}]},{"name":"tail","type":"long"}]}
```
**Expected behavior**
All four reads succeed: `schema_json` and `explicit_json` describe the same
schema, so the reader schema is identical to the writer schema in every case.
**Actual behavior**
```text
no reader schema: Ok(2)
reader schema = schema_json: Err("AvroError(\"Parser error: Union
branch index 1 not resolvable by reader schema\")")
reader schema = explicit_json: Ok(2)
nested record, reader = rec_json: Err("ParseError(\"Record name mismatch
writer=Bw, reader=Bw\")")
```
Without the root `"namespace": "ns"` (so no nested type inherits a
namespace), the read with the reader schema succeeds.
Reproduced on `main` (5db01b1fa) and on the 60.0.0 release.
**Root cause**
`resolve_records` checks names with `ensure_names_match`, passing each
record's own `namespace` attribute but not the enclosing namespace, which is
available as its `namespace` parameter:
https://github.com/apache/arrow-rs/blob/5db01b1fa7ba6f70d5f63abf31c48fabed40929b/arrow-avro/src/codec.rs#L2280-L2294
`ensure_names_match` → `names_match` → `full_name_set` builds the full name
with `make_full_name(name, ns, None)`, i.e. with no enclosing namespace, so the
writer's `Bw` (explicit `namespace: "ns"`) becomes `ns.Bw` and the reader's
`Bw` (inherited) becomes `Bw`:
https://github.com/apache/arrow-rs/blob/5db01b1fa7ba6f70d5f63abf31c48fabed40929b/arrow-avro/src/codec.rs#L1273-L1282
`resolve_enums` and `resolve_fixed` call `ensure_names_match` the same way:
https://github.com/apache/arrow-rs/blob/5db01b1fa7ba6f70d5f63abf31c48fabed40929b/arrow-avro/src/codec.rs#L2090-L2098
https://github.com/apache/arrow-rs/blob/5db01b1fa7ba6f70d5f63abf31c48fabed40929b/arrow-avro/src/codec.rs#L2199-L2207
In a union, the failed name check makes `find_best_union_match` skip the
reader branch, so the writer branch has no mapping and the decoder fails when
it reads a value in that branch:
https://github.com/apache/arrow-rs/blob/5db01b1fa7ba6f70d5f63abf31c48fabed40929b/arrow-avro/src/codec.rs#L1973-L1997
https://github.com/apache/arrow-rs/blob/5db01b1fa7ba6f70d5f63abf31c48fabed40929b/arrow-avro/src/reader/record.rs#L2222
`make_full_name` already accepts an enclosing namespace (it is used that way
for union branch keys and the canonical form); the name check needs the
enclosing namespace of each side passed through. Note that inside
`resolve_records` the fields are resolved with `reader_ns` as the enclosing
namespace for both writer and reader field types, so the writer side's
enclosing namespace may need to be tracked separately for the case where writer
and reader root namespaces differ.
**Additional context**
The specification's schema-resolution rules match records, enums and fixed
by unqualified name ("both schemas are records with the same (unqualified)
name"), so full-name comparison is stricter than the specification requires;
this report is about the narrower case where the full names are in fact equal.
Related: #11958
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
