zeroshade commented on PR #918: URL: https://github.com/apache/arrow-go/pull/918#issuecomment-4929415496
### How Arrow C++ handles run-end-encoded value nullability For context on the REE value-nullability discussion here, comparing against the C++ reference implementation: **C++ has no configurable value nullability.** `RunEndEncodedType` hard-wires the `values` child to `nullable=true` ([type.cc#L1307](https://github.com/apache/arrow/blob/30ca5a7ef262b28bacab50768531c26af5872d3c/cpp/src/arrow/type.cc#L1307-L1313)): ```cpp children_ = {std::make_shared<Field>("run_ends", std::move(run_end_type), false), std::make_shared<Field>("values", std::move(value_type), true)}; ``` - **DataType equality** compares child *data types* only — the values field's nullability is not part of REE type identity ([compare.cc#L883](https://github.com/apache/arrow/blob/30ca5a7ef262b28bacab50768531c26af5872d3c/cpp/src/arrow/compare.cc#L883-L887)): ```cpp result_ = left.value_type()->Equals(right.value_type()) && left.run_end_type()->Equals(right.run_end_type()); ``` - **Array equality** merges runs and compares the decoded `values` (validity included, since values are always nullable); it never consults `Field::nullable`. - **IPC** reconstructs `RunEndEncodedType(children[0]->type(), children[1]->type())` ([metadata_internal.cc#L423](https://github.com/apache/arrow/blob/30ca5a7ef262b28bacab50768531c26af5872d3c/cpp/src/arrow/ipc/metadata_internal.cc#L423-L432)) — the values-field nullable bit is dropped and re-fixed to nullable, so a non-nullable-values REE does not round-trip. - **Spec** ([Columnar.rst](https://github.com/apache/arrow/blob/30ca5a7ef262b28bacab50768531c26af5872d3c/docs/source/format/Columnar.rst#L1090-L1153)): `run_ends` cannot be null; `values` may be null. **Implications:** 1. Excluding `ValueNullable` from `arrow.TypeEqual` (as it is today) matches C++. Making it part of type identity would be stricter than the reference and breaks REE record/IPC construction in arrow-go (schema-vs-array type mismatch, since `ValueNullable` isn't set consistently and doesn't survive IPC). 2. More broadly, C++ never makes `Equal()` lenient based on a field's `Nullable` flag — it keeps equality strict and enforces schema nullability via `Validate()`. That is the direction taken in #833, and it's why the `equalOpts` mechanism in this PR diverges from the reference. This branch remains useful as the record of the alternative approach, but the C++-aligned path is `Validate()`-based (#833). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
