JingsongLi opened a new pull request, #54:
URL: https://github.com/apache/paimon-mosaic/pull/54
## Summary
Add STRUCT (ROW) type support where each field is expanded into an
**independent top-level column** with dot-separated naming (e.g., `info.name`,
`info.age`). These expanded columns participate in global name sorting and
bucket assignment, maximizing Mosaic's projection pushdown for wide nested
schemas.
Unlike ARRAY/MAP which expand within a bucket, STRUCT expansion happens at
the **schema level** — a STRUCT with 50 fields distributes those fields across
multiple buckets, so projecting a single field reads only the relevant
bucket(s).
## Design
```
Schema: id INT, info STRUCT<name UTF8, age INT> NULLABLE
Expanded columns (participate in bucket assignment):
id → bucket 0
info.__null__ → bucket 1 (STRUCT null bitmap)
info.age → bucket 1
info.name → bucket 2
```
Key decisions:
- **Dot-separated naming**: `info.name`, nested STRUCT recurses (`a.b.c`)
- **Nullable STRUCT**: virtual `__null__` Boolean column tracks STRUCT-level
null vs all-fields-null
- **Null propagation**: writer propagates STRUCT null to all child field
arrays
- **STRUCT fields can be ARRAY/MAP**: expanded as independent columns, then
each uses standard flattened storage in its bucket
- **Name conflict detection**: validate rejects schemas where expansion
creates duplicate column names
- **Serialization**: expanded columns stored directly (no STRUCT type byte
in file schema); reader reconstructs from dot-naming convention
- **Type byte 20**: reserved for STRUCT as element type in ARRAY/MAP (e.g.,
`ARRAY<STRUCT<...>>`)
## Changes
- **core/types.rs**: Type byte 20, validate STRUCT (recursive),
serialize/deserialize with field names
- **core/schema.rs**: `from_arrow` expands STRUCT, `StructMapping`,
`detect_struct_mappings`, conflict check
- **core/writer.rs**: `expand_batch_arrays` splits StructArray + null
propagation
- **core/reader.rs**: Reassemble StructArray from dot-named columns in
`read_columns`
- **docs/design.html**: STRUCT Type Storage section, TypeDescriptor table
- **Golden file**: `v1_with_struct.mosaic`
## Test Plan
- [x] Rust: 3 STRUCT tests (basic, nullable, with ARRAY field) + golden file
tests
- [x] Java: 1 STRUCT test (`testStructType`)
- [x] Python: 1 STRUCT test (`test_struct_type`)
- [x] C++: 1 STRUCT test (`test_struct_type`)
- [x] All existing ARRAY/MAP tests pass (252 Rust + 38 Python + 29 Java)
- [x] Clippy clean (`-D warnings`)
## Known Limitations
- STRUCT field order may change to alphabetical after roundtrip (original
order not preserved)
- STRUCT projection pushdown not yet implemented (projecting one field of a
STRUCT requires reading all sub-fields)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
