yihua opened a new issue, #867: URL: https://github.com/apache/hudi-rs/issues/867
## Summary The native writer checks a write batch against the table schema in two places with different rules. `append` uses `ensure_batch_schema_matches` (`crates/core/src/write/append.rs`), which compares field names and types and also accepts a batch whose Arrow to Avro to Arrow round trip equals the table schema. The rewrite verbs (`upsert`, MOR upsert, `overwrite`, `dynamic_partition_overwrite`) use `validate_write_input` (`crates/core/src/write/rewrite/validate.rs`), which calls `align_batch_to_schema` and requires exact names, types and order. A batch can therefore be accepted by one verb and refused by another for the same table, and every schema rule (such as nullability widening) has to be kept in step in both. ## Proposal Settle on one rule, matching what Java's writer accepts against the table schema, and move it into a single `pub(crate)` check in `write/mod.rs` that both paths call. This changes which batches one of the paths accepts, so it needs tests for the inputs that move from accepted to refused or the other way. Related: #851. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
