JingsongLi commented on code in PR #957:
URL: https://github.com/apache/paimon-rust/pull/957#discussion_r4205955050
##########
crates/paimon/src/table/table_write.rs:
##########
@@ -635,6 +635,38 @@ impl TableWrite {
self.table.schema().options(),
)?;
}
+ let config = PartialUpdateConfig::new(self.table.schema().options());
+ let batch = if !self.primary_key_indices.is_empty()
+ && config.is_enabled()
+ && config.remove_record_on_delete()
+ {
+ let batch = if batch
+ .schema()
+ .column_with_name(VALUE_KIND_FIELD_NAME)
+ .is_none()
+ {
+ Self::add_per_row_value_kind_column(
+ &batch,
+ vec![RowKind::Insert.to_value(); batch.num_rows()],
+ )?
+ } else {
+ batch
+ };
+ // NULL kinds are Inserts. Every buffered batch must allow them,
+ // including batches with synthesized Insert kinds.
+ let schema = batch.schema();
+ let kind_index = schema.index_of(VALUE_KIND_FIELD_NAME).unwrap();
+ let mut schema =
arrow_schema::SchemaBuilder::from(schema.as_ref());
+ Arc::make_mut(schema.field_mut(kind_index)).set_nullable(true);
Review Comment:
[P2] Replace NULL kinds with Insert before buffering input changelogs
Making the field nullable retains actual NULL values, while the input
changelog physical schema still requires a non-null _VALUE_KIND. With
merge-engine=partial-update, partial-update.remove-record-on-delete=true and
changelog-producer=input, I wrote an ordinary batch (id=1,value=10) followed by
a batch (id=2,value=20,_VALUE_KIND=NULL). Both write_arrow_batch calls succeed,
then prepare_commit fails: "Column _VALUE_KIND is declared as non-nullable but
contains null values". Key-value merging handles NULL as Insert for the data
file, but input changelog serialization copies the original [0,NULL] array into
its required kind field. The exact upstream normalizer control successfully
writes, commits and reads both Insert rows in this same mixed-schema scenario;
its first ordinary schema excludes the extra column and the changelog
synthesizes Inserts. Please materialize NULL kinds as RowKind::Insert during
this normalization, keep buffered schemas compatible, and test mixed nullable ki
nds with an input changelog through prepare/commit/read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
