zclllyybb commented on issue #65866:
URL: https://github.com/apache/doris/issues/65866#issuecomment-5041491285
Breakwater-GitHub-Analysis-Slot: slot_8ed64919122e
This content is generated by AI for reference only.
Initial code-path analysis:
I checked the reported commit `a78b707ed040c3206df99cf43ea3101ab2ba5b90` and
the current head of PR #62606 for the relevant BE storage writer code. I did
not run a local cluster reproduction, but the reported error matches a concrete
writer-path mismatch.
This looks like a real BE ROW-binlog handling bug for
`UPDATE_FLEXIBLE_COLUMNS`, not an IVM refresh-layer problem.
What happens:
1. When ROW binlog is enabled, the load creates a row-binlog sub-writer and
`SegmentFlusher::flush_single_block()` disables `VerticalSegmentWriter` for
that row-binlog write path. It then creates `RowBinlogSegmentWriter`.
2. `RowBinlogSegmentWriter::append_block()` treats every partial update as a
sparse fixed-column partial-update block and rejects:
```cpp
block->columns() <= source_schema->num_key_columns()
|| block->columns() >= source_schema->num_columns()
```
That is valid for fixed-column partial update, but not for flexible partial
update. Flexible partial update intentionally carries a full source-schema
block plus the skip-bitmap column, so `block->columns() ==
source_schema->num_columns()` is the expected shape.
3. The non-binlog flexible path is different:
`VerticalSegmentWriter::batch_block()` explicitly accepts `block->columns() ==
_tablet_schema->num_columns()`, and
`_append_block_with_flexible_partial_content()` uses the skip bitmap plus
historical-row lookup to build the full AFTER row before writing the segment.
4. For flexible mode, `PartialUpdateInfo::init()` does not populate
`update_cids` as a fixed sparse column list. It records all non-key columns in
`missing_cids`, while the per-row updated/missing cells are represented by the
skip bitmap. So the row-binlog writer cannot be fixed by only relaxing the
column-count check; it also needs flexible-specific skip-bitmap handling to
avoid writing placeholder values or double-writing columns.
This explains the matrix in the issue:
- Flexible partial update without ROW binlog succeeds because it uses the
vertical flexible path.
- Fixed-column partial update with ROW binlog succeeds because the
row-binlog writer's sparse-column assumption still holds.
- Flexible partial update with ROW binlog fails before it can build the full
AFTER/BEFORE images.
Suggested fix direction:
1. Add an explicit `is_flexible_partial_update()` branch in
`RowBinlogSegmentWriter::append_block()`.
2. Accept `block->columns() == source_schema->num_columns()` only for
flexible partial update.
3. For ROW binlog normal columns, build the completed AFTER block first
using skip-bitmap semantics equivalent to
`VerticalSegmentWriter::_append_block_with_flexible_partial_content()` /
`FlexibleReadPlan`, then append each row-binlog normal column exactly once from
that completed AFTER image.
4. Preserve `binlog.need_historical_value` / BEFORE-image behavior by using
historical rows for BEFORE columns, and verify op generation for existing-row
update, new-row insert, and delete rows.
5. Add a regression test for a MOW UNIQUE table with ROW binlog, skip
bitmap, and sequence column, covering both an existing-row flexible update and
a new-row insert in the same stream load. The test should also verify the
fixed-column ROW-binlog path remains unaffected and that row-binlog output has
the expected before/after data.
Useful extra validation material:
- Full Stream Load response JSON and BE log lines around the failing
label/txn.
- The tablet schema dump for both the base tablet and row-binlog tablet if
there is any schema-layout doubt.
- After a patch, dump the row-binlog rows for the repro to confirm omitted
flexible columns are filled from the historical row/defaults instead of
placeholder input values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
