Doris-Breakwater commented on issue #66450: URL: https://github.com/apache/doris/issues/66450#issuecomment-5180040901
Breakwater-GitHub-Analysis-Slot: slot_a59f1916a612 ## Preliminary assessment **Classification: expected behavior of fixed-column partial update, not evidence of a 4.0.11 regression. Confidence: high.** The issue currently has no labels. `partial_columns=true` is the backward-compatible spelling of `unique_key_update_mode=UPDATE_FIXED_COLUMNS`; the 4.0 Routine Load analyzer maps it explicitly to that mode ([source](https://github.com/apache/doris/blob/68261b8583982d09a8129556acdfdb457e132dba/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateRoutineLoadInfo.java#L205-L221)). In fixed-column mode, the `COLUMNS` clause defines one update-column set for the whole load job/batch. It is not recalculated from the keys present in each JSON object. The JSON reader then processes every declared JSONPath. When a path does not match, it calls `_fill_missing_column` ([source](https://github.com/apache/doris/blob/68261b8583982d09a8129556acdfdb457e132dba/be/src/vec/exec/format/json/new_json_reader.cpp#L1433-L1478)); for a nullable column with no configured default, that function inserts the nullable default, i.e. `NULL` ([source](https://github.com/apache/doris/blob/68261b8583982d09a8129556acdfdb457e132dba/be/src/vec/exec/format/json/new_json_reader.cpp#L1525-L1549)). Because `last_contact_time` is part of the job's fixed update set, that `NULL` participates in the update. This also explains why removing `FROM_MILLISECOND()` did not change the result and why replacing it with `IF(... IS NULL, '2099-01-01', ...)` wrote the sentinel: a load expression produces a value; it cannot produce the per-row "do not update this column" state. This matches the documented distinction: fixed partial update requires every row in a load to update the same columns, while flexible partial update determines the columns from each JSON row ([column-update documentation](https://doris.apache.org/docs/4.x/data-operate/update/partial-column-update/)). ## Answers to the questions 1. **Yes, this is expected under `partial_columns=true` / `UPDATE_FIXED_COLUMNS`.** 2. **Yes, a column included in the fixed import mapping participates even when its JSONPath is absent.** For a nullable column without a configured default, the extracted value is `NULL`. 3. **There is no supported way in one fixed-column Routine Load job to keep `jsonpaths` and make a mapped column conditional on per-message field presence.** Removing `last_contact_time` from both the input mapping and the update set makes that job never update the column, but cannot provide "update only when present" semantics. An `IF` expression cannot preserve the old value because it has no access to or skip marker for that stored value. 4. **For one Routine Load job whose rows have different update-column sets, `UPDATE_FLEXIBLE_COLUMNS` is the supported mode.** It records absent fields in a skip bitmap. Explicit JSON `null` remains an update to `NULL`; an absent key is skipped. If flexible mode cannot be used, the practical supported alternative is to route records with different fixed schemas to separate topics/jobs, with each job declaring only its own fixed update columns. ## Important migration constraints Routine Load intentionally rejects `UPDATE_FLEXIBLE_COLUMNS` when either `jsonpaths` or a `COLUMNS` clause is present ([validation](https://github.com/apache/doris/blob/68261b8583982d09a8129556acdfdb457e132dba/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateRoutineLoadInfo.java#L453-L479)). Therefore the shown job cannot simply change one property: - The table must be a Merge-on-Write Unique Key table with light schema change and the hidden skip-bitmap column, and it must not contain a `VARIANT` column. For an eligible existing table, the analyzer suggests `ALTER TABLE data_wework_customer ENABLE FEATURE "UPDATE_FLEXIBLE_COLUMNS";` ([table validation](https://github.com/apache/doris/blob/68261b8583982d09a8129556acdfdb457e132dba/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java#L3016-L3040)). Verify completion with `SHOW CREATE TABLE`. - Recreate the Routine Load job with `"format"="json"` and `"unique_key_update_mode"="UPDATE_FLEXIBLE_COLUMNS"`, without `partial_columns`, `jsonpaths`, `COLUMNS`, or `fuzzy_parse`. - Every JSON object must contain all non-auto-increment key columns. JSON key names must directly match Doris column names, and values must already be load-compatible. In this case, the producer/upstream transformation would need to convert camelCase names to the table's snake_case names and convert millisecond timestamps to values accepted by the target `DATETIME` columns, because the current JSONPath renaming and `FROM_MILLISECOND` mappings cannot be retained in flexible mode. ## Missing information / maintainer next step No BE log, load profile, or Doris query profile is needed to establish the current semantics. Before prescribing an exact migration, please provide: - the unabridged `SHOW CREATE TABLE data_wework_customer` output, to verify the skip-bitmap/light-schema-change prerequisites and absence of unsupported table features; - `SHOW CREATE ROUTINE LOAD customer` (or the complete actual job definition), because the issue's experiment mentions `temp_last_contact_time` while the displayed definition maps `last_contact_time` directly; - the full `SELECT VERSION()` output including the build/Git commit if exact binary-to-source verification is desired. The available public source has no `4.0.11` tag, so the code inspection above used public `branch-4.0` commit `68261b8583982d09a8129556acdfdb457e132dba` dated 2026-08-01. The reported behavior is fully consistent with that implementation. I recommend maintainers answer this as a documented fixed-vs-flexible mode distinction and close it as expected behavior; if support for per-row omission together with JSONPath renaming/load expressions is desired, that should be tracked as a separate enhancement because the current analyzer and reader explicitly exclude that combination. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
