Doris-Breakwater commented on issue #66072: URL: https://github.com/apache/doris/issues/66072#issuecomment-5086551285
Breakwater-GitHub-Analysis-Slot: slot_9d3b5a09b4ee This content is generated by AI for reference only. Initial assessment: this is not consistent with a silent write-path failure. The evidence shows that FE removed the partition from the table metadata first; the BE tablet and its rowsets then remained temporarily as orphaned storage and were cleaned later. Evidence from the report: - Tablet `1784016422686` was created successfully for table `1783304514830`, partition `1784016422685`, and rowset files were still written later on July 24. - `SHOW TABLET 1784016422686` could still resolve the stale tablet-inverted-index entry, but its `PartitionName` was `NULL`, and the generated `SHOW PROC` path failed with `Partition[1784016422685] does not exist`. This means the partition was already absent from the table's partition map; the files on disk did not make it query-visible. - On July 25, `ReportHandler.deleteFromBackend()` logged `failed add to meta` followed by `delete tablet ... because not found in meta`. These messages are a consequence of the missing partition, not the original cause. The 4.1.2 code explains the roughly one-day delay. A non-force partition drop removes the partition from the table and puts it in the catalog recycle bin while retaining its tablet metadata. When the recycle-bin entry expires, `CatalogRecycleBin` calls `onErasePartition`; a subsequent tablet report then treats the BE copy as orphaned and schedules deletion. The default `catalog_trash_expire_second` in this version is 86400 seconds. This lifecycle matches the screenshots closely and is independent of whether the table is AGGREGATE or DUPLICATE. The most likely trigger is auto-partition cleanup by `partition.retention_count` or `dynamic_partition.*`, although an explicit `DROP PARTITION`, `REPLACE PARTITION`/insert-overwrite operation, or another metadata operation must still be ruled out. For AUTO RANGE tables, `DynamicPartitionScheduler` deliberately creates non-force drop clauses for historical partitions beyond the retention count. Whether partition `1784016422685` was correctly classified as historical cannot be determined without its name/range, the partition-key value written by the load, the table properties, and the FE time-zone context. There is a relevant newer fix: [#65956](https://github.com/apache/doris/pull/65956), commit [`e69f225b92a`](https://github.com/apache/doris/commit/e69f225b92a32195f856205dba5ad3d5c7466472), snapshots auto-partition result metadata under the table read lock and returns a retry error when retention concurrently drops a just-created partition. The reported commit `aec169d2025` is the 4.1.2 tag target; this fix is absent from both 4.1.2 and 4.1.3, is present on `branch-4.1`, and is currently labeled for 4.1.4. It is relevant to this race shape, but it must not yet be treated as a complete fix for this report: it does not change retention selection and does not prevent a later, policy-driven drop after a successful load. Information needed to confirm the root trigger: 1. `SHOW CREATE TABLE` for each affected table, including all `partition.retention_count` and `dynamic_partition.*` properties; also provide any ALTER statements that changed those properties. 2. The partition name and range for partition ID `1784016422685`, plus representative source partition-key values from the successful loads. If the entry still exists, include the relevant row from `SHOW CATALOG RECYCLE BIN`. 3. Leader FE logs from before tablet creation until the first time the partition became unavailable, searched for partition ID `1784016422685`, tablet ID `1784016422686`, and messages including `recycle partition` and `succeed in dropping partition`. Please also include FE audit-log entries for `DROP/ALTER/REPLACE/INSERT OVERWRITE` in that interval and any leader-switch timeline. 4. The complete load responses and labels/transaction IDs (including `Status`, `TxnId`, loaded/filtered row counts), rather than only the writer's success summary. 5. Effective values of `catalog_trash_expire_second` and `dynamic_partition_check_interval_seconds`, FE JVM/system time zone, session/default `time_zone`, and the output of `SHOW FRONTENDS`. Recommended next steps: - Preserve the FE logs and affected BE tablet directories immediately; avoid force-dropping, manually deleting, or cleaning them while recovery is being assessed. - Check `SHOW CATALOG RECYCLE BIN` before the entry expires. If the partition is present, correct/disable the policy that dropped it first, then consider `RECOVER PARTITION <partition_name> FROM <db>.<table>`; otherwise the scheduler may drop it again. - If the missing partition is outside the configured retention window, the deletion is policy-driven and ingestion must reject/redirect such late data or the retention policy must be adjusted. - If it is a current or retained-range partition and there is no explicit metadata operation, treat this as a scheduler bug. With the requested logs and DDL, maintainers can reproduce the exact range/time-zone decision and determine whether an additional fix beyond #65956 is required. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
