gavinchou commented on PR #64167: URL: https://github.com/apache/doris/pull/64167#issuecomment-4849864872
Potential metadata durability issue in the CREATE TABLE path. For tenant-level colocate, `colocate_group` / `colocate_slave` are parsed and removed from table properties before the table is persisted. Then `OP_CREATE_TABLE` is logged first, and the tenant-level colocate add-table edit log is written later. If the master FE crashes or fails over after `OP_CREATE_TABLE` is durable but before the V2 colocate log is written, replay will recreate the table but will not rebuild `TenantLevelColocateTableIndex`. The table would then lose its tenant-level colocate semantics. Relevant code paths: - `PropertyAnalyzer` removes the V2 properties: https://github.com/apache/doris/blob/cde59482ce5a548a2652c3aead57096a9c832f22/fe/fe-core/src/main/java/org/apache/doris/common/util/PropertyAnalyzer.java#L784-L789 - table creation logs `OP_CREATE_TABLE` before V2 colocate membership logs: https://github.com/apache/doris/blob/cde59482ce5a548a2652c3aead57096a9c832f22/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java#L3318-L3400 - `replayCreateTable` recreates the table and inverted index, but does not reconstruct tenant-level colocate metadata: https://github.com/apache/doris/blob/cde59482ce5a548a2652c3aead57096a9c832f22/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java#L1322-L1366 Can we make table creation and tenant-level colocate membership durable as one replayable operation, or persist enough V2 information in the create-table log so replay can rebuild the index atomically? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
