Doris-Breakwater commented on issue #69251: URL: https://github.com/apache/doris/issues/69251#issuecomment-6258712231
Breakwater-GitHub-Analysis-Slot: slot_2264d8cc583b Initial triage: the reported refresh loop is consistent with the checksum implementation introduced by [#63894](https://github.com/apache/doris/pull/63894). This supports a remote metadata cache invalidation issue, conditional on the deployed builds containing that implementation. It does not establish a query correctness failure or quantify the performance impact. The issue currently has no labels; FE / Remote Doris Catalog would be the appropriate area for maintainer triage. **Version clarification is needed first.** The [Apache Doris 4.0.3 release](https://github.com/apache/doris/releases/tag/4.0.3-rc03) points to `4.0.3-rc03`. In that source, [`getOlapTableMeta()`](https://github.com/apache/doris/blob/e9096296b8b8fc9f44d43dbe58477dcbc7af5f7c/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java#L4776) compares partition visible version/time only, and `Partition.getMetaChecksum()` is absent. Please provide the exact FE build version and commit hash for both the querying cluster and every remote FE, including any backported changes. The analysis below refers to public commit `d2405af00a84c3be00e9f1287c30e018b66f3dcb`, not an assumption that the reported binaries match upstream 4.0.3. **Verified in source** - [`Partition.getMetaChecksum()`](https://github.com/apache/doris/blob/d2405af00a84c3be00e9f1287c30e018b66f3dcb/fe/fe-core/src/main/java/org/apache/doris/catalog/Partition.java#L268) hashes each replica's `pathHash`. A pathHash-only difference changes the checksum input even when visible version/time and replica placement are unchanged. - [`FrontendServiceImpl`](https://github.com/apache/doris/blob/d2405af00a84c3be00e9f1287c30e018b66f3dcb/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java#L5510) retains a partition in the updated set when the supplied checksum differs, then serializes its full JSON payload. [`FeServiceClient`](https://github.com/apache/doris/blob/d2405af00a84c3be00e9f1287c30e018b66f3dcb/fe/fe-core/src/main/java/org/apache/doris/datasource/doris/FeServiceClient.java#L263) stores the server-returned checksum and sends it on subsequent requests; its metadata RPC uses random FE selection. Thus alternating between two stable, different server checksums can repeatedly replace the cached partition. This applies to each mismatching partition; returning all 475 requires all 475 to mismatch. - The report RPC [requires the master](https://github.com/apache/doris/blob/d2405af00a84c3be00e9f1287c30e018b66f3dcb/fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java#L1073), and the [tablet report update path](https://github.com/apache/doris/blob/d2405af00a84c3be00e9f1287c30e018b66f3dcb/fe/fe-core/src/main/java/org/apache/doris/catalog/LocalTabletInvertedIndex.java#L355) assigns pathHash in memory without an EditLog operation at that assignment. In this revision the concrete field is in [`LocalReplica`](https://github.com/apache/doris/blob/d2405af00a84c3be00e9f1287c30e018b66f3dcb/fe/fe-core/src/main/java/org/apache/doris/catalog/LocalReplica.java#L53), defaults to `-1`, and lacks `@SerializedName`. The configured [Gson serialization exclusion](https://github.com/apache/doris/blob/d2405af00a84c3be00e9f1287c30e018b66f3dcb/fe/fe-common/src/main/java/org/apache/doris/persist/gson/GsonUtilsBase.java#L92) omits unannotated fields, so partition refresh does not transport this value. These mechanisms support the reported Leader/Follower difference; actual values on the affected FEs still require runtime evidence. **Semantics and suggested direction** `pathHash` is meaningful inside Doris: [`Tablet.getQueryableReplicas()`](https://github.com/apache/doris/blob/d2405af00a84c3be00e9f1287c30e018b66f3dcb/fe/fe-core/src/main/java/org/apache/doris/catalog/Tablet.java#L218) compares it with backend alive-disk hashes when choosing replicas. It should not be removed from native replica tracking or disk-health checks. However, a remote refresh cannot update a field excluded from its payload, so hashing that field currently introduces invalidation without conveying the corresponding state. For the existing remote payload contract, excluding pathHash from the remote partition checksum is the narrower candidate fix. Preserve invalidation for replica/backend identity changes, versions, and other relevant metadata: #63894 specifically prevents stale placement after balancing, even when partition visible version/time stays unchanged. Do not revert to version/time-only comparison. Audit other nonserialized checksum inputs, particularly `bad` and `isUserDrop()`, before claiming that removing pathHash eliminates every cross-FE mismatch. Synchronizing pathHash through EditLog/replay is a broader design change. It would need ordering, failover, and report-volume consideration, and would still leave the remote payload unable to carry the value. If remote queries are intended to consume disk placement/health, define and transport that state explicitly and test its consistency before choosing this approach. **Evidence and next steps** 1. Supply the build commits above, sanitized catalog properties showing the configured FE endpoints and `use_arrow_flight=false`, and minimal table DDL / refresh steps. Remove credentials. 2. Capture a sequential Leader → Follower → Leader refresh sequence with timestamps, target FE roles, table/partition IDs, visible version/time, request checksum, response checksum, and updated-partition count. For one representative partition, include the checksum input fields and a normalized payload comparison. Confirm the Follower has caught up and no load, DDL, balancing, or replica-health transition overlaps the comparison. 3. Add a regression test with identical partition payloads but different pathHashes across two FE responses; alternate endpoints while carrying forward the returned checksum. After warm-up, no partition payload should be resent solely for pathHash. Cover normal and temporary partitions, genuine replica relocation, partition rename/version changes, and older peers without checksum fields. The existing checksum unit test explicitly expects pathHash to change the checksum, so it must be reconciled with the chosen remote-cache semantics. This was a read-only source investigation; no cluster reproduction, benchmark, code change, or test execution was performed. The source establishes the mechanism, while the exact affected build and measured overhead remain unverified. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
