wangmingzhou1986 commented on issue #67898: URL: https://github.com/apache/doris/issues/67898#issuecomment-5673332383
@Asthenia0412 thanks — that matches what we measured. Confirming the fix from the release side, in case it helps the PR review. We upgraded the whole cluster (2 FE + 2 BE) from `doris-4.1.3-rc02-7126cf65d96` to `doris-4.1.4-rc04-ad35a140c7f` and re-ran the identical 20-query reproducer on freshly restarted BEs: | per BE, per 20 external-catalog queries | 4.1.3 | 4.1.4 | |---|---|---| | distinct `sdk-ScheduledExecutor-<n>` instances | **+40** | **+0** | | threads in `/proc/<pid>/task` | **+200** | **+1** (noise) | So the Paimon 1.3.1 → 1.4.2 bump is indeed what fixes it, and the three JNI-cleanup PRs suggested earlier in triage were not the relevant change. Since #67963 is the same bump for master, I'd expect it to close this. Two things we verified before upgrading that may save someone else the trouble: **1. Lake-format compatibility (1.4.2 reader over 1.3.1-written tables) holds.** Our tables are written by Flink CDC with Paimon 1.3.1. Before touching production we ran the 4.1.4 image as a throwaway `sleep` pod (no Doris process) and used its bundled Paimon 1.4.2 through the Java API to read production tables directly. Three static tables matched field-by-field, and a 57.8M-row primary-key table matched exactly when both engines were pinned to the same snapshot: `57,783,725` from `FOR VERSION AS OF 2333` on 4.1.3, and `57,783,725` from the 1.4.2 direct read. After the upgrade the same query still returns `57,783,725`. Worth noting for anyone repeating this: comparing row counts across engines on a *live* table is meaningless — ours commits every ~30 s, so each engine pins a different snapshot and the numbers differ in both directions. Pinning both sides to one snapshot is what makes the comparison valid. **2. FE and BE must be on the same Paimon version — the documented BE-then-FE order is an outage window for Paimon catalogs.** With BE on 4.1.4 and FE still on 4.1.3, *every* external-catalog query fails: ``` [INTERNAL_ERROR]failed to init reader, err: [JNI_ERROR]RuntimeException: java.io.InvalidClassException: org.apache.paimon.table.CatalogEnvironment; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = 2 ``` The FE serializes Paimon split/table objects for the BE to deserialize, and `CatalogEnvironment`'s `serialVersionUID` changed 1 → 2 between Paimon 1.3.1 and 1.4.2. Ours was down about 6 minutes. For deployments using Paimon external catalogs this is a planned outage rather than a transparent rolling upgrade — might be worth a line in the upgrade docs alongside this change. Happy to test anything else on 4.1.4 if useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
