wangmingzhou1986 commented on issue #67898: URL: https://github.com/apache/doris/issues/67898#issuecomment-5662329698
**Resolved in 4.1.4.** Upgraded the cluster today and the leak is gone. ## Measurement Same reproducer as before (20 × `SELECT COUNT(*)` on a Paimon primary-key table via external catalog), both BEs freshly restarted: | metric, per BE per 20 queries | 4.1.3 (`doris-4.1.3-rc02-7126cf65d96`) | 4.1.4 (`doris-4.1.4-rc04-ad35a140c7f`) | |---|---|---| | distinct `sdk-ScheduledExecutor-<n>` instances | **+40** | **+0** | | threads in `/proc/<pid>/task` | **+200** | **+1** (noise) | ``` BE threads before after Δ SDK instances before after Δ doris-be-0 1860 1861 +1 1 1 +0 doris-be2-0 1851 1852 +1 1 1 +0 ``` The SDK executor instance count stays at 1 on both BEs instead of growing by 2 per query. That is exactly the mechanism identified earlier (AWS SDK async clients created per Paimon scan and never `close()`d), so this issue can be closed as fixed in 4.1.4. One small note for anyone searching: the published image `apache/doris:be-4.1.4` self-reports as **`doris-4.1.4-rc04-ad35a140c7f`**. There is no separate `-rc04` tag on Docker Hub — the rc04 build *is* the 4.1.4 release. ## A compatibility caveat worth documenting for Paimon-catalog users 4.1.4 bumps the bundled Paimon from **1.3.1 to 1.4.2**, and the `java_extensions` directory is renamed `paimon-scanner` → `paimon-connector`. Two things followed from that in our environment: **1. Lake-format compatibility is fine.** Our tables are *written* by Paimon 1.3.1 (Flink CDC) and now *read* by 1.4.2. Before upgrading I verified this in isolation — a throwaway pod running the 4.1.4 image with `sleep` (no Doris process), using its bundled Paimon 1.4.2 through the Java API to read production tables directly: - 3 static tables matched **field-by-field** (11×24, 1×16, 2159×10) - a 57.8M-row primary-key table, both engines pinned to the **same snapshot**, matched exactly: `57,783,725` from `FOR VERSION AS OF 2333` on 4.1.3 and `57,783,725` from the 1.4.2 direct read. After the upgrade the same query still returns `57,783,725`. (Comparing row counts across engines on a *live* table is meaningless — the table commits every ~30 s, so each engine pins a different snapshot. Pinning both to one snapshot is what makes the comparison valid.) **2. FE and BE must be on the same Paimon version.** The documented upgrade order is BE first, then FE. During that window — BE on 4.1.4, FE still on 4.1.3 — **every external-catalog query fails**: ``` [INTERNAL_ERROR]failed to init reader, err: [JNI_ERROR]RuntimeException: java.io.InvalidClassException: org.apache.paimon.table.CatalogEnvironment; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = 2 ``` The FE serializes Paimon split/table objects and the BE deserializes them; `CatalogEnvironment`'s `serialVersionUID` changed 1 → 2 between Paimon 1.3.1 and 1.4.2. So for deployments using Paimon external catalogs, the BE-then-FE window is a **planned outage**, not a transparent rolling upgrade. Ours lasted about 6 minutes. It might be worth a line in the upgrade docs. Happy to run anything else if it would help confirm. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
