GJ100 opened a new issue, #65305: URL: https://github.com/apache/doris/issues/65305
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version Apache Doris 4.1.2 (JDK 17.0.19, `MaxDirectMemorySize=12g`) > Please replace with the exact output of `select version();` before submitting: `<fill: select version()>` ### What's Wrong? When a client keeps executing queries over **Arrow Flight SQL** (Python ADBC / `adbc_driver_flightsql`), the **FE Direct Memory grows monotonically and is never released**, until the FE can no longer allocate and every query fails with: ``` adbc_driver_manager.InternalError: INTERNAL: [FlightSQL] Unknown error: java.lang.OutOfMemoryError: Cannot reserve 4194304 bytes of direct buffer memory (allocated: 12883979585, limit: 12884901888) (Internal; Prepare) ``` Key characteristics: **1. Java Heap is healthy — the leak is in Direct Memory, not the heap.** `jstat -gcutil <fe_pid>` on the affected FE: ``` S0 S1 E O M CCS YGC YGCT FGC FGCT CGC CGCT GCT 0.00 61.42 0.10 17.66 99.69 98.94 12833 97.970 7 1.381 674882 3878.319 5658.417 ``` - Old Gen (`O`) stays low (~14–18%): no heap leak. - After we added `-XX:+ExplicitGCInvokesConcurrent`, Full GC (`FGC`) stays low, but **Concurrent GC (`CGC`) explodes to 600k+** — i.e. the JVM is constantly running GC trying (and failing) to reclaim Direct Memory. **2. Direct Memory reaches exactly `MaxDirectMemorySize` and stays there.** ``` allocated: 12883979585 ≈ limit: 12884901888 (12 GB) ``` **3. `io.netty.buffer.PoolSubpage` accumulates massively on the FE that serves Arrow Flight traffic.** `jcmd <fe_pid> GC.class_histogram` on the **abnormal** FE: ``` io.netty.buffer.PoolSubpage 265337 instances 19,104,264 bytes [Lio.netty.buffer.PoolSubpage; 3175 instances 6,355,568 bytes [B (byte arrays) 376158 instances 117,217,360 bytes ``` On a **healthy** FE (little/no Arrow Flight traffic), for comparison: ``` io.netty.buffer.PoolSubpage <fill: run jcmd GC.class_histogram on a healthy FE> instances ``` > In our environment the abnormal FE shows ~40–60x more `PoolSubpage` than idle FEs. Please fill in the healthy-FE numbers before submitting. **4. The failure always happens in the `Prepare` phase**, and it strongly correlates with **queries that error out**. Our FE `fe.audit.log` shows a large number of `State=ERR` Arrow Flight queries (note `Client=0.0.0.0:0`, i.e. Arrow Flight), e.g. querying non-existent tables: ``` |Client=0.0.0.0:0|User=prod|Db=prod1|State=ERR|ErrorCode=1105| ErrorMessage=errCode = 2, detailMessage = Table [xxx] does not exist in database [prod1].| ...|IsQuery=true|StmtType=SELECT|CommandType=Query| Stmt= select ... from xxx tmp1 join ... where ... ``` These error queries have `PeakMemoryBytes=0` and fail within ~1ms during analysis, yet the FE Direct Memory keeps climbing — suggesting buffers allocated on the Arrow Flight `Prepare`/error path are **not released when the statement fails**. ### What You Expected? Arrow Flight SQL statements — especially **failed / cancelled** ones — should release all Direct Memory (Arrow buffers / Netty PooledByteBuf) allocated during the `Prepare` phase, so that FE Direct Memory stays bounded under continuous query load. ### How to Reproduce? 1. Enable Arrow Flight SQL on FE/BE. 2. From an ADBC client, continuously execute a large number of **distinct** SQL statements over Arrow Flight (each statement text is unique — note Doris Arrow Flight does not support parameter binding, so clients inline literals, making every query a new `Prepare`). 3. Include a portion of statements that **error out** (e.g. `SELECT ... FROM a_nonexistent_table ...`). 4. Monitor FE Direct Memory (e.g. `jcmd <pid> GC.class_histogram` watching `io.netty.buffer.PoolSubpage`, and the JVM Direct Memory usage). Observed: FE Direct Memory increases monotonically and is not reclaimed by GC, eventually hitting `MaxDirectMemorySize` and failing every query with `Cannot reserve ... direct buffer memory (Internal; Prepare)`. ### Anything Else? **Client-side mitigations we already tried — none stop the growth**, which indicates the retention is on the FE server side, not a client connection leak: - Small Arrow connection pool (`maxconnections=4`). - Periodic whole-pool rebuild by time and by query count (close + recreate all Flight sessions). - Disabling the pool entirely (open a fresh connection per query, close it immediately after each query). - `-XX:+ExplicitGCInvokesConcurrent` + larger `MaxMetaspaceSize` — this only removed a Full-GC storm; Direct Memory still leaks. Even when the Flight session is explicitly closed after every single query, FE Direct Memory is not released — so the leaked buffers appear to be strongly referenced on the FE, not tied to session lifecycle. Environment: 3 FE nodes (32 GB each). Only the FE that actually receives Arrow Flight traffic accumulates the leak; the other FEs (same shared metadata) stay healthy — ruling out a catalog/tablet/replica metadata leak and pointing at the Arrow Flight request-processing path. Possibly related: - apache/arrow-java#50 ("Use prepared statement leads Memory leak", still open) - A very similar report in the Doris Chinese forum on 2.1.10 (FE Direct Memory keeps growing with Arrow Flight SQL until `create prepared statement failed, Failure allocating buffer` / `OutOfDirectMemoryError`). We could not find an existing PR that fixes this specific **FE Arrow Flight `Prepare`-phase Direct Memory leak** (the arrow memleak fixes we found, e.g. #51929, are BE-side and unrelated). Could the maintainers confirm whether this is known and in which version it is fixed? ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
