AniRamadoss opened a new issue, #4084: URL: https://github.com/apache/jena/issues/4084
### Version 4.x ### What happened? # Jena Fuseki: 1 MB output buffers are never pooled by the default Jetty ByteBufferPool, causing DirectMemory churn under load ## Summary Jena Fuseki (jena-fuseki-main) builds its Jetty `ServerConnector` without configuring a `ByteBufferPool`, so Jetty falls back to a default `ArrayByteBufferPool` whose maximum pooled buffer capacity is 65536 bytes (64 KB). At the same time, Fuseki configures the connector's `HttpConfiguration` with a 1 MB `outputBufferSize`. Because 1 MB is larger than the pool's 64 KB maximum capacity, every output buffer that Jetty acquires for an `HttpConnection` is allocated fresh and then dropped on release rather than being retained for reuse. The result is continuous allocation and garbage collection of 1 MB direct buffers with no pooling benefit. Under sustained high request throughput this drives elevated DirectMemory usage and GC pressure. This report describes how the pieces connect, the root cause, and options the maintainers may want to weigh for a fix on the Jena side. ## Environment - Jena Fuseki: jena-fuseki-main 4.10.0 - Jetty: 10.0.17 (the version Jena 4.10.0 declares via the `jetty-bom` import in its root `pom.xml`, see [`pom.xml` line 71](https://github.com/apache/jena/blob/jena-4.10.0/pom.xml#L71)) - Java: Java 17 (Amazon Corretto 17.0.19) - OS: Amazon Linux 2 ## Background: how the pieces fit together Three separate configuration points combine to produce this behavior. First, Fuseki provisions the Jetty connector in [`JettyHttps.java` around line 101](https://github.com/apache/jena/blob/jena4/jena-fuseki2/jena-fuseki-main/src/main/java/org/apache/jena/fuseki/main/JettyHttps.java#L101): ```java ServerConnector plainConnector = new ServerConnector(server, new HttpConnectionFactory(http_config)); ``` No `ByteBufferPool` is passed to the connector. When a Jetty `ServerConnector` (via `AbstractConnector`) is constructed without an explicit `ByteBufferPool`, it creates a default `ArrayByteBufferPool` (see [`AbstractConnector.java` line 198](https://github.com/jetty/jetty.project/blob/jetty-10.0.x/jetty-server/src/main/java/org/eclipse/jetty/server/AbstractConnector.java#L198)). That default `ArrayByteBufferPool` has a maximum pooled buffer capacity of 65536 bytes (64 KB), which is `DEFAULT_MAX_CAPACITY_BY_FACTOR * DEFAULT_FACTOR` = 16 * 4096 (see [`AbstractByteBufferPool.java` line 62](https://github.com/jetty/jetty.project/blob/jetty-10.0.x/jetty-io/src/main/java/org/eclipse/jetty/io/AbstractByteBufferPool.java#L62)). Second, the `HttpConfiguration` (`http_config` above) is built in [`JettyLib.httpConfiguration()` around line 92](https://github.com/apache/jena/blob/jena4/jena-fuseki2/jena-fuseki-main/src/main/java/org/apache/jena/fuseki/main/sys/JettyLib.java#L92), where it sets: - `setRequestHeaderSize(512 * 1024)` for 512 KB - `setOutputBufferSize(1024 * 1024)` for 1 MB (1,048,576 bytes) Third, each Jetty `HttpConnection` acquires buffers sized to the configured `outputBufferSize` (1 MB) from the connector's pool (see [`HttpConnection.java` line 779](https://github.com/jetty/jetty.project/blob/jetty-10.0.x/jetty-server/src/main/java/org/eclipse/jetty/server/HttpConnection.java#L779)). So the connector hands `HttpConnection` a pool that tops out at 64 KB, while `HttpConnection` repeatedly asks that pool for 1 MB buffers. ## Root Cause The mismatch between the configured 1 MB `outputBufferSize` and the default pool's 64 KB maximum capacity means the buffers can never be pooled. In [`ArrayByteBufferPool.acquire()`](https://github.com/jetty/jetty.project/blob/jetty-10.0.x/jetty-io/src/main/java/org/eclipse/jetty/io/ArrayByteBufferPool.java#L146), a request whose size maps to a bucket index outside the pool's range falls through to allocating a fresh, unpooled buffer, and the requested capacity is not clamped to the maximum, so an oversized request yields an oversized fresh buffer. However, [`ArrayByteBufferPool.release()`](https://github.com/jetty/jetty.project/blob/jetty-10.0.x/jetty-io/src/main/java/org/eclipse/jetty/io/ArrayByteBufferPool.java#L169) will not retain any buffer larger than the maximum capacity. Oversized buffers are discarded rather than being returned to the pool. The net effect is that every acquire of a 1 MB output buffer allocates fresh memory and every release throws it away. Under sustained or high TPS it produces heavy DirectMemory allocation churn and GC pressure. In my environment this contributed to memory growth and an OS-level OutOfMemoryError. Our situation was aggravated by a separate, environment-specific local misconfiguration in which DirectMemory could grow uncapped without any garbage collection cleanup. That local misconfiguration was an aggravating factor that led me to find this issue. ## Reproduction 1. Start Fuseki, which builds the connector as described above with a 1 MB output buffer size and the default `ByteBufferPool`. 2. Drive sustained HTTP traffic so that many `HttpConnection` instances acquire and release 1 MB output buffers. 3. Observe that the `ByteBufferPool` never retains these buffers, since its 64 KB maximum capacity is smaller than 1 MB, so DirectMemory allocation churns continuously with no reuse. A minimal Jetty-level reproduction is to acquire buffers larger than the pool's maximum capacity, release them, and repeat. Pooled memory never grows because oversized buffers are never retained on release. ## Suggested Fix / Recommendation According to Jetty, the buffer pools not reusing buffers but allowing provisioning of large buffers is an intentional design, so the fix belongs on consumers of the Jetty library ([See upstream Jetty ticket](https://github.com/jetty/jetty.project/issues/15464)). On the Jena side, this could be an explicit pool configuration or by keeping the pool capacity in sync with the configured output buffer size. The workaround I currently have is for Fuseki to explicitly configure the connector's `ByteBufferPool` with a maximum capacity at least as large as the configured output buffer size, so that the 1 MB buffers can actually be pooled and reused. The following configuration sets the pool maximum capacity to 2 MB, exceeding the 1 MB output buffer size: ```java ArrayByteBufferPool byteBufferPool = new ArrayByteBufferPool(0, 2048, 2 * 1024 * 1024, -1, -1, -1); server.addBean(byteBufferPool); ``` With this in place, the output buffers are pooled and reused rather than churned. I haven't checked if other versions of Apache Jena have this issue, but it may be worth verifying on 5.x as well. ## References - Related upstream Jetty issue: https://github.com/jetty/jetty.project/issues/15464 - Jena `pom.xml` line 71 (Jetty version, 10.0.17): https://github.com/apache/jena/blob/jena-4.10.0/pom.xml#L71 - Jena `JettyHttps.java` line 101 (connector provisioning): https://github.com/apache/jena/blob/jena4/jena-fuseki2/jena-fuseki-main/src/main/java/org/apache/jena/fuseki/main/JettyHttps.java#L101 - Jena `JettyLib.java` line 92 (1 MB output buffer size): https://github.com/apache/jena/blob/jena4/jena-fuseki2/jena-fuseki-main/src/main/java/org/apache/jena/fuseki/main/sys/JettyLib.java#L92 - Jetty `AbstractConnector.java` line 198 (default pool provisioning): https://github.com/jetty/jetty.project/blob/jetty-10.0.x/jetty-server/src/main/java/org/eclipse/jetty/server/AbstractConnector.java#L198 - Jetty `HttpConnection.java` line 779 (acquires output buffer sized to `outputBufferSize`): https://github.com/jetty/jetty.project/blob/jetty-10.0.x/jetty-server/src/main/java/org/eclipse/jetty/server/HttpConnection.java#L779 - Jetty `AbstractByteBufferPool.java` line 62 (max capacity defaults to 65536 bytes): https://github.com/jetty/jetty.project/blob/jetty-10.0.x/jetty-io/src/main/java/org/eclipse/jetty/io/AbstractByteBufferPool.java#L62 - Jetty `ArrayByteBufferPool.java` line 146 (`acquire()`): https://github.com/jetty/jetty.project/blob/jetty-10.0.x/jetty-io/src/main/java/org/eclipse/jetty/io/ArrayByteBufferPool.java#L146 - Jetty `ArrayByteBufferPool.java` line 169 (`release()` discarding oversized buffers): https://github.com/jetty/jetty.project/blob/jetty-10.0.x/jetty-io/src/main/java/org/eclipse/jetty/io/ArrayByteBufferPool.java#L169 ### Relevant output and stacktrace ```shell ``` ### Are you interested in making a pull request? Maybe -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
