gortiz opened a new pull request, #17674: URL: https://github.com/apache/pinot/pull/17674
This PR adds a new machinery to inspect, log and monitor the memory used by Netty. As is explained on the NettyInstance javadoc, a Pinot installation has between 2 and 3 copies of the Netty code: 1. The version we shade. This is present in production but not when running Pinot without shading (i.e., tests or a QuickStart started from the IDE or with Maven) 2. The version shade by gRPC 3. Optionally, an unshaded version, which is present when 1 is not present and may also be included by a third-party library Netty uses static attributes to keep some important information, like whether it can use off-heap memory or not, or how much memory is being allocated. Given we have 2-3 copies of Netty, we have 2-3 **independent** copies of these attributes. The applied shade doesn't just change package of the classes, it also changes the system properties we need to set in order to customize Netty. We always need to set at least one JAVA_OPT in order to let Netty use offheap memory in Java 21: `io.netty.tryReflectionSetAccessible` As a corollary, for each JAVA_OPT we plan to use to customize Netty we need to provide that option 2-3 times with different prefixes. Given that different copies of the same classes are involved here, this is a very error-prone process. This is why we introduce a new class, NettyInstance, that offers clean access to the most important Netty properties. This class can be instantiated to access a specific Netty copy. Another class, NettyInspector, is used to add some checks on the state of important properties of each known NettyInstance. Right now, the only check we have is whether it uses onheap of offheap memory, but more may be added in the future. This PR also adds a 2 new metrics per NettyInstance: how much memory is being used and how much memory that instance can use. This is important **because by default each independent Netty copy will consume as many direct memory as specified by `XX:MaxDirectMemorySize`, which means we may be consuming a maximum of: 1. `MaxDirectMemorySize` bytes by normal ByteBuffers 2. `MaxDirectMemorySize` bytes by unshade/pinot-shaded Netty (used in SSE) 3. `MaxDirectMemorySize` bytes by grpc-shaded Netty (used in MSE) Finally, this PR also adds logs when Brokers and Servers start. Specifically: - For each NettyInstance, we add a warning log if we are using on-heap memory. - For each NettyInstance, we add an info log indicating how much memory it is using (usually 0) and how much memory it can use. - A single node indicating the sum of off-heap memory all NettyInstances are using and can use. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
