snazy commented on PR #490: URL: https://github.com/apache/polaris/pull/490#issuecomment-2732340120
(Just commenting in general) Background: Nessie's cache keeps _serialized_ representations of the cached object to limit the _amount_ of _reachable_ (from GC roots) Java objects - it also helps to calculate the effective heap usage and literally use "number of bytes" in the weigher. As an optimization, we added an additional `SoftReference` to the Java object as an optimization. That optimization looked quite solid in tests - but it's been proven to cause issues in (longer running) production systems. [This is a prominent example](https://issues.apache.org/jira/browse/CASSANDRA-11206) why an excessive amount of (reachable) Java objects is bad. Generally, there are _many_ factors that influence the behavior of a JVM/GC combination. Naming some: * age of objects / object generation * whether objects are referenced/reachable from only one thread or multiple threads * number of totally reachable objects * kind of reference (hard/soft/weak/phantom) * system load * kind of GC (know which GC is used in all relevant environments) - G1/Shenandoah/Parallel/Z/Serial/... * GC configuration (pro tip: do NOT tune GC unless you really really have to and proven that it works) * number of CPUs / NUMA * hardware (of course) About testing: * Do not call `Runtime.gc()` during tests - you don't know what it's really doing * Use tools like `jmc`, IntelliJ's profiler or other tools to monitor heap usage * Use multi-threaded tests * Push it to the limits / (try to) break it intentionally * Small/concise tests should be JMH benchmarks (it's got a lot of profilers) * Gatling for longer running tests, JVM stats monitored by external tools -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
