Hi, +HeapDumpOnOutOfMemroyError is not provided as default jvm args, I've comment that out, just to provide convenience for DBAs when they want to use this parameter.
On Tue, Nov 14, 2023 at 2:00 PM ZhangJian He <shoot...@gmail.com> wrote: > +1 for ExitOnOutOfMemoryError. > +HeapDumpOnOutOfMemroyError may produce very large files. How to clean up > these old files? WDYT? > > Thanks > ZhangJian He > > > On Tue, 14 Nov 2023 at 12:04, Gaofei Cao <caogao...@apache.org> wrote: > > > +1, > > > > `-XX:+ExitOnOutOfMemoryError` parameter can avoid the loss of some > > key threads, it will be beneficial to the system. > > If the IoTDB cluster is deployed on k8s, this parameter is more > > indispensable, because k8s can dispatch another pod to replace this > > OOM node rapidly. > > Besides, i think we can add the usage of `-XX:+ExitOnOutOfMemoryError` > > and `-XX:+HeapDumpOnOutOfMemoryError` in the user/DBA manual, which is > > important to find the root cause of OOM. > > > > Best, > > ---------------------- > > Gaofei Cao > > > > Yuan Tian <jackietie...@gmail.com> 于2023年11月13日周一 19:52写道： > > > > > > Hi all, > > > > > > Recently, we found in some real user cases that when OOM occurs in the > > > DataNode process (although we should ensure that OOM does not happen, > but > > > we all know that bugs will always exist), some threads(e.g. rpc > listening > > > threads) may exit unexpectedly which may cause some strange things to > > > happen. For example, if the heartbeat listening thread on the DataNode > > > unexpectedly exits due to OOM, and then the OOM recovers on its own > (some > > > large queries end, or some compaction tasks end), but this thread will > > > never exist again, causing the DataNode to remain in unknown state, > > because > > > the ConfigNode can no longer contact it via heartbeat. > > > > > > Therefore, we feel that OOM is a high-risk error, and we should let the > > > process exit directly to avoid the loss of some key threads. > > > > > > And I did an experiment and found that -XX:+ExitOnOutOfMemoryError and > > > -XX:+HeapDumpOnOutOfMemoryError do not conflict which means that we can > > > keep both in jvm args and when OOM happens, it will firstly dump the > heap > > > memory and then exit. > > > > > > I've made this change in my pr( > > https://github.com/apache/iotdb/pull/11531). > > > > > > What do you think? > > > > > > > > > > > > > > > Best, > > > ---------------------- > > > Yuan Tian > > >
