Hello, I'm using ZK based discovery for my 6 node grid. Its been working smoothly for a while until suddenly my ZK node went OOM. Turns out there were 1000s of znodes, many with data about ~1M + there were suddenly a lot of stuff ZK requests (tx log was huge).
One symptom on the grid to notes is that when this happened my nodes were heavily stalling (this is a separate issue to discuss - they're stalling with lots of high JVM pauses but GC logs appear alright) and were also getting heavy write from DataStreamers. I see the joinData znode having many 1000s of persistent children. I'd like to undersstand why so many znodes were created under 'jd' and what's the best way to prevent this and clean up these child nodes under jd. Thanks, Abhishek