Hello! I think this is expected. Ignite needs to convert an entry (800G array) into packets to send to other nodes or put to persistence, where it will allocate another array (800G more).
I don't think you can do anything here, short of chunking your data. Regards, -- Ilya Kasnacheev пт, 13 нояб. 2020 г. в 17:04, Kalin Katev <ka...@reasonance.de>: > Hi, > > I am sorry for responding so late. I can only send screenshots of the heap > dump as seen in visualvm. I took screenshots of 2 different byte arrays, > one is 800mb large, the other 1gb. Every cached value in ignite (which > itself is 800mb) creates a tuple of these arrays. I hope this helps, or you > can maybe give me an idea which references are crucial, as there are too > many for me to investigate. > > Thank you! > > Kalin Katev > Resonance GmbH > > Am Di., 3. Nov. 2020 um 15:56 Uhr schrieb Ilya Kasnacheev < > ilya.kasnach...@gmail.com>: > >> Hello! >> >> 800MB entry is far above of the entry size that we ever expected to see. >> Even brief holding of these entries on heap will cause problems for you, as >> well as sending them over communication. >> >> I recommend splitting entries into chunks, maybe. That's what IGFS did >> basically, we decided to ax it, but you can still use that approach. >> >> Having said that, if you can check the heap dump to see where are these >> stuck byte arrays referenced from, I may check it. >> >> Regards, >> -- >> Ilya Kasnacheev >> >> >> вт, 3 нояб. 2020 г. в 11:48, Kalin Katev <ka...@reasonance.de>: >> >>> Hi, >>> >>> not too long ago I tested apache ignite for my use case on Open JDK 11. >>> The use case consists of writing cache entries with values going up to >>> 800MB in size, the data itself being a simple string. After writing 5 >>> caches entries, 800 MB each, I noticed my Heap space exploding up to 11GB, >>> while the entries themselves were written off-heap. The histogram of the >>> heap dump shows that there are 5 tuples of byte[] arrays with size 800MB >>> and 1000MB that are left dangling on heap. I am very curious if I did >>> something wrong or if there indeed is an issue in Ignite. All details can >>> be seen on >>> https://stackoverflow.com/questions/64550479/possible-memory-leak-in-apache-ignite >>> >>> Should I create a jira ticket for this issue? >>> >>> Best regards, >>> Kalin Katev >>> >>> Resonance GmbH >>> >>
