[ https://issues.apache.org/jira/browse/IGNITE-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054655#comment-17054655 ]
Sunghan Suh commented on IGNITE-12543: -------------------------------------- Hello, [~Pavlukhin]. I'm a leader of an open source development group in Samsung. As [~redcomet] explained, we suffered from this issue in a mission critical system. I know limitations in the previous comment, but the patch we have uploaded can fix our prolem with ease and it doesn't affect other codes. Please review [GitHub Pull Request #7403|https://github.com/apache/ignite/pull/7403] positively, and let us end it gracefully. Thank you for your times. > When put List<List<SomeObject>>, the data was increased much larger. > -------------------------------------------------------------------- > > Key: IGNITE-12543 > URL: https://issues.apache.org/jira/browse/IGNITE-12543 > Project: Ignite > Issue Type: Bug > Components: thin client > Affects Versions: 2.6 > Reporter: LEE PYUNG BEOM > Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > I use Ignite 2.6 version of Java Thin Client. > > When I put data in the form List<List<SomeObject>>, > The size of the original 200KB data was increased to 50MB when inquired by > Ignite servers. > On the Heap Dump, the list element was repeatedly accumulated, increasing the > data size. > > When I checked org.apacheignite.internal.binary.BinaryWriterExImpl.java > doWriteBinaryObject() method, > {code:java} > // org.apacheignite.internal.binary.BinaryWriterExImpl.java > public void doWriteBinaryObject(@Nullable BinaryObjectImpl po) { > if (po == null) > out.writeByte(GridBinaryMarshaller.NULL); > else { > byte[] poArr = po.array(); > out.unsafeEnsure(1 + 4 + poArr.length +4); > out.unsafeWriteByte(GridBinaryMarshaller.BINARY_OBJ); > out.unsafeWriteInt(poArr.length); > out.writeByteArray(poArr); > out.unsafeWriteInt(po.start()); > } > } > {code} > > The current Ignite implementation for storing data in the form > List<List<Some_Objectject>> is: > In the Marshalling stage, for example, data the size of List(5 > members)<List(10 members)<Some_Object(size:200 KB)> is: > As many as 10*5 of the list's elements are duplicated. > If the above data contains five objects of 200KB size, ten by one, > 50 iterations are stored and 200K*10**5 = 100MB of data is used for cache and > transfer. > As a result of this increase in data size, it is confirmed that the failure > of OOM, GC, etc. is caused by occupying Heap memory. > Unnecessarily redundant data is used for cache storage and network transport. > When looking up cache data, only some of the data at the top is read based on > file location information from the entire data, so that normal data is > retrieved. > The way we're implemented today is safe from basic behavior, but we're > wasting memory and network unnecessarily using inefficient algorithms > This can have very serious consequences. Please check. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)