Hi Vladimir, We are using 2 server nodes of 5 GB each. Also keeping backup count 1.
We used binarylizable and raw mode which shrinked size but SQL queries are not running on raw mode. Is there any way we can run SQL queries with raw mode or we could use any external serializer with which SQL queries could run? On 17 Feb 2016 23:54, "Vladimir Ozerov [via Apache Ignite Users]" < ml-node+s70518n305...@n6.nabble.com> wrote: > Hi vidhu, > > If your object have 101 int field, each instance serialized with > BinaryMarshaller (the default one) will be an array of 731 byte. This means > 1.4GB of heap for 2M objects. How many nodes are there in your application? > Do you have backups? It possible pleas share your configuration and source > code. > > Using Binarylizable interface and raw mode could shrink size of > serialized object to somewhat about 450 bytes per instance. It looks > strange to me that you cannot get objects serialized in this way form > cache. Again, you source code will help us to understand what the problem > is. > > Vladimir. > > On Wed, Feb 17, 2016 at 8:18 AM, vidhu <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=3056&i=0>> wrote: > >> Thanks for quick reply. >> >> I mistakenly write 20 million to earlier but objects I am putting in >> cache are 2 million which means .9 GB approx. >> >> Sorry for putting wrong number of objects earlier. >> >> I am starting 2 server nodes of 5 GB each and a client node of 5 gb to >> load data into cache. >> >> Apart from this cache I am having 2 more caches whose size combined is >> less than 400 MB. >> On 17 Feb 2016 10:44, "vkulichenko [via Apache Ignite Users]" <[hidden >> email] <http:///user/SendEmail.jtp?type=node&node=3051&i=0>> wrote: >> >>> Hi vidhu, >>> >>> First of all, can you please subscribe to the mailing list so that >>> community receives email notifications? Please follow this simple >>> instruction: >>> http://apache-ignite-users.70518.x6.nabble.com/mailing_list/MailingListOptions.jtp?forum=1 >>> >>> vidhu wrote >>> I am facing a problem where my heap reaches to about 2.8 gb when I am >>> storing 20 million objects. Size of each object is approx. 462 bytes. >>> Structure of my object is a simple pojo having 101 fields each of int >>> type. >>> >>> I tried using following 3 marshalling techniques: >>> >>> 1.) Default marshaller of Ignite. (Size of heap ~ 2.8 gb, looking with >>> Visual VM find that most of objects on heap are byte array.) >>> >>> 2.) Optimized marshaller. (Again size was ~ 2.8 gb, looking with Visual >>> VM find that this time size of objects on heap of byte array is half + >>> there are lot of objects of class that I put in cache. ) >>> >>> 3.) Tried implementing Binarylizable in my class that I am storing in >>> cache. I used rawWriter. This time size reduced considerably but I was not >>> able to read values out of cache, (getting null on read). >>> >>> I am implementing this in java version 1.8. >>> >>> 20,000,000 * 462 bytes is already around 8.6GB of memory, so 2.8 is not >>> enough even without Ignite's overhead. Did you try to allocate more heap >>> memory and/or start more nodes? With your sizings I would start with two >>> nodes, 8GB of memory per node. >>> >>> As for options you tried, I would recommend to use default binary >>> marshaller because it's much more compact. Also you should take a look at >>> offheap memory [1]. It can be useful if you're going to have a lot of data. >>> >>> [1] https://apacheignite.readme.io/docs/off-heap-memory >>> >>> -Val >>> >>> ------------------------------ >>> If you reply to this email, your message will be added to the discussion >>> below: >>> >>> http://apache-ignite-users.70518.x6.nabble.com/Huge-heap-size-for-ignite-cache-when-storing-20-million-objects-of-small-size-tp3049p3050.html >>> To unsubscribe from Huge heap size for ignite cache when storing 20 >>> million objects of small size, click here. >>> NAML >>> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>> >> >> ------------------------------ >> View this message in context: Re: Huge heap size for ignite cache when >> storing 20 million objects of small size >> <http://apache-ignite-users.70518.x6.nabble.com/Huge-heap-size-for-ignite-cache-when-storing-20-million-objects-of-small-size-tp3049p3051.html> >> >> Sent from the Apache Ignite Users mailing list archive >> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com. >> > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > > http://apache-ignite-users.70518.x6.nabble.com/Huge-heap-size-for-ignite-cache-when-storing-20-million-objects-of-small-size-tp3049p3056.html > To unsubscribe from Huge heap size for ignite cache when storing 20 > million objects of small size, click here > <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3049&code=bWV2aWRodXNoYXJtYUBnbWFpbC5jb218MzA0OXwtMTAzMjgwMDMwOQ==> > . > NAML > <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Huge-heap-size-for-ignite-cache-when-storing-20-million-objects-of-small-size-tp3049p3057.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.
