Hello! I can't help myself but wonder how large of a benefit will it give. I have checked the ticket description, it looks the proposed scheme is elaborate and benefit for non-extreme binary objects rather tiny.
WDYT? Regards, -- Ilya Kasnacheev пн, 18 мая 2020 г. в 22:54, steve.hostett...@gmail.com < steve.hostett...@gmail.com>: > Hello igniters, > > while I would like to help on the calcite because H2 optimiser (or the lack > thereof) is really killing us, I think that it would be wiser to start by > contributing on something easier. > > Therefore I will tackle another problem that we have which is the memory > consumption. I stumbled upon this IEP > > https://cwiki.apache.org/confluence/display/IGNITE/IEP-2%3A+Binary+object+format+improvements > < > https://cwiki.apache.org/confluence/display/IGNITE/IEP-2%3A+Binary+object+format+improvements> > > that is about optimising the binary marshaller. > > The low hanging fruit seemed to be the null compaction so I decided to > start > with it. Though I am sure I do see some hidden complexity. > > Here a couple of questions: > - Can I assign myself IGNITE-6499 and attach a patch? > - Who can I contact to help with the review. In the following page > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute > <https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute> > there is no one assigned for marshalling. > > On the details: > The compression is disabled by default as it is not compatible with objects > previously marshalled. > > My approach was to go a bit beyond the JIRA. No only do I remove the > indexes > to null fields in the footer, I also remove the 0x65 in the objects. I did > not remove them fro the collections and arrays because they are using > absolute positioning. > > I gain between 5% to 20% depending of my test cases. Obviously the smaller > the object and the higher the number of nulls, the higher the compression > rate. > > Based on that I can quite easily add var int compression which is > IGNITE-6418 and should significantly increase the compression rate with a > lot of integers and longs when only using small numbers. > > Next step is to add JMH micro-benchmark to check the impact in terms of > performances. > > > Example on a simple object w/ null compaction > > Length=55 FooterPosition=50 > 0x67 // ValueType > 0x01 // FormatVersion > 0x2b 0x00 //Flags userType=true hasSchema=true offset=1 compactFooter=true > 0x78 0x66 0xbe 0x44 //TypeId > 0xf9 0xcd 0x07 0x57 //Hashcode > 0x37 0x00 0x00 0x00 //Length > 0x3d 0xa8 0x15 0xe4 //SchemaId > 0x32 0x00 0x00 0x00 //Footer position = 50 > 0x03 0x01 0x00 0x00 0x00 0x03 0x01 0x00 0x00 0x00 0x09 0x03 0x00 0x00 0x00 > 0x61 0x62 0x63 0x09 0x03 0x00 0x00 0x00 0x61 0x62 0x63 > Footer length=5 > 0x18 0x1d 0x22 0x2a 0x47 > > and w/o null compaction > Length=60 FooterPosition=53 > 0x67 // ValueType > 0x01 // FormatVersion > 0x2b 0x00 //Flags userType=true hasSchema=true offset=1 compactFooter=true > 0x78 0x66 0xbe 0x44 //TypeId > 0xa4 0x43 0x0e 0xf5 //Hashcode > 0x3c 0x00 0x00 0x00 //Length > 0x3d 0xa8 0x15 0xe4 //SchemaId > 0x35 0x00 0x00 0x00 //Footer position = 53 > 0x03 0x01 0x00 0x00 0x00 0x03 0x01 0x00 0x00 0x00 0x09 0x03 0x00 0x00 0x00 > 0x61 0x62 0x63 0x65 0x65 0x65 0x09 0x03 0x00 0x00 0x00 0x61 0x62 0x63 > Footer length=7 > 0x18 0x1d 0x22 0x2a 0x2b 0x2c 0x2d > > > > > -- > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ >
