Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Dmitriy, Alex, thanks for clarification, this is that i already mention in ticket. I think this approach correct. Try to provide PR soon. thanks! > >--- Forwarded message --- >From: "Alex Plehanov" < plehanov.a...@gmail.com > >To: dev@ignite.apache.org >Cc: arzamas...@mail.ru >Subject: Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench. >Date: Tue, 21 Aug 2018 23:17:20 +0300 > >Dmitriy, yes, you correctly understood my idea. > >вт, 21 авг. 2018 г. в 22:03, Vladimir Ozerov < voze...@gridgain.com >: > >> Guys, >> >> Do we have any numbers for real workloads? Or at least Yardstick. >> >> вт, 21 авг. 2018 г. в 21:04, Dmitriy Pavlov < dpavlov@gmail.com >: >> >> > Let us double-check current options. Maybe I'm mistaken about Alex >> > Plehanov's idea. >> > >> > Let's introduce the following variables and functions: >> > data - our block. >> > OldImpl(data) = X >> > NewImpl(data) = Y >> > NewImpl(data) ^ C = Y ^ C, where C is constant, ^ is bitwise XOR >> operation. >> > And NewImpl(data) ^ C = X = OldImpl(data) >> > >> > So new and old implementations give us X in all cases, we don't need >> any >> > compatibility mode. >> > >> > Let's try to adopt new fast implementation with C constant. >> > >> > Evgeniy, would you like to try? >> > >> > вт, 21 авг. 2018 г. в 15:51, Sergey Kozlov < skoz...@gridgain.com >: >> > >> > > Alex >> > > >> > > In that case the data becomes in unpredictable state (mix of new and >> old >> > > methods) and there's a chance that some partitions will be never >> touched >> > > and never converted. It will force us always support old compression. >> > > I suppose the explicit conversion that clearly say what is happening >> now >> > > and may be warn that the db files should backed up before version >> > upgrade. >> > > >> > > Also I think AI 3.0 will have set of incompatible changes that give >> us >> a >> > > chance to add an upgrade procedure where compression method update >> among >> > > other stuff will take into account. >> > > >> > > On Tue, Aug 21, 2018 at 11:58 AM, Alex Plehanov < >> plehanov.a...@gmail.com >> > > >> > > wrote: >> > > >> > > > Sergey, converting data also force us to introduce some flag to WAL >> > > > segments/records or convert WAL segments too. WAL segments can be >> > > archived, >> > > > they also can be compressed. >> > > > IMO we better should not introduce any compatibility modes, left >> data >> > as >> > > is >> > > > and always just convert crc value returned by zip.CRC32 to old >> format >> > > (xor >> > > > it) at runtime. >> > > > >> > > > 2018-08-21 0:12 GMT+03:00 Sergey Kozlov < skoz...@gridgain.com >: >> > > > >> > > > > Dmitriy >> > > > > >> > > > > Due to significant improvement and to reduce the number supported >> > > > > modes/options would be good to convert the data at the moment of >> > > upgrade. >> > > > > >> > > > > On Tue, Aug 21, 2018 at 12:03 AM, Dmitriy Setrakyan < >> > > > dsetrak...@apache.org >> > > > > > >> > > > > wrote: >> > > > > >> > > > > > Sergey, that was precisely my comment in the ticket: >> > > > > > >> > > > > > Can we add this option without breaking compatibility with >> previous >> > > > > > page/storage formats? If not, then this should support both >> > > > > implementation. >> > > > > > The default should be the new fastest implementation, but if we >> > > > encounter >> > > > > > the older, slower one, then we should print out a warning in >> the >> > log >> > > > and >> > > > > > automatically switch to the older implementation. >> > > > > > >> > > > > > On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov < >> > skoz...@gridgain.com >> > > > >> > > > > > wrote: >> > > > > > >> > > > > > > Hi Igniters >> > >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Dmitriy, yes, you correctly understood my idea. вт, 21 авг. 2018 г. в 22:03, Vladimir Ozerov : > Guys, > > Do we have any numbers for real workloads? Or at least Yardstick. > > вт, 21 авг. 2018 г. в 21:04, Dmitriy Pavlov : > > > Let us double-check current options. Maybe I'm mistaken about Alex > > Plehanov's idea. > > > > Let's introduce the following variables and functions: > > data - our block. > > OldImpl(data) = X > > NewImpl(data) = Y > > NewImpl(data) ^ C = Y ^ C, where C is constant, ^ is bitwise XOR > operation. > > And NewImpl(data) ^ C = X = OldImpl(data) > > > > So new and old implementations give us X in all cases, we don't need any > > compatibility mode. > > > > Let's try to adopt new fast implementation with C constant. > > > > Evgeniy, would you like to try? > > > > вт, 21 авг. 2018 г. в 15:51, Sergey Kozlov : > > > > > Alex > > > > > > In that case the data becomes in unpredictable state (mix of new and > old > > > methods) and there's a chance that some partitions will be never > touched > > > and never converted. It will force us always support old compression. > > > I suppose the explicit conversion that clearly say what is happening > now > > > and may be warn that the db files should backed up before version > > upgrade. > > > > > > Also I think AI 3.0 will have set of incompatible changes that give us > a > > > chance to add an upgrade procedure where compression method update > among > > > other stuff will take into account. > > > > > > On Tue, Aug 21, 2018 at 11:58 AM, Alex Plehanov < > plehanov.a...@gmail.com > > > > > > wrote: > > > > > > > Sergey, converting data also force us to introduce some flag to WAL > > > > segments/records or convert WAL segments too. WAL segments can be > > > archived, > > > > they also can be compressed. > > > > IMO we better should not introduce any compatibility modes, left data > > as > > > is > > > > and always just convert crc value returned by zip.CRC32 to old format > > > (xor > > > > it) at runtime. > > > > > > > > 2018-08-21 0:12 GMT+03:00 Sergey Kozlov : > > > > > > > > > Dmitriy > > > > > > > > > > Due to significant improvement and to reduce the number supported > > > > > modes/options would be good to convert the data at the moment of > > > upgrade. > > > > > > > > > > On Tue, Aug 21, 2018 at 12:03 AM, Dmitriy Setrakyan < > > > > dsetrak...@apache.org > > > > > > > > > > > wrote: > > > > > > > > > > > Sergey, that was precisely my comment in the ticket: > > > > > > > > > > > > Can we add this option without breaking compatibility with > previous > > > > > > page/storage formats? If not, then this should support both > > > > > implementation. > > > > > > The default should be the new fastest implementation, but if we > > > > encounter > > > > > > the older, slower one, then we should print out a warning in the > > log > > > > and > > > > > > automatically switch to the older implementation. > > > > > > > > > > > > On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov < > > skoz...@gridgain.com > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Igniters > > > > > > > > > > > > > > I suppose that'll break compatibility for LFS (PDS). > > > > > > > > > > > > > > Do we plan to provide a migration guide w/o data loss for > upgrade > > > AI > > > > > 2.x > > > > > > to > > > > > > > 3.0? > > > > > > > > > > > > > > On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan < > > > > > > dsetrak...@apache.org > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > I commented in the ticket: https://issues.apache.org/ > > > > > > > > jira/browse/IGNITE-9272 > > > > > > > > > > > > > > > > It if can integrate it correctly, according to my comment, in > > 2.7 > > > > > > > release, > > > > > > > > it would be great. Otherwise, let's plan this change for 3.0 > > > > release. > > > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev < > > > > > > > > eduard.shangar...@gmail.com> wrote: > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > I have checked the benchmark and it shows great performance > > > boost > > > > > on > > > > > > my > > > > > > > > > laptop! > > > > > > > > > > > > > > > > > > +1 for this change. > > > > > > > > > > > > > > > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov < > > > > > > dpavlov@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Evgeniy, > > > > > > > > > > > > > > > > > > > > Thank you. I see that the ticket is unassigned. > > > > > > > > > > > > > > > > > > > > Would you like to contribute PR to be macro-benchmarked > > with > > > > > > Ignite? > > > > > > > > > > > > > > > > > > > > Sincerely, > > > > > > > > > > Dmitriy Pavlov > > > > > > > > > > > > > > > > > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский > > > > > > > > > > : > > > > > > > > > > > > > > > > > > > > > I fill the ticket, bench code attached there. > > > > > > > > > > >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Guys, Do we have any numbers for real workloads? Or at least Yardstick. вт, 21 авг. 2018 г. в 21:04, Dmitriy Pavlov : > Let us double-check current options. Maybe I'm mistaken about Alex > Plehanov's idea. > > Let's introduce the following variables and functions: > data - our block. > OldImpl(data) = X > NewImpl(data) = Y > NewImpl(data) ^ C = Y ^ C, where C is constant, ^ is bitwise XOR operation. > And NewImpl(data) ^ C = X = OldImpl(data) > > So new and old implementations give us X in all cases, we don't need any > compatibility mode. > > Let's try to adopt new fast implementation with C constant. > > Evgeniy, would you like to try? > > вт, 21 авг. 2018 г. в 15:51, Sergey Kozlov : > > > Alex > > > > In that case the data becomes in unpredictable state (mix of new and old > > methods) and there's a chance that some partitions will be never touched > > and never converted. It will force us always support old compression. > > I suppose the explicit conversion that clearly say what is happening now > > and may be warn that the db files should backed up before version > upgrade. > > > > Also I think AI 3.0 will have set of incompatible changes that give us a > > chance to add an upgrade procedure where compression method update among > > other stuff will take into account. > > > > On Tue, Aug 21, 2018 at 11:58 AM, Alex Plehanov > > > wrote: > > > > > Sergey, converting data also force us to introduce some flag to WAL > > > segments/records or convert WAL segments too. WAL segments can be > > archived, > > > they also can be compressed. > > > IMO we better should not introduce any compatibility modes, left data > as > > is > > > and always just convert crc value returned by zip.CRC32 to old format > > (xor > > > it) at runtime. > > > > > > 2018-08-21 0:12 GMT+03:00 Sergey Kozlov : > > > > > > > Dmitriy > > > > > > > > Due to significant improvement and to reduce the number supported > > > > modes/options would be good to convert the data at the moment of > > upgrade. > > > > > > > > On Tue, Aug 21, 2018 at 12:03 AM, Dmitriy Setrakyan < > > > dsetrak...@apache.org > > > > > > > > > wrote: > > > > > > > > > Sergey, that was precisely my comment in the ticket: > > > > > > > > > > Can we add this option without breaking compatibility with previous > > > > > page/storage formats? If not, then this should support both > > > > implementation. > > > > > The default should be the new fastest implementation, but if we > > > encounter > > > > > the older, slower one, then we should print out a warning in the > log > > > and > > > > > automatically switch to the older implementation. > > > > > > > > > > On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov < > skoz...@gridgain.com > > > > > > > > wrote: > > > > > > > > > > > Hi Igniters > > > > > > > > > > > > I suppose that'll break compatibility for LFS (PDS). > > > > > > > > > > > > Do we plan to provide a migration guide w/o data loss for upgrade > > AI > > > > 2.x > > > > > to > > > > > > 3.0? > > > > > > > > > > > > On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan < > > > > > dsetrak...@apache.org > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > I commented in the ticket: https://issues.apache.org/ > > > > > > > jira/browse/IGNITE-9272 > > > > > > > > > > > > > > It if can integrate it correctly, according to my comment, in > 2.7 > > > > > > release, > > > > > > > it would be great. Otherwise, let's plan this change for 3.0 > > > release. > > > > > > > > > > > > > > D. > > > > > > > > > > > > > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev < > > > > > > > eduard.shangar...@gmail.com> wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > I have checked the benchmark and it shows great performance > > boost > > > > on > > > > > my > > > > > > > > laptop! > > > > > > > > > > > > > > > > +1 for this change. > > > > > > > > > > > > > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov < > > > > > dpavlov@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Evgeniy, > > > > > > > > > > > > > > > > > > Thank you. I see that the ticket is unassigned. > > > > > > > > > > > > > > > > > > Would you like to contribute PR to be macro-benchmarked > with > > > > > Ignite? > > > > > > > > > > > > > > > > > > Sincerely, > > > > > > > > > Dmitriy Pavlov > > > > > > > > > > > > > > > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский > > > > > > > > > : > > > > > > > > > > > > > > > > > > > I fill the ticket, bench code attached there. > > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-9272 > > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >Has anyone else run the benchmark and reproduced the > > > > performance > > > > > > > > > > >difference? > > > > > > > > > > > > > > > > > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov < > > > > > > > > dpavlov@gmail.com > > > > > > > > > > > > > > > > > > > > >wrote: > > > > > > > > > > > > > > > > > > > > > >>
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Let us double-check current options. Maybe I'm mistaken about Alex Plehanov's idea. Let's introduce the following variables and functions: data - our block. OldImpl(data) = X NewImpl(data) = Y NewImpl(data) ^ C = Y ^ C, where C is constant, ^ is bitwise XOR operation. And NewImpl(data) ^ C = X = OldImpl(data) So new and old implementations give us X in all cases, we don't need any compatibility mode. Let's try to adopt new fast implementation with C constant. Evgeniy, would you like to try? вт, 21 авг. 2018 г. в 15:51, Sergey Kozlov : > Alex > > In that case the data becomes in unpredictable state (mix of new and old > methods) and there's a chance that some partitions will be never touched > and never converted. It will force us always support old compression. > I suppose the explicit conversion that clearly say what is happening now > and may be warn that the db files should backed up before version upgrade. > > Also I think AI 3.0 will have set of incompatible changes that give us a > chance to add an upgrade procedure where compression method update among > other stuff will take into account. > > On Tue, Aug 21, 2018 at 11:58 AM, Alex Plehanov > wrote: > > > Sergey, converting data also force us to introduce some flag to WAL > > segments/records or convert WAL segments too. WAL segments can be > archived, > > they also can be compressed. > > IMO we better should not introduce any compatibility modes, left data as > is > > and always just convert crc value returned by zip.CRC32 to old format > (xor > > it) at runtime. > > > > 2018-08-21 0:12 GMT+03:00 Sergey Kozlov : > > > > > Dmitriy > > > > > > Due to significant improvement and to reduce the number supported > > > modes/options would be good to convert the data at the moment of > upgrade. > > > > > > On Tue, Aug 21, 2018 at 12:03 AM, Dmitriy Setrakyan < > > dsetrak...@apache.org > > > > > > > wrote: > > > > > > > Sergey, that was precisely my comment in the ticket: > > > > > > > > Can we add this option without breaking compatibility with previous > > > > page/storage formats? If not, then this should support both > > > implementation. > > > > The default should be the new fastest implementation, but if we > > encounter > > > > the older, slower one, then we should print out a warning in the log > > and > > > > automatically switch to the older implementation. > > > > > > > > On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov > > > > > wrote: > > > > > > > > > Hi Igniters > > > > > > > > > > I suppose that'll break compatibility for LFS (PDS). > > > > > > > > > > Do we plan to provide a migration guide w/o data loss for upgrade > AI > > > 2.x > > > > to > > > > > 3.0? > > > > > > > > > > On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan < > > > > dsetrak...@apache.org > > > > > > > > > > > wrote: > > > > > > > > > > > I commented in the ticket: https://issues.apache.org/ > > > > > > jira/browse/IGNITE-9272 > > > > > > > > > > > > It if can integrate it correctly, according to my comment, in 2.7 > > > > > release, > > > > > > it would be great. Otherwise, let's plan this change for 3.0 > > release. > > > > > > > > > > > > D. > > > > > > > > > > > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev < > > > > > > eduard.shangar...@gmail.com> wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I have checked the benchmark and it shows great performance > boost > > > on > > > > my > > > > > > > laptop! > > > > > > > > > > > > > > +1 for this change. > > > > > > > > > > > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov < > > > > dpavlov@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Evgeniy, > > > > > > > > > > > > > > > > Thank you. I see that the ticket is unassigned. > > > > > > > > > > > > > > > > Would you like to contribute PR to be macro-benchmarked with > > > > Ignite? > > > > > > > > > > > > > > > > Sincerely, > > > > > > > > Dmitriy Pavlov > > > > > > > > > > > > > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский > > > > > > > > : > > > > > > > > > > > > > > > > > I fill the ticket, bench code attached there. > > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-9272 > > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > > > > > > > > >Has anyone else run the benchmark and reproduced the > > > performance > > > > > > > > > >difference? > > > > > > > > > > > > > > > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov < > > > > > > > dpavlov@gmail.com > > > > > > > > > > > > > > > > > > >wrote: > > > > > > > > > > > > > > > > > > > >> It depends. > > > > > > > > > >> > > > > > > > > > >> CRC is a CPU-intensive operation, while WAL logging and > > page > > > > > store > > > > > > > > write > > > > > > > > > >> are mostly about IO speed. > > > > > > > > > >> > > > > > > > > > >> In the same time, it can make the huge impact on > machines > > > with > > > > > > fast > > > > > > > IO > > > > > > > > > >> and > > > > > > > > > >> slow CPU. So if we can apply change
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Alex In that case the data becomes in unpredictable state (mix of new and old methods) and there's a chance that some partitions will be never touched and never converted. It will force us always support old compression. I suppose the explicit conversion that clearly say what is happening now and may be warn that the db files should backed up before version upgrade. Also I think AI 3.0 will have set of incompatible changes that give us a chance to add an upgrade procedure where compression method update among other stuff will take into account. On Tue, Aug 21, 2018 at 11:58 AM, Alex Plehanov wrote: > Sergey, converting data also force us to introduce some flag to WAL > segments/records or convert WAL segments too. WAL segments can be archived, > they also can be compressed. > IMO we better should not introduce any compatibility modes, left data as is > and always just convert crc value returned by zip.CRC32 to old format (xor > it) at runtime. > > 2018-08-21 0:12 GMT+03:00 Sergey Kozlov : > > > Dmitriy > > > > Due to significant improvement and to reduce the number supported > > modes/options would be good to convert the data at the moment of upgrade. > > > > On Tue, Aug 21, 2018 at 12:03 AM, Dmitriy Setrakyan < > dsetrak...@apache.org > > > > > wrote: > > > > > Sergey, that was precisely my comment in the ticket: > > > > > > Can we add this option without breaking compatibility with previous > > > page/storage formats? If not, then this should support both > > implementation. > > > The default should be the new fastest implementation, but if we > encounter > > > the older, slower one, then we should print out a warning in the log > and > > > automatically switch to the older implementation. > > > > > > On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov > > > wrote: > > > > > > > Hi Igniters > > > > > > > > I suppose that'll break compatibility for LFS (PDS). > > > > > > > > Do we plan to provide a migration guide w/o data loss for upgrade AI > > 2.x > > > to > > > > 3.0? > > > > > > > > On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan < > > > dsetrak...@apache.org > > > > > > > > > wrote: > > > > > > > > > I commented in the ticket: https://issues.apache.org/ > > > > > jira/browse/IGNITE-9272 > > > > > > > > > > It if can integrate it correctly, according to my comment, in 2.7 > > > > release, > > > > > it would be great. Otherwise, let's plan this change for 3.0 > release. > > > > > > > > > > D. > > > > > > > > > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev < > > > > > eduard.shangar...@gmail.com> wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I have checked the benchmark and it shows great performance boost > > on > > > my > > > > > > laptop! > > > > > > > > > > > > +1 for this change. > > > > > > > > > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov < > > > dpavlov@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Hi Evgeniy, > > > > > > > > > > > > > > Thank you. I see that the ticket is unassigned. > > > > > > > > > > > > > > Would you like to contribute PR to be macro-benchmarked with > > > Ignite? > > > > > > > > > > > > > > Sincerely, > > > > > > > Dmitriy Pavlov > > > > > > > > > > > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский > > > > > > > : > > > > > > > > > > > > > > > I fill the ticket, bench code attached there. > > > > > > > > https://issues.apache.org/jira/browse/IGNITE-9272 > > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > > > > > >Has anyone else run the benchmark and reproduced the > > performance > > > > > > > > >difference? > > > > > > > > > > > > > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov < > > > > > > dpavlov@gmail.com > > > > > > > > > > > > > > > > >wrote: > > > > > > > > > > > > > > > > > >> It depends. > > > > > > > > >> > > > > > > > > >> CRC is a CPU-intensive operation, while WAL logging and > page > > > > store > > > > > > > write > > > > > > > > >> are mostly about IO speed. > > > > > > > > >> > > > > > > > > >> In the same time, it can make the huge impact on machines > > with > > > > > fast > > > > > > IO > > > > > > > > >> and > > > > > > > > >> slow CPU. So if we can apply change proposed by Evgeniy > and > > > > Alexey > > > > > > it > > > > > > > > >> could > > > > > > > > >> benefit performance because we save CPU. Later we can use > > it's > > > > > power > > > > > > > in > > > > > > > > a > > > > > > > > >> more efficient manner (e.g. with compression). > > > > > > > > >> > > > > > > > > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov < > > > > yzhda...@apache.org > > > > > >: > > > > > > > > >> > > > > > > > > >> > Guys, what time in % does crc calculation take in WAL > > > logging > > > > > > > process? > > > > > > > > >> > > > > > > > > > >> > --Yakov > > > > > > > > >> > > > > > > > > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov < > > > > > dpavlov@gmail.com > > > > > > > >: > > > > > > > > >> > > > > > > > > > >> > > Hi Alex, thank you for this idea. > > > > > > > > >> >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Sergey, converting data also force us to introduce some flag to WAL segments/records or convert WAL segments too. WAL segments can be archived, they also can be compressed. IMO we better should not introduce any compatibility modes, left data as is and always just convert crc value returned by zip.CRC32 to old format (xor it) at runtime. 2018-08-21 0:12 GMT+03:00 Sergey Kozlov : > Dmitriy > > Due to significant improvement and to reduce the number supported > modes/options would be good to convert the data at the moment of upgrade. > > On Tue, Aug 21, 2018 at 12:03 AM, Dmitriy Setrakyan > > wrote: > > > Sergey, that was precisely my comment in the ticket: > > > > Can we add this option without breaking compatibility with previous > > page/storage formats? If not, then this should support both > implementation. > > The default should be the new fastest implementation, but if we encounter > > the older, slower one, then we should print out a warning in the log and > > automatically switch to the older implementation. > > > > On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov > > wrote: > > > > > Hi Igniters > > > > > > I suppose that'll break compatibility for LFS (PDS). > > > > > > Do we plan to provide a migration guide w/o data loss for upgrade AI > 2.x > > to > > > 3.0? > > > > > > On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan < > > dsetrak...@apache.org > > > > > > > wrote: > > > > > > > I commented in the ticket: https://issues.apache.org/ > > > > jira/browse/IGNITE-9272 > > > > > > > > It if can integrate it correctly, according to my comment, in 2.7 > > > release, > > > > it would be great. Otherwise, let's plan this change for 3.0 release. > > > > > > > > D. > > > > > > > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev < > > > > eduard.shangar...@gmail.com> wrote: > > > > > > > > > Hi, > > > > > > > > > > I have checked the benchmark and it shows great performance boost > on > > my > > > > > laptop! > > > > > > > > > > +1 for this change. > > > > > > > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov < > > dpavlov@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi Evgeniy, > > > > > > > > > > > > Thank you. I see that the ticket is unassigned. > > > > > > > > > > > > Would you like to contribute PR to be macro-benchmarked with > > Ignite? > > > > > > > > > > > > Sincerely, > > > > > > Dmitriy Pavlov > > > > > > > > > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский > > > > > > : > > > > > > > > > > > > > I fill the ticket, bench code attached there. > > > > > > > https://issues.apache.org/jira/browse/IGNITE-9272 > > > > > > > Thanks! > > > > > > > > > > > > > > > > > > > > > >Has anyone else run the benchmark and reproduced the > performance > > > > > > > >difference? > > > > > > > > > > > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov < > > > > > dpavlov@gmail.com > > > > > > > > > > > > > > >wrote: > > > > > > > > > > > > > > > >> It depends. > > > > > > > >> > > > > > > > >> CRC is a CPU-intensive operation, while WAL logging and page > > > store > > > > > > write > > > > > > > >> are mostly about IO speed. > > > > > > > >> > > > > > > > >> In the same time, it can make the huge impact on machines > with > > > > fast > > > > > IO > > > > > > > >> and > > > > > > > >> slow CPU. So if we can apply change proposed by Evgeniy and > > > Alexey > > > > > it > > > > > > > >> could > > > > > > > >> benefit performance because we save CPU. Later we can use > it's > > > > power > > > > > > in > > > > > > > a > > > > > > > >> more efficient manner (e.g. with compression). > > > > > > > >> > > > > > > > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov < > > > yzhda...@apache.org > > > > >: > > > > > > > >> > > > > > > > >> > Guys, what time in % does crc calculation take in WAL > > logging > > > > > > process? > > > > > > > >> > > > > > > > > >> > --Yakov > > > > > > > >> > > > > > > > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov < > > > > dpavlov@gmail.com > > > > > > >: > > > > > > > >> > > > > > > > > >> > > Hi Alex, thank you for this idea. > > > > > > > >> > > > > > > > > > >> > > Evgeniy, Alex, would you like to submit the patch with > > > > bypassing > > > > > > > >> > > implementation differences to keep compatibility? > > > > > > > >> > > > > > > > > > >> > > Sincerely, > > > > > > > >> > > Dmitriy Pavlov > > > > > > > >> > > > > > > > > > >> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov < > > > > > > > plehanov.a...@gmail.com >: > > > > > > > >> > > > > > > > > > >> > > > Hello, Igniters! > > > > > > > >> > > > > > > > > > > >> > > > In java8 java.lang.zip.CRC32 methods become intrinsic, > > > > > moreover > > > > > > > new > > > > > > > >> > > > "update" method, which use ByteBuffer was introduced. > > > Since > > > > we > > > > > > > >> moved > > > > > > > >> to > > > > > > > >> > > > java8, perhaps we really can get performance boost by > > > using > > > > > > > >> standard > > > > > > > >> > > > java.lang.zip.CRC32 instead of PureJavaCrc32. > > >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Dmitriy Due to significant improvement and to reduce the number supported modes/options would be good to convert the data at the moment of upgrade. On Tue, Aug 21, 2018 at 12:03 AM, Dmitriy Setrakyan wrote: > Sergey, that was precisely my comment in the ticket: > > Can we add this option without breaking compatibility with previous > page/storage formats? If not, then this should support both implementation. > The default should be the new fastest implementation, but if we encounter > the older, slower one, then we should print out a warning in the log and > automatically switch to the older implementation. > > On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov > wrote: > > > Hi Igniters > > > > I suppose that'll break compatibility for LFS (PDS). > > > > Do we plan to provide a migration guide w/o data loss for upgrade AI 2.x > to > > 3.0? > > > > On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan < > dsetrak...@apache.org > > > > > wrote: > > > > > I commented in the ticket: https://issues.apache.org/ > > > jira/browse/IGNITE-9272 > > > > > > It if can integrate it correctly, according to my comment, in 2.7 > > release, > > > it would be great. Otherwise, let's plan this change for 3.0 release. > > > > > > D. > > > > > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev < > > > eduard.shangar...@gmail.com> wrote: > > > > > > > Hi, > > > > > > > > I have checked the benchmark and it shows great performance boost on > my > > > > laptop! > > > > > > > > +1 for this change. > > > > > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov < > dpavlov@gmail.com> > > > > wrote: > > > > > > > > > Hi Evgeniy, > > > > > > > > > > Thank you. I see that the ticket is unassigned. > > > > > > > > > > Would you like to contribute PR to be macro-benchmarked with > Ignite? > > > > > > > > > > Sincerely, > > > > > Dmitriy Pavlov > > > > > > > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский > > > > > : > > > > > > > > > > > I fill the ticket, bench code attached there. > > > > > > https://issues.apache.org/jira/browse/IGNITE-9272 > > > > > > Thanks! > > > > > > > > > > > > > > > > > > >Has anyone else run the benchmark and reproduced the performance > > > > > > >difference? > > > > > > > > > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov < > > > > dpavlov@gmail.com > > > > > > > > > > > > >wrote: > > > > > > > > > > > > > >> It depends. > > > > > > >> > > > > > > >> CRC is a CPU-intensive operation, while WAL logging and page > > store > > > > > write > > > > > > >> are mostly about IO speed. > > > > > > >> > > > > > > >> In the same time, it can make the huge impact on machines with > > > fast > > > > IO > > > > > > >> and > > > > > > >> slow CPU. So if we can apply change proposed by Evgeniy and > > Alexey > > > > it > > > > > > >> could > > > > > > >> benefit performance because we save CPU. Later we can use it's > > > power > > > > > in > > > > > > a > > > > > > >> more efficient manner (e.g. with compression). > > > > > > >> > > > > > > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov < > > yzhda...@apache.org > > > >: > > > > > > >> > > > > > > >> > Guys, what time in % does crc calculation take in WAL > logging > > > > > process? > > > > > > >> > > > > > > > >> > --Yakov > > > > > > >> > > > > > > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov < > > > dpavlov@gmail.com > > > > > >: > > > > > > >> > > > > > > > >> > > Hi Alex, thank you for this idea. > > > > > > >> > > > > > > > > >> > > Evgeniy, Alex, would you like to submit the patch with > > > bypassing > > > > > > >> > > implementation differences to keep compatibility? > > > > > > >> > > > > > > > > >> > > Sincerely, > > > > > > >> > > Dmitriy Pavlov > > > > > > >> > > > > > > > > >> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov < > > > > > > plehanov.a...@gmail.com >: > > > > > > >> > > > > > > > > >> > > > Hello, Igniters! > > > > > > >> > > > > > > > > > >> > > > In java8 java.lang.zip.CRC32 methods become intrinsic, > > > > moreover > > > > > > new > > > > > > >> > > > "update" method, which use ByteBuffer was introduced. > > Since > > > we > > > > > > >> moved > > > > > > >> to > > > > > > >> > > > java8, perhaps we really can get performance boost by > > using > > > > > > >> standard > > > > > > >> > > > java.lang.zip.CRC32 instead of PureJavaCrc32. > > > > > > >> > > > > > > > > > >> > > > About compatibility: looks like PureJavaCrc32 implements > > the > > > > > same > > > > > > >> > > algorithm > > > > > > >> > > > as java.lang.zip.CRC32. These two implementations uses > the > > > > same > > > > > > >> > > polynomial > > > > > > >> > > > and the same initial value. The only difference is final > > xor > > > > > mask > > > > > > >> > > > (0x for java.lang.zip.CRC32). So, we can easily > > > > convert > > > > > > >> from > > > > > > >> > > > PureJavaCrc32 > > > > > > >> > > > to standard CRC32 and vice versa, using this expression: > > > crc32 > > > > > ^= > > > > > > >> > > > 0x > > > > > > >> > > > > > >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Igniters, let's come back to the idea from Alex Plehanov. We can try to provide custom xor mask at the end of CRC calc. We can do a compatible solution using fast implementation, can't we? Sincerely, Dmitriy Pavlov вт, 21 авг. 2018 г. в 0:04, Dmitriy Setrakyan : > Sergey, that was precisely my comment in the ticket: > > Can we add this option without breaking compatibility with previous > page/storage formats? If not, then this should support both implementation. > The default should be the new fastest implementation, but if we encounter > the older, slower one, then we should print out a warning in the log and > automatically switch to the older implementation. > > On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov > wrote: > > > Hi Igniters > > > > I suppose that'll break compatibility for LFS (PDS). > > > > Do we plan to provide a migration guide w/o data loss for upgrade AI 2.x > to > > 3.0? > > > > On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan < > dsetrak...@apache.org > > > > > wrote: > > > > > I commented in the ticket: https://issues.apache.org/ > > > jira/browse/IGNITE-9272 > > > > > > It if can integrate it correctly, according to my comment, in 2.7 > > release, > > > it would be great. Otherwise, let's plan this change for 3.0 release. > > > > > > D. > > > > > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev < > > > eduard.shangar...@gmail.com> wrote: > > > > > > > Hi, > > > > > > > > I have checked the benchmark and it shows great performance boost on > my > > > > laptop! > > > > > > > > +1 for this change. > > > > > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov < > dpavlov@gmail.com> > > > > wrote: > > > > > > > > > Hi Evgeniy, > > > > > > > > > > Thank you. I see that the ticket is unassigned. > > > > > > > > > > Would you like to contribute PR to be macro-benchmarked with > Ignite? > > > > > > > > > > Sincerely, > > > > > Dmitriy Pavlov > > > > > > > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский > > > > > : > > > > > > > > > > > I fill the ticket, bench code attached there. > > > > > > https://issues.apache.org/jira/browse/IGNITE-9272 > > > > > > Thanks! > > > > > > > > > > > > > > > > > > >Has anyone else run the benchmark and reproduced the performance > > > > > > >difference? > > > > > > > > > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov < > > > > dpavlov@gmail.com > > > > > > > > > > > > >wrote: > > > > > > > > > > > > > >> It depends. > > > > > > >> > > > > > > >> CRC is a CPU-intensive operation, while WAL logging and page > > store > > > > > write > > > > > > >> are mostly about IO speed. > > > > > > >> > > > > > > >> In the same time, it can make the huge impact on machines with > > > fast > > > > IO > > > > > > >> and > > > > > > >> slow CPU. So if we can apply change proposed by Evgeniy and > > Alexey > > > > it > > > > > > >> could > > > > > > >> benefit performance because we save CPU. Later we can use it's > > > power > > > > > in > > > > > > a > > > > > > >> more efficient manner (e.g. with compression). > > > > > > >> > > > > > > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov < > > yzhda...@apache.org > > > >: > > > > > > >> > > > > > > >> > Guys, what time in % does crc calculation take in WAL > logging > > > > > process? > > > > > > >> > > > > > > > >> > --Yakov > > > > > > >> > > > > > > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov < > > > dpavlov@gmail.com > > > > > >: > > > > > > >> > > > > > > > >> > > Hi Alex, thank you for this idea. > > > > > > >> > > > > > > > > >> > > Evgeniy, Alex, would you like to submit the patch with > > > bypassing > > > > > > >> > > implementation differences to keep compatibility? > > > > > > >> > > > > > > > > >> > > Sincerely, > > > > > > >> > > Dmitriy Pavlov > > > > > > >> > > > > > > > > >> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov < > > > > > > plehanov.a...@gmail.com >: > > > > > > >> > > > > > > > > >> > > > Hello, Igniters! > > > > > > >> > > > > > > > > > >> > > > In java8 java.lang.zip.CRC32 methods become intrinsic, > > > > moreover > > > > > > new > > > > > > >> > > > "update" method, which use ByteBuffer was introduced. > > Since > > > we > > > > > > >> moved > > > > > > >> to > > > > > > >> > > > java8, perhaps we really can get performance boost by > > using > > > > > > >> standard > > > > > > >> > > > java.lang.zip.CRC32 instead of PureJavaCrc32. > > > > > > >> > > > > > > > > > >> > > > About compatibility: looks like PureJavaCrc32 implements > > the > > > > > same > > > > > > >> > > algorithm > > > > > > >> > > > as java.lang.zip.CRC32. These two implementations uses > the > > > > same > > > > > > >> > > polynomial > > > > > > >> > > > and the same initial value. The only difference is final > > xor > > > > > mask > > > > > > >> > > > (0x for java.lang.zip.CRC32). So, we can easily > > > > convert > > > > > > >> from > > > > > > >> > > > PureJavaCrc32 > > > > > > >> > > > to standard CRC32 and vice versa, using this expression: > > > crc32 > > > > > ^= >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Sergey, that was precisely my comment in the ticket: Can we add this option without breaking compatibility with previous page/storage formats? If not, then this should support both implementation. The default should be the new fastest implementation, but if we encounter the older, slower one, then we should print out a warning in the log and automatically switch to the older implementation. On Mon, Aug 20, 2018 at 1:58 PM, Sergey Kozlov wrote: > Hi Igniters > > I suppose that'll break compatibility for LFS (PDS). > > Do we plan to provide a migration guide w/o data loss for upgrade AI 2.x to > 3.0? > > On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan > > wrote: > > > I commented in the ticket: https://issues.apache.org/ > > jira/browse/IGNITE-9272 > > > > It if can integrate it correctly, according to my comment, in 2.7 > release, > > it would be great. Otherwise, let's plan this change for 3.0 release. > > > > D. > > > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev < > > eduard.shangar...@gmail.com> wrote: > > > > > Hi, > > > > > > I have checked the benchmark and it shows great performance boost on my > > > laptop! > > > > > > +1 for this change. > > > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov > > > wrote: > > > > > > > Hi Evgeniy, > > > > > > > > Thank you. I see that the ticket is unassigned. > > > > > > > > Would you like to contribute PR to be macro-benchmarked with Ignite? > > > > > > > > Sincerely, > > > > Dmitriy Pavlov > > > > > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский > > > > : > > > > > > > > > I fill the ticket, bench code attached there. > > > > > https://issues.apache.org/jira/browse/IGNITE-9272 > > > > > Thanks! > > > > > > > > > > > > > > > >Has anyone else run the benchmark and reproduced the performance > > > > > >difference? > > > > > > > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov < > > > dpavlov@gmail.com > > > > > > > > > > >wrote: > > > > > > > > > > > >> It depends. > > > > > >> > > > > > >> CRC is a CPU-intensive operation, while WAL logging and page > store > > > > write > > > > > >> are mostly about IO speed. > > > > > >> > > > > > >> In the same time, it can make the huge impact on machines with > > fast > > > IO > > > > > >> and > > > > > >> slow CPU. So if we can apply change proposed by Evgeniy and > Alexey > > > it > > > > > >> could > > > > > >> benefit performance because we save CPU. Later we can use it's > > power > > > > in > > > > > a > > > > > >> more efficient manner (e.g. with compression). > > > > > >> > > > > > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov < > yzhda...@apache.org > > >: > > > > > >> > > > > > >> > Guys, what time in % does crc calculation take in WAL logging > > > > process? > > > > > >> > > > > > > >> > --Yakov > > > > > >> > > > > > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov < > > dpavlov@gmail.com > > > > >: > > > > > >> > > > > > > >> > > Hi Alex, thank you for this idea. > > > > > >> > > > > > > > >> > > Evgeniy, Alex, would you like to submit the patch with > > bypassing > > > > > >> > > implementation differences to keep compatibility? > > > > > >> > > > > > > > >> > > Sincerely, > > > > > >> > > Dmitriy Pavlov > > > > > >> > > > > > > > >> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov < > > > > > plehanov.a...@gmail.com >: > > > > > >> > > > > > > > >> > > > Hello, Igniters! > > > > > >> > > > > > > > > >> > > > In java8 java.lang.zip.CRC32 methods become intrinsic, > > > moreover > > > > > new > > > > > >> > > > "update" method, which use ByteBuffer was introduced. > Since > > we > > > > > >> moved > > > > > >> to > > > > > >> > > > java8, perhaps we really can get performance boost by > using > > > > > >> standard > > > > > >> > > > java.lang.zip.CRC32 instead of PureJavaCrc32. > > > > > >> > > > > > > > > >> > > > About compatibility: looks like PureJavaCrc32 implements > the > > > > same > > > > > >> > > algorithm > > > > > >> > > > as java.lang.zip.CRC32. These two implementations uses the > > > same > > > > > >> > > polynomial > > > > > >> > > > and the same initial value. The only difference is final > xor > > > > mask > > > > > >> > > > (0x for java.lang.zip.CRC32). So, we can easily > > > convert > > > > > >> from > > > > > >> > > > PureJavaCrc32 > > > > > >> > > > to standard CRC32 and vice versa, using this expression: > > crc32 > > > > ^= > > > > > >> > > > 0x > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > 2018-08-14 0:19 GMT+03:00 Eduard Shangareev < > > > > > >> > eduard.shangar...@gmail.com > > > > > >> > > >: > > > > > >> > > > > > > > > >> > > > > Evgeniy, > > > > > >> > > > > > > > > > >> > > > > Could you share benchmark code? And please share what > > > version > > > > of > > > > > >> JVM > > > > > >> > > > > you have used. > > > > > >> > > > > > > > > > >> > > > > On Mon, Aug 13, 2018 at 10:44 PM Zhenya > > > > > >> < arzamas...@mail.ru.invalid > > > > > >> > > > > > > >> > > > > wrote: > > > > > >> > > > > > > > > > >> >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Hi Igniters I suppose that'll break compatibility for LFS (PDS). Do we plan to provide a migration guide w/o data loss for upgrade AI 2.x to 3.0? On Mon, Aug 20, 2018 at 11:46 PM, Dmitriy Setrakyan wrote: > I commented in the ticket: https://issues.apache.org/ > jira/browse/IGNITE-9272 > > It if can integrate it correctly, according to my comment, in 2.7 release, > it would be great. Otherwise, let's plan this change for 3.0 release. > > D. > > On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev < > eduard.shangar...@gmail.com> wrote: > > > Hi, > > > > I have checked the benchmark and it shows great performance boost on my > > laptop! > > > > +1 for this change. > > > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov > > wrote: > > > > > Hi Evgeniy, > > > > > > Thank you. I see that the ticket is unassigned. > > > > > > Would you like to contribute PR to be macro-benchmarked with Ignite? > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский > > > : > > > > > > > I fill the ticket, bench code attached there. > > > > https://issues.apache.org/jira/browse/IGNITE-9272 > > > > Thanks! > > > > > > > > > > > > >Has anyone else run the benchmark and reproduced the performance > > > > >difference? > > > > > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov < > > dpavlov@gmail.com > > > > > > > > >wrote: > > > > > > > > > >> It depends. > > > > >> > > > > >> CRC is a CPU-intensive operation, while WAL logging and page store > > > write > > > > >> are mostly about IO speed. > > > > >> > > > > >> In the same time, it can make the huge impact on machines with > fast > > IO > > > > >> and > > > > >> slow CPU. So if we can apply change proposed by Evgeniy and Alexey > > it > > > > >> could > > > > >> benefit performance because we save CPU. Later we can use it's > power > > > in > > > > a > > > > >> more efficient manner (e.g. with compression). > > > > >> > > > > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov < yzhda...@apache.org > >: > > > > >> > > > > >> > Guys, what time in % does crc calculation take in WAL logging > > > process? > > > > >> > > > > > >> > --Yakov > > > > >> > > > > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov < > dpavlov@gmail.com > > > >: > > > > >> > > > > > >> > > Hi Alex, thank you for this idea. > > > > >> > > > > > > >> > > Evgeniy, Alex, would you like to submit the patch with > bypassing > > > > >> > > implementation differences to keep compatibility? > > > > >> > > > > > > >> > > Sincerely, > > > > >> > > Dmitriy Pavlov > > > > >> > > > > > > >> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov < > > > > plehanov.a...@gmail.com >: > > > > >> > > > > > > >> > > > Hello, Igniters! > > > > >> > > > > > > > >> > > > In java8 java.lang.zip.CRC32 methods become intrinsic, > > moreover > > > > new > > > > >> > > > "update" method, which use ByteBuffer was introduced. Since > we > > > > >> moved > > > > >> to > > > > >> > > > java8, perhaps we really can get performance boost by using > > > > >> standard > > > > >> > > > java.lang.zip.CRC32 instead of PureJavaCrc32. > > > > >> > > > > > > > >> > > > About compatibility: looks like PureJavaCrc32 implements the > > > same > > > > >> > > algorithm > > > > >> > > > as java.lang.zip.CRC32. These two implementations uses the > > same > > > > >> > > polynomial > > > > >> > > > and the same initial value. The only difference is final xor > > > mask > > > > >> > > > (0x for java.lang.zip.CRC32). So, we can easily > > convert > > > > >> from > > > > >> > > > PureJavaCrc32 > > > > >> > > > to standard CRC32 and vice versa, using this expression: > crc32 > > > ^= > > > > >> > > > 0x > > > > >> > > > > > > > >> > > > > > > > >> > > > 2018-08-14 0:19 GMT+03:00 Eduard Shangareev < > > > > >> > eduard.shangar...@gmail.com > > > > >> > > >: > > > > >> > > > > > > > >> > > > > Evgeniy, > > > > >> > > > > > > > > >> > > > > Could you share benchmark code? And please share what > > version > > > of > > > > >> JVM > > > > >> > > > > you have used. > > > > >> > > > > > > > > >> > > > > On Mon, Aug 13, 2018 at 10:44 PM Zhenya > > > > >> < arzamas...@mail.ru.invalid > > > > >> > > > > > >> > > > > wrote: > > > > >> > > > > > > > > >> > > > > > I think it would break backward compatibility, as > Nikolay > > > > >> mentioned > > > > >> > > > above > > > > >> > > > > > we would take exception here: > > > > >> > > > > > > > > > >> > > > > > [1] > > > > >> > > > > > > > > > >> > > > > > https://github.com/apache/ignite/blob/master/modules/ > > > > >> > > > > core/src/main/java/org/apache/ignite/internal/processors/ > > > > >> > > > > cache/persistence/file/FilePageStore.java#L372 > > > > >> > > > > > > > > > >> > > > > > thats why i question for community thoughts here. > > > > >> > > > > > > > > > >> > > > > > > Hi Evgeniy, > > > > >> > > > > > > > > > > >> > > > > > > would you like to submit a patch with CRC32 > > implementation > > > > >> > change? > > > > >> > > > > > > > > > > >> >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
I commented in the ticket: https://issues.apache.org/jira/browse/IGNITE-9272 It if can integrate it correctly, according to my comment, in 2.7 release, it would be great. Otherwise, let's plan this change for 3.0 release. D. On Mon, Aug 20, 2018 at 3:50 AM, Eduard Shangareev < eduard.shangar...@gmail.com> wrote: > Hi, > > I have checked the benchmark and it shows great performance boost on my > laptop! > > +1 for this change. > > On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov > wrote: > > > Hi Evgeniy, > > > > Thank you. I see that the ticket is unassigned. > > > > Would you like to contribute PR to be macro-benchmarked with Ignite? > > > > Sincerely, > > Dmitriy Pavlov > > > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский > > : > > > > > I fill the ticket, bench code attached there. > > > https://issues.apache.org/jira/browse/IGNITE-9272 > > > Thanks! > > > > > > > > > >Has anyone else run the benchmark and reproduced the performance > > > >difference? > > > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov < > dpavlov@gmail.com > > > > > > >wrote: > > > > > > > >> It depends. > > > >> > > > >> CRC is a CPU-intensive operation, while WAL logging and page store > > write > > > >> are mostly about IO speed. > > > >> > > > >> In the same time, it can make the huge impact on machines with fast > IO > > > >> and > > > >> slow CPU. So if we can apply change proposed by Evgeniy and Alexey > it > > > >> could > > > >> benefit performance because we save CPU. Later we can use it's power > > in > > > a > > > >> more efficient manner (e.g. with compression). > > > >> > > > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov < yzhda...@apache.org >: > > > >> > > > >> > Guys, what time in % does crc calculation take in WAL logging > > process? > > > >> > > > > >> > --Yakov > > > >> > > > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov < dpavlov@gmail.com > > >: > > > >> > > > > >> > > Hi Alex, thank you for this idea. > > > >> > > > > > >> > > Evgeniy, Alex, would you like to submit the patch with bypassing > > > >> > > implementation differences to keep compatibility? > > > >> > > > > > >> > > Sincerely, > > > >> > > Dmitriy Pavlov > > > >> > > > > > >> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov < > > > plehanov.a...@gmail.com >: > > > >> > > > > > >> > > > Hello, Igniters! > > > >> > > > > > > >> > > > In java8 java.lang.zip.CRC32 methods become intrinsic, > moreover > > > new > > > >> > > > "update" method, which use ByteBuffer was introduced. Since we > > > >> moved > > > >> to > > > >> > > > java8, perhaps we really can get performance boost by using > > > >> standard > > > >> > > > java.lang.zip.CRC32 instead of PureJavaCrc32. > > > >> > > > > > > >> > > > About compatibility: looks like PureJavaCrc32 implements the > > same > > > >> > > algorithm > > > >> > > > as java.lang.zip.CRC32. These two implementations uses the > same > > > >> > > polynomial > > > >> > > > and the same initial value. The only difference is final xor > > mask > > > >> > > > (0x for java.lang.zip.CRC32). So, we can easily > convert > > > >> from > > > >> > > > PureJavaCrc32 > > > >> > > > to standard CRC32 and vice versa, using this expression: crc32 > > ^= > > > >> > > > 0x > > > >> > > > > > > >> > > > > > > >> > > > 2018-08-14 0:19 GMT+03:00 Eduard Shangareev < > > > >> > eduard.shangar...@gmail.com > > > >> > > >: > > > >> > > > > > > >> > > > > Evgeniy, > > > >> > > > > > > > >> > > > > Could you share benchmark code? And please share what > version > > of > > > >> JVM > > > >> > > > > you have used. > > > >> > > > > > > > >> > > > > On Mon, Aug 13, 2018 at 10:44 PM Zhenya > > > >> < arzamas...@mail.ru.invalid > > > >> > > > > >> > > > > wrote: > > > >> > > > > > > > >> > > > > > I think it would break backward compatibility, as Nikolay > > > >> mentioned > > > >> > > > above > > > >> > > > > > we would take exception here: > > > >> > > > > > > > > >> > > > > > [1] > > > >> > > > > > > > > >> > > > > > https://github.com/apache/ignite/blob/master/modules/ > > > >> > > > > core/src/main/java/org/apache/ignite/internal/processors/ > > > >> > > > > cache/persistence/file/FilePageStore.java#L372 > > > >> > > > > > > > > >> > > > > > thats why i question for community thoughts here. > > > >> > > > > > > > > >> > > > > > > Hi Evgeniy, > > > >> > > > > > > > > > >> > > > > > > would you like to submit a patch with CRC32 > implementation > > > >> > change? > > > >> > > > > > > > > > >> > > > > > > Sincerely, > > > >> > > > > > > Dmitriy Pavlov > > > >> > > > > > > > > > >> > > > > > > пн, 13 авг. 2018 г. в 22:08, Евгений Станиловский > > > >> > > > > > > < arzamas...@mail.ru.invalid >: > > > >> > > > > > > > > > >> > > > > > >> Hi, igniters, i wrote a simple bench, looks like > > > >> PureJavaCrc32 > > > >> > has > > > >> > > > > > >> performance problems in compatible with zip.CRC32. > > > >> > > > > > >> > > > >> > > > > > >> Benchmark Mode Cnt Score Error Units > > > >> > > > > > >>
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Hi, I have checked the benchmark and it shows great performance boost on my laptop! +1 for this change. On Tue, Aug 14, 2018 at 9:01 PM Dmitriy Pavlov wrote: > Hi Evgeniy, > > Thank you. I see that the ticket is unassigned. > > Would you like to contribute PR to be macro-benchmarked with Ignite? > > Sincerely, > Dmitriy Pavlov > > вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский > : > > > I fill the ticket, bench code attached there. > > https://issues.apache.org/jira/browse/IGNITE-9272 > > Thanks! > > > > > > >Has anyone else run the benchmark and reproduced the performance > > >difference? > > > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov < dpavlov@gmail.com > > > > >wrote: > > > > > >> It depends. > > >> > > >> CRC is a CPU-intensive operation, while WAL logging and page store > write > > >> are mostly about IO speed. > > >> > > >> In the same time, it can make the huge impact on machines with fast IO > > >> and > > >> slow CPU. So if we can apply change proposed by Evgeniy and Alexey it > > >> could > > >> benefit performance because we save CPU. Later we can use it's power > in > > a > > >> more efficient manner (e.g. with compression). > > >> > > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov < yzhda...@apache.org >: > > >> > > >> > Guys, what time in % does crc calculation take in WAL logging > process? > > >> > > > >> > --Yakov > > >> > > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov < dpavlov@gmail.com > >: > > >> > > > >> > > Hi Alex, thank you for this idea. > > >> > > > > >> > > Evgeniy, Alex, would you like to submit the patch with bypassing > > >> > > implementation differences to keep compatibility? > > >> > > > > >> > > Sincerely, > > >> > > Dmitriy Pavlov > > >> > > > > >> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov < > > plehanov.a...@gmail.com >: > > >> > > > > >> > > > Hello, Igniters! > > >> > > > > > >> > > > In java8 java.lang.zip.CRC32 methods become intrinsic, moreover > > new > > >> > > > "update" method, which use ByteBuffer was introduced. Since we > > >> moved > > >> to > > >> > > > java8, perhaps we really can get performance boost by using > > >> standard > > >> > > > java.lang.zip.CRC32 instead of PureJavaCrc32. > > >> > > > > > >> > > > About compatibility: looks like PureJavaCrc32 implements the > same > > >> > > algorithm > > >> > > > as java.lang.zip.CRC32. These two implementations uses the same > > >> > > polynomial > > >> > > > and the same initial value. The only difference is final xor > mask > > >> > > > (0x for java.lang.zip.CRC32). So, we can easily convert > > >> from > > >> > > > PureJavaCrc32 > > >> > > > to standard CRC32 and vice versa, using this expression: crc32 > ^= > > >> > > > 0x > > >> > > > > > >> > > > > > >> > > > 2018-08-14 0:19 GMT+03:00 Eduard Shangareev < > > >> > eduard.shangar...@gmail.com > > >> > > >: > > >> > > > > > >> > > > > Evgeniy, > > >> > > > > > > >> > > > > Could you share benchmark code? And please share what version > of > > >> JVM > > >> > > > > you have used. > > >> > > > > > > >> > > > > On Mon, Aug 13, 2018 at 10:44 PM Zhenya > > >> < arzamas...@mail.ru.invalid > > >> > > > >> > > > > wrote: > > >> > > > > > > >> > > > > > I think it would break backward compatibility, as Nikolay > > >> mentioned > > >> > > > above > > >> > > > > > we would take exception here: > > >> > > > > > > > >> > > > > > [1] > > >> > > > > > > > >> > > > > > https://github.com/apache/ignite/blob/master/modules/ > > >> > > > > core/src/main/java/org/apache/ignite/internal/processors/ > > >> > > > > cache/persistence/file/FilePageStore.java#L372 > > >> > > > > > > > >> > > > > > thats why i question for community thoughts here. > > >> > > > > > > > >> > > > > > > Hi Evgeniy, > > >> > > > > > > > > >> > > > > > > would you like to submit a patch with CRC32 implementation > > >> > change? > > >> > > > > > > > > >> > > > > > > Sincerely, > > >> > > > > > > Dmitriy Pavlov > > >> > > > > > > > > >> > > > > > > пн, 13 авг. 2018 г. в 22:08, Евгений Станиловский > > >> > > > > > > < arzamas...@mail.ru.invalid >: > > >> > > > > > > > > >> > > > > > >> Hi, igniters, i wrote a simple bench, looks like > > >> PureJavaCrc32 > > >> > has > > >> > > > > > >> performance problems in compatible with zip.CRC32. > > >> > > > > > >> > > >> > > > > > >> Benchmark Mode Cnt Score Error Units > > >> > > > > > >> BenchmarkCRC.Crc32 avgt 5 1088914.540 ± 368851.822 ns/op > > >> > > > > > >> BenchmarkCRC.pureJavaCrc32 avgt 5 6619408.049 ± > 3746712.210 > > >> > ns/op > > >> > > > > > >> > > >> > > > > > >> thoughts? > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > -- > > Евгений Станиловский > > >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Hi Evgeniy, Thank you. I see that the ticket is unassigned. Would you like to contribute PR to be macro-benchmarked with Ignite? Sincerely, Dmitriy Pavlov вт, 14 авг. 2018 г. в 20:57, Евгений Станиловский : > I fill the ticket, bench code attached there. > https://issues.apache.org/jira/browse/IGNITE-9272 > Thanks! > > > >Has anyone else run the benchmark and reproduced the performance > >difference? > > > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov < dpavlov@gmail.com > > >wrote: > > > >> It depends. > >> > >> CRC is a CPU-intensive operation, while WAL logging and page store write > >> are mostly about IO speed. > >> > >> In the same time, it can make the huge impact on machines with fast IO > >> and > >> slow CPU. So if we can apply change proposed by Evgeniy and Alexey it > >> could > >> benefit performance because we save CPU. Later we can use it's power in > a > >> more efficient manner (e.g. with compression). > >> > >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov < yzhda...@apache.org >: > >> > >> > Guys, what time in % does crc calculation take in WAL logging process? > >> > > >> > --Yakov > >> > > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov < dpavlov@gmail.com >: > >> > > >> > > Hi Alex, thank you for this idea. > >> > > > >> > > Evgeniy, Alex, would you like to submit the patch with bypassing > >> > > implementation differences to keep compatibility? > >> > > > >> > > Sincerely, > >> > > Dmitriy Pavlov > >> > > > >> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov < > plehanov.a...@gmail.com >: > >> > > > >> > > > Hello, Igniters! > >> > > > > >> > > > In java8 java.lang.zip.CRC32 methods become intrinsic, moreover > new > >> > > > "update" method, which use ByteBuffer was introduced. Since we > >> moved > >> to > >> > > > java8, perhaps we really can get performance boost by using > >> standard > >> > > > java.lang.zip.CRC32 instead of PureJavaCrc32. > >> > > > > >> > > > About compatibility: looks like PureJavaCrc32 implements the same > >> > > algorithm > >> > > > as java.lang.zip.CRC32. These two implementations uses the same > >> > > polynomial > >> > > > and the same initial value. The only difference is final xor mask > >> > > > (0x for java.lang.zip.CRC32). So, we can easily convert > >> from > >> > > > PureJavaCrc32 > >> > > > to standard CRC32 and vice versa, using this expression: crc32 ^= > >> > > > 0x > >> > > > > >> > > > > >> > > > 2018-08-14 0:19 GMT+03:00 Eduard Shangareev < > >> > eduard.shangar...@gmail.com > >> > > >: > >> > > > > >> > > > > Evgeniy, > >> > > > > > >> > > > > Could you share benchmark code? And please share what version of > >> JVM > >> > > > > you have used. > >> > > > > > >> > > > > On Mon, Aug 13, 2018 at 10:44 PM Zhenya > >> < arzamas...@mail.ru.invalid > >> > > >> > > > > wrote: > >> > > > > > >> > > > > > I think it would break backward compatibility, as Nikolay > >> mentioned > >> > > > above > >> > > > > > we would take exception here: > >> > > > > > > >> > > > > > [1] > >> > > > > > > >> > > > > > https://github.com/apache/ignite/blob/master/modules/ > >> > > > > core/src/main/java/org/apache/ignite/internal/processors/ > >> > > > > cache/persistence/file/FilePageStore.java#L372 > >> > > > > > > >> > > > > > thats why i question for community thoughts here. > >> > > > > > > >> > > > > > > Hi Evgeniy, > >> > > > > > > > >> > > > > > > would you like to submit a patch with CRC32 implementation > >> > change? > >> > > > > > > > >> > > > > > > Sincerely, > >> > > > > > > Dmitriy Pavlov > >> > > > > > > > >> > > > > > > пн, 13 авг. 2018 г. в 22:08, Евгений Станиловский > >> > > > > > > < arzamas...@mail.ru.invalid >: > >> > > > > > > > >> > > > > > >> Hi, igniters, i wrote a simple bench, looks like > >> PureJavaCrc32 > >> > has > >> > > > > > >> performance problems in compatible with zip.CRC32. > >> > > > > > >> > >> > > > > > >> Benchmark Mode Cnt Score Error Units > >> > > > > > >> BenchmarkCRC.Crc32 avgt 5 1088914.540 ± 368851.822 ns/op > >> > > > > > >> BenchmarkCRC.pureJavaCrc32 avgt 5 6619408.049 ± 3746712.210 > >> > ns/op > >> > > > > > >> > >> > > > > > >> thoughts? > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > -- > Евгений Станиловский >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
I fill the ticket, bench code attached there. https://issues.apache.org/jira/browse/IGNITE-9272 Thanks! >Has anyone else run the benchmark and reproduced the performance >difference? > >On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov < dpavlov@gmail.com > >wrote: > >> It depends. >> >> CRC is a CPU-intensive operation, while WAL logging and page store write >> are mostly about IO speed. >> >> In the same time, it can make the huge impact on machines with fast IO >> and >> slow CPU. So if we can apply change proposed by Evgeniy and Alexey it >> could >> benefit performance because we save CPU. Later we can use it's power in a >> more efficient manner (e.g. with compression). >> >> вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov < yzhda...@apache.org >: >> >> > Guys, what time in % does crc calculation take in WAL logging process? >> > >> > --Yakov >> > >> > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov < dpavlov@gmail.com >: >> > >> > > Hi Alex, thank you for this idea. >> > > >> > > Evgeniy, Alex, would you like to submit the patch with bypassing >> > > implementation differences to keep compatibility? >> > > >> > > Sincerely, >> > > Dmitriy Pavlov >> > > >> > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov < plehanov.a...@gmail.com >: >> > > >> > > > Hello, Igniters! >> > > > >> > > > In java8 java.lang.zip.CRC32 methods become intrinsic, moreover new >> > > > "update" method, which use ByteBuffer was introduced. Since we >> moved >> to >> > > > java8, perhaps we really can get performance boost by using >> standard >> > > > java.lang.zip.CRC32 instead of PureJavaCrc32. >> > > > >> > > > About compatibility: looks like PureJavaCrc32 implements the same >> > > algorithm >> > > > as java.lang.zip.CRC32. These two implementations uses the same >> > > polynomial >> > > > and the same initial value. The only difference is final xor mask >> > > > (0x for java.lang.zip.CRC32). So, we can easily convert >> from >> > > > PureJavaCrc32 >> > > > to standard CRC32 and vice versa, using this expression: crc32 ^= >> > > > 0x >> > > > >> > > > >> > > > 2018-08-14 0:19 GMT+03:00 Eduard Shangareev < >> > eduard.shangar...@gmail.com >> > > >: >> > > > >> > > > > Evgeniy, >> > > > > >> > > > > Could you share benchmark code? And please share what version of >> JVM >> > > > > you have used. >> > > > > >> > > > > On Mon, Aug 13, 2018 at 10:44 PM Zhenya >> < arzamas...@mail.ru.invalid >> > >> > > > > wrote: >> > > > > >> > > > > > I think it would break backward compatibility, as Nikolay >> mentioned >> > > > above >> > > > > > we would take exception here: >> > > > > > >> > > > > > [1] >> > > > > > >> > > > > > https://github.com/apache/ignite/blob/master/modules/ >> > > > > core/src/main/java/org/apache/ignite/internal/processors/ >> > > > > cache/persistence/file/FilePageStore.java#L372 >> > > > > > >> > > > > > thats why i question for community thoughts here. >> > > > > > >> > > > > > > Hi Evgeniy, >> > > > > > > >> > > > > > > would you like to submit a patch with CRC32 implementation >> > change? >> > > > > > > >> > > > > > > Sincerely, >> > > > > > > Dmitriy Pavlov >> > > > > > > >> > > > > > > пн, 13 авг. 2018 г. в 22:08, Евгений Станиловский >> > > > > > > < arzamas...@mail.ru.invalid >: >> > > > > > > >> > > > > > >> Hi, igniters, i wrote a simple bench, looks like >> PureJavaCrc32 >> > has >> > > > > > >> performance problems in compatible with zip.CRC32. >> > > > > > >> >> > > > > > >> Benchmark Mode Cnt Score Error Units >> > > > > > >> BenchmarkCRC.Crc32 avgt 5 1088914.540 ± 368851.822 ns/op >> > > > > > >> BenchmarkCRC.pureJavaCrc32 avgt 5 6619408.049 ± 3746712.210 >> > ns/op >> > > > > > >> >> > > > > > >> thoughts? >> > > > > > >> > > > > >> > > > >> > > >> > -- Евгений Станиловский
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Has anyone else run the benchmark and reproduced the performance difference? On Tue, Aug 14, 2018 at 8:16 AM, Dmitriy Pavlov wrote: > It depends. > > CRC is a CPU-intensive operation, while WAL logging and page store write > are mostly about IO speed. > > In the same time, it can make the huge impact on machines with fast IO and > slow CPU. So if we can apply change proposed by Evgeniy and Alexey it could > benefit performance because we save CPU. Later we can use it's power in a > more efficient manner (e.g. with compression). > > вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov : > > > Guys, what time in % does crc calculation take in WAL logging process? > > > > --Yakov > > > > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov : > > > > > Hi Alex, thank you for this idea. > > > > > > Evgeniy, Alex, would you like to submit the patch with bypassing > > > implementation differences to keep compatibility? > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov : > > > > > > > Hello, Igniters! > > > > > > > > In java8 java.lang.zip.CRC32 methods become intrinsic, moreover new > > > > "update" method, which use ByteBuffer was introduced. Since we moved > to > > > > java8, perhaps we really can get performance boost by using standard > > > > java.lang.zip.CRC32 instead of PureJavaCrc32. > > > > > > > > About compatibility: looks like PureJavaCrc32 implements the same > > > algorithm > > > > as java.lang.zip.CRC32. These two implementations uses the same > > > polynomial > > > > and the same initial value. The only difference is final xor mask > > > > (0x for java.lang.zip.CRC32). So, we can easily convert from > > > > PureJavaCrc32 > > > > to standard CRC32 and vice versa, using this expression: crc32 ^= > > > > 0x > > > > > > > > > > > > 2018-08-14 0:19 GMT+03:00 Eduard Shangareev < > > eduard.shangar...@gmail.com > > > >: > > > > > > > > > Evgeniy, > > > > > > > > > > Could you share benchmark code? And please share what version of > JVM > > > > > you have used. > > > > > > > > > > On Mon, Aug 13, 2018 at 10:44 PM Zhenya > > > > > > wrote: > > > > > > > > > > > I think it would break backward compatibility, as Nikolay > mentioned > > > > above > > > > > > we would take exception here: > > > > > > > > > > > > [1] > > > > > > > > > > > > https://github.com/apache/ignite/blob/master/modules/ > > > > > core/src/main/java/org/apache/ignite/internal/processors/ > > > > > cache/persistence/file/FilePageStore.java#L372 > > > > > > > > > > > > thats why i question for community thoughts here. > > > > > > > > > > > > > Hi Evgeniy, > > > > > > > > > > > > > > would you like to submit a patch with CRC32 implementation > > change? > > > > > > > > > > > > > > Sincerely, > > > > > > > Dmitriy Pavlov > > > > > > > > > > > > > > пн, 13 авг. 2018 г. в 22:08, Евгений Станиловский > > > > > > > : > > > > > > > > > > > > > >> Hi, igniters, i wrote a simple bench, looks like PureJavaCrc32 > > has > > > > > > >> performance problems in compatible with zip.CRC32. > > > > > > >> > > > > > > >> Benchmark Mode Cnt Score Error Units > > > > > > >> BenchmarkCRC.Crc32 avgt 5 1088914.540 ± 368851.822 ns/op > > > > > > >> BenchmarkCRC.pureJavaCrc32 avgt 5 6619408.049 ± 3746712.210 > > ns/op > > > > > > >> > > > > > > >> thoughts? > > > > > > > > > > > > > > > > > > > > >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
It depends. CRC is a CPU-intensive operation, while WAL logging and page store write are mostly about IO speed. In the same time, it can make the huge impact on machines with fast IO and slow CPU. So if we can apply change proposed by Evgeniy and Alexey it could benefit performance because we save CPU. Later we can use it's power in a more efficient manner (e.g. with compression). вт, 14 авг. 2018 г. в 14:03, Yakov Zhdanov : > Guys, what time in % does crc calculation take in WAL logging process? > > --Yakov > > 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov : > > > Hi Alex, thank you for this idea. > > > > Evgeniy, Alex, would you like to submit the patch with bypassing > > implementation differences to keep compatibility? > > > > Sincerely, > > Dmitriy Pavlov > > > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov : > > > > > Hello, Igniters! > > > > > > In java8 java.lang.zip.CRC32 methods become intrinsic, moreover new > > > "update" method, which use ByteBuffer was introduced. Since we moved to > > > java8, perhaps we really can get performance boost by using standard > > > java.lang.zip.CRC32 instead of PureJavaCrc32. > > > > > > About compatibility: looks like PureJavaCrc32 implements the same > > algorithm > > > as java.lang.zip.CRC32. These two implementations uses the same > > polynomial > > > and the same initial value. The only difference is final xor mask > > > (0x for java.lang.zip.CRC32). So, we can easily convert from > > > PureJavaCrc32 > > > to standard CRC32 and vice versa, using this expression: crc32 ^= > > > 0x > > > > > > > > > 2018-08-14 0:19 GMT+03:00 Eduard Shangareev < > eduard.shangar...@gmail.com > > >: > > > > > > > Evgeniy, > > > > > > > > Could you share benchmark code? And please share what version of JVM > > > > you have used. > > > > > > > > On Mon, Aug 13, 2018 at 10:44 PM Zhenya > > > > wrote: > > > > > > > > > I think it would break backward compatibility, as Nikolay mentioned > > > above > > > > > we would take exception here: > > > > > > > > > > [1] > > > > > > > > > > https://github.com/apache/ignite/blob/master/modules/ > > > > core/src/main/java/org/apache/ignite/internal/processors/ > > > > cache/persistence/file/FilePageStore.java#L372 > > > > > > > > > > thats why i question for community thoughts here. > > > > > > > > > > > Hi Evgeniy, > > > > > > > > > > > > would you like to submit a patch with CRC32 implementation > change? > > > > > > > > > > > > Sincerely, > > > > > > Dmitriy Pavlov > > > > > > > > > > > > пн, 13 авг. 2018 г. в 22:08, Евгений Станиловский > > > > > > : > > > > > > > > > > > >> Hi, igniters, i wrote a simple bench, looks like PureJavaCrc32 > has > > > > > >> performance problems in compatible with zip.CRC32. > > > > > >> > > > > > >> Benchmark Mode Cnt Score Error Units > > > > > >> BenchmarkCRC.Crc32 avgt 5 1088914.540 ± 368851.822 ns/op > > > > > >> BenchmarkCRC.pureJavaCrc32 avgt 5 6619408.049 ± 3746712.210 > ns/op > > > > > >> > > > > > >> thoughts? > > > > > > > > > > > > > > >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Guys, what time in % does crc calculation take in WAL logging process? --Yakov 2018-08-14 13:37 GMT+03:00 Dmitriy Pavlov : > Hi Alex, thank you for this idea. > > Evgeniy, Alex, would you like to submit the patch with bypassing > implementation differences to keep compatibility? > > Sincerely, > Dmitriy Pavlov > > вт, 14 авг. 2018 г. в 12:06, Alex Plehanov : > > > Hello, Igniters! > > > > In java8 java.lang.zip.CRC32 methods become intrinsic, moreover new > > "update" method, which use ByteBuffer was introduced. Since we moved to > > java8, perhaps we really can get performance boost by using standard > > java.lang.zip.CRC32 instead of PureJavaCrc32. > > > > About compatibility: looks like PureJavaCrc32 implements the same > algorithm > > as java.lang.zip.CRC32. These two implementations uses the same > polynomial > > and the same initial value. The only difference is final xor mask > > (0x for java.lang.zip.CRC32). So, we can easily convert from > > PureJavaCrc32 > > to standard CRC32 and vice versa, using this expression: crc32 ^= > > 0x > > > > > > 2018-08-14 0:19 GMT+03:00 Eduard Shangareev >: > > > > > Evgeniy, > > > > > > Could you share benchmark code? And please share what version of JVM > > > you have used. > > > > > > On Mon, Aug 13, 2018 at 10:44 PM Zhenya > > > wrote: > > > > > > > I think it would break backward compatibility, as Nikolay mentioned > > above > > > > we would take exception here: > > > > > > > > [1] > > > > > > > > https://github.com/apache/ignite/blob/master/modules/ > > > core/src/main/java/org/apache/ignite/internal/processors/ > > > cache/persistence/file/FilePageStore.java#L372 > > > > > > > > thats why i question for community thoughts here. > > > > > > > > > Hi Evgeniy, > > > > > > > > > > would you like to submit a patch with CRC32 implementation change? > > > > > > > > > > Sincerely, > > > > > Dmitriy Pavlov > > > > > > > > > > пн, 13 авг. 2018 г. в 22:08, Евгений Станиловский > > > > > : > > > > > > > > > >> Hi, igniters, i wrote a simple bench, looks like PureJavaCrc32 has > > > > >> performance problems in compatible with zip.CRC32. > > > > >> > > > > >> Benchmark Mode Cnt Score Error Units > > > > >> BenchmarkCRC.Crc32 avgt 5 1088914.540 ± 368851.822 ns/op > > > > >> BenchmarkCRC.pureJavaCrc32 avgt 5 6619408.049 ± 3746712.210 ns/op > > > > >> > > > > >> thoughts? > > > > > > > > > >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Hi Alex, thank you for this idea. Evgeniy, Alex, would you like to submit the patch with bypassing implementation differences to keep compatibility? Sincerely, Dmitriy Pavlov вт, 14 авг. 2018 г. в 12:06, Alex Plehanov : > Hello, Igniters! > > In java8 java.lang.zip.CRC32 methods become intrinsic, moreover new > "update" method, which use ByteBuffer was introduced. Since we moved to > java8, perhaps we really can get performance boost by using standard > java.lang.zip.CRC32 instead of PureJavaCrc32. > > About compatibility: looks like PureJavaCrc32 implements the same algorithm > as java.lang.zip.CRC32. These two implementations uses the same polynomial > and the same initial value. The only difference is final xor mask > (0x for java.lang.zip.CRC32). So, we can easily convert from > PureJavaCrc32 > to standard CRC32 and vice versa, using this expression: crc32 ^= > 0x > > > 2018-08-14 0:19 GMT+03:00 Eduard Shangareev : > > > Evgeniy, > > > > Could you share benchmark code? And please share what version of JVM > > you have used. > > > > On Mon, Aug 13, 2018 at 10:44 PM Zhenya > > wrote: > > > > > I think it would break backward compatibility, as Nikolay mentioned > above > > > we would take exception here: > > > > > > [1] > > > > > > https://github.com/apache/ignite/blob/master/modules/ > > core/src/main/java/org/apache/ignite/internal/processors/ > > cache/persistence/file/FilePageStore.java#L372 > > > > > > thats why i question for community thoughts here. > > > > > > > Hi Evgeniy, > > > > > > > > would you like to submit a patch with CRC32 implementation change? > > > > > > > > Sincerely, > > > > Dmitriy Pavlov > > > > > > > > пн, 13 авг. 2018 г. в 22:08, Евгений Станиловский > > > > : > > > > > > > >> Hi, igniters, i wrote a simple bench, looks like PureJavaCrc32 has > > > >> performance problems in compatible with zip.CRC32. > > > >> > > > >> Benchmark Mode Cnt Score Error Units > > > >> BenchmarkCRC.Crc32 avgt 5 1088914.540 ± 368851.822 ns/op > > > >> BenchmarkCRC.pureJavaCrc32 avgt 5 6619408.049 ± 3746712.210 ns/op > > > >> > > > >> thoughts? > > > > > >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Hello, Igniters! In java8 java.lang.zip.CRC32 methods become intrinsic, moreover new "update" method, which use ByteBuffer was introduced. Since we moved to java8, perhaps we really can get performance boost by using standard java.lang.zip.CRC32 instead of PureJavaCrc32. About compatibility: looks like PureJavaCrc32 implements the same algorithm as java.lang.zip.CRC32. These two implementations uses the same polynomial and the same initial value. The only difference is final xor mask (0x for java.lang.zip.CRC32). So, we can easily convert from PureJavaCrc32 to standard CRC32 and vice versa, using this expression: crc32 ^= 0x 2018-08-14 0:19 GMT+03:00 Eduard Shangareev : > Evgeniy, > > Could you share benchmark code? And please share what version of JVM > you have used. > > On Mon, Aug 13, 2018 at 10:44 PM Zhenya > wrote: > > > I think it would break backward compatibility, as Nikolay mentioned above > > we would take exception here: > > > > [1] > > > > https://github.com/apache/ignite/blob/master/modules/ > core/src/main/java/org/apache/ignite/internal/processors/ > cache/persistence/file/FilePageStore.java#L372 > > > > thats why i question for community thoughts here. > > > > > Hi Evgeniy, > > > > > > would you like to submit a patch with CRC32 implementation change? > > > > > > Sincerely, > > > Dmitriy Pavlov > > > > > > пн, 13 авг. 2018 г. в 22:08, Евгений Станиловский > > > : > > > > > >> Hi, igniters, i wrote a simple bench, looks like PureJavaCrc32 has > > >> performance problems in compatible with zip.CRC32. > > >> > > >> Benchmark Mode Cnt Score Error Units > > >> BenchmarkCRC.Crc32 avgt 5 1088914.540 ± 368851.822 ns/op > > >> BenchmarkCRC.pureJavaCrc32 avgt 5 6619408.049 ± 3746712.210 ns/op > > >> > > >> thoughts? > > >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Evgeniy, Could you share benchmark code? And please share what version of JVM you have used. On Mon, Aug 13, 2018 at 10:44 PM Zhenya wrote: > I think it would break backward compatibility, as Nikolay mentioned above > we would take exception here: > > [1] > > https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/file/FilePageStore.java#L372 > > thats why i question for community thoughts here. > > > Hi Evgeniy, > > > > would you like to submit a patch with CRC32 implementation change? > > > > Sincerely, > > Dmitriy Pavlov > > > > пн, 13 авг. 2018 г. в 22:08, Евгений Станиловский > > : > > > >> Hi, igniters, i wrote a simple bench, looks like PureJavaCrc32 has > >> performance problems in compatible with zip.CRC32. > >> > >> Benchmark Mode Cnt Score Error Units > >> BenchmarkCRC.Crc32 avgt 5 1088914.540 ± 368851.822 ns/op > >> BenchmarkCRC.pureJavaCrc32 avgt 5 6619408.049 ± 3746712.210 ns/op > >> > >> thoughts? >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
I think it would break backward compatibility, as Nikolay mentioned above we would take exception here: [1] https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/file/FilePageStore.java#L372 thats why i question for community thoughts here. Hi Evgeniy, would you like to submit a patch with CRC32 implementation change? Sincerely, Dmitriy Pavlov пн, 13 авг. 2018 г. в 22:08, Евгений Станиловский : Hi, igniters, i wrote a simple bench, looks like PureJavaCrc32 has performance problems in compatible with zip.CRC32. Benchmark Mode Cnt Score Error Units BenchmarkCRC.Crc32 avgt 5 1088914.540 ± 368851.822 ns/op BenchmarkCRC.pureJavaCrc32 avgt 5 6619408.049 ± 3746712.210 ns/op thoughts?
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Hi Evgeniy, would you like to submit a patch with CRC32 implementation change? Sincerely, Dmitriy Pavlov пн, 13 авг. 2018 г. в 22:08, Евгений Станиловский : > Hi, igniters, i wrote a simple bench, looks like PureJavaCrc32 has > performance problems in compatible with zip.CRC32. > > Benchmark Mode Cnt Score Error Units > BenchmarkCRC.Crc32 avgt 5 1088914.540 ± 368851.822 ns/op > BenchmarkCRC.pureJavaCrc32 avgt 5 6619408.049 ± 3746712.210 ns/op > > thoughts? >
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Hello, Denis. AFAIK CRC calculated: 1. For each page that are readed or written to the persistence store [1]. 2. For each read or write of WAL record [2]. [1] https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/file/FilePageStore.java#L372 [2] https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/serializer/RecordV1Serializer.java#L422 В Пн, 13/08/2018 в 12:10 -0700, Denis Magda пишет: > Hello Evgeniy, > > What is PureJavaCrc32 used for in Ignite? Is it on a critical path > somewhere? > > -- > Denis > > On Mon, Aug 13, 2018 at 12:08 PM Евгений Станиловский > wrote: > > > Hi, igniters, i wrote a simple bench, looks like PureJavaCrc32 has > > performance problems in compatible with zip.CRC32. > > > > Benchmark Mode Cnt Score Error Units > > BenchmarkCRC.Crc32 avgt 5 1088914.540 ± 368851.822 ns/op > > BenchmarkCRC.pureJavaCrc32 avgt 5 6619408.049 ± 3746712.210 ns/op > > > > thoughts? > > signature.asc Description: This is a digitally signed message part
Re: ignite PureJavaCrc32 vs java.util.zip.CRC32 bench.
Hello Evgeniy, What is PureJavaCrc32 used for in Ignite? Is it on a critical path somewhere? -- Denis On Mon, Aug 13, 2018 at 12:08 PM Евгений Станиловский wrote: > Hi, igniters, i wrote a simple bench, looks like PureJavaCrc32 has > performance problems in compatible with zip.CRC32. > > Benchmark Mode Cnt Score Error Units > BenchmarkCRC.Crc32 avgt 5 1088914.540 ± 368851.822 ns/op > BenchmarkCRC.pureJavaCrc32 avgt 5 6619408.049 ± 3746712.210 ns/op > > thoughts? >
