[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14338670#comment-14338670 ] Alexandre Dutra commented on CASSANDRA-8614: Thank you for the explanation and the pointer to CASSANDRA-8836. I was indeed trying to build with Intellij, following the instructions [here|http://wiki.apache.org/cassandra/RunningCassandraInIDEA]. In the meantime I'll use a JDK8. > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg >Assignee: Ariel Weisberg > Labels: performance > Attachments: 8614.patch, CRC32.class, Sample.java > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[
https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337943#comment-14337943
]
Ariel Weisberg commented on CASSANDRA-8614:
---
How are you building? Are you using build.xml or an IDE? build.xml includes a
dummy implementation of java.util.zip.CRC32 that has the missing method so it
compiles. Until CASSANDRA-8836 is done if you are using an IDE you will need to
have the compiler add -Xbootclasspath/p:${build.src.jdkoverride} where
build.src.jdkoverride is the path to src/jdkoverride.
> Select optimal CRC32 implementation at runtime
> --
>
> Key: CASSANDRA-8614
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8614
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
> Labels: performance
> Attachments: 8614.patch, CRC32.class, Sample.java
>
>
> JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec
> per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if
> I recall and it has a lookup table that evicts random cache lines every time
> it runs.
> In order to capture the benefit of that when it is available we can select a
> CRC32 implementation at startup in a static block.
> If JDK 8 is not what is running we can fall back to the existing
> PureJavaCRC32 implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[
https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337944#comment-14337944
]
Ariel Weisberg commented on CASSANDRA-8614:
---
How are you building? Are you using build.xml or an IDE? build.xml includes a
dummy implementation of java.util.zip.CRC32 that has the missing method so it
compiles. Until CASSANDRA-8836 is done if you are using an IDE you will need to
have the compiler add -Xbootclasspath/p:${build.src.jdkoverride} where
build.src.jdkoverride is the path to src/jdkoverride.
> Select optimal CRC32 implementation at runtime
> --
>
> Key: CASSANDRA-8614
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8614
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
> Labels: performance
> Attachments: 8614.patch, CRC32.class, Sample.java
>
>
> JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec
> per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if
> I recall and it has a lookup table that evicts random cache lines every time
> it runs.
> In order to capture the benefit of that when it is available we can select a
> CRC32 implementation at startup in a static block.
> If JDK 8 is not what is running we can fall back to the existing
> PureJavaCRC32 implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[
https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337918#comment-14337918
]
Alexandre Dutra commented on CASSANDRA-8614:
If I'm not mistaken it should still be possible to build/run Cassandra 3.0 with
JDK7.
However in {{CRC32Factory}} there is a call to
{{java.util.zip.CRC32#update(java.nio.ByteBuffer)}}.
This method has been introduced in Java 8. As a consequence It's currently
impossible to build the trunk with JDK7.
> Select optimal CRC32 implementation at runtime
> --
>
> Key: CASSANDRA-8614
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8614
> Project: Cassandra
> Issue Type: Improvement
> Components: Core
>Reporter: Ariel Weisberg
>Assignee: Ariel Weisberg
> Labels: performance
> Attachments: 8614.patch, CRC32.class, Sample.java
>
>
> JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec
> per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if
> I recall and it has a lookup table that evicts random cache lines every time
> it runs.
> In order to capture the benefit of that when it is available we can select a
> CRC32 implementation at startup in a static block.
> If JDK 8 is not what is running we can fall back to the existing
> PureJavaCRC32 implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14316419#comment-14316419 ] Benedict commented on CASSANDRA-8614: - Looks like you may have missed your JDK source file override to permit compilation with JDK8... > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg >Assignee: Ariel Weisberg > Labels: performance > Attachments: 8614.patch, Sample.java > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312830#comment-14312830 ] Benedict commented on CASSANDRA-8614: - I meant I was planning to do all of my commits post then, not that I would commit this to the 2.1 branch. > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg >Assignee: Ariel Weisberg > Labels: performance > Attachments: 8614.patch, Sample.java > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312825#comment-14312825 ] Jonathan Ellis commented on CASSANDRA-8614: --- This does look like something we should save for 3.0 to me. > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg >Assignee: Ariel Weisberg > Labels: performance > Attachments: 8614.patch, Sample.java > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312812#comment-14312812 ] Sylvain Lebresne commented on CASSANDRA-8614: - bq. i'll commit it with a raft of other upcoming commits I'll need to merge, probably post 2.1.3 to simplify that release process Any reason why this needs to go in the 2.1 branch? > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg >Assignee: Ariel Weisberg > Labels: performance > Attachments: 8614.patch, Sample.java > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312650#comment-14312650 ] Benedict commented on CASSANDRA-8614: - +1 with one nit: we tend (rightly or wrongly) to avoid the use of "final" as a local variable modifier except where necessary i'll commit it with a raft of other upcoming commits I'll need to merge, probably post 2.1.3 to simplify that release process > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg >Assignee: Ariel Weisberg > Labels: performance > Attachments: 8614.patch, Sample.java > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311593#comment-14311593 ] Ariel Weisberg commented on CASSANDRA-8614: --- There is some wonkiness with the .class file for the dummy CRC32 being necessary. You can compile the .java under src/jdkoverride with jdk7 to compile and test and then check it in. git diff doesn't include the binary .class file. In terms of picking the best CRC32 implementation the ticket and patch are ready for review. We might as well get the benefit of a a better CRC32 implementation where available in the situations where we are already choosing to use a CRC32. CASSANDRA-8684 covers widening the usage of CRC32 (or not) and can be addressed separately. The performance graphs from 8684 help advocate for selecting the intrinsic implementation when available. > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg >Assignee: Ariel Weisberg > Labels: performance > Attachments: 8614.patch, Sample.java > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14311331#comment-14311331 ] Benedict commented on CASSANDRA-8614: - I'm not sure what the status is of this patch following the prior comments: is it ready for review? does it make sense? > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg >Assignee: Ariel Weisberg > Labels: performance > Attachments: 8614.patch, Sample.java > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14292129#comment-14292129 ] Ariel Weisberg commented on CASSANDRA-8614: --- I think I may be wrong. There is some wonkiness with the build picking up JDK 8 some of the time even though the first JDK in my path is 7. I think you do need the .class file for the fake CRC32 and not just the source. > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg >Assignee: Ariel Weisberg > Labels: performance > Attachments: 8614.patch, Sample.java > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14280634#comment-14280634 ] Benedict commented on CASSANDRA-8614: - Not sure what I keep pressing to accidentally adopt other people's tickets > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg >Assignee: Ariel Weisberg > Labels: performance > Attachments: 8614.patch, Sample.java > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277347#comment-14277347 ] Robert Stupp commented on CASSANDRA-8614: - The intrinsic is definitely not present in Java 7. zlib used in JDK8 is quite old (2010) - and support for Intel-CPU-CRC has been added to [zlib mid 2014 |https://github.com/jtkukunas/zlib/blob/master/crc_folding.c]. Maybe a newer zlib via ld_preload? > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg >Assignee: Benedict > Labels: performance > Attachments: 8614.patch, Sample.java > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277276#comment-14277276 ] Ariel Weisberg commented on CASSANDRA-8614: --- I am trying to disable the instruction so I can get the fallback and measure. I read in some places that Adler32 was also supposed to get accelerated, but I only see stuff for CRC32. Doesn't mean it doesn't do it on the sly, but we might get the speedup even using Adler32. The JVM is accepting -XX:-UseCLMUL -XX:-UseCRC32Intrinsic, but it doesn't seem to work. I keep getting 13 gigabytes/second. Something doesn't add up. The speed up going from 128-bytes to 1 megabyte is also suspicious. I also got 13 gigabytes/sec out of JDK 7. ??!!!?? I'm definitely not measuring what I think I'm measuring. > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg >Assignee: Benedict > Labels: performance > Attachments: 8614.patch, Sample.java > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277185#comment-14277185 ] Ariel Weisberg commented on CASSANDRA-8614: --- It seems like the fallback in JDK 8 is to a slicing table based implementation, but it isn't the slicing by 8 algorithm. It was considered not enough better. http://mail.openjdk.java.net/pipermail/aarch64-port-dev/2014-May/001113.html http://people.linaro.org/~edward.nevill/crc32/crc32.patch I wonder how safe it is to assume that hardware running C* will have it. It was introduced in 2010 http://en.wikipedia.org/wiki/CLMUL_instruction_set There is also the question of whether it is enabled in a virtualized environment. I will benchmark the fallback option and see how it compares to PureJavaCRC32. > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg >Assignee: Benedict > Labels: performance > Attachments: 8614.patch, Sample.java > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277162#comment-14277162 ] Ariel Weisberg commented on CASSANDRA-8614: --- If you switch outright how do you read tables that were written using the old checksum? I assumed the table metadata already contained the checksum type. > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg > Labels: performance > Attachments: 8614.patch, Sample.java > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277160#comment-14277160 ] Benedict commented on CASSANDRA-8614: - It would be great to run this on a non-modern system, or an AMD box, see what the differential is, > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg > Labels: performance > Attachments: 8614.patch, Sample.java > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277139#comment-14277139 ] Aleksey Yeschenko commented on CASSANDRA-8614: -- 3.0 will not require JDK8, but 3.1 will probably should. As things stand right now, the checksum algo depends strictly on the version in the descriptor, and those are static. We could probably encode the algo used in metadata, easily, so the current scheme is not a big deal of a limitation in itself, but I'd slightly prefer switching in 3.1, unconditionally. > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg > Labels: performance > Attachments: 8614.patch > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277131#comment-14277131 ] Benedict commented on CASSANDRA-8614: - Perhaps. But people might run C* with different VMs, and so result in a bad choice (i.e. might try JDK8, find it unstable, and rollback - but now have all their data with more expensive checksums). We should probably also detect the hardware to decide if running with native CRC is a good thing, versus Adler. Or perhaps perform some benchmarks on other hardware. It's likely still slower on hardware without the native instructions. > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg > Labels: performance > Attachments: 8614.patch > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277120#comment-14277120 ] Ariel Weisberg commented on CASSANDRA-8614: --- What release would we want this to go in? 3.0? Will 3.0 require JDK 8? Does it make sense to drive the choice for creating checksummed data based on what version of the JDK is running so people who use Java 8 at runtime can get the benefit even if Java 8 is not required? > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg > Labels: performance > Attachments: 8614.patch > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277117#comment-14277117 ] Benedict commented on CASSANDRA-8614: - bq. we -might want- probably should > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg > Labels: performance > Attachments: 8614.patch > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277115#comment-14277115 ] Aleksey Yeschenko commented on CASSANDRA-8614: -- Once we require JDK8, we might want to switch our sstable checksums back to CRC (from Adler), if intrinsic CRC32 happens to be faster (which it should). > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg > Labels: performance > Attachments: 8614.patch > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277113#comment-14277113 ] Benedict commented on CASSANDRA-8614: - I don't really mind. Both can be error prone - CompletableFuture could be included as a library in an older VM. Both are _unlikely_ to cause problems, and the worst outcome is a bad decision about performance, not a catastrophe. > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg > Labels: performance > Attachments: 8614.patch > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14276400#comment-14276400 ] Ariel Weisberg commented on CASSANDRA-8614: --- I checked Netty's PlatformDependent for how they did it and copied that. The question would be why they did it that way? Maybe to avoid error prone string parsing? I m fine with switching to parsing the java.version string. > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg > Labels: performance > Attachments: 8614.patch > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8614) Select optimal CRC32 implementation at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-8614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14276261#comment-14276261 ] Benedict commented on CASSANDRA-8614: - Why not check "java.version"? > Select optimal CRC32 implementation at runtime > -- > > Key: CASSANDRA-8614 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8614 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Ariel Weisberg > Labels: performance > Attachments: 8614.patch > > > JDK 8 has support for an intrinsic for CRC32 that runs at 12-13 gigabytes/sec > per core in my quick and dirty test. PureJavaCRC32 is < 800 megabytes/sec if > I recall and it has a lookup table that evicts random cache lines every time > it runs. > In order to capture the benefit of that when it is available we can select a > CRC32 implementation at startup in a static block. > If JDK 8 is not what is running we can fall back to the existing > PureJavaCRC32 implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
