[ https://issues.apache.org/jira/browse/CASSANDRA-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175035#comment-13175035 ]
Vijay edited comment on CASSANDRA-3610 at 12/22/11 8:30 PM: ------------------------------------------------------------ Ooops pasted the wrong data.... the above data is without any Heap settings and on Open JDK.. Plz see the below :) /usr/java/latest/jre/bin/java -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8G -Xmx8G -Xmn2G -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -jar TestCRC32Performance.jar ||bytes||PureJava MB/sec||Native MB/sec||Random PureJava MB/sec||Native MB/sec|| | 1 |121.124 |11.866 | | 2 |161.981 |23.851 | | 4 |204.718 |45.486 | | 8 |297.229 |76.296 | | 16 |379.268 |117.326 | | 32 |440.153 |157.711 | | 64 |468.143 |193.304 || PureJava |0-64 |272.921 MB/sec|| Native |0-64 |145.289 MB/sec| | 128 |500.006 |219.657 || PureJava |0-128 |367.816 MB/sec|| Native |0-128 |186.861 MB/sec| | 256 |511.572 |234.052 || PureJava |0-256 |432.433 MB/sec|| Native |0-256 |214.047 MB/sec| | 512 |517.550 |242.634 || PureJava |0-512 |474.074 MB/sec|| Native |0-512 |231.047 MB/sec| | 1024 |516.994 |246.424 || PureJava |0-1024 |498.055 MB/sec|| Native |0-1024 |241.056 MB/sec| | 2048 |518.095 |248.529 || PureJava |0-2048 |509.960 MB/sec|| Native |0-2048 |245.683 MB/sec| | 4096 |522.002 |249.755 || PureJava |0-4096 |518.226 MB/sec|| Native |0-4096 |248.062 MB/sec| | 8192 |522.795 |250.316 || PureJava |0-8192 |520.326 MB/sec|| Native |0-8192 |249.519 MB/sec| | 16384 |522.521 |250.484 || PureJava |0-16384 |522.480 MB/sec|| Native |0-16384 |250.002 MB/sec| | 32768 |521.098 |250.604 || PureJava |0-32768 |520.349 MB/sec|| Native |0-32768 |250.494 MB/sec| | 65536 |520.973 |250.837 || PureJava |0-65536 |520.392 MB/sec|| Native |0-65536 |249.063 MB/sec| | 131072 |510.129 |248.949 || PureJava |0-131072 |516.246 MB/sec|| Native |0-131072 |249.535 MB/sec| | 262144 |513.534 |249.506 || PureJava |0-262144 |514.407 MB/sec|| Native |0-262144 |250.617 MB/sec| | 524288 |519.554 |250.696 || PureJava |0-524288 |520.402 MB/sec|| Native |0-524288 |251.048 MB/sec| | 1048576 |519.559 |250.557 || PureJava |0-1048576 |520.403 MB/sec|| Native |0-1048576 |250.734 MB/sec| | 2097152 |519.259 |250.456 || PureJava |0-2097152 |519.337 MB/sec|| Native |0-2097152 |250.299 MB/sec| | 4194304 |518.649 |250.470 || PureJava |0-4194304 |518.495 MB/sec|| Native |0-4194304 |250.523 MB/sec| | 8388608 |501.986 |248.044 || PureJava |0-8388608 |509.521 MB/sec|| Native |0-8388608 |248.626 MB/sec| | 16777216 |508.201 |247.587 || PureJava |0-16777216 |505.258 MB/sec|| Native |0-16777216 |249.558 MB/sec| [vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$ /usr/java/latest/jre/bin/java -version java version "1.6.0_27" Java(TM) SE Runtime Environment (build 1.6.0_27-b07) Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode) [vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$ was (Author: vijay2...@yahoo.com): Ooops pasted the wrong data.... the above data is without any Heap settings hence GC becomes a bottleneck... Plz see the below :) /usr/java/latest/jre/bin/java -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8G -Xmx8G -Xmn2G -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -jar TestCRC32Performance.jar ||bytes||PureJava MB/sec||Native MB/sec||Random PureJava MB/sec||Native MB/sec|| | 1 |121.124 |11.866 | | 2 |161.981 |23.851 | | 4 |204.718 |45.486 | | 8 |297.229 |76.296 | | 16 |379.268 |117.326 | | 32 |440.153 |157.711 | | 64 |468.143 |193.304 || PureJava |0-64 |272.921 MB/sec|| Native |0-64 |145.289 MB/sec| | 128 |500.006 |219.657 || PureJava |0-128 |367.816 MB/sec|| Native |0-128 |186.861 MB/sec| | 256 |511.572 |234.052 || PureJava |0-256 |432.433 MB/sec|| Native |0-256 |214.047 MB/sec| | 512 |517.550 |242.634 || PureJava |0-512 |474.074 MB/sec|| Native |0-512 |231.047 MB/sec| | 1024 |516.994 |246.424 || PureJava |0-1024 |498.055 MB/sec|| Native |0-1024 |241.056 MB/sec| | 2048 |518.095 |248.529 || PureJava |0-2048 |509.960 MB/sec|| Native |0-2048 |245.683 MB/sec| | 4096 |522.002 |249.755 || PureJava |0-4096 |518.226 MB/sec|| Native |0-4096 |248.062 MB/sec| | 8192 |522.795 |250.316 || PureJava |0-8192 |520.326 MB/sec|| Native |0-8192 |249.519 MB/sec| | 16384 |522.521 |250.484 || PureJava |0-16384 |522.480 MB/sec|| Native |0-16384 |250.002 MB/sec| | 32768 |521.098 |250.604 || PureJava |0-32768 |520.349 MB/sec|| Native |0-32768 |250.494 MB/sec| | 65536 |520.973 |250.837 || PureJava |0-65536 |520.392 MB/sec|| Native |0-65536 |249.063 MB/sec| | 131072 |510.129 |248.949 || PureJava |0-131072 |516.246 MB/sec|| Native |0-131072 |249.535 MB/sec| | 262144 |513.534 |249.506 || PureJava |0-262144 |514.407 MB/sec|| Native |0-262144 |250.617 MB/sec| | 524288 |519.554 |250.696 || PureJava |0-524288 |520.402 MB/sec|| Native |0-524288 |251.048 MB/sec| | 1048576 |519.559 |250.557 || PureJava |0-1048576 |520.403 MB/sec|| Native |0-1048576 |250.734 MB/sec| | 2097152 |519.259 |250.456 || PureJava |0-2097152 |519.337 MB/sec|| Native |0-2097152 |250.299 MB/sec| | 4194304 |518.649 |250.470 || PureJava |0-4194304 |518.495 MB/sec|| Native |0-4194304 |250.523 MB/sec| | 8388608 |501.986 |248.044 || PureJava |0-8388608 |509.521 MB/sec|| Native |0-8388608 |248.626 MB/sec| | 16777216 |508.201 |247.587 || PureJava |0-16777216 |505.258 MB/sec|| Native |0-16777216 |249.558 MB/sec| [vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$ /usr/java/latest/jre/bin/java -version java version "1.6.0_27" Java(TM) SE Runtime Environment (build 1.6.0_27-b07) Java HotSpot(TM) 64-Bit Server VM (build 20.2-b06, mixed mode) [vijay_tcasstest@vijay_tcass--1a-i-aad629c8 ~]$ > Checksum improvement for CompressedRandomAccessReader > ----------------------------------------------------- > > Key: CASSANDRA-3610 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3610 > Project: Cassandra > Issue Type: Improvement > Components: Core > Affects Versions: 1.1 > Environment: JVM > Reporter: Vijay > Assignee: Vijay > Priority: Minor > Fix For: 1.1 > > Attachments: 0001-use-pure-java-CRC32-v2.patch, > 0001-use-pure-java-CRC32.patch > > > When compression is on, Currently we see checksum taking up about 40% of the > CPU more than snappy library. > Looks like hadoop solved it by implementing their own checksum, we can either > use it or implement something like that. > http://images.slidesharecdn.com/1toddlipconyanpeichen-cloudera-hadoopandperformance-final-111110132228-phpapp01-slide-15-768.jpg?1321043717 > in our test env it provided 50% improvement over native implementation which > uses jni to call the OS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira