[jira] [Updated] (CASSANDRA-3623) use MMapedBuffer in CompressedSegmentedFile.getSegment
[ https://issues.apache.org/jira/browse/CASSANDRA-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-3623: -- Reviewer: yukim (was: xedin) I'll ask Yuki to take a look. > use MMapedBuffer in CompressedSegmentedFile.getSegment > -- > > Key: CASSANDRA-3623 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3623 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 1.1 >Reporter: Vijay >Assignee: Vijay > Labels: compression > Fix For: 1.1 > > Attachments: 0001-MMaped-Compression-segmented-file-v2.patch, > 0001-MMaped-Compression-segmented-file-v3.patch, > 0001-MMaped-Compression-segmented-file.patch, > 0002-tests-for-MMaped-Compression-segmented-file-v2.patch, > 0002-tests-for-MMaped-Compression-segmented-file-v3.patch, CRC+MMapIO.xlsx, > MMappedIO-Performance.docx > > > CompressedSegmentedFile.getSegment seem to open a new file and doesnt seem to > use the MMap and hence a higher CPU on the nodes and higher latencies on > reads. > This ticket is to implement the TODO mentioned in CompressedRandomAccessReader > // TODO refactor this to separate concept of "buffer to avoid lots of read() > syscalls" and "compression buffer" > but i think a separate class for the Buffer will be better. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3623) use MMapedBuffer in CompressedSegmentedFile.getSegment
[ https://issues.apache.org/jira/browse/CASSANDRA-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay updated CASSANDRA-3623: - Attachment: CRC+MMapIO.xlsx MMappedIO-Performance.docx Done, 1) fixed the data for 10K 2) rebased 3610 Thanks! > use MMapedBuffer in CompressedSegmentedFile.getSegment > -- > > Key: CASSANDRA-3623 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3623 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 1.1 >Reporter: Vijay >Assignee: Vijay > Labels: compression > Fix For: 1.1 > > Attachments: 0001-MMaped-Compression-segmented-file-v2.patch, > 0001-MMaped-Compression-segmented-file-v3.patch, > 0001-MMaped-Compression-segmented-file.patch, > 0002-tests-for-MMaped-Compression-segmented-file-v2.patch, > 0002-tests-for-MMaped-Compression-segmented-file-v3.patch, CRC+MMapIO.xlsx, > MMappedIO-Performance.docx > > > CompressedSegmentedFile.getSegment seem to open a new file and doesnt seem to > use the MMap and hence a higher CPU on the nodes and higher latencies on > reads. > This ticket is to implement the TODO mentioned in CompressedRandomAccessReader > // TODO refactor this to separate concept of "buffer to avoid lots of read() > syscalls" and "compression buffer" > but i think a separate class for the Buffer will be better. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3623) use MMapedBuffer in CompressedSegmentedFile.getSegment
[ https://issues.apache.org/jira/browse/CASSANDRA-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay updated CASSANDRA-3623: - Attachment: (was: CRC+MMapIO.xlsx) > use MMapedBuffer in CompressedSegmentedFile.getSegment > -- > > Key: CASSANDRA-3623 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3623 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 1.1 >Reporter: Vijay >Assignee: Vijay > Labels: compression > Fix For: 1.1 > > Attachments: 0001-MMaped-Compression-segmented-file-v2.patch, > 0001-MMaped-Compression-segmented-file-v3.patch, > 0001-MMaped-Compression-segmented-file.patch, > 0002-tests-for-MMaped-Compression-segmented-file-v2.patch, > 0002-tests-for-MMaped-Compression-segmented-file-v3.patch > > > CompressedSegmentedFile.getSegment seem to open a new file and doesnt seem to > use the MMap and hence a higher CPU on the nodes and higher latencies on > reads. > This ticket is to implement the TODO mentioned in CompressedRandomAccessReader > // TODO refactor this to separate concept of "buffer to avoid lots of read() > syscalls" and "compression buffer" > but i think a separate class for the Buffer will be better. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3623) use MMapedBuffer in CompressedSegmentedFile.getSegment
[ https://issues.apache.org/jira/browse/CASSANDRA-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay updated CASSANDRA-3623: - Attachment: (was: MMappedIO-Performance.docx) > use MMapedBuffer in CompressedSegmentedFile.getSegment > -- > > Key: CASSANDRA-3623 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3623 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 1.1 >Reporter: Vijay >Assignee: Vijay > Labels: compression > Fix For: 1.1 > > Attachments: 0001-MMaped-Compression-segmented-file-v2.patch, > 0001-MMaped-Compression-segmented-file-v3.patch, > 0001-MMaped-Compression-segmented-file.patch, > 0002-tests-for-MMaped-Compression-segmented-file-v2.patch, > 0002-tests-for-MMaped-Compression-segmented-file-v3.patch > > > CompressedSegmentedFile.getSegment seem to open a new file and doesnt seem to > use the MMap and hence a higher CPU on the nodes and higher latencies on > reads. > This ticket is to implement the TODO mentioned in CompressedRandomAccessReader > // TODO refactor this to separate concept of "buffer to avoid lots of read() > syscalls" and "compression buffer" > but i think a separate class for the Buffer will be better. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3623) use MMapedBuffer in CompressedSegmentedFile.getSegment
[ https://issues.apache.org/jira/browse/CASSANDRA-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay updated CASSANDRA-3623: - Attachment: CRC+MMapIO.xlsx MMappedIO-Performance.docx 0002-tests-for-MMaped-Compression-segmented-file-v3.patch 0001-MMaped-Compression-segmented-file-v3.patch Alright i think i found the the missing peace: 1) Plz reapply v2 from CASSANDRA-3611 (which also depends on CASSANDRA-3610) 2) Plz reapply v3 which has the mark() (this seem to be used by range slice and Stress tool does it). The Test Results are attached, let me know in case of any questions... the performance seem to be better. I Used stress test so we are in the same page, and when the Column size or the range of columns to be fetched increases the performance gets better (rebuffers) > use MMapedBuffer in CompressedSegmentedFile.getSegment > -- > > Key: CASSANDRA-3623 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3623 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 1.1 >Reporter: Vijay >Assignee: Vijay > Labels: compression > Fix For: 1.1 > > Attachments: 0001-MMaped-Compression-segmented-file-v2.patch, > 0001-MMaped-Compression-segmented-file-v3.patch, > 0001-MMaped-Compression-segmented-file.patch, > 0002-tests-for-MMaped-Compression-segmented-file-v2.patch, > 0002-tests-for-MMaped-Compression-segmented-file-v3.patch, CRC+MMapIO.xlsx, > MMappedIO-Performance.docx > > > CompressedSegmentedFile.getSegment seem to open a new file and doesnt seem to > use the MMap and hence a higher CPU on the nodes and higher latencies on > reads. > This ticket is to implement the TODO mentioned in CompressedRandomAccessReader > // TODO refactor this to separate concept of "buffer to avoid lots of read() > syscalls" and "compression buffer" > but i think a separate class for the Buffer will be better. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3623) use MMapedBuffer in CompressedSegmentedFile.getSegment
[ https://issues.apache.org/jira/browse/CASSANDRA-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay updated CASSANDRA-3623: - Attachment: 0002-tests-for-MMaped-Compression-segmented-file-v2.patch 0001-MMaped-Compression-segmented-file-v2.patch Attached patch has optimization on memcpy which the earlier one didnt. Performance: Current trunk: 400+ms Avg Removing CRC (CASSANDRA-3611): 200+ms Avg With this patch: 100+ms Avg > use MMapedBuffer in CompressedSegmentedFile.getSegment > -- > > Key: CASSANDRA-3623 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3623 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 1.1 >Reporter: Vijay >Assignee: Vijay > Labels: compression > Fix For: 1.1 > > Attachments: 0001-MMaped-Compression-segmented-file-v2.patch, > 0001-MMaped-Compression-segmented-file.patch, > 0002-tests-for-MMaped-Compression-segmented-file-v2.patch > > > CompressedSegmentedFile.getSegment seem to open a new file and doesnt seem to > use the MMap and hence a higher CPU on the nodes and higher latencies on > reads. > This ticket is to implement the TODO mentioned in CompressedRandomAccessReader > // TODO refactor this to separate concept of "buffer to avoid lots of read() > syscalls" and "compression buffer" > but i think a separate class for the Buffer will be better. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3623) use MMapedBuffer in CompressedSegmentedFile.getSegment
[ https://issues.apache.org/jira/browse/CASSANDRA-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis updated CASSANDRA-3623: -- Reviewer: xedin Labels: compression (was: ) > use MMapedBuffer in CompressedSegmentedFile.getSegment > -- > > Key: CASSANDRA-3623 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3623 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 1.1 >Reporter: Vijay >Assignee: Vijay > Labels: compression > Fix For: 1.1 > > Attachments: 0001-MMaped-Compression-segmented-file.patch > > > CompressedSegmentedFile.getSegment seem to open a new file and doesnt seem to > use the MMap and hence a higher CPU on the nodes and higher latencies on > reads. > This ticket is to implement the TODO mentioned in CompressedRandomAccessReader > // TODO refactor this to separate concept of "buffer to avoid lots of read() > syscalls" and "compression buffer" > but i think a separate class for the Buffer will be better. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3623) use MMapedBuffer in CompressedSegmentedFile.getSegment
[ https://issues.apache.org/jira/browse/CASSANDRA-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vijay updated CASSANDRA-3623: - Attachment: 0001-MMaped-Compression-segmented-file.patch Attached allows mmaped io on compressed SST's. We basically ignore the boundaries and split the files based on the chunks. > use MMapedBuffer in CompressedSegmentedFile.getSegment > -- > > Key: CASSANDRA-3623 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3623 > Project: Cassandra > Issue Type: Improvement > Components: Core >Affects Versions: 1.1 >Reporter: Vijay >Assignee: Vijay > Fix For: 1.1 > > Attachments: 0001-MMaped-Compression-segmented-file.patch > > > CompressedSegmentedFile.getSegment seem to open a new file and doesnt seem to > use the MMap and hence a higher CPU on the nodes and higher latencies on > reads. > This ticket is to implement the TODO mentioned in CompressedRandomAccessReader > // TODO refactor this to separate concept of "buffer to avoid lots of read() > syscalls" and "compression buffer" > but i think a separate class for the Buffer will be better. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
