[jira] [Commented] (CASSANDRA-4165) Generate Digest file for compressed SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13931848#comment-13931848 ] Jonathan Ellis commented on CASSANDRA-4165: --- committed > Generate Digest file for compressed SSTables > > > Key: CASSANDRA-4165 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4165 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Marcus Eriksson >Assignee: Jonathan Ellis >Priority: Minor > Labels: performance > Fix For: 2.1 beta2 > > Attachments: 0001-Generate-digest-for-compressed-files-as-well.patch, > 0002-dont-do-crc-and-add-digests-for-compressed-files.txt, 4165-rebased.txt > > > We use the generated *Digest.sha1-files to verify backups, would be nice if > they were generated for compressed sstables as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-4165) Generate Digest file for compressed SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13931589#comment-13931589 ] Marcus Eriksson commented on CASSANDRA-4165: rebased and a fix here: https://github.com/krummas/cassandra/commits/jbellis/4165-3 (the Digest component for compressed files was not written) > Generate Digest file for compressed SSTables > > > Key: CASSANDRA-4165 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4165 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Marcus Eriksson >Assignee: Jonathan Ellis >Priority: Minor > Labels: performance > Fix For: 2.1 beta2 > > Attachments: 0001-Generate-digest-for-compressed-files-as-well.patch, > 0002-dont-do-crc-and-add-digests-for-compressed-files.txt, 4165-rebased.txt > > > We use the generated *Digest.sha1-files to verify backups, would be nice if > they were generated for compressed sstables as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-4165) Generate Digest file for compressed SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13930913#comment-13930913 ] Jonathan Ellis commented on CASSANDRA-4165: --- Can you review that branch, [~krummas]? > Generate Digest file for compressed SSTables > > > Key: CASSANDRA-4165 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4165 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Marcus Eriksson >Assignee: Jonathan Ellis >Priority: Minor > Labels: performance > Fix For: 2.1 beta2 > > Attachments: 0001-Generate-digest-for-compressed-files-as-well.patch, > 0002-dont-do-crc-and-add-digests-for-compressed-files.txt, 4165-rebased.txt > > > We use the generated *Digest.sha1-files to verify backups, would be nice if > they were generated for compressed sstables as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-4165) Generate Digest file for compressed SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13856081#comment-13856081 ] Jonathan Ellis commented on CASSANDRA-4165: --- https://github.com/jbellis/cassandra/commits/4165-3 refactors and switches to an adler32 digest, WDYT? > Generate Digest file for compressed SSTables > > > Key: CASSANDRA-4165 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4165 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Marcus Eriksson >Priority: Minor > Labels: performance > Fix For: 2.1 > > Attachments: 0001-Generate-digest-for-compressed-files-as-well.patch, > 0002-dont-do-crc-and-add-digests-for-compressed-files.txt, 4165-rebased.txt > > > We use the generated *Digest.sha1-files to verify backups, would be nice if > they were generated for compressed sstables as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4165) Generate Digest file for compressed SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855628#comment-13855628 ] Radovan Zvoncek commented on CASSANDRA-4165: So I used Marcuse's patch as a basis and added checks preventing computation/writing of CRC components for compressed files. I'm still worried a bit about backwards compatibility though. I did some experiments and didn't encounter any problems. However, I can't fully comprehend the impact of this change and might have missed something. > Generate Digest file for compressed SSTables > > > Key: CASSANDRA-4165 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4165 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Marcus Eriksson >Priority: Minor > Labels: performance > Fix For: 2.1 > > Attachments: 0001-Generate-digest-for-compressed-files-as-well.patch, > 0002-dont-do-crc-and-add-digests-for-compressed-files.txt, 4165-rebased.txt > > > We use the generated *Digest.sha1-files to verify backups, would be nice if > they were generated for compressed sstables as well. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-4165) Generate Digest file for compressed SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853810#comment-13853810 ] Radovan Zvoncek commented on CASSANDRA-4165: Any update on this? I have written a patch for CASSANDRA-5791 but it assumes having digests for compressed files available. > Generate Digest file for compressed SSTables > > > Key: CASSANDRA-4165 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4165 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Minor > Labels: performance > Fix For: 2.1 > > Attachments: 0001-Generate-digest-for-compressed-files-as-well.patch, > 4165-rebased.txt > > > We use the generated *Digest.sha1-files to verify backups, would be nice if > they were generated for compressed sstables as well. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (CASSANDRA-4165) Generate Digest file for compressed SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13774636#comment-13774636 ] Jonathan Ellis commented on CASSANDRA-4165: --- So we have # Block-level CRC, in a CRC component, for uncompressed files # File-level hash, in a Digest component, for uncompressed files # Block-level CRC, inline in the Data component, for compressed files # [proposed] file-level hash, in a Digest component, for compressed files So at the least we should clean up the patch here to not also generate a CRC component for compressed files. Not sure what else we can do to clean it up ... I'm not sure I'm a fan of the CRC component to begin with, but removing it (in favor of a CRCOnlyCompression, for instance) is out of scope here. > Generate Digest file for compressed SSTables > > > Key: CASSANDRA-4165 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4165 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Minor > Attachments: 0001-Generate-digest-for-compressed-files-as-well.patch, > 4165-rebased.txt > > > We use the generated *Digest.sha1-files to verify backups, would be nice if > they were generated for compressed sstables as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4165) Generate Digest file for compressed SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773543#comment-13773543 ] Vijay commented on CASSANDRA-4165: -- Hi Jonathan, 3648 actually adds block level CRC for uncompressed files and writes to a separate file (CRC.db), and uses it during the streaming parts of the file to validate before streaming (not during normal reads). Hence we need 2 Checksums during the flush 1 for blocks and the md5 for the whole file. > Generate Digest file for compressed SSTables > > > Key: CASSANDRA-4165 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4165 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Minor > Attachments: 0001-Generate-digest-for-compressed-files-as-well.patch, > 4165-rebased.txt > > > We use the generated *Digest.sha1-files to verify backups, would be nice if > they were generated for compressed sstables as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4165) Generate Digest file for compressed SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257534#comment-13257534 ] Sylvain Lebresne commented on CASSANDRA-4165: - We don't really have discussed it more than the reasoning Jonathan explained :). But if it's for external tools, is it still useful to have it computed during the sstable write (i.e, you could generate the sha1 yourself before backupping the file in the first place)? Not that it's much work for us to do it (well, except for the added cpu usage during sstable write maybe). > Generate Digest file for compressed SSTables > > > Key: CASSANDRA-4165 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4165 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson >Priority: Minor > Attachments: 0001-Generate-digest-for-compressed-files-as-well.patch > > > We use the generated *Digest.sha1-files to verify backups, would be nice if > they were generated for compressed sstables as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4165) Generate Digest file for compressed SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256576#comment-13256576 ] Marcus Eriksson commented on CASSANDRA-4165: if you have already discussed this, just ignore this issue, ill maintain the patch internally > Generate Digest file for compressed SSTables > > > Key: CASSANDRA-4165 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4165 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson >Priority: Minor > Attachments: 0001-Generate-digest-for-compressed-files-as-well.patch > > > We use the generated *Digest.sha1-files to verify backups, would be nice if > they were generated for compressed sstables as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4165) Generate Digest file for compressed SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256566#comment-13256566 ] Marcus Eriksson commented on CASSANDRA-4165: yes, but when building external tools (like our backup validation thing), it would be nice to not have special cases for compressed cfs > Generate Digest file for compressed SSTables > > > Key: CASSANDRA-4165 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4165 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson >Priority: Minor > Attachments: 0001-Generate-digest-for-compressed-files-as-well.patch > > > We use the generated *Digest.sha1-files to verify backups, would be nice if > they were generated for compressed sstables as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4165) Generate Digest file for compressed SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256560#comment-13256560 ] Jonathan Ellis commented on CASSANDRA-4165: --- The thinking was, compressed sstables have a per-block checksum, so there's no need to have the less-granular sha. > Generate Digest file for compressed SSTables > > > Key: CASSANDRA-4165 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4165 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson >Priority: Minor > Attachments: 0001-Generate-digest-for-compressed-files-as-well.patch > > > We use the generated *Digest.sha1-files to verify backups, would be nice if > they were generated for compressed sstables as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
