[jira] [Commented] (CASSANDRA-4165) Generate Digest file for compressed SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256576#comment-13256576 ] Marcus Eriksson commented on CASSANDRA-4165: if you have already discussed this, just ignore this issue, ill maintain the patch internally > Generate Digest file for compressed SSTables > > > Key: CASSANDRA-4165 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4165 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson >Priority: Minor > Attachments: 0001-Generate-digest-for-compressed-files-as-well.patch > > > We use the generated *Digest.sha1-files to verify backups, would be nice if > they were generated for compressed sstables as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4165) Generate Digest file for compressed SSTables
[ https://issues.apache.org/jira/browse/CASSANDRA-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13256566#comment-13256566 ] Marcus Eriksson commented on CASSANDRA-4165: yes, but when building external tools (like our backup validation thing), it would be nice to not have special cases for compressed cfs > Generate Digest file for compressed SSTables > > > Key: CASSANDRA-4165 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4165 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson >Priority: Minor > Attachments: 0001-Generate-digest-for-compressed-files-as-well.patch > > > We use the generated *Digest.sha1-files to verify backups, would be nice if > they were generated for compressed sstables as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3794) Change ColumnFamily identifiers to be UUIDs instead of sequential Integers.
[ https://issues.apache.org/jira/browse/CASSANDRA-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194545#comment-13194545 ] Marcus Eriksson commented on CASSANDRA-3794: duplicate of CASSANDRA-1983 right? > Change ColumnFamily identifiers to be UUIDs instead of sequential Integers. > --- > > Key: CASSANDRA-3794 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3794 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Pavel Yaskevich >Assignee: Pavel Yaskevich >Priority: Minor > Fix For: 1.2 > > > Change ColumnFamily identifiers to be UUIDs instead of sequential Integers. > Would be useful in the situation when nodes simultaneously trying to create > ColumnFamilies. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3624) Hinted Handoff - related OOM
[ https://issues.apache.org/jira/browse/CASSANDRA-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13175302#comment-13175302 ] Marcus Eriksson commented on CASSANDRA-3624: not yet, i might be able to do it today though this happened in a production cluster so i will have to try to reproduce somewhere else > Hinted Handoff - related OOM > > > Key: CASSANDRA-3624 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3624 > Project: Cassandra > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Marcus Eriksson >Assignee: Jonathan Ellis > Labels: hintedhandoff > Fix For: 1.0.7 > > Attachments: 3624.txt > > > One of our nodes had collected alot of hints for another node, so when the > dead node came back and the row mutations were read back from disk, the node > died with an OOM-exception (and kept dying after restart, even with increased > heap (from 8G to 12G)). The heap dump contained alot of SuperColumns and our > application does not use those (but HH does). > I'm guessing that each mutation is big so that PAGE_SIZE* does > not fit in memory (will check this tomorrow) > A simple fix (if my assumption above is correct) would be to reduce the > PAGE_SIZE in HintedHandOffManager.java to something like 10 (or even 1?) to > reduce the memory pressure. The performance hit would be small since we are > doing the hinted handoff throttle delay sleep before sending every *mutation* > anyway (not every page), thoughts? > If anyone runs in to the same problem, I got the node started again by simply > removing the HintsColumnFamily* files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2749) fine-grained control over data directories
[ https://issues.apache.org/jira/browse/CASSANDRA-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170467#comment-13170467 ] Marcus Eriksson commented on CASSANDRA-2749: sounds great (both just supporting new-style-layout and limiting names to 32chars) guess we need to supply a tool to rename sstables files if anyone is on longer names? and rolling upgrades are out of the question then right? (maybe the already are?) > fine-grained control over data directories > -- > > Key: CASSANDRA-2749 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2749 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Jonathan Ellis >Priority: Minor > Fix For: 1.1 > > Attachments: > 0001-Make-it-possible-to-put-column-families-in-subdirect.patch, > 0001-add-new-directory-layout.patch, > 0001-non-backwards-compatible-patch-for-2749-putting-cfs-.patch.gz, > 0002-fix-unit-tests.patch, 2749.tar.gz, 2749_backwards_compatible_v1.patch, > 2749_backwards_compatible_v2.patch, 2749_backwards_compatible_v3.patch, > 2749_backwards_compatible_v4.patch, > 2749_backwards_compatible_v4_rebase1.patch, 2749_not_backwards.tar.gz, > 2749_proper.tar.gz > > > Currently Cassandra supports multiple data directories but no way to control > what sstables are placed where. Particularly for systems with mixed SSDs and > rotational disks, it would be nice to pin frequently accessed columnfamilies > to the SSDs. > Postgresql does this with tablespaces > (http://www.postgresql.org/docs/9.0/static/manage-ag-tablespaces.html) but we > should probably avoid using that name because of confusing similarity to > "keyspaces." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2749) fine-grained control over data directories
[ https://issues.apache.org/jira/browse/CASSANDRA-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13163570#comment-13163570 ] Marcus Eriksson commented on CASSANDRA-2749: agree that the file structure is all over the place, that's why this patch was so incredibly painful to do seems i only did the disk space checking in sub directories in the backwards compatible patch i had on algorithm for detecting what kind of file it is in the old backwards compatible patch, it iterates over all data directories and figures out which data directory the file is in, then it knows the keyspace is the next part of the filename, and can check if there are files or directories in that directory. but i agree, it is incredibly ugly, your modifications make it alot better, and yes, this would be a good time to do this properly, not like me just trying to do a minimal patch that "works". regarding keyspaces in file names, sure, why not, guess having a header with this info in the file is out of the question, then the only meta data we have is the file name, right? A problem could be if we want to do CASSANDRA-1983 later, that would increase the file name length even more. > fine-grained control over data directories > -- > > Key: CASSANDRA-2749 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2749 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Jonathan Ellis >Priority: Minor > Fix For: 1.1 > > Attachments: > 0001-Make-it-possible-to-put-column-families-in-subdirect.patch, > 0001-add-new-directory-layout.patch, > 0001-non-backwards-compatible-patch-for-2749-putting-cfs-.patch.gz, > 0002-fix-unit-tests.patch, 2749.tar.gz, 2749_backwards_compatible_v1.patch, > 2749_backwards_compatible_v2.patch, 2749_backwards_compatible_v3.patch, > 2749_backwards_compatible_v4.patch, > 2749_backwards_compatible_v4_rebase1.patch, 2749_not_backwards.tar.gz, > 2749_proper.tar.gz > > > Currently Cassandra supports multiple data directories but no way to control > what sstables are placed where. Particularly for systems with mixed SSDs and > rotational disks, it would be nice to pin frequently accessed columnfamilies > to the SSDs. > Postgresql does this with tablespaces > (http://www.postgresql.org/docs/9.0/static/manage-ag-tablespaces.html) but we > should probably avoid using that name because of confusing similarity to > "keyspaces." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3459) Remove CF name from SSTable file
[ https://issues.apache.org/jira/browse/CASSANDRA-3459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144309#comment-13144309 ] Marcus Eriksson commented on CASSANDRA-3459: ah yeah agree, otherwise it would have to support both naming schemes again.. ill have a look if it is a big patch > Remove CF name from SSTable file > > > Key: CASSANDRA-3459 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3459 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson >Priority: Minor > Fix For: 1.1 > > > When CASSANDRA-2749 is done, it is unnecessary to keep the CF name in the > filename of the sstable files (this is already in the directory name). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2749) fine-grained control over data directories
[ https://issues.apache.org/jira/browse/CASSANDRA-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127329#comment-13127329 ] Marcus Eriksson commented on CASSANDRA-2749: Biggest part left is figuring out how to estimate disk space to know where to write an sstable, ill work on that this weekend Other than that small cleanups (codestyle), unit tests and adding the config parameter > fine-grained control over data directories > -- > > Key: CASSANDRA-2749 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2749 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Jonathan Ellis >Priority: Minor > Fix For: 1.1 > > Attachments: > 0001-Make-it-possible-to-put-column-families-in-subdirect.patch, > 2749_backwards_compatible_v1.patch, 2749_backwards_compatible_v2.patch > > > Currently Cassandra supports multiple data directories but no way to control > what sstables are placed where. Particularly for systems with mixed SSDs and > rotational disks, it would be nice to pin frequently accessed columnfamilies > to the SSDs. > Postgresql does this with tablespaces > (http://www.postgresql.org/docs/9.0/static/manage-ag-tablespaces.html) but we > should probably avoid using that name because of confusing similarity to > "keyspaces." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2749) fine-grained control over data directories
[ https://issues.apache.org/jira/browse/CASSANDRA-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126910#comment-13126910 ] Marcus Eriksson commented on CASSANDRA-2749: ah that is a leftover from my first non-backwards-compatible patch it works even without the cf in the new File(...) > fine-grained control over data directories > -- > > Key: CASSANDRA-2749 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2749 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Jonathan Ellis >Priority: Minor > Fix For: 1.1 > > Attachments: > 0001-Make-it-possible-to-put-column-families-in-subdirect.patch, > 2749_backwards_compatible_v1.patch, 2749_backwards_compatible_v2.patch > > > Currently Cassandra supports multiple data directories but no way to control > what sstables are placed where. Particularly for systems with mixed SSDs and > rotational disks, it would be nice to pin frequently accessed columnfamilies > to the SSDs. > Postgresql does this with tablespaces > (http://www.postgresql.org/docs/9.0/static/manage-ag-tablespaces.html) but we > should probably avoid using that name because of confusing similarity to > "keyspaces." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2749) fine-grained control over data directories
[ https://issues.apache.org/jira/browse/CASSANDRA-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124288#comment-13124288 ] Marcus Eriksson commented on CASSANDRA-2749: Anyone got a suggestion on how to do the available disk space checking? Reason in that now we cannot simply check which dataFileLocation that has the most available disk space, we need to check the subdirs where files are actually written. My naive first approach would be to simply change DatabaseDescriptor.getDataFileLocationForTable(..) to take a column family name, which we ought to always know when calling that method (and a quick check seems to say that only compactions might prove a problem for this approach). Comments? > fine-grained control over data directories > -- > > Key: CASSANDRA-2749 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2749 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Jonathan Ellis >Priority: Minor > Fix For: 1.1 > > Attachments: > 0001-Make-it-possible-to-put-column-families-in-subdirect.patch, > 2749_backwards_compatible_v1.patch, 2749_backwards_compatible_v2.patch > > > Currently Cassandra supports multiple data directories but no way to control > what sstables are placed where. Particularly for systems with mixed SSDs and > rotational disks, it would be nice to pin frequently accessed columnfamilies > to the SSDs. > Postgresql does this with tablespaces > (http://www.postgresql.org/docs/9.0/static/manage-ag-tablespaces.html) but we > should probably avoid using that name because of confusing similarity to > "keyspaces." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2749) fine-grained control over data directories
[ https://issues.apache.org/jira/browse/CASSANDRA-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118867#comment-13118867 ] Marcus Eriksson commented on CASSANDRA-2749: agree, gonna look at how much pain it is to actually build though wont need alot of downtime if it is done offline, can rsync files into the subdir right before stopping the node > fine-grained control over data directories > -- > > Key: CASSANDRA-2749 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2749 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Jonathan Ellis >Priority: Minor > Fix For: 1.1 > > Attachments: > 0001-Make-it-possible-to-put-column-families-in-subdirect.patch > > > Currently Cassandra supports multiple data directories but no way to control > what sstables are placed where. Particularly for systems with mixed SSDs and > rotational disks, it would be nice to pin frequently accessed columnfamilies > to the SSDs. > Postgresql does this with tablespaces > (http://www.postgresql.org/docs/9.0/static/manage-ag-tablespaces.html) but we > should probably avoid using that name because of confusing similarity to > "keyspaces." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2749) fine-grained control over data directories
[ https://issues.apache.org/jira/browse/CASSANDRA-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118859#comment-13118859 ] Marcus Eriksson commented on CASSANDRA-2749: Ok, i'll update the patch > fine-grained control over data directories > -- > > Key: CASSANDRA-2749 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2749 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Jonathan Ellis >Priority: Minor > Fix For: 1.1 > > Attachments: > 0001-Make-it-possible-to-put-column-families-in-subdirect.patch > > > Currently Cassandra supports multiple data directories but no way to control > what sstables are placed where. Particularly for systems with mixed SSDs and > rotational disks, it would be nice to pin frequently accessed columnfamilies > to the SSDs. > Postgresql does this with tablespaces > (http://www.postgresql.org/docs/9.0/static/manage-ag-tablespaces.html) but we > should probably avoid using that name because of confusing similarity to > "keyspaces." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2749) fine-grained control over data directories
[ https://issues.apache.org/jira/browse/CASSANDRA-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118696#comment-13118696 ] Marcus Eriksson commented on CASSANDRA-2749: It is not backwards compatible, i figured it was not worth the extra complexity, i imagine the "upgrade"-path to be: 1. shut down node 2. edit config file 3. symlink in SSDs and move the files into the subdirectories 4. start node Of course, this could be done on startup by cassandra itself, but providing a shell-script that does it keeps it simple, what do you think? I really dont want the complexity with sstable files in two places at the same time. My solution simply changed the Descriptor class to return paths to sstable files in subdirs, and then fix everything that was affected. Many of the classes modified were due to assumptions in the unit tests. I'm currently refactoring the Descriptor class to make it cleaner and a central place to glob for files etc. Expect a patch tonight or tomorrow. I will of course change DatabaseDescriptor.getPerCFDirectory() to read the config file, kept it hard coded to make testing simpler. > fine-grained control over data directories > -- > > Key: CASSANDRA-2749 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2749 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Jonathan Ellis >Priority: Minor > Fix For: 1.1 > > Attachments: > 0001-Make-it-possible-to-put-column-families-in-subdirect.patch > > > Currently Cassandra supports multiple data directories but no way to control > what sstables are placed where. Particularly for systems with mixed SSDs and > rotational disks, it would be nice to pin frequently accessed columnfamilies > to the SSDs. > Postgresql does this with tablespaces > (http://www.postgresql.org/docs/9.0/static/manage-ag-tablespaces.html) but we > should probably avoid using that name because of confusing similarity to > "keyspaces." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2749) fine-grained control over data directories
[ https://issues.apache.org/jira/browse/CASSANDRA-2749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118017#comment-13118017 ] Marcus Eriksson commented on CASSANDRA-2749: btw, yes, i did include the data files in the patch on purpose, they need to be in subdirs for some of the unit tests to work > fine-grained control over data directories > -- > > Key: CASSANDRA-2749 > URL: https://issues.apache.org/jira/browse/CASSANDRA-2749 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Jonathan Ellis >Priority: Minor > Attachments: > 0001-Make-it-possible-to-put-column-families-in-subdirect.patch > > > Currently Cassandra supports multiple data directories but no way to control > what sstables are placed where. Particularly for systems with mixed SSDs and > rotational disks, it would be nice to pin frequently accessed columnfamilies > to the SSDs. > Postgresql does this with tablespaces > (http://www.postgresql.org/docs/9.0/static/manage-ag-tablespaces.html) but we > should probably avoid using that name because of confusing similarity to > "keyspaces." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3263) Whitespace in SimpleSeedProvider string makes seed ignored
[ https://issues.apache.org/jira/browse/CASSANDRA-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115683#comment-13115683 ] Marcus Eriksson commented on CASSANDRA-3263: of course i meant that only 1.2.3.4 will became a seed in the example in the description.. > Whitespace in SimpleSeedProvider string makes seed ignored > -- > > Key: CASSANDRA-3263 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3263 > Project: Cassandra > Issue Type: Bug > Components: Core >Affects Versions: 0.8.4, 0.8.5, 0.8.6 >Reporter: Marcus Eriksson >Priority: Minor > Fix For: 1.0.0 > > Attachments: 3263.txt > > > If a seeds given to SimpleSeedProvider contains whitespace, the seed will be > ignored > for example "1.2.3.4, 5.6.7.8" will only make 5.6.7.8 a seed. > patch simply trim()s the host. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
