[jira] [Commented] (CASSANDRA-8160) CF level option to call posix_fadvise for sstables on creation and startup
[ https://issues.apache.org/jira/browse/CASSANDRA-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339018#comment-14339018 ] Matt Stump commented on CASSANDRA-8160: --- After learning how the new implementation of the in memory stuff works with DSE I don't think it would be a good idea to move forward with this ticket. Using mlock seems like a much better way to tackle this problem. CF level option to call posix_fadvise for sstables on creation and startup -- Key: CASSANDRA-8160 URL: https://issues.apache.org/jira/browse/CASSANDRA-8160 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Matt Stump Assignee: Branimir Lambov Priority: Minor Fix For: 2.1.4 Attachments: trunk-8160.txt We should have a CF level configuration with will result in posix_fadvise being called for sstables for that CF. It should be called on node startup and for new sstables. This should be configurable per CF to allow for some CFs to be prioritized above others. Not sure if we should use POSIX_FADV_SEQUENTIAL or POSIX_FADV_WILLNEED. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8336) Quarantine nodes after receiving the gossip shutdown message
[ https://issues.apache.org/jira/browse/CASSANDRA-8336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339022#comment-14339022 ] Brandon Williams commented on CASSANDRA-8336: - bq. If hit 'Unable to gossip with any seeds’ on replace, it shuts down the gossiper. Do you have the stacktrace where this is happening? I have a feeling we're going to end up in checked exception hell trying to fix this since we throw RuntimeException there (to avoid such a hell, in fact.) Quarantine nodes after receiving the gossip shutdown message Key: CASSANDRA-8336 URL: https://issues.apache.org/jira/browse/CASSANDRA-8336 Project: Cassandra Issue Type: Bug Components: Core Reporter: Brandon Williams Assignee: Brandon Williams Fix For: 2.0.13 Attachments: 8336-v2.txt, 8336-v3.txt, 8336.txt In CASSANDRA-3936 we added a gossip shutdown announcement. The problem here is that this isn't sufficient; you can still get TOEs and have to wait on the FD to figure things out. This happens due to gossip propagation time and variance; if node X shuts down and sends the message to Y, but Z has a greater gossip version than Y for X and has not yet received the message, it can initiate gossip with Y and thus mark X alive again. I propose quarantining to solve this, however I feel it should be a -D parameter you have to specify, so as not to destroy current dev and test practices, since this will mean a node that shuts down will not be able to restart until the quarantine expires. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8700) replace the wiki with docs in the git repo
[ https://issues.apache.org/jira/browse/CASSANDRA-8700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339139#comment-14339139 ] Jonathan Ellis commented on CASSANDRA-8700: --- Sure, but I'm throwing that project back at you. :) bq. Jon, can you put together an outline of what you have in mind for minimum viable docs? replace the wiki with docs in the git repo -- Key: CASSANDRA-8700 URL: https://issues.apache.org/jira/browse/CASSANDRA-8700 Project: Cassandra Issue Type: Improvement Components: Documentation website Reporter: Jon Haddad Priority: Minor The wiki as it stands is pretty terrible. It takes several minutes to apply a single update, and as a result, it's almost never updated. The information there has very little context as to what version it applies to. Most people I've talked to that try to use the information they find there find it is more confusing than helpful. I'd like to propose that instead of using the wiki, the doc directory in the cassandra repo be used for docs (already used for CQL3 spec) in a format that can be built to a variety of output formats like HTML / epub / etc. I won't start the bikeshedding on which markup format is preferable - but there are several options that can work perfectly fine. I've personally use sphinx w/ restructured text, and markdown. Both can build easily and as an added bonus be pushed to readthedocs (or something similar) automatically. For an example, see cqlengine's documentation, which I think is already significantly better than the wiki: http://cqlengine.readthedocs.org/en/latest/ In addition to being overall easier to maintain, putting the documentation in the git repo adds context, since it evolves with the versions of Cassandra. If the wiki were kept even remotely up to date, I wouldn't bother with this, but not having at least some basic documentation in the repo, or anywhere associated with the project, is frustrating. For reference, the last 3 updates were: 1/15/15 - updating committers list 1/08/15 - updating contributers and how to contribute 12/16/14 - added a link to CQL docs from wiki frontpage (by me) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8867) Get whole keyspace size
Tamar Nirenberg created CASSANDRA-8867: -- Summary: Get whole keyspace size Key: CASSANDRA-8867 URL: https://issues.apache.org/jira/browse/CASSANDRA-8867 Project: Cassandra Issue Type: Task Reporter: Tamar Nirenberg Priority: Trivial Hi, Is there a way to find the size (bytes) of the whole keyspace? cfstats seem to supply it column families. Does cfstat size reflects what is on 1 node - or all Ring? Thanks, Tamar -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8390) The process cannot access the file because it is being used by another process
[
https://issues.apache.org/jira/browse/CASSANDRA-8390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338277#comment-14338277
]
Amichai Rothman commented on CASSANDRA-8390:
Can someone please reopen this issue? Multiple users can recreate it, but being
closed we can't vote for it, and it may be overlooked by devs...
The process cannot access the file because it is being used by another process
--
Key: CASSANDRA-8390
URL: https://issues.apache.org/jira/browse/CASSANDRA-8390
Project: Cassandra
Issue Type: Bug
Reporter: Ilya Komolkin
Assignee: Joshua McKenzie
Fix For: 2.1.3
Attachments: CassandraDiedWithDiskAccessModeStandardLogs.7z, FSD.PNG,
NoHostAvailableLogs.zip
{code}21:46:27.810 [NonPeriodicTasks:1] ERROR o.a.c.service.CassandraDaemon -
Exception in thread Thread[NonPeriodicTasks:1,5,main]
org.apache.cassandra.io.FSWriteError: java.nio.file.FileSystemException:
E:\Upsource_12391\data\cassandra\data\kernel\filechangehistory_t-a277b560764611e48c8e4915424c75fe\kernel-filechangehistory_t-ka-33-Index.db:
The process cannot access the file because it is being used by another
process.
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:135)
~[cassandra-all-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:121)
~[cassandra-all-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:113)
~[cassandra-all-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableDeletingTask.run(SSTableDeletingTask.java:94)
~[cassandra-all-2.1.1.jar:2.1.1]
at
org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:664)
~[cassandra-all-2.1.1.jar:2.1.1]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_71]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
~[na:1.7.0_71]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
~[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_71]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: java.nio.file.FileSystemException:
E:\Upsource_12391\data\cassandra\data\kernel\filechangehistory_t-a277b560764611e48c8e4915424c75fe\kernel-filechangehistory_t-ka-33-Index.db:
The process cannot access the file because it is being used by another
process.
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
~[na:1.7.0_71]
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
~[na:1.7.0_71]
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
~[na:1.7.0_71]
at
sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:269)
~[na:1.7.0_71]
at
sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
~[na:1.7.0_71]
at java.nio.file.Files.delete(Files.java:1079) ~[na:1.7.0_71]
at
org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:131)
~[cassandra-all-2.1.1.jar:2.1.1]
... 11 common frames omitted{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7212) Allow to switch user within CQLSH session
[ https://issues.apache.org/jira/browse/CASSANDRA-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14338314#comment-14338314 ] Anuja Mandlecha commented on CASSANDRA-7212: I am taking up this bug and I think the syntax should be USE keyspace [USER] WITH [PASSWORD] wherein the password should be hidden (using asterisks) as in postgres Please let me know your thoughts on this. Allow to switch user within CQLSH session - Key: CASSANDRA-7212 URL: https://issues.apache.org/jira/browse/CASSANDRA-7212 Project: Cassandra Issue Type: Improvement Components: API Environment: [cqlsh 4.1.1 | Cassandra 2.0.7.31 | CQL spec 3.1.1 | Thrift protocol 19.39.0] Reporter: Jose Martinez Poblete Labels: cqlsh Once a user is logged into CQLSH, it is not possible to switch to another user without exiting and relaunch This is a feature offered in postgres and probably other databases: http://secure.encivasolutions.com/knowledgebase.php?action=displayarticleid=1126 Perhaps this could be implemented on CQLSH as part of the USE directive: USE Keyspace [USER] [password] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8873) Add PropertySeedProvider
Albert P Tobey created CASSANDRA-8873: - Summary: Add PropertySeedProvider Key: CASSANDRA-8873 URL: https://issues.apache.org/jira/browse/CASSANDRA-8873 Project: Cassandra Issue Type: New Feature Components: Core Reporter: Albert P Tobey Priority: Minor Attachments: PropertySeedProvider.java Add a PropertySeedProvider that allows administrators to set a seed on the command line with -Dcassandra.seeds=127.0.0.1,127.0.0.2 instead of rewriting cassandra.yaml. It looks like the yaml parser expects there to always be a parameters: option on seeds, so unless we change it to be optional, there needs to be a dummy map or the yaml will not parse, e.g. seed_provider: - class_name: org.apache.cassandra.locator.PropertySeedProvider parameters: - stub: this is required for the yaml parser and is ignored -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8366) Repair grows data on nodes, causes load to become unbalanced
[
https://issues.apache.org/jira/browse/CASSANDRA-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339436#comment-14339436
]
Yuki Morishita commented on CASSANDRA-8366:
---
You should use {{cfs.selectAndReference}} instead of looping and referencing in
CompactionManager change.
Especially {{Refs.ref}} throws exception when referencing failed.
{{selectAndReference}} uses {{tryRef}} which retries until reference is
acquired.
Otherwise +1.
Repair grows data on nodes, causes load to become unbalanced
Key: CASSANDRA-8366
URL: https://issues.apache.org/jira/browse/CASSANDRA-8366
Project: Cassandra
Issue Type: Bug
Environment: 4 node cluster
2.1.2 Cassandra
Inserts and reads are done with CQL driver
Reporter: Jan Karlsson
Assignee: Marcus Eriksson
Attachments: 0001-8366.patch, results-1000-inc-repairs.txt,
results-1750_inc_repair.txt, results-500_1_inc_repairs.txt,
results-500_2_inc_repairs.txt,
results-500_full_repair_then_inc_repairs.txt,
results-500_inc_repairs_not_parallel.txt,
run1_with_compact_before_repair.log, run2_no_compact_before_repair.log,
run3_no_compact_before_repair.log, test.sh, testv2.sh
There seems to be something weird going on when repairing data.
I have a program that runs 2 hours which inserts 250 random numbers and reads
250 times per second. It creates 2 keyspaces with SimpleStrategy and RF of 3.
I use size-tiered compaction for my cluster.
After those 2 hours I run a repair and the load of all nodes goes up. If I
run incremental repair the load goes up alot more. I saw the load shoot up 8
times the original size multiple times with incremental repair. (from 2G to
16G)
with node 9 8 7 and 6 the repro procedure looked like this:
(Note that running full repair first is not a requirement to reproduce.)
{noformat}
After 2 hours of 250 reads + 250 writes per second:
UN 9 583.39 MB 256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1
UN 8 584.01 MB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1
UN 7 583.72 MB 256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1
UN 6 583.84 MB 256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1
Repair -pr -par on all nodes sequentially
UN 9 746.29 MB 256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1
UN 8 751.02 MB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1
UN 7 748.89 MB 256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1
UN 6 758.34 MB 256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1
repair -inc -par on all nodes sequentially
UN 9 2.41 GB256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1
UN 8 2.53 GB256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1
UN 7 2.6 GB 256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1
UN 6 2.17 GB256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1
after rolling restart
UN 9 1.47 GB256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1
UN 8 1.5 GB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1
UN 7 2.46 GB256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1
UN 6 1.19 GB256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1
compact all nodes sequentially
UN 9 989.99 MB 256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1
UN 8 994.75 MB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1
UN 7 1.46 GB256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1
UN 6 758.82 MB 256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1
repair -inc -par on all nodes sequentially
UN 9 1.98 GB256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1
UN 8 2.3 GB 256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1
UN 7 3.71 GB256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1
UN 6 1.68 GB256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1
restart once more
UN 9 2 GB 256 ? 28220962-26ae-4eeb-8027-99f96e377406 rack1
UN 8 2.05 GB256 ? f2de6ea1-de88-4056-8fde-42f9c476a090 rack1
UN 7 4.1 GB 256 ? 2b6b5d66-13c8-43d8-855c-290c0f3c3a0b rack1
UN 6 1.68 GB256 ? b8bd67f1-a816-46ff-b4a4-136ad5af6d4b rack1
{noformat}
Is there something im missing or is this strange behavior?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6246) EPaxos
[ https://issues.apache.org/jira/browse/CASSANDRA-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339463#comment-14339463 ] Blake Eggleston commented on CASSANDRA-6246: I just merged some commits into my CASSANDRA-6246 branch. This is the initial implementation of the epoch, instance gc, streaming/repair, read repair, and failure recovery logic. I also have a dtests fork that tests it here: https://github.com/bdeggleston/cassandra-dtest/tree/CASSANDRA-6246. I still have some items I need to complete before submitting review, but nothing major (relative to this stuff). Mostly cleaning up and refining things. EPaxos -- Key: CASSANDRA-6246 URL: https://issues.apache.org/jira/browse/CASSANDRA-6246 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Jonathan Ellis Assignee: Blake Eggleston Priority: Minor One reason we haven't optimized our Paxos implementation with Multi-paxos is that Multi-paxos requires leader election and hence, a period of unavailability when the leader dies. EPaxos is a Paxos variant that requires (1) less messages than multi-paxos, (2) is particularly useful across multiple datacenters, and (3) allows any node to act as coordinator: http://sigops.org/sosp/sosp13/papers/p358-moraru.pdf However, there is substantial additional complexity involved if we choose to implement it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8865) DROP INDEX name case sensitivity causing errors in cass upgrade 2.0.10 to 2.1.3
[
https://issues.apache.org/jira/browse/CASSANDRA-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-8865:
---
Reproduced In: 2.1.3
Fix Version/s: 2.1.4
DROP INDEX name case sensitivity causing errors in cass upgrade 2.0.10 to
2.1.3
---
Key: CASSANDRA-8865
URL: https://issues.apache.org/jira/browse/CASSANDRA-8865
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Amazon, single node, ubuntu 14.04, jdk 7
Reporter: Constance Eustace
Fix For: 2.1.4
We are upgrading our dev cluster. 2.0.10 to 2.1.3
The indexes are behaving very strangely.
create index definition_bundle__BundleDefSkuIDXTest on
definition_bundle.entity_bundledef(e_entlinks) ;
definition_bundle select column_name, index_name, index_options, index_type,
component_index from system.schema_columns where keyspace_name =
'definition_bundle' and columnfamily_name = 'entity_bundledef';
column_name | index_name | index_options |
index_type | component_index
-++---++-
bundle_sku | definition_bundle__BundleDefSkuIDX |{} |
COMPOSITES | 1
e_entid | null | null |
null |null
e_entlinks | definition_bundle__bundledefskuidxtest |{} |
COMPOSITES | 1
NOTICE THE AUTO-DOWNCASE of our newly created index. The index that already
existed is NOT AUTO-DOWNCASED. I don't know if this is recent or not.
We cannot drop the mixed case index. Nodetool index reconstruction did not
work. Indexes are doing very weird things.
Hm. UPDATE:
This did successfully delete the index:
drop index definition_bundle__BundleDefSkuIDX;
Anyway, it looks like there is some upcase/downcase assumptions not being
properly done somewhere, either in upgrades or similar stuff.
We will probably drop our indexes and recreate them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-8863) $JVM_EXTRA_OPTS not properly sourced
[
https://issues.apache.org/jira/browse/CASSANDRA-8863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Shuler resolved CASSANDRA-8863.
---
Resolution: Won't Fix
That line is in cassandra-env.sh to append JVM_EXTRA_OPTS that were exported in
a shell, or you could actually add a {{JVM_EXTRA_OPTS=-D..., -D... line in
cassandra-env.sh. /etc/default/cassandra is not the correct place to add JVM
options.
FYI, I'm removing the export of options added to /etc/default/cassandra in 8821
and basically commenting the same - cassandra-env.sh is the configuration file
to modify or add JVM options.
$JVM_EXTRA_OPTS not properly sourced
Key: CASSANDRA-8863
URL: https://issues.apache.org/jira/browse/CASSANDRA-8863
Project: Cassandra
Issue Type: Bug
Components: Config
Reporter: Pablo RUTH
Assignee: Michael Shuler
Priority: Minor
Attachments: jvm-extra-opts.patch
JVM options set in /etc/default/cassandra whit $JVM_EXTRA_OPTS are not
properly sourced. The default file is sourced in the init script with the
cassandra-env.sh. But in /usr/sbin/cassandra only cassandra-env.sh is
sourced, not default so $JVM_EXTRA_OPTS are not applied to cassandra process.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8576) Primary Key Pushdown For Hadoop
[ https://issues.apache.org/jira/browse/CASSANDRA-8576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339343#comment-14339343 ] Brandon Williams commented on CASSANDRA-8576: - LGTM, can you attach a version for trunk as well? Primary Key Pushdown For Hadoop --- Key: CASSANDRA-8576 URL: https://issues.apache.org/jira/browse/CASSANDRA-8576 Project: Cassandra Issue Type: Improvement Components: Hadoop Reporter: Russell Alexander Spitzer Assignee: Alex Liu Fix For: 2.1.4 Attachments: 8576-2.1-branch.txt I've heard reports from several users that they would like to have predicate pushdown functionality for hadoop (Hive in particular) based services. Example usecase Table with wide partitions, one per customer Application team has HQL they would like to run on a single customer Currently time to complete scales with number of customers since Input Format can't pushdown primary key predicate Current implementation requires a full table scan (since it can't recognize that a single partition was specified) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8863) $JVM_EXTRA_OPTS not properly sourced
[
https://issues.apache.org/jira/browse/CASSANDRA-8863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339369#comment-14339369
]
Michael Shuler commented on CASSANDRA-8863:
---
I think I will probably be adding a commented-out {{JVM_EXTRA_OPTS=}} line in
cassandra-env.sh to make this clear.
$JVM_EXTRA_OPTS not properly sourced
Key: CASSANDRA-8863
URL: https://issues.apache.org/jira/browse/CASSANDRA-8863
Project: Cassandra
Issue Type: Bug
Components: Config
Reporter: Pablo RUTH
Assignee: Michael Shuler
Priority: Minor
Attachments: jvm-extra-opts.patch
JVM options set in /etc/default/cassandra whit $JVM_EXTRA_OPTS are not
properly sourced. The default file is sourced in the init script with the
cassandra-env.sh. But in /usr/sbin/cassandra only cassandra-env.sh is
sourced, not default so $JVM_EXTRA_OPTS are not applied to cassandra process.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (CASSANDRA-8150) Revaluate Default JVM tuning parameters
[ https://issues.apache.org/jira/browse/CASSANDRA-8150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-8150: Comment: was deleted (was: Dear sender, I am on a short break and be back on 03-03-2015. Your email will not be forwarded. Best regards, Hans van der Linde - ATTENTION: The information in this electronic mail message is private and confidential, and only intended for the addressee. Should you receive this message by mistake, you are hereby notified that any disclosure, reproduction, distribution or use of this message is strictly prohibited. Please inform the sender by reply transmission and delete the message without copying or opening it. Messages and attachments are scanned for all viruses known. If this message contains password-protected attachments, the files have NOT been scanned for viruses by the ING mail domain. Always scan attachments before opening them. - ) Revaluate Default JVM tuning parameters --- Key: CASSANDRA-8150 URL: https://issues.apache.org/jira/browse/CASSANDRA-8150 Project: Cassandra Issue Type: Improvement Components: Config Reporter: Matt Stump Assignee: Brandon Williams Attachments: upload.png It's been found that the old twitter recommendations of 100m per core up to 800m is harmful and should no longer be used. Instead the formula used should be 1/3 or 1/4 max heap with a max of 2G. 1/3 or 1/4 is debatable and I'm open to suggestions. If I were to hazard a guess 1/3 is probably better for releases greater than 2.1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8874) running out of FD, and causing clients hang when dropping a keyspace with many CF with many sstables
[
https://issues.apache.org/jira/browse/CASSANDRA-8874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philip Thompson updated CASSANDRA-8874:
---
Reproduced In: 2.0.11
Fix Version/s: 2.0.13
running out of FD, and causing clients hang when dropping a keyspace with
many CF with many sstables
Key: CASSANDRA-8874
URL: https://issues.apache.org/jira/browse/CASSANDRA-8874
Project: Cassandra
Issue Type: Bug
Reporter: Jackson Chung
Fix For: 2.0.13
we already set number of file descriptors to 10 for c* usage, and
confirmed that from /proc/$cass_pid/limits
we have 16 nodes, 2 DC, each node stores about 600GB to 1TB data; ec2, i2-2xl
instances, raid0 the 2 disks
we use both hector and datastax drivers, and there are many clients
connecting to the cluster.
1 day we dropped a keyspace (that our app no longer use), which has a good
amount of CFs, with some of them use leveledbcompaction and have some good
amount of sstables... and our app went down. CPU/load avg were high and we
couldn't even ssh to them. We have to force a reboot, and restart 2 of the
C*, that was filled (hundreds of thousands) of errors of too many open files
C* 2.0.11
{noformat}$ grep -ic caused by.*too many open file system.log.*
system.log.1:0
system.log.10:18659
system.log.11:17539
system.log.12:18941
system.log.13:18936
system.log.14:18601
system.log.15:18933
system.log.16:18937
system.log.17:18954
system.log.18:18892
system.log.19:18942
system.log.2:0
system.log.20:18977
system.log.21:18977
system.log.22:18852
system.log.23:18978
system.log.24:18978
system.log.25:18978
system.log.26:18978
system.log.27:18978
system.log.28:18978
system.log.29:18978
system.log.3:654
system.log.30:18978
system.log.31:18978
system.log.32:18978
system.log.33:18977
system.log.34:18978
system.log.35:18978
system.log.36:17943
system.log.37:18867
system.log.38:15082
system.log.39:17766
system.log.4:17932
system.log.40:18029
system.log.41:18890
system.log.42:18048
system.log.43:18812
system.log.44:18787
system.log.45:18962
system.log.46:18978
system.log.47:18978
system.log.48:18978
system.log.49:18978
system.log.5:15284
system.log.50:18978
system.log.6:17180
system.log.7:17286
system.log.8:18651
system.log.9:17720
{noformat}
all the logs are from that day..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8821) Errors in JVM_OPTS and cassandra_parms environment vars
[ https://issues.apache.org/jira/browse/CASSANDRA-8821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Shuler updated CASSANDRA-8821: -- Attachment: (was: 8821.txt) Errors in JVM_OPTS and cassandra_parms environment vars --- Key: CASSANDRA-8821 URL: https://issues.apache.org/jira/browse/CASSANDRA-8821 Project: Cassandra Issue Type: Bug Environment: Ubuntu 14.04 LTS amd64 Reporter: Terry Moschou Assignee: Michael Shuler Priority: Minor Fix For: 3.0, 2.0.13, 2.1.4 Attachments: 8821_2.0.txt, 8821_2.1.txt Repos: deb http://www.apache.org/dist/cassandra/debian 21x main deb-src http://www.apache.org/dist/cassandra/debian 21x main The cassandra init script /etc/init.d/cassandra is sourcing the environment file /etc/cassandra/cassandra-env.sh twice. Once directly from the init script, and again inside /usr/sbin/cassandra The result is arguments in JVM_OPTS are duplicated. Further the JVM opt -XX:CMSWaitDuration=1 is defined twice if jvm = 1.7.60. Also, for the environment variable CASSANDRA_CONF used in this context -XX:CompileCommandFile=$CASSANDRA_CONF/hotspot_compiler is undefined when /etc/cassandra/cassandra-env.sh is sourced from the init script. Lastly the variable cassandra_storagedir is undefined in /usr/sbin/cassandra when used in this context -Dcassandra.storagedir=$cassandra_storagedir -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8821) Errors in JVM_OPTS and cassandra_parms environment vars
[ https://issues.apache.org/jira/browse/CASSANDRA-8821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Shuler updated CASSANDRA-8821: -- Attachment: 8821_2.1.txt 8821_2.0.txt Errors in JVM_OPTS and cassandra_parms environment vars --- Key: CASSANDRA-8821 URL: https://issues.apache.org/jira/browse/CASSANDRA-8821 Project: Cassandra Issue Type: Bug Environment: Ubuntu 14.04 LTS amd64 Reporter: Terry Moschou Assignee: Michael Shuler Priority: Minor Fix For: 3.0, 2.0.13, 2.1.4 Attachments: 8821_2.0.txt, 8821_2.1.txt Repos: deb http://www.apache.org/dist/cassandra/debian 21x main deb-src http://www.apache.org/dist/cassandra/debian 21x main The cassandra init script /etc/init.d/cassandra is sourcing the environment file /etc/cassandra/cassandra-env.sh twice. Once directly from the init script, and again inside /usr/sbin/cassandra The result is arguments in JVM_OPTS are duplicated. Further the JVM opt -XX:CMSWaitDuration=1 is defined twice if jvm = 1.7.60. Also, for the environment variable CASSANDRA_CONF used in this context -XX:CompileCommandFile=$CASSANDRA_CONF/hotspot_compiler is undefined when /etc/cassandra/cassandra-env.sh is sourced from the init script. Lastly the variable cassandra_storagedir is undefined in /usr/sbin/cassandra when used in this context -Dcassandra.storagedir=$cassandra_storagedir -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8863) $JVM_EXTRA_OPTS not properly sourced
[
https://issues.apache.org/jira/browse/CASSANDRA-8863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339339#comment-14339339
]
Michael Shuler edited comment on CASSANDRA-8863 at 2/26/15 10:57 PM:
-
That line is in cassandra-env.sh to append JVM_EXTRA_OPTS that were exported in
a shell, or you could actually add a {{JVM_EXTRA_OPTS=-D..., -D...}} line in
cassandra-env.sh. /etc/default/cassandra is not the correct place to add JVM
options.
FYI, I'm removing the export of options added to /etc/default/cassandra in 8821
and basically commenting the same - cassandra-env.sh is the configuration file
to modify or add JVM options.
was (Author: mshuler):
That line is in cassandra-env.sh to append JVM_EXTRA_OPTS that were exported in
a shell, or you could actually add a {{JVM_EXTRA_OPTS=-D..., -D... line in
cassandra-env.sh. /etc/default/cassandra is not the correct place to add JVM
options.
FYI, I'm removing the export of options added to /etc/default/cassandra in 8821
and basically commenting the same - cassandra-env.sh is the configuration file
to modify or add JVM options.
$JVM_EXTRA_OPTS not properly sourced
Key: CASSANDRA-8863
URL: https://issues.apache.org/jira/browse/CASSANDRA-8863
Project: Cassandra
Issue Type: Bug
Components: Config
Reporter: Pablo RUTH
Assignee: Michael Shuler
Priority: Minor
Attachments: jvm-extra-opts.patch
JVM options set in /etc/default/cassandra whit $JVM_EXTRA_OPTS are not
properly sourced. The default file is sourced in the init script with the
cassandra-env.sh. But in /usr/sbin/cassandra only cassandra-env.sh is
sourced, not default so $JVM_EXTRA_OPTS are not applied to cassandra process.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8821) Errors in JVM_OPTS and cassandra_parms environment vars
[ https://issues.apache.org/jira/browse/CASSANDRA-8821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Shuler updated CASSANDRA-8821: -- Attachment: 8821.txt Errors in JVM_OPTS and cassandra_parms environment vars --- Key: CASSANDRA-8821 URL: https://issues.apache.org/jira/browse/CASSANDRA-8821 Project: Cassandra Issue Type: Bug Environment: Ubuntu 14.04 LTS amd64 Reporter: Terry Moschou Assignee: Michael Shuler Priority: Minor Fix For: 3.0, 2.0.13, 2.1.4 Attachments: 8821.txt Repos: deb http://www.apache.org/dist/cassandra/debian 21x main deb-src http://www.apache.org/dist/cassandra/debian 21x main The cassandra init script /etc/init.d/cassandra is sourcing the environment file /etc/cassandra/cassandra-env.sh twice. Once directly from the init script, and again inside /usr/sbin/cassandra The result is arguments in JVM_OPTS are duplicated. Further the JVM opt -XX:CMSWaitDuration=1 is defined twice if jvm = 1.7.60. Also, for the environment variable CASSANDRA_CONF used in this context -XX:CompileCommandFile=$CASSANDRA_CONF/hotspot_compiler is undefined when /etc/cassandra/cassandra-env.sh is sourced from the init script. Lastly the variable cassandra_storagedir is undefined in /usr/sbin/cassandra when used in this context -Dcassandra.storagedir=$cassandra_storagedir -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8836) Factor out CRC32Ex into a separate maven dependency
[ https://issues.apache.org/jira/browse/CASSANDRA-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339438#comment-14339438 ] T Jake Luciani commented on CASSANDRA-8836: --- Patch here https://github.com/tjake/cassandra/tree/8836-cec32ex Factor out CRC32Ex into a separate maven dependency --- Key: CASSANDRA-8836 URL: https://issues.apache.org/jira/browse/CASSANDRA-8836 Project: Cassandra Issue Type: Improvement Reporter: Ariel Weisberg Assignee: T Jake Luciani Fix For: 3.0 Attachments: 8836.patch The current arrangement works from the CLI, but is inconvenient for developer using Java 7 from an IDE. They have to configure the override class the way build.xml does when compiling. If we refactored http://pastebin.com/Z5NAEhzr and the interface it needs to compile http://pastebin.com/tCEvuETA into a separate maven dependency and removed CRC32Ex from CRC32Factory it wouldn't trip up IDEs. They would just add all the jars under lib and move on with life. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8836) Factor out CRC32Ex into a separate maven dependency
[ https://issues.apache.org/jira/browse/CASSANDRA-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339438#comment-14339438 ] T Jake Luciani edited comment on CASSANDRA-8836 at 2/26/15 11:58 PM: - Patch here https://github.com/tjake/cassandra/tree/8836-cec32ex github project here https://github.com/tjake/crc32ex was (Author: tjake): Patch here https://github.com/tjake/cassandra/tree/8836-cec32ex Factor out CRC32Ex into a separate maven dependency --- Key: CASSANDRA-8836 URL: https://issues.apache.org/jira/browse/CASSANDRA-8836 Project: Cassandra Issue Type: Improvement Reporter: Ariel Weisberg Assignee: T Jake Luciani Fix For: 3.0 Attachments: 8836.patch The current arrangement works from the CLI, but is inconvenient for developer using Java 7 from an IDE. They have to configure the override class the way build.xml does when compiling. If we refactored http://pastebin.com/Z5NAEhzr and the interface it needs to compile http://pastebin.com/tCEvuETA into a separate maven dependency and removed CRC32Ex from CRC32Factory it wouldn't trip up IDEs. They would just add all the jars under lib and move on with life. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-3124) java heap limit for nodetool
[ https://issues.apache.org/jira/browse/CASSANDRA-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339292#comment-14339292 ] Vladimir commented on CASSANDRA-3124: - Today I tested 100 nodes cassandra cluster. `nodetool status` works but `nodetool ring' crashed with Out Of Memory: ./application/lib/apache-cassandra-2.0.10/bin/nodetool ring Note: Ownership information does not include topology; for complete information, specify a keyspace Datacenter: datacenter1 == Exception in thread main java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Unknown Source) at java.util.ArrayList.grow(Unknown Source) at java.util.ArrayList.ensureExplicitCapacity(Unknown Source) at java.util.ArrayList.ensureCapacityInternal(Unknown Source) at java.util.ArrayList.addAll(Unknown Source) at org.apache.cassandra.tools.NodeCmd.printDc(NodeCmd.java:325) at org.apache.cassandra.tools.NodeCmd.printRing(NodeCmd.java:292) at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:1224) I had to set up higher limit, 320MB at the script. Do you really think 32 Mb is enough or something wrong on my side? nodetool ring | wc -l 24842 ring.txt java heap limit for nodetool Key: CASSANDRA-3124 URL: https://issues.apache.org/jira/browse/CASSANDRA-3124 Project: Cassandra Issue Type: Improvement Components: Tools Environment: not important Reporter: Zenek Kraweznik Assignee: Zenek Kraweznik Priority: Minor Fix For: 1.0.1 by defaull (from debian package) # nodetool Error occurred during initialization of VM Could not reserve enough space for object heap Could not create the Java virtual machine. # and: --- /usr/bin/nodetool.old 2011-09-02 14:15:14.228152799 +0200 +++ /usr/bin/nodetool 2011-09-02 14:14:28.745154552 +0200 @@ -55,7 +55,7 @@ ;; esac -$JAVA -cp $CLASSPATH -Dstorage-config=$CASSANDRA_CONF \ +$JAVA -Xmx32m -cp $CLASSPATH -Dstorage-config=$CASSANDRA_CONF \ -Dlog4j.configuration=log4j-tools.properties \ org.apache.cassandra.tools.NodeCmd $@ after every upgrade i had to add limit manually. I think it's good idea to add it by default ;) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8150) Revaluate Default JVM tuning parameters
[ https://issues.apache.org/jira/browse/CASSANDRA-8150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339381#comment-14339381 ] Brandon Williams commented on CASSANDRA-8150: - The next step here is to have someone on [~enigmacurry]'s team do some comparison runs on cstar and proceed with any empirical evidence that provides. Revaluate Default JVM tuning parameters --- Key: CASSANDRA-8150 URL: https://issues.apache.org/jira/browse/CASSANDRA-8150 Project: Cassandra Issue Type: Improvement Components: Config Reporter: Matt Stump Assignee: Brandon Williams Attachments: upload.png It's been found that the old twitter recommendations of 100m per core up to 800m is harmful and should no longer be used. Instead the formula used should be 1/3 or 1/4 max heap with a max of 2G. 1/3 or 1/4 is debatable and I'm open to suggestions. If I were to hazard a guess 1/3 is probably better for releases greater than 2.1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8874) running out of FD, and causing clients hang when dropping a keyspace with many CF with many sstables
Jackson Chung created CASSANDRA-8874:
Summary: running out of FD, and causing clients hang when dropping
a keyspace with many CF with many sstables
Key: CASSANDRA-8874
URL: https://issues.apache.org/jira/browse/CASSANDRA-8874
Project: Cassandra
Issue Type: Bug
Reporter: Jackson Chung
we already set number of file descriptors to 10 for c* usage, and confirmed
that from /proc/$cass_pid/limits
we have 16 nodes, 2 DC, each node stores about 600GB to 1TB data; ec2, i2-2xl
instances, raid0 the 2 disks
we use both hector and datastax drivers, and there are many clients connecting
to the cluster.
1 day we dropped a keyspace (that our app no longer use), which has a good
amount of CFs, with some of them use leveledbcompaction and have some good
amount of sstables... and our app went down. CPU/load avg were high and we
couldn't even ssh to them. We have to force a reboot, and restart 2 of the C*,
that was filled (hundreds of thousands) of errors of too many open files
C* 2.0.11
{noformat}$ grep -ic caused by.*too many open file system.log.*
system.log.1:0
system.log.10:18659
system.log.11:17539
system.log.12:18941
system.log.13:18936
system.log.14:18601
system.log.15:18933
system.log.16:18937
system.log.17:18954
system.log.18:18892
system.log.19:18942
system.log.2:0
system.log.20:18977
system.log.21:18977
system.log.22:18852
system.log.23:18978
system.log.24:18978
system.log.25:18978
system.log.26:18978
system.log.27:18978
system.log.28:18978
system.log.29:18978
system.log.3:654
system.log.30:18978
system.log.31:18978
system.log.32:18978
system.log.33:18977
system.log.34:18978
system.log.35:18978
system.log.36:17943
system.log.37:18867
system.log.38:15082
system.log.39:17766
system.log.4:17932
system.log.40:18029
system.log.41:18890
system.log.42:18048
system.log.43:18812
system.log.44:18787
system.log.45:18962
system.log.46:18978
system.log.47:18978
system.log.48:18978
system.log.49:18978
system.log.5:15284
system.log.50:18978
system.log.6:17180
system.log.7:17286
system.log.8:18651
system.log.9:17720
{noformat}
all the logs are from that day..
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8757) IndexSummaryBuilder should construct itself offheap, and share memory between the result of each build() invocation
[
https://issues.apache.org/jira/browse/CASSANDRA-8757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339391#comment-14339391
]
Ariel Weisberg commented on CASSANDRA-8757:
---
I didn't understand the offset business so the comment probably doesn't provide
the right context. After agonizing for a while I figured out what it meant. It
could be lack of sleep.
It might be clearer if it described the mismatch between in memory and on disk
(in a way I grok). The in-memory representation is a set of offsets into a
separate zero indexed array while the disk based representation is a set of
offsets to entries appended after the offsets section so every offset needs to
be recalculated.
I think serialization point didn't parse for me as being the point in the
file after the offsets.
{quote}
because we serialize/deserialize in native
+// int/long format,
{quote}
And that doesn't seem to be the cause of this mess. It's not the native
int/long formatness of it. It's that the offsets are into array and the two
have to be flattened into one file.
SSTableReader line 747 random semi-colon, IndexSummaryBuilder line 216 extra
semi-colon.
SafeMemoryWriter has no unit test.
Otherwise I am +1
IndexSummaryBuilder should construct itself offheap, and share memory between
the result of each build() invocation
---
Key: CASSANDRA-8757
URL: https://issues.apache.org/jira/browse/CASSANDRA-8757
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Benedict
Assignee: Benedict
Fix For: 2.1.4
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7802) Need to export JVM_OPTS from init.d script
[ https://issues.apache.org/jira/browse/CASSANDRA-7802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339364#comment-14339364 ] Michael Shuler commented on CASSANDRA-7802: --- [~mattrobenolt] just an FYI that this export of the JVM_OPTS is going to be removed from the init script and /etc/default/cassandra is going to get some additional comments that conf/cassandra-env.sh is really the correct place to add/modify your JVM options. Need to export JVM_OPTS from init.d script -- Key: CASSANDRA-7802 URL: https://issues.apache.org/jira/browse/CASSANDRA-7802 Project: Cassandra Issue Type: Bug Components: Packaging Environment: Debian/Ubuntu Reporter: Matt Robenolt Assignee: Matt Robenolt Priority: Minor Labels: patch, qa-resolved Fix For: 2.0.11, 2.1.0 Attachments: 42.diff Since 2.0, the init.d script was refactored and requires JVM variables to be exported for them to actually be picked up and used. In this case, JVM_OPTS never gets exported, so user defined variables from /etc/default/cassandra are never applied. This also affects the latest 2.1 rc, and I assume all previous versions. Pull request: https://github.com/apache/cassandra/pull/42 Diff: https://github.com/apache/cassandra/pull/42.diff Patch: https://github.com/apache/cassandra/pull/42.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8150) Revaluate Default JVM tuning parameters
[ https://issues.apache.org/jira/browse/CASSANDRA-8150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339387#comment-14339387 ] Hans van der Linde commented on CASSANDRA-8150: --- Dear sender, I am on a short break and be back on 03-03-2015. Your email will not be forwarded. Best regards, Hans van der Linde - ATTENTION: The information in this electronic mail message is private and confidential, and only intended for the addressee. Should you receive this message by mistake, you are hereby notified that any disclosure, reproduction, distribution or use of this message is strictly prohibited. Please inform the sender by reply transmission and delete the message without copying or opening it. Messages and attachments are scanned for all viruses known. If this message contains password-protected attachments, the files have NOT been scanned for viruses by the ING mail domain. Always scan attachments before opening them. - Revaluate Default JVM tuning parameters --- Key: CASSANDRA-8150 URL: https://issues.apache.org/jira/browse/CASSANDRA-8150 Project: Cassandra Issue Type: Improvement Components: Config Reporter: Matt Stump Assignee: Brandon Williams Attachments: upload.png It's been found that the old twitter recommendations of 100m per core up to 800m is harmful and should no longer be used. Instead the formula used should be 1/3 or 1/4 max heap with a max of 2G. 1/3 or 1/4 is debatable and I'm open to suggestions. If I were to hazard a guess 1/3 is probably better for releases greater than 2.1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8821) Errors in JVM_OPTS and cassandra_parms environment vars
[ https://issues.apache.org/jira/browse/CASSANDRA-8821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Shuler updated CASSANDRA-8821: -- Fix Version/s: 2.0.13 3.0 Errors in JVM_OPTS and cassandra_parms environment vars --- Key: CASSANDRA-8821 URL: https://issues.apache.org/jira/browse/CASSANDRA-8821 Project: Cassandra Issue Type: Bug Environment: Ubuntu 14.04 LTS amd64 Reporter: Terry Moschou Assignee: Michael Shuler Priority: Minor Fix For: 3.0, 2.0.13, 2.1.4 Repos: deb http://www.apache.org/dist/cassandra/debian 21x main deb-src http://www.apache.org/dist/cassandra/debian 21x main The cassandra init script /etc/init.d/cassandra is sourcing the environment file /etc/cassandra/cassandra-env.sh twice. Once directly from the init script, and again inside /usr/sbin/cassandra The result is arguments in JVM_OPTS are duplicated. Further the JVM opt -XX:CMSWaitDuration=1 is defined twice if jvm = 1.7.60. Also, for the environment variable CASSANDRA_CONF used in this context -XX:CompileCommandFile=$CASSANDRA_CONF/hotspot_compiler is undefined when /etc/cassandra/cassandra-env.sh is sourced from the init script. Lastly the variable cassandra_storagedir is undefined in /usr/sbin/cassandra when used in this context -Dcassandra.storagedir=$cassandra_storagedir -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8516) NEW_NODE topology event emitted instead of MOVED_NODE in a one node cluster
[
https://issues.apache.org/jira/browse/CASSANDRA-8516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339489#comment-14339489
]
Stefania commented on CASSANDRA-8516:
-
At [~thobbs], I think that the problematic call to {{updateNormalTokens}} is in
SS.{{setTokens()}}:
Is it possible to rely on {{StorageService.operatingMode}} ({{Mode.MOVING}})
instead of or along with the moving tokens? I works if we call move on the
local node but does it get called in other contexts too (e.g. Gossip)?
Also, I reproduced the problem with {{nodetool move 123}} but I kept on getting
the exception below, which I could only bypass it by commenting it out:
{code}
if (getTokenMetadata().getTokens(localAddress).size() 1)
{
logger.error(Invalid request to move(Token); This node has more than
one token and cannot be moved thusly.);
throw new UnsupportedOperationException(This node has more than one
token and cannot be moved thusly.);
}
{code}
Could I have the correct command or config to reproduce this issue on 2.1 along
with any other suggested tests (single node / multiple node clusters), and do
we need to test drivers too or is it enough to check that we call {{onMove}}
rather than {{onJoinCluster}} in SS.{{handleStateNormal}}.
Thanks!
NEW_NODE topology event emitted instead of MOVED_NODE in a one node cluster
---
Key: CASSANDRA-8516
URL: https://issues.apache.org/jira/browse/CASSANDRA-8516
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Tyler Hobbs
Assignee: Stefania
Priority: Minor
Fix For: 2.0.13
As discovered in CASSANDRA-8373, when you move a node in a single-node
cluster, a {{NEW_NODE}} event is generated instead of a {{MOVED_NODE}} event.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8516) NEW_NODE topology event emitted instead of MOVED_NODE in a one node cluster
[
https://issues.apache.org/jira/browse/CASSANDRA-8516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339560#comment-14339560
]
Brandon Williams commented on CASSANDRA-8516:
-
bq. Also, I reproduced the problem with nodetool move 123 but I kept on getting
the exception below
You need to set num_tokens: 1 in the yaml, vnodes don't support move.
NEW_NODE topology event emitted instead of MOVED_NODE in a one node cluster
---
Key: CASSANDRA-8516
URL: https://issues.apache.org/jira/browse/CASSANDRA-8516
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Tyler Hobbs
Assignee: Stefania
Priority: Minor
Fix For: 2.0.13
As discovered in CASSANDRA-8373, when you move a node in a single-node
cluster, a {{NEW_NODE}} event is generated instead of a {{MOVED_NODE}} event.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6246) EPaxos
[ https://issues.apache.org/jira/browse/CASSANDRA-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339572#comment-14339572 ] sankalp kohli commented on CASSANDRA-6246: -- Let me take a look EPaxos -- Key: CASSANDRA-6246 URL: https://issues.apache.org/jira/browse/CASSANDRA-6246 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Jonathan Ellis Assignee: Blake Eggleston Priority: Minor One reason we haven't optimized our Paxos implementation with Multi-paxos is that Multi-paxos requires leader election and hence, a period of unavailability when the leader dies. EPaxos is a Paxos variant that requires (1) less messages than multi-paxos, (2) is particularly useful across multiple datacenters, and (3) allows any node to act as coordinator: http://sigops.org/sosp/sosp13/papers/p358-moraru.pdf However, there is substantial additional complexity involved if we choose to implement it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8801) Decommissioned nodes are willing to rejoin the cluster if restarted
[
https://issues.apache.org/jira/browse/CASSANDRA-8801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339724#comment-14339724
]
Michael Shuler commented on CASSANDRA-8801:
---
I think CASSANDRA-5780 could be resolved at the same time.
Decommissioned nodes are willing to rejoin the cluster if restarted
---
Key: CASSANDRA-8801
URL: https://issues.apache.org/jira/browse/CASSANDRA-8801
Project: Cassandra
Issue Type: Bug
Components: Core
Reporter: Eric Stevens
Assignee: Brandon Williams
Fix For: 3.0
This issue comes from the Cassandra user group.
If a node which was successfully decommissioned gets restarted with its data
directory in tact, it will rejoin the cluster immediately going to {{UN}} and
beginning to serve client requests.
This is wrong - the node has consistency issues, having missed any writes
while it was offline because no hinted handoffs were being kept. And in the
best case scenario (it's spotted and remediated immediately), near-100%
overstreaming will still occur.
Also, whatever reasons the operator had for decommissioning the node would
presumably still be valid, so this action may threaten cluster stability if
the node is underpowered or suffering hardware issues.
But what elevates this to critical is that if the node had been offline
longer than gc_grace_seconds, it may cause permanent and unrecoverable
consistency issues due to data resurrection.
h3. Recommendation:
A node should remember that it was decommissioned and refuse to rejoin a
cluster without at least a -Dflag forcing it to.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8860) Too many java.util.HashMap$Entry objects in heap
[ https://issues.apache.org/jira/browse/CASSANDRA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339580#comment-14339580 ] Benedict edited comment on CASSANDRA-8860 at 2/27/15 2:10 AM: -- One way to do this might be, for each reader: * find its overlaps; ** for each overlap, find _its_ set of overlaps, but discard these unless they're both larger than the one we've most recently kept, _and_ the estimated win is sufficient * remove all of the (first layer of) overlaps from the readers we visit, and continue with the next remaining reader This gives us linear space complexity, but possibly still quadratic time complexity in the worst case, which is trickier to improve. was (Author: benedict): One way to do this might be, for each table: * find the overlaps; ** for each overlap, find _its_ set of overlaps, but keep only the largest where the estimated win is sufficient * remove all of the (first layer of) overlaps from the files we visit This gives us linear space complexity, but possibly still quadratic time complexity in the worst case, which is trickier to improve. Too many java.util.HashMap$Entry objects in heap Key: CASSANDRA-8860 URL: https://issues.apache.org/jira/browse/CASSANDRA-8860 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.1.3, jdk 1.7u51 Reporter: Phil Yang Assignee: Marcus Eriksson Fix For: 2.1.4 Attachments: cassandra-env.sh, cassandra.yaml, jmap.txt, jstack.txt While I upgrading my cluster to 2.1.3, I find some nodes (not all) may have GC issue after the node restarting successfully. Old gen grows very fast and most of the space can not be recycled after setting its status to normal immediately. The qps of both reading and writing are very low and there is no heavy compaction. Jmap result seems strange that there are too many java.util.HashMap$Entry objects in heap, where in my experience the [B is usually the No1. If I downgrade it to 2.1.1, this issue will not appear. I uploaded conf files and jstack/jmap outputs. I'll upload heap dump if someone need it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8860) Too many java.util.HashMap$Entry objects in heap
[ https://issues.apache.org/jira/browse/CASSANDRA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339580#comment-14339580 ] Benedict commented on CASSANDRA-8860: - One way to do this might be, for each table: * find the overlaps; ** for each overlap, find _its_ set of overlaps, but keep only the largest where the estimated win is sufficient * remove all of the (first layer of) overlaps from the files we visit This gives us linear space complexity, but possibly still quadratic time complexity in the worst case, which is trickier to improve. Too many java.util.HashMap$Entry objects in heap Key: CASSANDRA-8860 URL: https://issues.apache.org/jira/browse/CASSANDRA-8860 Project: Cassandra Issue Type: Bug Environment: Cassandra 2.1.3, jdk 1.7u51 Reporter: Phil Yang Assignee: Marcus Eriksson Fix For: 2.1.4 Attachments: cassandra-env.sh, cassandra.yaml, jmap.txt, jstack.txt While I upgrading my cluster to 2.1.3, I find some nodes (not all) may have GC issue after the node restarting successfully. Old gen grows very fast and most of the space can not be recycled after setting its status to normal immediately. The qps of both reading and writing are very low and there is no heavy compaction. Jmap result seems strange that there are too many java.util.HashMap$Entry objects in heap, where in my experience the [B is usually the No1. If I downgrade it to 2.1.1, this issue will not appear. I uploaded conf files and jstack/jmap outputs. I'll upload heap dump if someone need it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
