[jira] [Resolved] (CASSANDRA-8567) Functions like length() and trim() on cqlsh table fields
[ https://issues.apache.org/jira/browse/CASSANDRA-8567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylvain Lebresne resolved CASSANDRA-8567. - Resolution: Won't Fix We might add some {{length}} and {{trim}} method at some point, though even if we don't, UDF in 3.0 will allow you to define them. But what you're really asking here is to be able to use functions in {{ORDER BY}}, and that is not going to happen. The reason is that we have no better strategy server side than to read everything and sort it in memory afterwards before returning to the client. Which 1) is no faster than letting you the client do the sorting client side and 2) doesn't work with paging at all and thus makes it likely to OOM as soon as the amount of data to order is not small. For those reason, we prefer letting clients do the sorting themselves post-query if they wants to (but we understand that for cqlsh it's not as nice as you'd like. Though if you really need quick and dirty post-query sorting, piping the output of cqlsh into sort (the unix utility) isn't terribly hard). Alternatively, if you need results in that sort order often, you should store the data in a table whose clustering is {{length(trim(field)}}. > Functions like length() and trim() on cqlsh table fields > > > Key: CASSANDRA-8567 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8567 > Project: Cassandra > Issue Type: Wish > Components: Core >Reporter: Rekha Joshi > > It would be nice to be able to order by length of field values. > A function like length(field) and trim(field) on cqlsh > To enable do something like say - select * from where > = order by length(trim()) desc; > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[ https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265852#comment-14265852 ] Sylvain Lebresne commented on CASSANDRA-8325: - For the sake of clarification, being from DataStax doesn't entitled anyone to make calls regarding Apache Cassandra (having a good track record of contributions, which usually means you're a committer, is probably what entitled you to make calls, though decisions are ideally agreed upon collectively). > Cassandra 2.1.x fails to start on FreeBSD (JVM crash) > - > > Key: CASSANDRA-8325 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8325 > Project: Cassandra > Issue Type: Bug > Environment: FreeBSD 10.0 with openjdk version "1.7.0_71", 64-Bit > Server VM >Reporter: Leonid Shalupov > Attachments: hs_err_pid1856.log, system.log, unsafeCopy1.txt, > untested_8325.patch > > > See attached error file after JVM crash > {quote} > FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu > Jan 16 22:34:59 UTC 2014 > r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 > {quote} > {quote} > % java -version > openjdk version "1.7.0_71" > OpenJDK Runtime Environment (build 1.7.0_71-b14) > OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode) > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8547) Make RangeTombstone.Tracker.isDeleted() faster
[ https://issues.apache.org/jira/browse/CASSANDRA-8547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-8547: Reviewer: Sylvain Lebresne (was: Benedict) > Make RangeTombstone.Tracker.isDeleted() faster > -- > > Key: CASSANDRA-8547 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8547 > Project: Cassandra > Issue Type: Improvement > Components: Core > Environment: 2.0.11 >Reporter: Dominic Letz >Assignee: Dominic Letz > Fix For: 2.1.3 > > Attachments: Selection_044.png, cassandra-2.0.11-8547.txt, > cassandra-2.1-8547.txt, rangetombstone.tracker.txt > > > During compaction and repairs with many tombstones an exorbitant amount of > time is spend in RangeTombstone.Tracker.isDeleted(). > The amount of time spend there can be so big that compactions and repairs > look "stalled" and the time remaining time estimated frozen at the same value > for days. > Using visualvm I've been sample profiling the code during execution and both > in Compaction as well as during repairs found this. (point in time backtraces > attached) > Looking at the code the problem is obviously the linear scanning: > {code} > public boolean isDeleted(Column column) > { > for (RangeTombstone tombstone : ranges) > { > if (comparator.compare(column.name(), tombstone.min) >= 0 > && comparator.compare(column.name(), tombstone.max) <= 0 > && tombstone.maxTimestamp() >= column.timestamp()) > { > return true; > } > } > return false; > } > {code} > I would like to propose to change this and instead use a sorted list (e.g. > RangeTombstoneList) here instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8547) Make RangeTombstone.Tracker.isDeleted() faster
[ https://issues.apache.org/jira/browse/CASSANDRA-8547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265891#comment-14265891 ] Benedict commented on CASSANDRA-8547: - This looks like something [~slebresne] should take a look at. > Make RangeTombstone.Tracker.isDeleted() faster > -- > > Key: CASSANDRA-8547 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8547 > Project: Cassandra > Issue Type: Improvement > Components: Core > Environment: 2.0.11 >Reporter: Dominic Letz >Assignee: Dominic Letz > Fix For: 2.1.3 > > Attachments: Selection_044.png, cassandra-2.0.11-8547.txt, > cassandra-2.1-8547.txt, rangetombstone.tracker.txt > > > During compaction and repairs with many tombstones an exorbitant amount of > time is spend in RangeTombstone.Tracker.isDeleted(). > The amount of time spend there can be so big that compactions and repairs > look "stalled" and the time remaining time estimated frozen at the same value > for days. > Using visualvm I've been sample profiling the code during execution and both > in Compaction as well as during repairs found this. (point in time backtraces > attached) > Looking at the code the problem is obviously the linear scanning: > {code} > public boolean isDeleted(Column column) > { > for (RangeTombstone tombstone : ranges) > { > if (comparator.compare(column.name(), tombstone.min) >= 0 > && comparator.compare(column.name(), tombstone.max) <= 0 > && tombstone.maxTimestamp() >= column.timestamp()) > { > return true; > } > } > return false; > } > {code} > I would like to propose to change this and instead use a sorted list (e.g. > RangeTombstoneList) here instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8552) Large compactions run out of off-heap RAM
[ https://issues.apache.org/jira/browse/CASSANDRA-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265898#comment-14265898 ] Benedict commented on CASSANDRA-8552: - [~philipthompson] could we try to reproduce ourselves, since Brent has navigated around the problem for the moment? It would be great to figure this out and fix for 2.1.3 > Large compactions run out of off-heap RAM > - > > Key: CASSANDRA-8552 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8552 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 14.4 > AWS EC2 > 12 m1.xlarge nodes [4 cores, 16GB RAM, 1TB storage (251GB Used)] > Java build 1.7.0_55-b13 and build 1.8.0_25-b17 >Reporter: Brent Haines >Assignee: Benedict >Priority: Blocker > Fix For: 2.1.3 > > Attachments: Screen Shot 2015-01-02 at 9.36.11 PM.png, fhandles.log, > freelog.log, lsof.txt, meminfo.txt, sysctl.txt, system.log > > > We have a large table of storing, effectively event logs and a pair of > denormalized tables for indexing. > When updating from 2.0 to 2.1 we saw performance improvements, but some > random and silent crashes during nightly repairs. We lost a node (totally > corrupted) and replaced it. That node has never stabilized -- it simply can't > finish the compactions. > Smaller compactions finish. Larger compactions, like these two never finish - > {code} > pending tasks: 48 >compaction type keyspace table completed total > unit progress > Compaction data stories 16532973358 75977993784 > bytes 21.76% > Compaction data stories_by_text 10593780658 38555048812 > bytes 27.48% > Active compaction remaining time : 0h10m51s > {code} > We are not getting exceptions and are not running out of heap space. The > Ubuntu OOM killer is reaping the process after all of the memory is consumed. > We watch memory in the opscenter console and it will grow. If we turn off the > OOM killer for the process, it will run until everything else is killed > instead and then the kernel panics. > We have the following settings configured: > 2G Heap > 512M New > {code} > memtable_heap_space_in_mb: 1024 > memtable_offheap_space_in_mb: 1024 > memtable_allocation_type: heap_buffers > commitlog_total_space_in_mb: 2048 > concurrent_compactors: 1 > compaction_throughput_mb_per_sec: 128 > {code} > The compaction strategy is leveled (these are read-intensive tables that are > rarely updated) > I have tried every setting, every option and I have the system where the MTBF > is about an hour now, but we never finish compacting because there are some > large compactions pending. None of the GC tools or settings help because it > is not a GC problem. It is an off-heap memory problem. > We are getting these messages in our syslog > {code} > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219527] BUG: Bad page map in > process java pte:0320 pmd:2d6fa5067 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219545] addr:7fb820be3000 > vm_flags:0870 anon_vma: (null) mapping: (null) > index:7fb820be3 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219556] CPU: 3 PID: 27344 > Comm: java Tainted: GB3.13.0-24-generic #47-Ubuntu > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219559] 880028510e40 > 88020d43da98 81715ac4 7fb820be3000 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219565] 88020d43dae0 > 81174183 0320 0007fb820be3 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219568] 8802d6fa5f18 > 0320 7fb820be3000 7fb820be4000 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219572] Call Trace: > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219584] [] > dump_stack+0x45/0x56 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219591] [] > print_bad_pte+0x1a3/0x250 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219594] [] > vm_normal_page+0x69/0x80 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219598] [] > unmap_page_range+0x3bb/0x7f0 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219602] [] > unmap_single_vma+0x81/0xf0 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219605] [] > unmap_vmas+0x49/0x90 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219610] [] > exit_mmap+0x9c/0x170 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219617] [] > ? __delayacct_add_tsk+0x153/0x170 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219621] [] > mmput+0x5c/0x120 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219625] [] > do_exit+0x26c/0xa50 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219631] [] > ? __unqueue_futex+0x31/0x60 > Jan 2 07:06:00 ip-1
[jira] [Commented] (CASSANDRA-6246) EPaxos
[ https://issues.apache.org/jira/browse/CASSANDRA-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265909#comment-14265909 ] sankalp kohli commented on CASSANDRA-6246: -- I am a little confused as in how you will use epoch to make sure instances are executed on all replicas when incrementing? > EPaxos > -- > > Key: CASSANDRA-6246 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6246 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jonathan Ellis >Assignee: Blake Eggleston >Priority: Minor > > One reason we haven't optimized our Paxos implementation with Multi-paxos is > that Multi-paxos requires leader election and hence, a period of > unavailability when the leader dies. > EPaxos is a Paxos variant that requires (1) less messages than multi-paxos, > (2) is particularly useful across multiple datacenters, and (3) allows any > node to act as coordinator: > http://sigops.org/sosp/sosp13/papers/p358-moraru.pdf > However, there is substantial additional complexity involved if we choose to > implement it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[ https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265911#comment-14265911 ] Benedict commented on CASSANDRA-8325: - [~graham sanderson] thanks for the New Year cheer and putting together another great patch. My slight concern with this approach is that it pollutes the instruction cache, so we do need to run some performance tests. These codepaths _are_ pretty sensitive to these effects. Ultimately, I think a follow up ticket might be sensible that performs all of this work natively, since we should not need such code duplication. > Cassandra 2.1.x fails to start on FreeBSD (JVM crash) > - > > Key: CASSANDRA-8325 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8325 > Project: Cassandra > Issue Type: Bug > Environment: FreeBSD 10.0 with openjdk version "1.7.0_71", 64-Bit > Server VM >Reporter: Leonid Shalupov > Attachments: hs_err_pid1856.log, system.log, unsafeCopy1.txt, > untested_8325.patch > > > See attached error file after JVM crash > {quote} > FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu > Jan 16 22:34:59 UTC 2014 > r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 > {quote} > {quote} > % java -version > openjdk version "1.7.0_71" > OpenJDK Runtime Environment (build 1.7.0_71-b14) > OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode) > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8546) RangeTombstoneList becoming bottleneck on tombstone heavy tasks
[ https://issues.apache.org/jira/browse/CASSANDRA-8546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-8546: Assignee: (was: Benedict) > RangeTombstoneList becoming bottleneck on tombstone heavy tasks > --- > > Key: CASSANDRA-8546 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8546 > Project: Cassandra > Issue Type: Improvement > Components: Core > Environment: 2.0.11 / 2.1 >Reporter: Dominic Letz > Fix For: 2.1.3 > > Attachments: cassandra-2.0.11-8546.txt, cassandra-2.1-8546.txt, > rangetombstonelist_compaction.png, rangetombstonelist_mutation.png, > rangetombstonelist_read.png, tombstone_test.tgz > > > I would like to propose a change of the data structure used in the > RangeTombstoneList to store and insert tombstone ranges to something with at > least O(log N) insert in the middle and at near O(1) and start AND end. Here > is why: > When having tombstone heavy work-loads the current implementation of > RangeTombstoneList becomes a bottleneck with slice queries. > Scanning the number of tombstones up to the default maximum (100k) can take > up to 3 minutes of how addInternal() scales on insertion of middle and start > elements. > The attached test shows that with 50k deletes from both sides of a range. > INSERT 1...11 > flush() > DELETE 1...5 > DELETE 11...6 > While one direction performs ok (~400ms on my notebook): > {code} > SELECT * FROM timeseries WHERE name = 'a' ORDER BY timestamp DESC LIMIT 1 > {code} > The other direction underperforms (~7seconds on my notebook) > {code} > SELECT * FROM timeseries WHERE name = 'a' ORDER BY timestamp ASC LIMIT 1 > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8568) Impose new API on data tracker modifications that makes correct usage obvious and imposes safety
Benedict created CASSANDRA-8568: --- Summary: Impose new API on data tracker modifications that makes correct usage obvious and imposes safety Key: CASSANDRA-8568 URL: https://issues.apache.org/jira/browse/CASSANDRA-8568 Project: Cassandra Issue Type: Bug Reporter: Benedict DataTracker has become a bit of a quagmire, and not at all obvious to interface with, with many subtly different modifiers. I suspect it is still subtly broken, especially around error recovery. I propose piggy-backing on CASSANDRA-7705 to offer RAII (and GC-enforced, for those situations where a try/finally block isn't possible) objects that have transactional behaviour, and with few simple declarative methods that can be composed simply to provide all of the functionality we currently need. See CASSANDRA-8399 for context -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-8506) Improve management of DataTracker, esp. compacting
[ https://issues.apache.org/jira/browse/CASSANDRA-8506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict resolved CASSANDRA-8506. - Resolution: Duplicate > Improve management of DataTracker, esp. compacting > -- > > Key: CASSANDRA-8506 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8506 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Benedict > Fix For: 3.0 > > > Building on CASSANDRA-7705, we can use debuggable ref counting to manage the > marking and unmarking of compaction state, so that we can quickly track down > errors. > We should also simplify the logic wrt rewriters, by ignoring the descriptor > type, perhaps for all sets. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-7438) Serializing Row cache alternative (Fully off heap)
[ https://issues.apache.org/jira/browse/CASSANDRA-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14257721#comment-14257721 ] Robert Stupp edited comment on CASSANDRA-7438 at 1/6/15 10:08 AM: -- I had the opportunity to test OHC on a big machine. First: it works - very happy about that :) Some things I want to notice: * high number of segments do not have any really measurable influence (default of 2* # of cores is fine) * throughput heavily depends on serialization (hash entry size) - Java8 gave about 10% to 15% improvement in some tests (either on {{Unsafe.copyMemory}} or something related like JNI barrier) * the number of entries per bucket stays pretty low with the default load factor of .75 - vast majority has 0 or 1 entries, some 2 or 3 and few up to 8 Issue (not solvable yet): It works great for hash entries to approx. 64kB with good to great throughput. Above that barrier it first works good but after some time the system spends a huge amount of CPU time (~95%) in {{malloc()}} / {{free()}} (with jemalloc, Unsafe.allocate is not worth discussing at all on Linux). I tried to add some „memory buffer cache“ that caches free’d hash entries for reuse. But it turned out that in the end it would be too complex if done right. The current implementation is still in the code, but must be explicitly enabled with a system property. Workloads with small entries and high number of threads easily trigger Linux OOM protection (that kills the process). Please note that it works with large hash entries - but throughput drops dramatically to just a few thousand writes per second. Some numbers (value sizes have gaussian distribution). Had to do these tests in a hurry because I had to give back the machine. Code used during these tests is tagged as {{0.1-SNAP-Bench}} in git. Throughput is limited by {{malloc()}} / {{free()}} and most tests did only use 50% of available CPU capacity (on _c3.8xlarge_ - 32 cores, Intel Xeon E5-2680v2 @2.8GHz, 64GB). * -1k..200k value size, 32 threads, 1M keys, 90% read ratio, 32GB: 22k writes/sec, 200k reads/sec, ~8k evictions/sec, write: 8ms (99perc), read: 3ms(99perc)- * -1k..64k value size, 500 threads, 1M keys, 90% read ratio, 32GB: 55k writes/sec, 499k reads/sec, ~2k evictions/sec, write: .1ms (99perc), read: .03ms(99perc)- * -1k..64k value size, 500 threads, 1M keys, 50% read ratio, 32GB: 195k writes/sec, 195k reads/sec, ~9k evictions/sec, write: .2ms (99perc), read: .1ms(99perc)- * -1k..64k value size, 500 threads, 1M keys, 10% read ratio, 32GB: 185k writes/sec, 20k reads/sec, ~7k evictions/sec, write: 4ms (99perc), read: .07ms(99perc)- * -1k..16k value size, 500 threads, 5M keys, 90% read ratio, 32GB: 110k writes/sec, 1M reads/sec, 30k evictions/sec, write: .04ms (99perc), read: .01ms(99perc)- * -1k..16k value size, 500 threads, 5M keys, 50% read ratio, 32GB: 420k writes/sec, 420k reads/sec, 125k evictions/sec, write: .06ms (99perc), read: .01ms(99perc)- * -1k..16k value size, 500 threads, 5M keys, 10% read ratio, 32GB: 435k writes/sec, 48k reads/sec, 130k evictions/sec, write: .06ms (99perc), read: .01ms(99perc)- * -1k..4k value size, 500 threads, 20M keys, 90% read ratio, 32GB: 140k writes/sec, 1.25M reads/sec, 50k evictions/sec, write: .02ms (99perc), read: .005ms(99perc)- * -1k..4k value size, 500 threads, 20M keys, 50% read ratio, 32GB: 530k writes/sec, 530k reads/sec, 220k evictions/sec, write: .04ms (99perc), read: .005ms(99perc)- * -1k..4k value size, 500 threads, 20M keys, 10% read ratio, 32GB: 665k writes/sec, 74k reads/sec, 250k evcictions/sec, write: .04ms (99perc), read: .005ms(99perc)- Command line to execute the benchmark: {code} java -jar ohc-benchmark/target/ohc-benchmark-0.1-SNAPSHOT.jar -rkd 'uniform(1..2000)' -wkd 'uniform(1..2000)' -vs 'gaussian(1024..4096,2)' -r .1 -cap 320 -d 86400 -t 500 -dr 8 -r = read rate -d = duration -t = # of threads -dr = # of driver threads that feed the worker threads -rkd = read key distribution -wkd = write key distribution -vs = value size -cap = capacity {code} Sample bucket histogram from 20M test: {code} [0..0]: 8118604 [1..1]: 5892298 [2..2]: 2138308 [3..3]: 518089 [4..4]: 94441 [5..5]: 13672 [6..6]: 1599 [7..7]: 189 [8..9]: 16 {code} After trapping into that memory management issue with varying allocation sized of some few kB to several MB, I think that it’s still worth to work on an own off-heap memory management. Maybe some block-based approach (fixed or variable). But that’s out of the scope of this ticket. EDIT: The problem with high system-CPU usage only persists on systems with multiple CPUs. Cross check with the second CPU socket disabled - calling the benchmark with {{taskset 0x3ff java -jar ...}} does not show 95% system CPU usage. EDIT2: Marked benchmark values as invalid (see my comment on 01/
[jira] [Updated] (CASSANDRA-8194) Reading from Auth table should not be in the request path
[ https://issues.apache.org/jira/browse/CASSANDRA-8194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sankalp kohli updated CASSANDRA-8194: - Assignee: Vishy Kasar > Reading from Auth table should not be in the request path > - > > Key: CASSANDRA-8194 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8194 > Project: Cassandra > Issue Type: Improvement >Reporter: Vishy Kasar >Assignee: Vishy Kasar >Priority: Minor > Fix For: 2.0.12, 3.0 > > Attachments: 8194-V2.patch, 8194.patch, CacheTest2.java > > > We use PasswordAuthenticator and PasswordAuthorizer. The system_auth has a RF > of 10 per DC over 2 DCs. The permissions_validity_in_ms is 5 minutes. > We still have few thousand requests failing each day with the trace below. > The reason for this is read cache request realizing that cached entry has > expired and doing a blocking request to refresh cache. > We should have cache refreshed periodically only in the back ground. The user > request should simply look at the cache and not try to refresh it. > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 0 responses. > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) > at com.google.common.cache.LocalCache.get(LocalCache.java:3990) > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:292) > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172) > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165) > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149) > at > org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:75) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:102) > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:113) > at > org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1735) > at > org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4162) > at > org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4150) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) > at > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:206) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 0 responses. > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:256) > at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84) > at > org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) > at > org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:68) > at org.apache.cassandra.service.ClientState$1.load(ClientState.java:278) > at org.apache.cassandra.service.ClientState$1.load(ClientState.java:275) > at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) > ... 19 more > Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation > timed out - received only 0 responses. > at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) > at > org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:943) > at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:828) > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:140) > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:245) > ... 28 more > ERROR [Thrift:17232] 2014-10-24 05:06:51,004 CustomTThreadPoolServer.java > (line 224) Error occurred during pro
[jira] [Commented] (CASSANDRA-7438) Serializing Row cache alternative (Fully off heap)
[ https://issues.apache.org/jira/browse/CASSANDRA-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265947#comment-14265947 ] Robert Stupp commented on CASSANDRA-7438: - The latest (just checked in) benchmark implementation gives much better results. Using {{com.codahale.metrics.Timer#time(java.util.concurrent.Callable)}} eliminates use of {{System.nanoTime()}} or {{ThreadMXBean.getCurrentThreadCpuTime()}} - it can directly use its internal clock. The benchmark {{java -jar ohc-benchmark/target/ohc-benchmark-0.2-SNAPSHOT.jar -rkd 'gaussian(1..2000,2)' -wkd 'gaussian(1..2000,2)' -vs 'gaussian(1024..4096,2)' -r .9 -cap 16 -d 30 -t 30}} improved from 800k reads to 3.3M reads per second w/ 8 cores). So yes - benchmark was measuring its own mad code. Due to that I edited my previous comment with the benchmark results since those are invalid now. I've added a (yet simple) JMH benchmark as a separate module. This one can cause high system CPU usage - at operation rates of 2M per second or more (8 cores). I think these rates are really fine. Note: these rates cannot be achieved in production since then you'll obviously have to pay for (de)serialization, too. So we want to address these topics as follow-up: * own off-heap allocator * C* ability to access off-heap cached rows * C* ability to serialize hot keys directly from off-heap (might be a minor win since it's triggered not that often) * per-table knob to control whether to add to row-cache on writes -- I strongly believe that this is a useful feature (maybe LHF) on workloads where read and written data work on different (row} keys. * investigate if counter-cache can benefit * investigate if key-cache can benefit bq. You could start with it outside and publish to maven central and if there an issue getting patches applied quickly we can always fork it in C*. OK bq. pluggable row cache Then I'll start with that - just make row-cache pluggable and the implementation configurable. Note: JNA has a synchronized block that's executed at every call - version 4.2.0 fixes this (don't know when it will be released). > Serializing Row cache alternative (Fully off heap) > -- > > Key: CASSANDRA-7438 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7438 > Project: Cassandra > Issue Type: Improvement > Components: Core > Environment: Linux >Reporter: Vijay >Assignee: Vijay > Labels: performance > Fix For: 3.0 > > Attachments: 0001-CASSANDRA-7438.patch, tests.zip > > > Currently SerializingCache is partially off heap, keys are still stored in > JVM heap as BB, > * There is a higher GC costs for a reasonably big cache. > * Some users have used the row cache efficiently in production for better > results, but this requires careful tunning. > * Overhead in Memory for the cache entries are relatively high. > So the proposal for this ticket is to move the LRU cache logic completely off > heap and use JNI to interact with cache. We might want to ensure that the new > implementation match the existing API's (ICache), and the implementation > needs to have safe memory access, low overhead in memory and less memcpy's > (As much as possible). > We might also want to make this cache configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7438) Serializing Row cache alternative (Fully off heap)
[ https://issues.apache.org/jira/browse/CASSANDRA-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Stupp updated CASSANDRA-7438: Reviewer: Ariel Weisberg (was: Robert Stupp) Assignee: Robert Stupp (was: Vijay) > Serializing Row cache alternative (Fully off heap) > -- > > Key: CASSANDRA-7438 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7438 > Project: Cassandra > Issue Type: Improvement > Components: Core > Environment: Linux >Reporter: Vijay >Assignee: Robert Stupp > Labels: performance > Fix For: 3.0 > > Attachments: 0001-CASSANDRA-7438.patch, tests.zip > > > Currently SerializingCache is partially off heap, keys are still stored in > JVM heap as BB, > * There is a higher GC costs for a reasonably big cache. > * Some users have used the row cache efficiently in production for better > results, but this requires careful tunning. > * Overhead in Memory for the cache entries are relatively high. > So the proposal for this ticket is to move the LRU cache logic completely off > heap and use JNI to interact with cache. We might want to ensure that the new > implementation match the existing API's (ICache), and the implementation > needs to have safe memory access, low overhead in memory and less memcpy's > (As much as possible). > We might also want to make this cache configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8163) Re-introduce DESCRIBE permission
[ https://issues.apache.org/jira/browse/CASSANDRA-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265958#comment-14265958 ] Ben Laplanche commented on CASSANDRA-8163: -- The ability for users to see the table names for other users keyspaces is blocking us from using a cluster in a multi-tenant setup, which for some clients and the nature of their work can be prohibitive in using Cassandra. > Re-introduce DESCRIBE permission > > > Key: CASSANDRA-8163 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8163 > Project: Cassandra > Issue Type: Improvement >Reporter: Vishy Kasar >Priority: Minor > > We have a cluster like this: > project1_keyspace > table101 > table102 > project2_keyspace > table201 > table202 > We have set up following users and grants: > project1_user has all access to project1_keyspace > project2_user has all access to project2_keyspace > However project1_user can still do a 'describe schema' and get the schema for > project2_keyspace as well. We do not want project1_user to have any knowledge > for project2 in any way (cqlsh/java-driver etc) . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8558) deleted row still can be selected out
[ https://issues.apache.org/jira/browse/CASSANDRA-8558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265965#comment-14265965 ] Benedict commented on CASSANDRA-8558: - This is nothing to do with either change as far as I can tell. Somewhere inbetween those two changes something else was presumably broken, that is unrelated to either of them. It's possible that it was broken before either, in fact. Somewhere amongst them we changed drop behaviour to introduce flushing of dirty tables, and this flushing causes the problem. In fact we probably have a much worse problem than this appears. I can illicit this behaviour with a simple call to nodetool flush. The deletion records are flushed to disk, and appear in their respective sstables, but are not being returned by the IndexedSliceReader that queries them. A new test to find this behaviour would be simpler: CREATE KEYSPACE space1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1}; CREATE TABLE space1.table1(a int, b int, c text,primary key(a,b)); INSERT INTO space1.table1(a,b,c) VALUES(1,1,'1'); // nodetool flush DELETE FROM space1.table3 where a=1 and b=1; // nodetool flush SELECT * FROM space1.table3 where a=1 and b=1; This is much more fundamentally broken than this ticket suggests, but I'm probably not the best person to investigate, since it looks to be a problem with IndexedSliceReader. Hopefully a git bisect with the updated test will blame a suitable candidate next time around :) (assuming it isn't somehow still my fault) > deleted row still can be selected out > - > > Key: CASSANDRA-8558 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8558 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: 2.1.2 > java version "1.7.0_55" >Reporter: zhaoyan >Assignee: Benedict > Fix For: 2.1.3 > > > first > {code}CREATE KEYSPACE space1 WITH replication = {'class': 'SimpleStrategy', > 'replication_factor': 3}; > CREATE TABLE space1.table3(a int, b int, c text,primary key(a,b)); > CREATE KEYSPACE space2 WITH replication = {'class': 'SimpleStrategy', > 'replication_factor': 3};{code} > second > {code}CREATE TABLE space2.table1(a int, b int, c int, primary key(a,b)); > CREATE TABLE space2.table2(a int, b int, c int, primary key(a,b)); > INSERT INTO space1.table3(a,b,c) VALUES(1,1,'1'); > drop table space2.table1; > DELETE FROM space1.table3 where a=1 and b=1; > drop table space2.table2; > select * from space1.table3 where a=1 and b=1;{code} > you will find that the row (a=1 and b=1) in space1.table3 is not deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8558) deleted row still can be selected out
[ https://issues.apache.org/jira/browse/CASSANDRA-8558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-8558: Assignee: Philip Thompson (was: Benedict) > deleted row still can be selected out > - > > Key: CASSANDRA-8558 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8558 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: 2.1.2 > java version "1.7.0_55" >Reporter: zhaoyan >Assignee: Philip Thompson > Fix For: 2.1.3 > > > first > {code}CREATE KEYSPACE space1 WITH replication = {'class': 'SimpleStrategy', > 'replication_factor': 3}; > CREATE TABLE space1.table3(a int, b int, c text,primary key(a,b)); > CREATE KEYSPACE space2 WITH replication = {'class': 'SimpleStrategy', > 'replication_factor': 3};{code} > second > {code}CREATE TABLE space2.table1(a int, b int, c int, primary key(a,b)); > CREATE TABLE space2.table2(a int, b int, c int, primary key(a,b)); > INSERT INTO space1.table3(a,b,c) VALUES(1,1,'1'); > drop table space2.table1; > DELETE FROM space1.table3 where a=1 and b=1; > drop table space2.table2; > select * from space1.table3 where a=1 and b=1;{code} > you will find that the row (a=1 and b=1) in space1.table3 is not deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8558) deleted row still can be selected out
[ https://issues.apache.org/jira/browse/CASSANDRA-8558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265965#comment-14265965 ] Benedict edited comment on CASSANDRA-8558 at 1/6/15 12:12 PM: -- This is nothing to do with either change as far as I can tell. Somewhere inbetween those two changes something else was presumably broken, that is unrelated to either of them. It's possible that it was broken before either, in fact. Somewhere amongst them we changed drop behaviour to introduce flushing of dirty tables, and this flushing causes the problem. In fact we probably have a much worse problem than this appears. I can illicit this behaviour with a simple call to nodetool flush. The deletion records are flushed to disk, and appear in their respective sstables, but are not being returned by the IndexedSliceReader that queries them. A new test to find this behaviour would be simpler: CREATE KEYSPACE space1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1}; CREATE TABLE space1.table1(a int, b int, c text,primary key(a,b)); INSERT INTO space1.table1(a,b,c) VALUES(1,1,'1'); // nodetool flush DELETE FROM space1.table1 where a=1 and b=1; // nodetool flush SELECT * FROM space1.table1 where a=1 and b=1; This is much more fundamentally broken than this ticket suggests, but I'm probably not the best person to investigate, since it looks to be a problem with IndexedSliceReader. Hopefully a git bisect with the updated test will blame a suitable candidate next time around :) (assuming it isn't somehow still my fault) was (Author: benedict): This is nothing to do with either change as far as I can tell. Somewhere inbetween those two changes something else was presumably broken, that is unrelated to either of them. It's possible that it was broken before either, in fact. Somewhere amongst them we changed drop behaviour to introduce flushing of dirty tables, and this flushing causes the problem. In fact we probably have a much worse problem than this appears. I can illicit this behaviour with a simple call to nodetool flush. The deletion records are flushed to disk, and appear in their respective sstables, but are not being returned by the IndexedSliceReader that queries them. A new test to find this behaviour would be simpler: CREATE KEYSPACE space1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1}; CREATE TABLE space1.table1(a int, b int, c text,primary key(a,b)); INSERT INTO space1.table1(a,b,c) VALUES(1,1,'1'); // nodetool flush DELETE FROM space1.table3 where a=1 and b=1; // nodetool flush SELECT * FROM space1.table3 where a=1 and b=1; This is much more fundamentally broken than this ticket suggests, but I'm probably not the best person to investigate, since it looks to be a problem with IndexedSliceReader. Hopefully a git bisect with the updated test will blame a suitable candidate next time around :) (assuming it isn't somehow still my fault) > deleted row still can be selected out > - > > Key: CASSANDRA-8558 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8558 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: 2.1.2 > java version "1.7.0_55" >Reporter: zhaoyan >Assignee: Philip Thompson > Fix For: 2.1.3 > > > first > {code}CREATE KEYSPACE space1 WITH replication = {'class': 'SimpleStrategy', > 'replication_factor': 3}; > CREATE TABLE space1.table3(a int, b int, c text,primary key(a,b)); > CREATE KEYSPACE space2 WITH replication = {'class': 'SimpleStrategy', > 'replication_factor': 3};{code} > second > {code}CREATE TABLE space2.table1(a int, b int, c int, primary key(a,b)); > CREATE TABLE space2.table2(a int, b int, c int, primary key(a,b)); > INSERT INTO space1.table3(a,b,c) VALUES(1,1,'1'); > drop table space2.table1; > DELETE FROM space1.table3 where a=1 and b=1; > drop table space2.table2; > select * from space1.table3 where a=1 and b=1;{code} > you will find that the row (a=1 and b=1) in space1.table3 is not deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7124) Use JMX Notifications to Indicate Success/Failure of Long-Running Operations
[ https://issues.apache.org/jira/browse/CASSANDRA-7124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266033#comment-14266033 ] Rajanarayanan Thottuvaikkatumana commented on CASSANDRA-7124: - [~yukim], Did you get a chance to look at the changes made? Any comments? Thanks > Use JMX Notifications to Indicate Success/Failure of Long-Running Operations > > > Key: CASSANDRA-7124 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7124 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Tyler Hobbs >Assignee: Rajanarayanan Thottuvaikkatumana >Priority: Minor > Labels: lhf > Fix For: 3.0 > > Attachments: 7124-wip.txt, cassandra-trunk-compact-7124.txt, > cassandra-trunk-decommission-7124.txt > > > If {{nodetool cleanup}} or some other long-running operation takes too long > to complete, you'll see an error like the one in CASSANDRA-2126, so you can't > tell if the operation completed successfully or not. CASSANDRA-4767 fixed > this for repairs with JMX notifications. We should do something similar for > nodetool cleanup, compact, decommission, move, relocate, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8194) Reading from Auth table should not be in the request path
[ https://issues.apache.org/jira/browse/CASSANDRA-8194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266056#comment-14266056 ] Sam Tunnicliffe commented on CASSANDRA-8194: While there is a window during which a stale set of permissions is used, under normal operation I don't think this *should* present too many practical problems. Refresh is triggered by the first lookup after permisions_validity_in_ms, so we'll continue to use the stale set between that point and when that refresh actually completes. Outside of tests though, clients have no visibility/expectation about the precise load or expiry timings, so this shouldn't usually matter. My concern would be performing every IAuthorizer.authorize call on a single thread using StorageService.tasks instead of distributing them across client request threads could cause a backlog and allow the window to grow unacceptably (plus, these tasks will also be contending with other users of the shared executor). The point about the proliferation of threads and executors is valid, but maybe there's a case for a dedicated executor here. We could make it a TPE with a default poolsize of 1 but allow that to be increased via a system property if necessary. What may be more of an issue is that we'll continue to serve the stale perms as long as the refresh fails completely due to IAuthorizer.authorize throwing some exception. This shouldn't really happen with CassandraAuthorizer, but other IAuthorizer impls could well encounter errors when fetching perms. To guard against that, we can force an invalidation if the ListenableFutureTask encounters an exception. That would pretty much maintain current behaviour, with the client receiving an error response while the refresh fails (actually, the authorize calls after an error would serve stale perms until the exception is thrown & caught, but all subsequent calls would fail as per current behaviour). I've attached a v3 with this second change, what are your thoughts on reverting back to a dedicated executor for cache refresh? Also, as I mentioned, tests do have concrete expectations about expiry of permissions and so this breaks auth_test.py:TestAuth.permissions_caching_test. I've pushed a fix [here|https://github.com/beobal/cassandra-dtest/tree/8194] and I'll open a PR shortly. > Reading from Auth table should not be in the request path > - > > Key: CASSANDRA-8194 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8194 > Project: Cassandra > Issue Type: Improvement >Reporter: Vishy Kasar >Assignee: Vishy Kasar >Priority: Minor > Fix For: 2.0.12, 3.0 > > Attachments: 8194-V2.patch, 8194.patch, CacheTest2.java > > > We use PasswordAuthenticator and PasswordAuthorizer. The system_auth has a RF > of 10 per DC over 2 DCs. The permissions_validity_in_ms is 5 minutes. > We still have few thousand requests failing each day with the trace below. > The reason for this is read cache request realizing that cached entry has > expired and doing a blocking request to refresh cache. > We should have cache refreshed periodically only in the back ground. The user > request should simply look at the cache and not try to refresh it. > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 0 responses. > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) > at com.google.common.cache.LocalCache.get(LocalCache.java:3990) > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:292) > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172) > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165) > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149) > at > org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:75) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:102) > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:113) > at > org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1735) > at > org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4162) > at > org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4150) > at org.ap
[jira] [Updated] (CASSANDRA-8194) Reading from Auth table should not be in the request path
[ https://issues.apache.org/jira/browse/CASSANDRA-8194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-8194: --- Attachment: 8194-V3.txt > Reading from Auth table should not be in the request path > - > > Key: CASSANDRA-8194 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8194 > Project: Cassandra > Issue Type: Improvement >Reporter: Vishy Kasar >Assignee: Vishy Kasar >Priority: Minor > Fix For: 2.0.12, 3.0 > > Attachments: 8194-V2.patch, 8194-V3.txt, 8194.patch, CacheTest2.java > > > We use PasswordAuthenticator and PasswordAuthorizer. The system_auth has a RF > of 10 per DC over 2 DCs. The permissions_validity_in_ms is 5 minutes. > We still have few thousand requests failing each day with the trace below. > The reason for this is read cache request realizing that cached entry has > expired and doing a blocking request to refresh cache. > We should have cache refreshed periodically only in the back ground. The user > request should simply look at the cache and not try to refresh it. > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 0 responses. > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258) > at com.google.common.cache.LocalCache.get(LocalCache.java:3990) > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994) > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878) > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:292) > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:172) > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:165) > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:149) > at > org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:75) > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:102) > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:113) > at > org.apache.cassandra.thrift.CassandraServer.execute_cql3_query(CassandraServer.java:1735) > at > org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4162) > at > org.apache.cassandra.thrift.Cassandra$Processor$execute_cql3_query.getResult(Cassandra.java:4150) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34) > at > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:206) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 0 responses. > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:256) > at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84) > at > org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) > at > org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:68) > at org.apache.cassandra.service.ClientState$1.load(ClientState.java:278) > at org.apache.cassandra.service.ClientState$1.load(ClientState.java:275) > at > com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589) > at > com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374) > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337) > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252) > ... 19 more > Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation > timed out - received only 0 responses. > at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:105) > at > org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:943) > at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:828) > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:140) > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:245) > ... 28 more > ERROR [Thrift:17232] 2014-10-24 05:06:51,004 CustomTThreadPoolServer.java > (line 224) Error
[jira] [Updated] (CASSANDRA-6983) DirectoriesTest fails when run as root
[ https://issues.apache.org/jira/browse/CASSANDRA-6983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Boudreault updated CASSANDRA-6983: --- Tester: Alan Boudreault > DirectoriesTest fails when run as root > -- > > Key: CASSANDRA-6983 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6983 > Project: Cassandra > Issue Type: Bug > Components: Tests >Reporter: Brandon Williams >Assignee: Ryan McGuire >Priority: Minor > Fix For: 2.0.12 > > > When you run the DirectoriesTest as a normal user, it passes because it fails > to create the 'bad' directory: > {noformat} > [junit] - Standard Error - > [junit] ERROR 16:16:18,111 Failed to create > /tmp/cassandra4119802552776680052unittest/ks/bad directory > [junit] WARN 16:16:18,112 Blacklisting > /tmp/cassandra4119802552776680052unittest/ks/bad for writes > [junit] - --- > {noformat} > But when you run the test as root, it succeeds in making the directory, > causing an assertion failure that it's unwritable: > {noformat} > [junit] Testcase: > testDiskFailurePolicy_best_effort(org.apache.cassandra.db.DirectoriesTest): > FAILED > [junit] > [junit] junit.framework.AssertionFailedError: > [junit] at > org.apache.cassandra.db.DirectoriesTest.testDiskFailurePolicy_best_effort(DirectoriesTest.java:199) > {noformat} > It seems to me that we shouldn't be relying on failing the make the > directory. If we're just going to test a nonexistent dir, why try to make > one at all? And if that is supposed to succeed, then we have a problem with > either the test or blacklisting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-6983) DirectoriesTest fails when run as root
[ https://issues.apache.org/jira/browse/CASSANDRA-6983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Boudreault updated CASSANDRA-6983: --- Attachment: 6983-v1.patch As a non-root user perspective, the test is ok. We have to call the Directories.create() if we want to get the internal cassandra disk policy mechanism triggered. If we remove that fake directory creation, the test will fail since the it is not marked as an unwritable BlacklistedDirectory. The real issue with that test using the root user is that we cannot set the directory unwritable using Java. Internally, Java will simply re-set the write permission for the operation since the root user is allowed to do everything. Please correct me if I'm wrong here about that Java behavior. At this point, we have two options to fix this test: 1. Do not call Directories.create(), but rather set the directory unwritable manually via the BlacklistedDirectories class. I don't especially like this option since it doesn't really trigger the best_effort internal mechanism. 2. Simulate a fake directory creation by throwing an exception. This will trigger the internal handleFSError and the best_effort mechanism. This will work for the root user since we are not playing anymore with the filesystem. I've attached a tentative patch with this solution. Let me know what you think or if you have any suggestion. > DirectoriesTest fails when run as root > -- > > Key: CASSANDRA-6983 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6983 > Project: Cassandra > Issue Type: Bug > Components: Tests >Reporter: Brandon Williams >Assignee: Ryan McGuire >Priority: Minor > Fix For: 2.0.12 > > Attachments: 6983-v1.patch > > > When you run the DirectoriesTest as a normal user, it passes because it fails > to create the 'bad' directory: > {noformat} > [junit] - Standard Error - > [junit] ERROR 16:16:18,111 Failed to create > /tmp/cassandra4119802552776680052unittest/ks/bad directory > [junit] WARN 16:16:18,112 Blacklisting > /tmp/cassandra4119802552776680052unittest/ks/bad for writes > [junit] - --- > {noformat} > But when you run the test as root, it succeeds in making the directory, > causing an assertion failure that it's unwritable: > {noformat} > [junit] Testcase: > testDiskFailurePolicy_best_effort(org.apache.cassandra.db.DirectoriesTest): > FAILED > [junit] > [junit] junit.framework.AssertionFailedError: > [junit] at > org.apache.cassandra.db.DirectoriesTest.testDiskFailurePolicy_best_effort(DirectoriesTest.java:199) > {noformat} > It seems to me that we shouldn't be relying on failing the make the > directory. If we're just going to test a nonexistent dir, why try to make > one at all? And if that is supposed to succeed, then we have a problem with > either the test or blacklisting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8303) Provide "strict mode" for CQL Queries
[ https://issues.apache.org/jira/browse/CASSANDRA-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266101#comment-14266101 ] Sam Tunnicliffe commented on CASSANDRA-8303: It seems to me that this could be a reasonably natural fit as an extension to the existing authz system. If we do go down that route there are a couple of things to consider. What we're really talking about here are restrictions on the permissions a particular user may have been granted. For instance, you could grant SELECT on ks1.cf1 to user foo, but add a restriction such that multipartition queries are not allowed. My first question then is are there other restrictions that would be useful but which don't fit this definition? For restrictions which do fit that definition, it seems sensible to use a similar resource hierarchy as permissions. i.e. a restriction can be applied at the table level, keyspace level (which trickles down to apply to all tables in the ks), or at the root level. The alternative implied by the syntax in the previous comments suggests that a restriction should always be applied at the root level (i.e. for all keyspaces & tables). How inheritance is handled in that hierarchy of resources should be from the bottom upwards, working from the most specific permission/restriction.For instance, in the case where user x is granted SELECT without any restriction on ks1 and the same user is also granted SELECT with the MULTI_PARTITION restriction on ks1.cf1 it seems clear that MP queries from user x should only be disallowed on the one table, not all tables in ks1. Also, the unrestricted grant should not trump the more specific, restricted one and allow MP queries on that table. In the reverse scenario, where the restriction is at the keyspace level, but there is also a non-restricted grant for the table, the behaviour should also be reversed, user x can make MP queries on the 1 table, but on no others in the keyspace. As noted, managing all this at the user level could be overly burdensome but CASSANDRA-7653 should relieve most of that extra administrative load. However, it would involve some additional complexity in handling resolution of permissions and restrictions. During the authz process when we resolve permissions for a user on a particular resource, the most specific permission whether granted directly or inherited through role membership should apply, along with any of its restrictions. If there are multiple inherited or direct grants for the exact same resource (i.e. at the same level in the resource hierarchy) then we would merge them, unioning any restrictions, e.g {noformat} role x is granted SELECT on ks1 with MULTI_PARTITION restriction role y is granted SELECT on ks1.cf1, unrestricted role y should be allowed to SELECT from any table in ks1, but perform MP queries on ks1.cf1 only role z is granted both role x and role y role z should be allowed to SELECT from any table in ks1, but perform MP queries on ks1.cf1 only role a is granted SELECT on ks1, unrestricted role z is granted roles x, y, a role z should be allowed to SELECT from any table in ks1, but perform MP queries on ks1.cf1 only {noformat} So, I think this could fit nicely into authz for 3.0 but we may also want to think about adding options to cassandra.yaml to enforce the same restrictions without requiring authz (& authn). Obviously, that would be much more of a blunt instrument - enforcing a given restriction for all queries when enabled, though it would mean that it could be turned on on a per-node/per-DC basis. I don't see a problem with supporting both cql and yaml based approaches so long as we define an order of precedence. > Provide "strict mode" for CQL Queries > - > > Key: CASSANDRA-8303 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8303 > Project: Cassandra > Issue Type: Improvement >Reporter: Anupam Arora > Fix For: 3.0 > > > Please provide a "strict mode" option in cassandra that will kick out any CQL > queries that are expensive, e.g. any query with ALLOWS FILTERING, > multi-partition queries, secondary index queries, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-6983) DirectoriesTest fails when run as root
[ https://issues.apache.org/jira/browse/CASSANDRA-6983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-6983: Reviewer: Yuki Morishita Assignee: Alan Boudreault (was: Ryan McGuire) > DirectoriesTest fails when run as root > -- > > Key: CASSANDRA-6983 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6983 > Project: Cassandra > Issue Type: Bug > Components: Tests >Reporter: Brandon Williams >Assignee: Alan Boudreault >Priority: Minor > Fix For: 2.0.12 > > Attachments: 6983-v1.patch > > > When you run the DirectoriesTest as a normal user, it passes because it fails > to create the 'bad' directory: > {noformat} > [junit] - Standard Error - > [junit] ERROR 16:16:18,111 Failed to create > /tmp/cassandra4119802552776680052unittest/ks/bad directory > [junit] WARN 16:16:18,112 Blacklisting > /tmp/cassandra4119802552776680052unittest/ks/bad for writes > [junit] - --- > {noformat} > But when you run the test as root, it succeeds in making the directory, > causing an assertion failure that it's unwritable: > {noformat} > [junit] Testcase: > testDiskFailurePolicy_best_effort(org.apache.cassandra.db.DirectoriesTest): > FAILED > [junit] > [junit] junit.framework.AssertionFailedError: > [junit] at > org.apache.cassandra.db.DirectoriesTest.testDiskFailurePolicy_best_effort(DirectoriesTest.java:199) > {noformat} > It seems to me that we shouldn't be relying on failing the make the > directory. If we're just going to test a nonexistent dir, why try to make > one at all? And if that is supposed to succeed, then we have a problem with > either the test or blacklisting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8558) deleted row still can be selected out
[ https://issues.apache.org/jira/browse/CASSANDRA-8558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams updated CASSANDRA-8558: Priority: Blocker (was: Major) > deleted row still can be selected out > - > > Key: CASSANDRA-8558 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8558 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: 2.1.2 > java version "1.7.0_55" >Reporter: zhaoyan >Assignee: Philip Thompson >Priority: Blocker > Fix For: 2.1.3 > > > first > {code}CREATE KEYSPACE space1 WITH replication = {'class': 'SimpleStrategy', > 'replication_factor': 3}; > CREATE TABLE space1.table3(a int, b int, c text,primary key(a,b)); > CREATE KEYSPACE space2 WITH replication = {'class': 'SimpleStrategy', > 'replication_factor': 3};{code} > second > {code}CREATE TABLE space2.table1(a int, b int, c int, primary key(a,b)); > CREATE TABLE space2.table2(a int, b int, c int, primary key(a,b)); > INSERT INTO space1.table3(a,b,c) VALUES(1,1,'1'); > drop table space2.table1; > DELETE FROM space1.table3 where a=1 and b=1; > drop table space2.table2; > select * from space1.table3 where a=1 and b=1;{code} > you will find that the row (a=1 and b=1) in space1.table3 is not deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[1/3] cassandra git commit: Use lexical comparison for java revisions
Repository: cassandra Updated Branches: refs/heads/cassandra-2.1 1af8ea5ef -> e83304ddf refs/heads/trunk dc0102a8e -> 0f024a619 Use lexical comparison for java revisions Patch by Michael Shuler, reviewed by brandonwilliams for CASSANDRA-8315 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e83304dd Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e83304dd Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e83304dd Branch: refs/heads/cassandra-2.1 Commit: e83304ddfee1377417b334d393c61671895d8ed5 Parents: 1af8ea5 Author: Brandon Williams Authored: Tue Jan 6 08:24:48 2015 -0600 Committer: Brandon Williams Committed: Tue Jan 6 08:24:48 2015 -0600 -- conf/cassandra-env.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/e83304dd/conf/cassandra-env.sh -- diff --git a/conf/cassandra-env.sh b/conf/cassandra-env.sh index 191fb7e..755f962 100644 --- a/conf/cassandra-env.sh +++ b/conf/cassandra-env.sh @@ -99,7 +99,7 @@ if [ "$JVM_VERSION" \< "1.7" ] ; then exit 1; fi -if [ "$JVM_VERSION" \< "1.8" ] && [ "$JVM_PATCH_VERSION" -lt "25" ] ; then +if [ "$JVM_VERSION" \< "1.8" ] && [ "$JVM_PATCH_VERSION" \< "25" ] ; then echo "Cassandra 2.0 and later require Java 7u25 or later." exit 1; fi
[3/3] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/0f024a61 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/0f024a61 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/0f024a61 Branch: refs/heads/trunk Commit: 0f024a6192f185b93439ae53a69ce113b363 Parents: dc0102a e83304d Author: Brandon Williams Authored: Tue Jan 6 08:25:56 2015 -0600 Committer: Brandon Williams Committed: Tue Jan 6 08:25:56 2015 -0600 -- conf/cassandra-env.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/0f024a61/conf/cassandra-env.sh --
[2/3] cassandra git commit: Use lexical comparison for java revisions
Use lexical comparison for java revisions Patch by Michael Shuler, reviewed by brandonwilliams for CASSANDRA-8315 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e83304dd Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e83304dd Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e83304dd Branch: refs/heads/trunk Commit: e83304ddfee1377417b334d393c61671895d8ed5 Parents: 1af8ea5 Author: Brandon Williams Authored: Tue Jan 6 08:24:48 2015 -0600 Committer: Brandon Williams Committed: Tue Jan 6 08:24:48 2015 -0600 -- conf/cassandra-env.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/e83304dd/conf/cassandra-env.sh -- diff --git a/conf/cassandra-env.sh b/conf/cassandra-env.sh index 191fb7e..755f962 100644 --- a/conf/cassandra-env.sh +++ b/conf/cassandra-env.sh @@ -99,7 +99,7 @@ if [ "$JVM_VERSION" \< "1.7" ] ; then exit 1; fi -if [ "$JVM_VERSION" \< "1.8" ] && [ "$JVM_PATCH_VERSION" -lt "25" ] ; then +if [ "$JVM_VERSION" \< "1.8" ] && [ "$JVM_PATCH_VERSION" \< "25" ] ; then echo "Cassandra 2.0 and later require Java 7u25 or later." exit 1; fi
[jira] [Commented] (CASSANDRA-7653) Add role based access control to Cassandra
[ https://issues.apache.org/jira/browse/CASSANDRA-7653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266182#comment-14266182 ] Mike Adamson commented on CASSANDRA-7653: - I have a couple of initial thoughts on this. # Is the IAuthenticator.constructInitialSaslToken method really necessary? The only usage of this is from login methods that are only going to use the plain text sasl implementation offered by the PasswordAuthenticator so they could build the initial token themselves. # Is there any way of not having the Option enum? This fixes the options that an Authenticator can support and doesn't allow any 3rd party to have different options but still work with the CQL grammar. Could we have some similar to the replication strategies? Or perhaps keep the Option enum but have an option of EXTENSION (or other name) that would allow the passing in of a json set of extension options. > Add role based access control to Cassandra > -- > > Key: CASSANDRA-7653 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7653 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Mike Adamson >Assignee: Sam Tunnicliffe > Fix For: 3.0 > > Attachments: 7653.patch, CQLSmokeTest.java, cql_smoke_test.py > > > The current authentication model supports granting permissions to individual > users. While this is OK for small or medium organizations wanting to > implement authorization, it does not work well in large organizations because > of the overhead of having to maintain the permissions for each user. > Introducing roles into the authentication model would allow sets of > permissions to be controlled in one place as a role and then the role granted > to users. Roles should also be able to be granted to other roles to allow > hierarchical sets of permissions to be built up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8448) "Comparison method violates its general contract" in AbstractEndpointSnitch
[ https://issues.apache.org/jira/browse/CASSANDRA-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266191#comment-14266191 ] Brandon Williams commented on CASSANDRA-8448: - ISTM we already do that: {noformat} ArrayList sortedScores = new ArrayList<>(subsnitchOrderedScores); Collections.sort(sortedScores); Iterator sortedScoreIterator = sortedScores.iterator(); for (Double subsnitchScore : subsnitchOrderedScores) { if (subsnitchScore > (sortedScoreIterator.next() * (1.0 + BADNESS_THRESHOLD))) { sortByProximityWithScore(address, addresses); return; } } {noformat} So I'm not sure what to do (and have no clear repro steps so I can't even try.) A simple workaround would be disabling badness, though. > "Comparison method violates its general contract" in AbstractEndpointSnitch > --- > > Key: CASSANDRA-8448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8448 > Project: Cassandra > Issue Type: Bug >Reporter: J.B. Langston >Assignee: Brandon Williams > > Seen in both 1.2 and 2.0. The error is occurring here: > https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/locator/AbstractEndpointSnitch.java#L49 > {code} > ERROR [Thrift:9] 2014-12-04 20:12:28,732 CustomTThreadPoolServer.java (line > 219) Error occurred during processing of message. > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.IllegalArgumentException: Comparison method violates its general > contract! > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199) > at com.google.common.cache.LocalCache.get(LocalCache.java:3932) > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936) > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806) > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:352) > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:224) > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:218) > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:202) > at > org.apache.cassandra.thrift.CassandraServer.createMutationList(CassandraServer.java:822) > at > org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:954) > at com.datastax.bdp.server.DseServer.batch_mutate(DseServer.java:576) > at > org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3922) > at > org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3906) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.IllegalArgumentException: Comparison method violates its > general contract! > at java.util.TimSort.mergeHi(TimSort.java:868) > at java.util.TimSort.mergeAt(TimSort.java:485) > at java.util.TimSort.mergeCollapse(TimSort.java:410) > at java.util.TimSort.sort(TimSort.java:214) > at java.util.TimSort.sort(TimSort.java:173) > at java.util.Arrays.sort(Arrays.java:659) > at java.util.Collections.sort(Collections.java:217) > at > org.apache.cassandra.locator.AbstractEndpointSnitch.sortByProximity(AbstractEndpointSnitch.java:49) > at > org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithScore(DynamicEndpointSnitch.java:157) > at > org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithBadness(DynamicEndpointSnitch.java:186) > at > org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximity(DynamicEndpointSnitch.java:151) > at > org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1408) > at > org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1402) > at > org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:148) > at > org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1223) > at > org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1165) > at > org.apache.cassandra.cq
[jira] [Updated] (CASSANDRA-8558) deleted row still can be selected out
[ https://issues.apache.org/jira/browse/CASSANDRA-8558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8558: --- Assignee: Sylvain Lebresne (was: Philip Thompson) > deleted row still can be selected out > - > > Key: CASSANDRA-8558 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8558 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: 2.1.2 > java version "1.7.0_55" >Reporter: zhaoyan >Assignee: Sylvain Lebresne >Priority: Blocker > Fix For: 2.1.3 > > > first > {code}CREATE KEYSPACE space1 WITH replication = {'class': 'SimpleStrategy', > 'replication_factor': 3}; > CREATE TABLE space1.table3(a int, b int, c text,primary key(a,b)); > CREATE KEYSPACE space2 WITH replication = {'class': 'SimpleStrategy', > 'replication_factor': 3};{code} > second > {code}CREATE TABLE space2.table1(a int, b int, c int, primary key(a,b)); > CREATE TABLE space2.table2(a int, b int, c int, primary key(a,b)); > INSERT INTO space1.table3(a,b,c) VALUES(1,1,'1'); > drop table space2.table1; > DELETE FROM space1.table3 where a=1 and b=1; > drop table space2.table2; > select * from space1.table3 where a=1 and b=1;{code} > you will find that the row (a=1 and b=1) in space1.table3 is not deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8558) deleted row still can be selected out
[ https://issues.apache.org/jira/browse/CASSANDRA-8558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266195#comment-14266195 ] Philip Thompson commented on CASSANDRA-8558: Thanks for the better test. The new bisect was much cleaner, the only failure I saw was the expected one highlighted by the reporter. Here is the bisect output: {code} 362cc05352ec67e707e0ac790732e96a15e63f6b is the first bad commit commit 362cc05352ec67e707e0ac790732e96a15e63f6b Author: Sylvain Lebresne Date: Tue Oct 29 11:03:52 2013 +0100 Push composites support in the storage engine patch by slebresne; reviewed by benedict for CASSANDRA-5417 :100644 100644 c0d0f0d243f0454c1b1926957634bb52165295aa dcc7e33b064e52d245d8c6ba5de887c74e0f0a00 M CHANGES.txt :04 04 da620b5b36e8ba6b97cdb8efe4f690692a85e15e 5dd1113af54ad7b90dd8694e462483bac6e7f985 M src :04 04 1fe1db9fca3bf03946828cdc917a0aef5bf7a9e1 3bf0563361febc821a357cf61b5a0760f5a8eca4 M test {code} > deleted row still can be selected out > - > > Key: CASSANDRA-8558 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8558 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: 2.1.2 > java version "1.7.0_55" >Reporter: zhaoyan >Assignee: Philip Thompson >Priority: Blocker > Fix For: 2.1.3 > > > first > {code}CREATE KEYSPACE space1 WITH replication = {'class': 'SimpleStrategy', > 'replication_factor': 3}; > CREATE TABLE space1.table3(a int, b int, c text,primary key(a,b)); > CREATE KEYSPACE space2 WITH replication = {'class': 'SimpleStrategy', > 'replication_factor': 3};{code} > second > {code}CREATE TABLE space2.table1(a int, b int, c int, primary key(a,b)); > CREATE TABLE space2.table2(a int, b int, c int, primary key(a,b)); > INSERT INTO space1.table3(a,b,c) VALUES(1,1,'1'); > drop table space2.table1; > DELETE FROM space1.table3 where a=1 and b=1; > drop table space2.table2; > select * from space1.table3 where a=1 and b=1;{code} > you will find that the row (a=1 and b=1) in space1.table3 is not deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8558) deleted row still can be selected out
[ https://issues.apache.org/jira/browse/CASSANDRA-8558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8558: --- Tester: Philip Thompson > deleted row still can be selected out > - > > Key: CASSANDRA-8558 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8558 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: 2.1.2 > java version "1.7.0_55" >Reporter: zhaoyan >Assignee: Sylvain Lebresne >Priority: Blocker > Fix For: 2.1.3 > > > first > {code}CREATE KEYSPACE space1 WITH replication = {'class': 'SimpleStrategy', > 'replication_factor': 3}; > CREATE TABLE space1.table3(a int, b int, c text,primary key(a,b)); > CREATE KEYSPACE space2 WITH replication = {'class': 'SimpleStrategy', > 'replication_factor': 3};{code} > second > {code}CREATE TABLE space2.table1(a int, b int, c int, primary key(a,b)); > CREATE TABLE space2.table2(a int, b int, c int, primary key(a,b)); > INSERT INTO space1.table3(a,b,c) VALUES(1,1,'1'); > drop table space2.table1; > DELETE FROM space1.table3 where a=1 and b=1; > drop table space2.table2; > select * from space1.table3 where a=1 and b=1;{code} > you will find that the row (a=1 and b=1) in space1.table3 is not deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8558) deleted row still can be selected out
[ https://issues.apache.org/jira/browse/CASSANDRA-8558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266220#comment-14266220 ] Benedict commented on CASSANDRA-8558: - I had a quick debug, and it appears EOCs are the problem: the range tombstone has an EOC of start, but the SliceQuery has an EOC of NONE, and the SimpleBlockFetcher skips over every atom <= the slice start. But the name of a range tombstone is only its start, so this is considered prior to the slice's start, even though it stretches across the slice's range. > deleted row still can be selected out > - > > Key: CASSANDRA-8558 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8558 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: 2.1.2 > java version "1.7.0_55" >Reporter: zhaoyan >Assignee: Sylvain Lebresne >Priority: Blocker > Fix For: 2.1.3 > > > first > {code}CREATE KEYSPACE space1 WITH replication = {'class': 'SimpleStrategy', > 'replication_factor': 3}; > CREATE TABLE space1.table3(a int, b int, c text,primary key(a,b)); > CREATE KEYSPACE space2 WITH replication = {'class': 'SimpleStrategy', > 'replication_factor': 3};{code} > second > {code}CREATE TABLE space2.table1(a int, b int, c int, primary key(a,b)); > CREATE TABLE space2.table2(a int, b int, c int, primary key(a,b)); > INSERT INTO space1.table3(a,b,c) VALUES(1,1,'1'); > drop table space2.table1; > DELETE FROM space1.table3 where a=1 and b=1; > drop table space2.table2; > select * from space1.table3 where a=1 and b=1;{code} > you will find that the row (a=1 and b=1) in space1.table3 is not deleted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6246) EPaxos
[ https://issues.apache.org/jira/browse/CASSANDRA-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266223#comment-14266223 ] Blake Eggleston commented on CASSANDRA-6246: By using the existing epaxos ordering constraints. Incrementing the epoch is done by an instance which takes all unacknowledged instances as dependencies for the token range it's incrementing the epoch for. The epoch can only be incremented if all previous instances have also been executed. > EPaxos > -- > > Key: CASSANDRA-6246 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6246 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jonathan Ellis >Assignee: Blake Eggleston >Priority: Minor > > One reason we haven't optimized our Paxos implementation with Multi-paxos is > that Multi-paxos requires leader election and hence, a period of > unavailability when the leader dies. > EPaxos is a Paxos variant that requires (1) less messages than multi-paxos, > (2) is particularly useful across multiple datacenters, and (3) allows any > node to act as coordinator: > http://sigops.org/sosp/sosp13/papers/p358-moraru.pdf > However, there is substantial additional complexity involved if we choose to > implement it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-6246) EPaxos
[ https://issues.apache.org/jira/browse/CASSANDRA-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266223#comment-14266223 ] Blake Eggleston edited comment on CASSANDRA-6246 at 1/6/15 3:11 PM: By using the existing epaxos ordering constraints. Incrementing the epoch is done by an instance which takes all unacknowledged instances as dependencies for the token range it's incrementing the epoch for. The epoch can only be incremented if all previous instances have also been executed. I pushed up some commits that add the epoch functionality yesterday if you'd like to take a look: https://github.com/bdeggleston/cassandra/tree/CASSANDRA-6246 was (Author: bdeggleston): By using the existing epaxos ordering constraints. Incrementing the epoch is done by an instance which takes all unacknowledged instances as dependencies for the token range it's incrementing the epoch for. The epoch can only be incremented if all previous instances have also been executed. > EPaxos > -- > > Key: CASSANDRA-6246 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6246 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Jonathan Ellis >Assignee: Blake Eggleston >Priority: Minor > > One reason we haven't optimized our Paxos implementation with Multi-paxos is > that Multi-paxos requires leader election and hence, a period of > unavailability when the leader dies. > EPaxos is a Paxos variant that requires (1) less messages than multi-paxos, > (2) is particularly useful across multiple datacenters, and (3) allows any > node to act as coordinator: > http://sigops.org/sosp/sosp13/papers/p358-moraru.pdf > However, there is substantial additional complexity involved if we choose to > implement it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-3025) PHP/PDO driver for Cassandra CQL
[ https://issues.apache.org/jira/browse/CASSANDRA-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266240#comment-14266240 ] Lex Lythius commented on CASSANDRA-3025: @Michael Yes, I've checked nearly all the drivers listed in that PlanetCassandra page. While Cassandra JIRA != DataStax, there is considerable overlapping between Apache and DataStax regarding Cassandra. My goal is to get some indication from either of the main driving forces behind Cassandra. @Alex's reply sheds some light on this matter: it will be a C++-built wrapper around the official C++ driver, which is native protocol-based. Will it be PDO-based as well? (with several C*-specific added features, to be sure). The one I've been contributing to is, but it uses the deprecated Thrift interface and, apart from a few bugs, it has no support for user-defined types and tuples. I will be happy to contribute in my pretty modest capacity as C++ developer to a C* PHP driver -- I would just like to know I work in the right direction. If this JIRA is not the right place to bring the issue to the table, where would that be? > PHP/PDO driver for Cassandra CQL > > > Key: CASSANDRA-3025 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3025 > Project: Cassandra > Issue Type: New Feature > Components: API >Reporter: Mikko Koppanen >Assignee: Mikko Koppanen > Labels: php > Attachments: pdo_cassandra-0.1.0.tgz, pdo_cassandra-0.1.1.tgz, > pdo_cassandra-0.1.2.tgz, pdo_cassandra-0.1.3.tgz, pdo_cassandra-0.2.0.tgz, > pdo_cassandra-0.2.1.tgz, php_test_results_20110818_2317.txt > > > Hello, > attached is the initial version of the PDO driver for Cassandra CQL language. > This is a native PHP extension written in what I would call a combination of > C and C++, due to PHP being C. The thrift API used is the C++. > The API looks roughly following: > {code} > $db = new PDO('cassandra:host=127.0.0.1;port=9160'); > $db->exec ("CREATE KEYSPACE mytest with strategy_class = 'SimpleStrategy' and > strategy_options:replication_factor=1;"); > $db->exec ("USE mytest"); > $db->exec ("CREATE COLUMNFAMILY users ( > my_key varchar PRIMARY KEY, > full_name varchar );"); > > $stmt = $db->prepare ("INSERT INTO users (my_key, full_name) VALUES (:key, > :full_name);"); > $stmt->execute (array (':key' => 'mikko', ':full_name' => 'Mikko K' )); > {code} > Currently prepared statements are emulated on the client side but I > understand that there is a plan to add prepared statements to Cassandra CQL > API as well. I will add this feature in to the extension as soon as they are > implemented. > Additional documentation can be found in github > https://github.com/mkoppanen/php-pdo_cassandra, in the form of rendered > MarkDown file. Tests are currently not included in the package file and they > can be found in the github for now as well. > I have created documentation in docbook format as well, but have not yet > rendered it. > Comments and feedback are welcome. > Thanks, > Mikko -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[6/6] cassandra git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/ffb7f649 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/ffb7f649 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/ffb7f649 Branch: refs/heads/trunk Commit: ffb7f6491d905cd2edaba3a710a1c0bb1f1bf797 Parents: 0f024a6 136042e Author: Brandon Williams Authored: Tue Jan 6 09:31:16 2015 -0600 Committer: Brandon Williams Committed: Tue Jan 6 09:31:16 2015 -0600 -- .../apache/cassandra/gms/FailureDetector.java | 8 +++ .../apache/cassandra/gms/ArrivalWindowTest.java | 24 +++- 2 files changed, 22 insertions(+), 10 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/ffb7f649/src/java/org/apache/cassandra/gms/FailureDetector.java --
[5/6] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/136042ec Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/136042ec Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/136042ec Branch: refs/heads/cassandra-2.1 Commit: 136042ec3985bbacfcf55333ef5668779d2744fe Parents: e83304d eb9c5bb Author: Brandon Williams Authored: Tue Jan 6 09:31:05 2015 -0600 Committer: Brandon Williams Committed: Tue Jan 6 09:31:05 2015 -0600 -- .../apache/cassandra/gms/FailureDetector.java | 8 +++ .../apache/cassandra/gms/ArrivalWindowTest.java | 24 +++- 2 files changed, 22 insertions(+), 10 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/136042ec/src/java/org/apache/cassandra/gms/FailureDetector.java --
[3/6] cassandra git commit: Improve FD logging when the arrival time is ignored.
Improve FD logging when the arrival time is ignored. Patch by Brandon Williams for CASSANDRA-8245 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/eb9c5bbc Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/eb9c5bbc Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/eb9c5bbc Branch: refs/heads/trunk Commit: eb9c5bbcfdaf498967052e02af65fa9f8167887d Parents: a223082 Author: Brandon Williams Authored: Tue Jan 6 09:30:07 2015 -0600 Committer: Brandon Williams Committed: Tue Jan 6 09:30:07 2015 -0600 -- .../apache/cassandra/gms/FailureDetector.java | 8 +++ .../apache/cassandra/gms/ArrivalWindowTest.java | 24 +++- 2 files changed, 22 insertions(+), 10 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb9c5bbc/src/java/org/apache/cassandra/gms/FailureDetector.java -- diff --git a/src/java/org/apache/cassandra/gms/FailureDetector.java b/src/java/org/apache/cassandra/gms/FailureDetector.java index 60729d3..e247e48 100644 --- a/src/java/org/apache/cassandra/gms/FailureDetector.java +++ b/src/java/org/apache/cassandra/gms/FailureDetector.java @@ -211,12 +211,12 @@ public class FailureDetector implements IFailureDetector, FailureDetectorMBean { // avoid adding an empty ArrivalWindow to the Map heartbeatWindow = new ArrivalWindow(SAMPLE_SIZE); -heartbeatWindow.add(now); +heartbeatWindow.add(now, ep); arrivalSamples.put(ep, heartbeatWindow); } else { -heartbeatWindow.add(now); +heartbeatWindow.add(now, ep); } } @@ -326,7 +326,7 @@ class ArrivalWindow } } -synchronized void add(long value) +synchronized void add(long value, InetAddress ep) { assert tLast >= 0; if (tLast > 0L) @@ -335,7 +335,7 @@ class ArrivalWindow if (interArrivalTime <= MAX_INTERVAL_IN_NANO) arrivalIntervals.add(interArrivalTime); else -logger.debug("Ignoring interval time of {}", interArrivalTime); +logger.debug("Ignoring interval time of {} for {}", interArrivalTime, ep); } else { http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb9c5bbc/test/unit/org/apache/cassandra/gms/ArrivalWindowTest.java -- diff --git a/test/unit/org/apache/cassandra/gms/ArrivalWindowTest.java b/test/unit/org/apache/cassandra/gms/ArrivalWindowTest.java index e678d86..511511b 100644 --- a/test/unit/org/apache/cassandra/gms/ArrivalWindowTest.java +++ b/test/unit/org/apache/cassandra/gms/ArrivalWindowTest.java @@ -25,6 +25,10 @@ import static org.junit.Assert.*; import org.junit.Test; +import java.lang.RuntimeException; +import java.net.InetAddress; +import java.net.UnknownHostException; + public class ArrivalWindowTest { @Test @@ -32,12 +36,20 @@ public class ArrivalWindowTest { final ArrivalWindow windowWithNano = new ArrivalWindow(4); final long toNano = 100L; - -windowWithNano.add(111 * toNano); -windowWithNano.add(222 * toNano); -windowWithNano.add(333 * toNano); -windowWithNano.add(444 * toNano); -windowWithNano.add(555 * toNano); +InetAddress ep; +try +{ +ep = InetAddress.getLocalHost(); +} +catch (UnknownHostException e) +{ +throw new RuntimeException(e); +} +windowWithNano.add(111 * toNano, ep); +windowWithNano.add(222 * toNano, ep); +windowWithNano.add(333 * toNano, ep); +windowWithNano.add(444 * toNano, ep); +windowWithNano.add(555 * toNano, ep); //all good assertEquals(1.0, windowWithNano.phi(666 * toNano), 0.01);
[2/6] cassandra git commit: Improve FD logging when the arrival time is ignored.
Improve FD logging when the arrival time is ignored. Patch by Brandon Williams for CASSANDRA-8245 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/eb9c5bbc Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/eb9c5bbc Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/eb9c5bbc Branch: refs/heads/cassandra-2.1 Commit: eb9c5bbcfdaf498967052e02af65fa9f8167887d Parents: a223082 Author: Brandon Williams Authored: Tue Jan 6 09:30:07 2015 -0600 Committer: Brandon Williams Committed: Tue Jan 6 09:30:07 2015 -0600 -- .../apache/cassandra/gms/FailureDetector.java | 8 +++ .../apache/cassandra/gms/ArrivalWindowTest.java | 24 +++- 2 files changed, 22 insertions(+), 10 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb9c5bbc/src/java/org/apache/cassandra/gms/FailureDetector.java -- diff --git a/src/java/org/apache/cassandra/gms/FailureDetector.java b/src/java/org/apache/cassandra/gms/FailureDetector.java index 60729d3..e247e48 100644 --- a/src/java/org/apache/cassandra/gms/FailureDetector.java +++ b/src/java/org/apache/cassandra/gms/FailureDetector.java @@ -211,12 +211,12 @@ public class FailureDetector implements IFailureDetector, FailureDetectorMBean { // avoid adding an empty ArrivalWindow to the Map heartbeatWindow = new ArrivalWindow(SAMPLE_SIZE); -heartbeatWindow.add(now); +heartbeatWindow.add(now, ep); arrivalSamples.put(ep, heartbeatWindow); } else { -heartbeatWindow.add(now); +heartbeatWindow.add(now, ep); } } @@ -326,7 +326,7 @@ class ArrivalWindow } } -synchronized void add(long value) +synchronized void add(long value, InetAddress ep) { assert tLast >= 0; if (tLast > 0L) @@ -335,7 +335,7 @@ class ArrivalWindow if (interArrivalTime <= MAX_INTERVAL_IN_NANO) arrivalIntervals.add(interArrivalTime); else -logger.debug("Ignoring interval time of {}", interArrivalTime); +logger.debug("Ignoring interval time of {} for {}", interArrivalTime, ep); } else { http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb9c5bbc/test/unit/org/apache/cassandra/gms/ArrivalWindowTest.java -- diff --git a/test/unit/org/apache/cassandra/gms/ArrivalWindowTest.java b/test/unit/org/apache/cassandra/gms/ArrivalWindowTest.java index e678d86..511511b 100644 --- a/test/unit/org/apache/cassandra/gms/ArrivalWindowTest.java +++ b/test/unit/org/apache/cassandra/gms/ArrivalWindowTest.java @@ -25,6 +25,10 @@ import static org.junit.Assert.*; import org.junit.Test; +import java.lang.RuntimeException; +import java.net.InetAddress; +import java.net.UnknownHostException; + public class ArrivalWindowTest { @Test @@ -32,12 +36,20 @@ public class ArrivalWindowTest { final ArrivalWindow windowWithNano = new ArrivalWindow(4); final long toNano = 100L; - -windowWithNano.add(111 * toNano); -windowWithNano.add(222 * toNano); -windowWithNano.add(333 * toNano); -windowWithNano.add(444 * toNano); -windowWithNano.add(555 * toNano); +InetAddress ep; +try +{ +ep = InetAddress.getLocalHost(); +} +catch (UnknownHostException e) +{ +throw new RuntimeException(e); +} +windowWithNano.add(111 * toNano, ep); +windowWithNano.add(222 * toNano, ep); +windowWithNano.add(333 * toNano, ep); +windowWithNano.add(444 * toNano, ep); +windowWithNano.add(555 * toNano, ep); //all good assertEquals(1.0, windowWithNano.phi(666 * toNano), 0.01);
[4/6] cassandra git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/136042ec Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/136042ec Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/136042ec Branch: refs/heads/trunk Commit: 136042ec3985bbacfcf55333ef5668779d2744fe Parents: e83304d eb9c5bb Author: Brandon Williams Authored: Tue Jan 6 09:31:05 2015 -0600 Committer: Brandon Williams Committed: Tue Jan 6 09:31:05 2015 -0600 -- .../apache/cassandra/gms/FailureDetector.java | 8 +++ .../apache/cassandra/gms/ArrivalWindowTest.java | 24 +++- 2 files changed, 22 insertions(+), 10 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/136042ec/src/java/org/apache/cassandra/gms/FailureDetector.java --
[1/6] cassandra git commit: Improve FD logging when the arrival time is ignored.
Repository: cassandra Updated Branches: refs/heads/cassandra-2.0 a22308294 -> eb9c5bbcf refs/heads/cassandra-2.1 e83304ddf -> 136042ec3 refs/heads/trunk 0f024a619 -> ffb7f6491 Improve FD logging when the arrival time is ignored. Patch by Brandon Williams for CASSANDRA-8245 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/eb9c5bbc Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/eb9c5bbc Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/eb9c5bbc Branch: refs/heads/cassandra-2.0 Commit: eb9c5bbcfdaf498967052e02af65fa9f8167887d Parents: a223082 Author: Brandon Williams Authored: Tue Jan 6 09:30:07 2015 -0600 Committer: Brandon Williams Committed: Tue Jan 6 09:30:07 2015 -0600 -- .../apache/cassandra/gms/FailureDetector.java | 8 +++ .../apache/cassandra/gms/ArrivalWindowTest.java | 24 +++- 2 files changed, 22 insertions(+), 10 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb9c5bbc/src/java/org/apache/cassandra/gms/FailureDetector.java -- diff --git a/src/java/org/apache/cassandra/gms/FailureDetector.java b/src/java/org/apache/cassandra/gms/FailureDetector.java index 60729d3..e247e48 100644 --- a/src/java/org/apache/cassandra/gms/FailureDetector.java +++ b/src/java/org/apache/cassandra/gms/FailureDetector.java @@ -211,12 +211,12 @@ public class FailureDetector implements IFailureDetector, FailureDetectorMBean { // avoid adding an empty ArrivalWindow to the Map heartbeatWindow = new ArrivalWindow(SAMPLE_SIZE); -heartbeatWindow.add(now); +heartbeatWindow.add(now, ep); arrivalSamples.put(ep, heartbeatWindow); } else { -heartbeatWindow.add(now); +heartbeatWindow.add(now, ep); } } @@ -326,7 +326,7 @@ class ArrivalWindow } } -synchronized void add(long value) +synchronized void add(long value, InetAddress ep) { assert tLast >= 0; if (tLast > 0L) @@ -335,7 +335,7 @@ class ArrivalWindow if (interArrivalTime <= MAX_INTERVAL_IN_NANO) arrivalIntervals.add(interArrivalTime); else -logger.debug("Ignoring interval time of {}", interArrivalTime); +logger.debug("Ignoring interval time of {} for {}", interArrivalTime, ep); } else { http://git-wip-us.apache.org/repos/asf/cassandra/blob/eb9c5bbc/test/unit/org/apache/cassandra/gms/ArrivalWindowTest.java -- diff --git a/test/unit/org/apache/cassandra/gms/ArrivalWindowTest.java b/test/unit/org/apache/cassandra/gms/ArrivalWindowTest.java index e678d86..511511b 100644 --- a/test/unit/org/apache/cassandra/gms/ArrivalWindowTest.java +++ b/test/unit/org/apache/cassandra/gms/ArrivalWindowTest.java @@ -25,6 +25,10 @@ import static org.junit.Assert.*; import org.junit.Test; +import java.lang.RuntimeException; +import java.net.InetAddress; +import java.net.UnknownHostException; + public class ArrivalWindowTest { @Test @@ -32,12 +36,20 @@ public class ArrivalWindowTest { final ArrivalWindow windowWithNano = new ArrivalWindow(4); final long toNano = 100L; - -windowWithNano.add(111 * toNano); -windowWithNano.add(222 * toNano); -windowWithNano.add(333 * toNano); -windowWithNano.add(444 * toNano); -windowWithNano.add(555 * toNano); +InetAddress ep; +try +{ +ep = InetAddress.getLocalHost(); +} +catch (UnknownHostException e) +{ +throw new RuntimeException(e); +} +windowWithNano.add(111 * toNano, ep); +windowWithNano.add(222 * toNano, ep); +windowWithNano.add(333 * toNano, ep); +windowWithNano.add(444 * toNano, ep); +windowWithNano.add(555 * toNano, ep); //all good assertEquals(1.0, windowWithNano.phi(666 * toNano), 0.01);
[jira] [Resolved] (CASSANDRA-8245) Cassandra nodes periodically die in 2-DC configuration
[ https://issues.apache.org/jira/browse/CASSANDRA-8245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Williams resolved CASSANDRA-8245. - Resolution: Fixed Fix Version/s: 2.1.3 2.0.12 I added the host to the logging. Since there's nothing else actionable here, I'm resolving this ticket. > Cassandra nodes periodically die in 2-DC configuration > -- > > Key: CASSANDRA-8245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8245 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Scientific Linux release 6.5 > java version "1.7.0_51" > Cassandra 2.0.9 >Reporter: Oleg Poleshuk >Assignee: Brandon Williams >Priority: Minor > Fix For: 2.0.12, 2.1.3 > > Attachments: stack1.txt, stack2.txt, stack3.txt, stack4.txt, > stack5.txt > > > We have 2 DCs with 3 nodes in each. > Second DC periodically has 1-2 nodes down. > Looks like it looses connectivity with another nodes and then Gossiper starts > to accumulate tasks until Cassandra dies with OOM. > WARN [MemoryMeter:1] 2014-08-12 14:34:59,803 Memtable.java (line 470) setting > live ratio to maximum of 64.0 instead of Infinity > WARN [GossipTasks:1] 2014-08-12 14:44:34,866 Gossiper.java (line 637) Gossip > stage has 1 pending tasks; skipping status check (no nodes will be marked > down) > WARN [GossipTasks:1] 2014-08-12 14:44:35,968 Gossiper.java (line 637) Gossip > stage has 4 pending tasks; skipping status check (no nodes will be marked > down) > WARN [GossipTasks:1] 2014-08-12 14:44:37,070 Gossiper.java (line 637) Gossip > stage has 8 pending tasks; skipping status check (no nodes will be marked > down) > WARN [GossipTasks:1] 2014-08-12 14:44:38,171 Gossiper.java (line 637) Gossip > stage has 11 pending tasks; skipping status check (no nodes will be marked > down) > ... > WARN [GossipTasks:1] 2014-10-06 21:42:51,575 Gossiper.java (line 637) Gossip > stage has 1014764 pending tasks; skipping status check (no nodes will be > marked down) > WARN [New I/O worker #13] 2014-10-06 21:54:27,010 Slf4JLogger.java (line 76) > Unexpected exception in the selector loop. > java.lang.OutOfMemoryError: Java heap space > Also those lines but not sure it is relevant: > DEBUG [GossipStage:1] 2014-08-12 11:33:18,801 FailureDetector.java (line 338) > Ignoring interval time of 2085963047 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8499) Ensure SSTableWriter cleans up properly after failure
[ https://issues.apache.org/jira/browse/CASSANDRA-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266284#comment-14266284 ] Marcus Eriksson commented on CASSANDRA-8499: 2.1 v2 seems to double-close the files, first when we switch the writer, then when we call abort(), running SSTableRewriterTest.testNumberOfFiles_abort() outputs this: WARN 15:43:17 close(81) failed, errno (9). > Ensure SSTableWriter cleans up properly after failure > - > > Key: CASSANDRA-8499 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8499 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Benedict >Assignee: Benedict > Fix For: 2.0.12 > > Attachments: 8499-20.txt, 8499-20v2, 8499-21.txt, 8499-21v2 > > > In 2.0 we do not free a bloom filter, in 2.1 we do not free a small piece of > offheap memory for writing compression metadata. In both we attempt to flush > the BF despite having encountered an exception, making the exception slow to > propagate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8541) References to non-existent/deprecated CqlPagingInputFormat in code
[ https://issues.apache.org/jira/browse/CASSANDRA-8541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266355#comment-14266355 ] Jeremiah Jordan commented on CASSANDRA-8541: Did you test this? I don't think you can just drop in replace CqlPagingInputFormat with CqlInputFormat > References to non-existent/deprecated CqlPagingInputFormat in code > -- > > Key: CASSANDRA-8541 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8541 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Reporter: Rekha Joshi >Assignee: Rekha Joshi > Labels: hadoop > Fix For: 2.0.12 > > Attachments: CASSANDRA-8541.txt > > > On Mac 10.9.5, Java 1.7, latest cassandra trunk - > References to non-existent/deprecated CqlPagingInputFormat in code. > As per Changes.txt/7570 both CqlPagingInputFormat and CqlPagingRecordReader > are removed, but lingering references in WordCount,CqlStorage.. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8325) Cassandra 2.1.x fails to start on FreeBSD (JVM crash)
[ https://issues.apache.org/jira/browse/CASSANDRA-8325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Shuler updated CASSANDRA-8325: -- Reproduced In: 2.1.2, 2.1.1 (was: 2.1.1, 2.1.2) Tester: Michael Shuler > Cassandra 2.1.x fails to start on FreeBSD (JVM crash) > - > > Key: CASSANDRA-8325 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8325 > Project: Cassandra > Issue Type: Bug > Environment: FreeBSD 10.0 with openjdk version "1.7.0_71", 64-Bit > Server VM >Reporter: Leonid Shalupov > Attachments: hs_err_pid1856.log, system.log, unsafeCopy1.txt, > untested_8325.patch > > > See attached error file after JVM crash > {quote} > FreeBSD xxx.intellij.net 10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789: Thu > Jan 16 22:34:59 UTC 2014 > r...@snap.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 > {quote} > {quote} > % java -version > openjdk version "1.7.0_71" > OpenJDK Runtime Environment (build 1.7.0_71-b14) > OpenJDK 64-Bit Server VM (build 24.71-b01, mixed mode) > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8457) nio MessagingService
[ https://issues.apache.org/jira/browse/CASSANDRA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266402#comment-14266402 ] Ariel Weisberg commented on CASSANDRA-8457: --- I think I stumbled onto what is going on based on Benedict's suggestion to disable TCP no delay. It looks like there is a small message performance issue. This is something I have seen before in EC2 where you can only send a surprisingly small number of messages in/out of a VM. I don't have the numbers from when I micro benchmarked it, but it is something like 450k messages with TCP no delay and a million or so without. Adding more sockets helps but it doesn't even double the number of messages in/out. Throwing more cores at the problem doesn't help you just end up with under utilized cores which matches the mysterious levels of starvation I was seeing in C* even though I was exposing sufficient concurrency. 14 servers nodes. 6 client nodes. 500 threads per client. Server started with "row_cache_size_in_mb" : "2000", "key_cache_size_in_mb":"500", "rpc_max_threads" : "1024", "rpc_min_threads" : "16", "native_transport_max_threads" : "1024" 8-gig old gen, 2 gig new gen. Client running CL=ALL and the same schema I have been using throughout this ticket. With no delay off First set of runs 390264 387958 392322 After replacing 10 instances 366579 365818 378221 No delay on 162987 Modified trunk to fix a bug batching messages and add a configurable window for coalescing multiple messages into a socket see https://github.com/aweisberg/cassandra/compare/f733996...49c6609 ||Coalesce window microseconds|Throughput|| |250| 502614| |200| 496206| |150| 487195| |100| 423415| |50| 326648| |25| 308175| |12| 292894| |6| 268456| |0| 153688| I did not expect get mileage out of coalescing at the application level but it works extremely well. CPU utilization is still low at 1800%. There seems to be less correlation between CPU utilization and throughput as I vary the coalescing window and throughput changes dramatically. I do see that core 0 is looking pretty saturated and is only 10% idle. That might be the next or actual bottleneck. What role this optimization plays at different cluster sizes is an important question. There hast to be a tipping point where coalescing stops working because not enough packets go to each end point at the same time. With vnodes it wouldn't be unusual to be communicating with a large number of other hosts right? It also takes a significant amount of additional latency to get the mileage at high levels of throughput, but at lower concurrency there is no benefit and it will probably show up as decreased throughput. It makes it tough to crank it up as a default. Either it is adaptive or most people don't get the benefit. At high levels of throughput it is a clear latency win. Latency is much lower for individual requests on average. Making this a config option is viable as a starting point. Possibly a separate option for local/remote DC coalescing. Ideally we could make it adapt to the workload. I am going to chase down what impact coalescing has at lower levels of concurrency so we can quantify the cost of turning it on. I'm also going to try and get to the bottom of all interrupts going to core 0. Maybe it is the real problem and coalescing is just a band aid to get more throughput. > nio MessagingService > > > Key: CASSANDRA-8457 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8457 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Jonathan Ellis >Assignee: Ariel Weisberg > Labels: performance > Fix For: 3.0 > > > Thread-per-peer (actually two each incoming and outbound) is a big > contributor to context switching, especially for larger clusters. Let's look > at switching to nio, possibly via Netty. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8457) nio MessagingService
[ https://issues.apache.org/jira/browse/CASSANDRA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266402#comment-14266402 ] Ariel Weisberg edited comment on CASSANDRA-8457 at 1/6/15 5:05 PM: --- I think I stumbled onto what is going on based on Benedict's suggestion to disable TCP no delay. It looks like there is a small message performance issue. This is something I have seen before in EC2 where you can only send a surprisingly small number of messages in/out of a VM. I don't have the numbers from when I micro benchmarked it, but it is something like 450k messages with TCP no delay and a million or so without. Adding more sockets helps but it doesn't even double the number of messages in/out. Throwing more cores at the problem doesn't help you just end up with under utilized cores which matches the mysterious levels of starvation I was seeing in C* even though I was exposing sufficient concurrency. 14 servers nodes. 6 client nodes. 500 threads per client. Server started with "row_cache_size_in_mb" : "2000", "key_cache_size_in_mb":"500", "rpc_max_threads" : "1024", "rpc_min_threads" : "16", "native_transport_max_threads" : "1024" 8-gig old gen, 2 gig new gen. Client running CL=ALL and the same schema I have been using throughout this ticket. With no delay off First set of runs 390264 387958 392322 After replacing 10 instances 366579 365818 378221 No delay on 162987 Modified trunk to fix a bug batching messages and add a configurable window for coalescing multiple messages into a socket see https://github.com/aweisberg/cassandra/compare/f733996...49c6609 ||Coalesce window microseconds|Throughput|| |250| 502614| |200| 496206| |150| 487195| |100| 423415| |50| 326648| |25| 308175| |12| 292894| |6| 268456| |0| 153688| I did not expect to get mileage out of coalescing at the application level but it works extremely well. CPU utilization is still low at 1800%. There seems to be less correlation between CPU utilization and throughput as I vary the coalescing window and throughput changes dramatically. I do see that core 0 is looking pretty saturated and is only 10% idle. That might be the next or actual bottleneck. What role this optimization plays at different cluster sizes is an important question. There has to be a tipping point where coalescing stops working because not enough packets go to each end point at the same time. With vnodes it wouldn't be unusual to be communicating with a large number of other hosts right? It also takes a significant amount of additional latency to get the mileage at high levels of throughput, but at lower concurrency there is no benefit and it will probably show up as decreased throughput. It makes it tough to crank it up as a default. Either it is adaptive or most people don't get the benefit. At high levels of throughput it is a clear latency win. Latency is much lower for individual requests on average. Making this a config option is viable as a starting point. Possibly a separate option for local/remote DC coalescing. Ideally we could make it adapt to the workload. I am going to chase down what impact coalescing has at lower levels of concurrency so we can quantify the cost of turning it on. I'm also going to try and get to the bottom of all interrupts going to core 0. Maybe it is the real problem and coalescing is just a band aid to get more throughput. was (Author: aweisberg): I think I stumbled onto what is going on based on Benedict's suggestion to disable TCP no delay. It looks like there is a small message performance issue. This is something I have seen before in EC2 where you can only send a surprisingly small number of messages in/out of a VM. I don't have the numbers from when I micro benchmarked it, but it is something like 450k messages with TCP no delay and a million or so without. Adding more sockets helps but it doesn't even double the number of messages in/out. Throwing more cores at the problem doesn't help you just end up with under utilized cores which matches the mysterious levels of starvation I was seeing in C* even though I was exposing sufficient concurrency. 14 servers nodes. 6 client nodes. 500 threads per client. Server started with "row_cache_size_in_mb" : "2000", "key_cache_size_in_mb":"500", "rpc_max_threads" : "1024", "rpc_min_threads" : "16", "native_transport_max_threads" : "1024" 8-gig old gen, 2 gig new gen. Client running CL=ALL and the same schema I have been using throughout this ticket. With no delay off First set of runs 390264 387958 392322 After replacing 10 instances 366579 365818 378221 No delay on 162987 Modified trunk to fix a bug batching messages and add a configurable window for coalescing multiple messages into a socket see https://github.com/aweisberg/cassandra/compa
[jira] [Commented] (CASSANDRA-8457) nio MessagingService
[ https://issues.apache.org/jira/browse/CASSANDRA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266422#comment-14266422 ] Ariel Weisberg commented on CASSANDRA-8457: --- https://forums.aws.amazon.com/thread.jspa?messageID=459260 This looks like a Xen/EC2 issue. I'll bet bare metal never has this issue because it has multiple interrupt vectors for NICs. The only work around is using multiple elastic network interfaces and doing something to make that practical for intra-cluster communication. > nio MessagingService > > > Key: CASSANDRA-8457 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8457 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Jonathan Ellis >Assignee: Ariel Weisberg > Labels: performance > Fix For: 3.0 > > > Thread-per-peer (actually two each incoming and outbound) is a big > contributor to context switching, especially for larger clusters. Let's look > at switching to nio, possibly via Netty. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7653) Add role based access control to Cassandra
[ https://issues.apache.org/jira/browse/CASSANDRA-7653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266424#comment-14266424 ] Sam Tunnicliffe commented on CASSANDRA-7653: I've been thinking about constuctInitialSaslToken too and it is truly unpleasant. It's there to support Thrift clients which could be sending arbitrary k/v pairs to a custom IAuthenticator via the login() call. So we have to support that somehow without changing the thrift interface. constructInitialSaslToken is one way to do it, but it sucks so my current plan is to replace it with a legacyAuthenticate() method which impls can decide whether to support or not (if they support Thrift and/or native protocol v1 authentication). On the second point, I think it's doable to support custom options using json syntax. Something like: {code} CREATE ROLE foo WITH PASSWORD 'bar' AND OPTIONS {'a' : 'aaa', 'b' : 1} NOSUPERUSER LOGIN; {code} > Add role based access control to Cassandra > -- > > Key: CASSANDRA-7653 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7653 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Mike Adamson >Assignee: Sam Tunnicliffe > Fix For: 3.0 > > Attachments: 7653.patch, CQLSmokeTest.java, cql_smoke_test.py > > > The current authentication model supports granting permissions to individual > users. While this is OK for small or medium organizations wanting to > implement authorization, it does not work well in large organizations because > of the overhead of having to maintain the permissions for each user. > Introducing roles into the authentication model would allow sets of > permissions to be controlled in one place as a role and then the role granted > to users. Roles should also be able to be granted to other roles to allow > hierarchical sets of permissions to be built up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8403) limit disregarded when paging with IN clause under certain conditions
[ https://issues.apache.org/jira/browse/CASSANDRA-8403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266425#comment-14266425 ] Benjamin Lerer commented on CASSANDRA-8403: --- I tried to reproduce the problem on a single node and on a 3 nodes cluster using cqlsh but without success. Could you tell me how many nodes you were using and how you interacted with Cassandra? > limit disregarded when paging with IN clause under certain conditions > - > > Key: CASSANDRA-8403 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8403 > Project: Cassandra > Issue Type: Bug >Reporter: Russ Hatch >Assignee: Benjamin Lerer >Priority: Minor > > This issue was originally reported on the python-driver userlist and > confirmed by [~aholmber] > When: > page_size < limit < data size, > the limit value is disregarded and all rows are paged back. > to repro: > create a table and populate it with two partitions > CREATE TABLE paging_test ( id int, value text, PRIMARY KEY (id, value) ) > Add data: in one partition create 10 rows, an in a second partition create 20 > rows > perform a query with page_size of 10 and a LIMIT of 20, like so: > SELECT * FROM paging_test where id in (1,2) LIMIT 20; > The limit is disregarded and three pages of 10 records each will be returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7875) Prepared statements using dropped indexes are not handled correctly
[ https://issues.apache.org/jira/browse/CASSANDRA-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266433#comment-14266433 ] Arindam commented on CASSANDRA-7875: Hello, Just getting the same error in 2.0.9. Is there any steps to resolution or upgrade is the only option? Thanks > Prepared statements using dropped indexes are not handled correctly > --- > > Key: CASSANDRA-7875 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7875 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Tyler Hobbs >Assignee: Tyler Hobbs >Priority: Minor > Fix For: 2.1.3 > > Attachments: repro.py > > > When select statements are prepared, we verify that the column restrictions > use indexes (where necessary). However, we don't perform a similar check > when the statement is executed, so it fails somewhere further down the line. > In this case, it hits an assertion: > {noformat} > java.lang.AssertionError: Sequential scan with filters is not supported (if > you just created an index, you need to wait for the creation to be propagated > to all nodes before querying it) > at > org.apache.cassandra.db.filter.ExtendedFilter$WithClauses.getExtraFilter(ExtendedFilter.java:259) > at > org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1759) > at > org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1709) > at > org.apache.cassandra.db.PagedRangeCommand.executeLocally(PagedRangeCommand.java:119) > at > org.apache.cassandra.service.StorageProxy$LocalRangeSliceRunnable.runMayThrow(StorageProxy.java:1394) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1936) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > {noformat} > During execution, we should check that the indexes still exist and provide a > better error if they do not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-7875) Prepared statements using dropped indexes are not handled correctly
[ https://issues.apache.org/jira/browse/CASSANDRA-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266433#comment-14266433 ] Arindam edited comment on CASSANDRA-7875 at 1/6/15 5:20 PM: Hello, Just getting the same error in 2.0.9. Is there any steps to resolution or upgrade is the only option? Note: The index is custom secondary index and it does exists as it has been created quite long back. The environment is having only 1 node cluster. Thanks Arindam was (Author: abose78): Hello, Just getting the same error in 2.0.9. Is there any steps to resolution or upgrade is the only option? Thanks > Prepared statements using dropped indexes are not handled correctly > --- > > Key: CASSANDRA-7875 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7875 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Tyler Hobbs >Assignee: Tyler Hobbs >Priority: Minor > Fix For: 2.1.3 > > Attachments: repro.py > > > When select statements are prepared, we verify that the column restrictions > use indexes (where necessary). However, we don't perform a similar check > when the statement is executed, so it fails somewhere further down the line. > In this case, it hits an assertion: > {noformat} > java.lang.AssertionError: Sequential scan with filters is not supported (if > you just created an index, you need to wait for the creation to be propagated > to all nodes before querying it) > at > org.apache.cassandra.db.filter.ExtendedFilter$WithClauses.getExtraFilter(ExtendedFilter.java:259) > at > org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1759) > at > org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1709) > at > org.apache.cassandra.db.PagedRangeCommand.executeLocally(PagedRangeCommand.java:119) > at > org.apache.cassandra.service.StorageProxy$LocalRangeSliceRunnable.runMayThrow(StorageProxy.java:1394) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1936) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > {noformat} > During execution, we should check that the indexes still exist and provide a > better error if they do not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8457) nio MessagingService
[ https://issues.apache.org/jira/browse/CASSANDRA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266422#comment-14266422 ] Ariel Weisberg edited comment on CASSANDRA-8457 at 1/6/15 5:31 PM: --- https://forums.aws.amazon.com/thread.jspa?messageID=459260 This looks like a Xen/EC2 issue. I'll bet bare metal never has this issue because it has multiple interrupt vectors for NICs. The only work around is using multiple elastic network interfaces and doing something to make that practical for intra-cluster communication. Apparently the right scale instances I am using don't have enhanced networking. To get enhanced networking I need to use VPC. I am not sure if starting instances with enhanced networking is possible via right scale, but I am going to find out. I don't know if enhanced networking addresses the interrupt vector issue. Will do more digging. was (Author: aweisberg): https://forums.aws.amazon.com/thread.jspa?messageID=459260 This looks like a Xen/EC2 issue. I'll bet bare metal never has this issue because it has multiple interrupt vectors for NICs. The only work around is using multiple elastic network interfaces and doing something to make that practical for intra-cluster communication. > nio MessagingService > > > Key: CASSANDRA-8457 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8457 > Project: Cassandra > Issue Type: New Feature > Components: Core >Reporter: Jonathan Ellis >Assignee: Ariel Weisberg > Labels: performance > Fix For: 3.0 > > > Thread-per-peer (actually two each incoming and outbound) is a big > contributor to context switching, especially for larger clusters. Let's look > at switching to nio, possibly via Netty. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7875) Prepared statements using dropped indexes are not handled correctly
[ https://issues.apache.org/jira/browse/CASSANDRA-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266463#comment-14266463 ] Aleksey Yeschenko commented on CASSANDRA-7875: -- Since 2.0 doesn't have CASSANDRA-7923 in, what you can do is re-prepare the statement. > Prepared statements using dropped indexes are not handled correctly > --- > > Key: CASSANDRA-7875 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7875 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Tyler Hobbs >Assignee: Tyler Hobbs >Priority: Minor > Fix For: 2.1.3 > > Attachments: repro.py > > > When select statements are prepared, we verify that the column restrictions > use indexes (where necessary). However, we don't perform a similar check > when the statement is executed, so it fails somewhere further down the line. > In this case, it hits an assertion: > {noformat} > java.lang.AssertionError: Sequential scan with filters is not supported (if > you just created an index, you need to wait for the creation to be propagated > to all nodes before querying it) > at > org.apache.cassandra.db.filter.ExtendedFilter$WithClauses.getExtraFilter(ExtendedFilter.java:259) > at > org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1759) > at > org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1709) > at > org.apache.cassandra.db.PagedRangeCommand.executeLocally(PagedRangeCommand.java:119) > at > org.apache.cassandra.service.StorageProxy$LocalRangeSliceRunnable.runMayThrow(StorageProxy.java:1394) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1936) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > {noformat} > During execution, we should check that the indexes still exist and provide a > better error if they do not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7875) Prepared statements using dropped indexes are not handled correctly
[ https://issues.apache.org/jira/browse/CASSANDRA-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266466#comment-14266466 ] Aleksey Yeschenko commented on CASSANDRA-7875: -- As for this ticket, I'm not sure that just adding an extra runtime check is the way to go, or sufficient. Since this event (dropping an index) should be very rare in production, we should probably just extend CASSANDRA-7910 logic here, and just invalidate the potentially affected statements. > Prepared statements using dropped indexes are not handled correctly > --- > > Key: CASSANDRA-7875 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7875 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Tyler Hobbs >Assignee: Tyler Hobbs >Priority: Minor > Fix For: 2.1.3 > > Attachments: repro.py > > > When select statements are prepared, we verify that the column restrictions > use indexes (where necessary). However, we don't perform a similar check > when the statement is executed, so it fails somewhere further down the line. > In this case, it hits an assertion: > {noformat} > java.lang.AssertionError: Sequential scan with filters is not supported (if > you just created an index, you need to wait for the creation to be propagated > to all nodes before querying it) > at > org.apache.cassandra.db.filter.ExtendedFilter$WithClauses.getExtraFilter(ExtendedFilter.java:259) > at > org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1759) > at > org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1709) > at > org.apache.cassandra.db.PagedRangeCommand.executeLocally(PagedRangeCommand.java:119) > at > org.apache.cassandra.service.StorageProxy$LocalRangeSliceRunnable.runMayThrow(StorageProxy.java:1394) > at > org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1936) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > {noformat} > During execution, we should check that the indexes still exist and provide a > better error if they do not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7139) Default concurrent_compactors is probably too high
[ https://issues.apache.org/jira/browse/CASSANDRA-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266473#comment-14266473 ] Catalin Alexandru Zamfir commented on CASSANDRA-7139: - Our set-up was RAID5 and the min (numberOfDisk, numberOfCores) would just be 2, when we have 40+ cores. The commented "concurrent_compactors" would be "2" meaning that a lot of SSTables are accumulating in high-cardinality tables (where the partition key is an UUID-type) because the compaction is limited to "2". Looking at "dstat" even if we've set compaction_throughput_in_mb_per_sec to 192 (spinning disk) the dstat -lrv1 disk write maxes out at 10MB/s. IMHO, the concurrent_compactors should be number_of_cores/compaction_throughput_in_mb_per_sec * 100 which in our case (40 cores) gives around 20/21 compactors. And on 8 cores (8/192 * 100 gives 4 concurrent compactors). > Default concurrent_compactors is probably too high > -- > > Key: CASSANDRA-7139 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7139 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Benedict >Assignee: Jonathan Ellis >Priority: Minor > Fix For: 2.1 rc1 > > Attachments: 7139.txt > > > The default number of concurrent compactors is probably too high for modern > hardware with spinning disks for storage: A modern blade can easily have 24+ > Cores, which would result in a default of 24 concurrent compactions. This not > only increases random IO, it also keeps around a lot of obsoleted files for > an unnecessarily long time, as each compaction keeps references to any > possibly overlapping files that it isn't itself compacting - but these can > have been obsoleted part way through by compactions that finished earlier. If > you factor in the default compaction throughput rate of 16Mb/s, anything but > a single default concurrent_compactor makes very little sense, as a single > thread should always be able to handle 16Mb/s, will cause less interference > with other processes, and permits obsoleted files to be immediately removed. > See [http://imgur.com/HDqhxFp] for a graph demonstrating the result of making > this change on a box with 24-cores and 8Tb of storage (first spike is default > settings) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7622) Implement virtual tables
[ https://issues.apache.org/jira/browse/CASSANDRA-7622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-7622: - Fix Version/s: (was: 3.0) 3.1 > Implement virtual tables > > > Key: CASSANDRA-7622 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7622 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Tupshin Harper > Fix For: 3.1 > > > There are a variety of reasons to want virtual tables, which would be any > table that would be backed by an API, rather than data explicitly managed and > stored as sstables. > One possible use case would be to expose JMX data through CQL as a > resurrection of CASSANDRA-3527. > Another is a more general framework to implement the ability to expose yaml > configuration information. So it would be an alternate approach to > CASSANDRA-7370. > A possible implementation would be in terms of CASSANDRA-7443, but I am not > presupposing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7139) Default concurrent_compactors is probably too high
[ https://issues.apache.org/jira/browse/CASSANDRA-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266493#comment-14266493 ] Benedict commented on CASSANDRA-7139: - This is only the default. You are recommended to tune this default based on your own system's behaviour. With modern SSDs and many cores, many concurrent compactors is a great idea. For spinning disk setups, it can be terrible, and we want to avoid terrible default decisions. Either way, I suspect the problem you are encountering is entirely different, i.e. that the default _compaction_throughput_mb_per_sec_ is 10, which would be why you are maxing out at exactly 10MB/s. > Default concurrent_compactors is probably too high > -- > > Key: CASSANDRA-7139 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7139 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Benedict >Assignee: Jonathan Ellis >Priority: Minor > Fix For: 2.1 rc1 > > Attachments: 7139.txt > > > The default number of concurrent compactors is probably too high for modern > hardware with spinning disks for storage: A modern blade can easily have 24+ > Cores, which would result in a default of 24 concurrent compactions. This not > only increases random IO, it also keeps around a lot of obsoleted files for > an unnecessarily long time, as each compaction keeps references to any > possibly overlapping files that it isn't itself compacting - but these can > have been obsoleted part way through by compactions that finished earlier. If > you factor in the default compaction throughput rate of 16Mb/s, anything but > a single default concurrent_compactor makes very little sense, as a single > thread should always be able to handle 16Mb/s, will cause less interference > with other processes, and permits obsoleted files to be immediately removed. > See [http://imgur.com/HDqhxFp] for a graph demonstrating the result of making > this change on a box with 24-cores and 8Tb of storage (first spike is default > settings) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8552) Large compactions run out of off-heap RAM
[ https://issues.apache.org/jira/browse/CASSANDRA-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266510#comment-14266510 ] Philip Thompson commented on CASSANDRA-8552: Yes, I will spin up a small cluster of Ubuntu 14.4 m1.xlarge nodes on EC2. > Large compactions run out of off-heap RAM > - > > Key: CASSANDRA-8552 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8552 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 14.4 > AWS EC2 > 12 m1.xlarge nodes [4 cores, 16GB RAM, 1TB storage (251GB Used)] > Java build 1.7.0_55-b13 and build 1.8.0_25-b17 >Reporter: Brent Haines >Assignee: Benedict >Priority: Blocker > Fix For: 2.1.3 > > Attachments: Screen Shot 2015-01-02 at 9.36.11 PM.png, fhandles.log, > freelog.log, lsof.txt, meminfo.txt, sysctl.txt, system.log > > > We have a large table of storing, effectively event logs and a pair of > denormalized tables for indexing. > When updating from 2.0 to 2.1 we saw performance improvements, but some > random and silent crashes during nightly repairs. We lost a node (totally > corrupted) and replaced it. That node has never stabilized -- it simply can't > finish the compactions. > Smaller compactions finish. Larger compactions, like these two never finish - > {code} > pending tasks: 48 >compaction type keyspace table completed total > unit progress > Compaction data stories 16532973358 75977993784 > bytes 21.76% > Compaction data stories_by_text 10593780658 38555048812 > bytes 27.48% > Active compaction remaining time : 0h10m51s > {code} > We are not getting exceptions and are not running out of heap space. The > Ubuntu OOM killer is reaping the process after all of the memory is consumed. > We watch memory in the opscenter console and it will grow. If we turn off the > OOM killer for the process, it will run until everything else is killed > instead and then the kernel panics. > We have the following settings configured: > 2G Heap > 512M New > {code} > memtable_heap_space_in_mb: 1024 > memtable_offheap_space_in_mb: 1024 > memtable_allocation_type: heap_buffers > commitlog_total_space_in_mb: 2048 > concurrent_compactors: 1 > compaction_throughput_mb_per_sec: 128 > {code} > The compaction strategy is leveled (these are read-intensive tables that are > rarely updated) > I have tried every setting, every option and I have the system where the MTBF > is about an hour now, but we never finish compacting because there are some > large compactions pending. None of the GC tools or settings help because it > is not a GC problem. It is an off-heap memory problem. > We are getting these messages in our syslog > {code} > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219527] BUG: Bad page map in > process java pte:0320 pmd:2d6fa5067 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219545] addr:7fb820be3000 > vm_flags:0870 anon_vma: (null) mapping: (null) > index:7fb820be3 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219556] CPU: 3 PID: 27344 > Comm: java Tainted: GB3.13.0-24-generic #47-Ubuntu > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219559] 880028510e40 > 88020d43da98 81715ac4 7fb820be3000 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219565] 88020d43dae0 > 81174183 0320 0007fb820be3 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219568] 8802d6fa5f18 > 0320 7fb820be3000 7fb820be4000 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219572] Call Trace: > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219584] [] > dump_stack+0x45/0x56 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219591] [] > print_bad_pte+0x1a3/0x250 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219594] [] > vm_normal_page+0x69/0x80 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219598] [] > unmap_page_range+0x3bb/0x7f0 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219602] [] > unmap_single_vma+0x81/0xf0 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219605] [] > unmap_vmas+0x49/0x90 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219610] [] > exit_mmap+0x9c/0x170 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219617] [] > ? __delayacct_add_tsk+0x153/0x170 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219621] [] > mmput+0x5c/0x120 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219625] [] > do_exit+0x26c/0xa50 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219631] [] > ? __unqueue_futex+0x31/0x60 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219634] [] > ? futex_wait+0x126/0x290 > Jan 2 07:06:0
[jira] [Commented] (CASSANDRA-8552) Large compactions run out of off-heap RAM
[ https://issues.apache.org/jira/browse/CASSANDRA-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266533#comment-14266533 ] Benedict commented on CASSANDRA-8552: - Thanks. I would try first off with a single node and simply see if it is enough to trigger the scenario by writing, say, 100G of data to the node. The simplest explanations shouldn't require a whole cluster; if we fail to reproduce we can try complicating the setup. > Large compactions run out of off-heap RAM > - > > Key: CASSANDRA-8552 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8552 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 14.4 > AWS EC2 > 12 m1.xlarge nodes [4 cores, 16GB RAM, 1TB storage (251GB Used)] > Java build 1.7.0_55-b13 and build 1.8.0_25-b17 >Reporter: Brent Haines >Assignee: Benedict >Priority: Blocker > Fix For: 2.1.3 > > Attachments: Screen Shot 2015-01-02 at 9.36.11 PM.png, fhandles.log, > freelog.log, lsof.txt, meminfo.txt, sysctl.txt, system.log > > > We have a large table of storing, effectively event logs and a pair of > denormalized tables for indexing. > When updating from 2.0 to 2.1 we saw performance improvements, but some > random and silent crashes during nightly repairs. We lost a node (totally > corrupted) and replaced it. That node has never stabilized -- it simply can't > finish the compactions. > Smaller compactions finish. Larger compactions, like these two never finish - > {code} > pending tasks: 48 >compaction type keyspace table completed total > unit progress > Compaction data stories 16532973358 75977993784 > bytes 21.76% > Compaction data stories_by_text 10593780658 38555048812 > bytes 27.48% > Active compaction remaining time : 0h10m51s > {code} > We are not getting exceptions and are not running out of heap space. The > Ubuntu OOM killer is reaping the process after all of the memory is consumed. > We watch memory in the opscenter console and it will grow. If we turn off the > OOM killer for the process, it will run until everything else is killed > instead and then the kernel panics. > We have the following settings configured: > 2G Heap > 512M New > {code} > memtable_heap_space_in_mb: 1024 > memtable_offheap_space_in_mb: 1024 > memtable_allocation_type: heap_buffers > commitlog_total_space_in_mb: 2048 > concurrent_compactors: 1 > compaction_throughput_mb_per_sec: 128 > {code} > The compaction strategy is leveled (these are read-intensive tables that are > rarely updated) > I have tried every setting, every option and I have the system where the MTBF > is about an hour now, but we never finish compacting because there are some > large compactions pending. None of the GC tools or settings help because it > is not a GC problem. It is an off-heap memory problem. > We are getting these messages in our syslog > {code} > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219527] BUG: Bad page map in > process java pte:0320 pmd:2d6fa5067 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219545] addr:7fb820be3000 > vm_flags:0870 anon_vma: (null) mapping: (null) > index:7fb820be3 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219556] CPU: 3 PID: 27344 > Comm: java Tainted: GB3.13.0-24-generic #47-Ubuntu > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219559] 880028510e40 > 88020d43da98 81715ac4 7fb820be3000 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219565] 88020d43dae0 > 81174183 0320 0007fb820be3 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219568] 8802d6fa5f18 > 0320 7fb820be3000 7fb820be4000 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219572] Call Trace: > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219584] [] > dump_stack+0x45/0x56 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219591] [] > print_bad_pte+0x1a3/0x250 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219594] [] > vm_normal_page+0x69/0x80 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219598] [] > unmap_page_range+0x3bb/0x7f0 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219602] [] > unmap_single_vma+0x81/0xf0 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219605] [] > unmap_vmas+0x49/0x90 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219610] [] > exit_mmap+0x9c/0x170 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219617] [] > ? __delayacct_add_tsk+0x153/0x170 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219621] [] > mmput+0x5c/0x120 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219625] [] > do_exit+0x26c/0xa50 > Jan 2 07:06:00
[jira] [Commented] (CASSANDRA-8303) Provide "strict mode" for CQL Queries
[ https://issues.apache.org/jira/browse/CASSANDRA-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266550#comment-14266550 ] Jonathan Shook commented on CASSANDRA-8303: --- It might be nice if the auth system was always in play (when that auth provider is set), but the system defaults are applied to a virtual role with a name like "defaults". This cleans up any layering questions by casting the yaml defaults into the authz conceptual model. If a user isn't assigned to another defined role, they should be automatically assigned to the defaults role. Otherwise, explaining the result of layering them, even with precedence, might become overly cumbersome. With it, you can use both. > Provide "strict mode" for CQL Queries > - > > Key: CASSANDRA-8303 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8303 > Project: Cassandra > Issue Type: Improvement >Reporter: Anupam Arora > Fix For: 3.0 > > > Please provide a "strict mode" option in cassandra that will kick out any CQL > queries that are expensive, e.g. any query with ALLOWS FILTERING, > multi-partition queries, secondary index queries, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8552) Large compactions run out of off-heap RAM
[ https://issues.apache.org/jira/browse/CASSANDRA-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8552: --- Reproduced In: 2.1.2, 2.1.1 (was: 2.1.1, 2.1.2) Tester: Alan Boudreault > Large compactions run out of off-heap RAM > - > > Key: CASSANDRA-8552 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8552 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 14.4 > AWS EC2 > 12 m1.xlarge nodes [4 cores, 16GB RAM, 1TB storage (251GB Used)] > Java build 1.7.0_55-b13 and build 1.8.0_25-b17 >Reporter: Brent Haines >Assignee: Benedict >Priority: Blocker > Fix For: 2.1.3 > > Attachments: Screen Shot 2015-01-02 at 9.36.11 PM.png, fhandles.log, > freelog.log, lsof.txt, meminfo.txt, sysctl.txt, system.log > > > We have a large table of storing, effectively event logs and a pair of > denormalized tables for indexing. > When updating from 2.0 to 2.1 we saw performance improvements, but some > random and silent crashes during nightly repairs. We lost a node (totally > corrupted) and replaced it. That node has never stabilized -- it simply can't > finish the compactions. > Smaller compactions finish. Larger compactions, like these two never finish - > {code} > pending tasks: 48 >compaction type keyspace table completed total > unit progress > Compaction data stories 16532973358 75977993784 > bytes 21.76% > Compaction data stories_by_text 10593780658 38555048812 > bytes 27.48% > Active compaction remaining time : 0h10m51s > {code} > We are not getting exceptions and are not running out of heap space. The > Ubuntu OOM killer is reaping the process after all of the memory is consumed. > We watch memory in the opscenter console and it will grow. If we turn off the > OOM killer for the process, it will run until everything else is killed > instead and then the kernel panics. > We have the following settings configured: > 2G Heap > 512M New > {code} > memtable_heap_space_in_mb: 1024 > memtable_offheap_space_in_mb: 1024 > memtable_allocation_type: heap_buffers > commitlog_total_space_in_mb: 2048 > concurrent_compactors: 1 > compaction_throughput_mb_per_sec: 128 > {code} > The compaction strategy is leveled (these are read-intensive tables that are > rarely updated) > I have tried every setting, every option and I have the system where the MTBF > is about an hour now, but we never finish compacting because there are some > large compactions pending. None of the GC tools or settings help because it > is not a GC problem. It is an off-heap memory problem. > We are getting these messages in our syslog > {code} > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219527] BUG: Bad page map in > process java pte:0320 pmd:2d6fa5067 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219545] addr:7fb820be3000 > vm_flags:0870 anon_vma: (null) mapping: (null) > index:7fb820be3 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219556] CPU: 3 PID: 27344 > Comm: java Tainted: GB3.13.0-24-generic #47-Ubuntu > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219559] 880028510e40 > 88020d43da98 81715ac4 7fb820be3000 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219565] 88020d43dae0 > 81174183 0320 0007fb820be3 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219568] 8802d6fa5f18 > 0320 7fb820be3000 7fb820be4000 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219572] Call Trace: > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219584] [] > dump_stack+0x45/0x56 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219591] [] > print_bad_pte+0x1a3/0x250 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219594] [] > vm_normal_page+0x69/0x80 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219598] [] > unmap_page_range+0x3bb/0x7f0 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219602] [] > unmap_single_vma+0x81/0xf0 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219605] [] > unmap_vmas+0x49/0x90 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219610] [] > exit_mmap+0x9c/0x170 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219617] [] > ? __delayacct_add_tsk+0x153/0x170 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219621] [] > mmput+0x5c/0x120 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219625] [] > do_exit+0x26c/0xa50 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219631] [] > ? __unqueue_futex+0x31/0x60 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219634] [] > ? futex_wait+0x126/0x290 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219640] []
[jira] [Updated] (CASSANDRA-8499) Ensure SSTableWriter cleans up properly after failure
[ https://issues.apache.org/jira/browse/CASSANDRA-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict updated CASSANDRA-8499: Attachment: 8499-21v3 Heh. Looks like we were previously instead at risk of leaking the directory file descriptor, if we aborted prior to closing. While the new behaviour was benign, a simple modification avoids it > Ensure SSTableWriter cleans up properly after failure > - > > Key: CASSANDRA-8499 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8499 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Benedict >Assignee: Benedict > Fix For: 2.0.12 > > Attachments: 8499-20.txt, 8499-20v2, 8499-21.txt, 8499-21v2, 8499-21v3 > > > In 2.0 we do not free a bloom filter, in 2.1 we do not free a small piece of > offheap memory for writing compression metadata. In both we attempt to flush > the BF despite having encountered an exception, making the exception slow to > propagate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8559) OOM caused by large tombstone warning.
[ https://issues.apache.org/jira/browse/CASSANDRA-8559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8559: --- Fix Version/s: 2.0.12 > OOM caused by large tombstone warning. > -- > > Key: CASSANDRA-8559 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8559 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: 2.0.11 / 2.1 >Reporter: Dominic Letz > Fix For: 2.0.12 > > Attachments: Selection_048.png, stacktrace.log > > > When running with high amount of tombstones the error message generation from > CASSANDRA-6117 can lead to out of memory situation with the default setting. > Attached a heapdump viewed in visualvm showing how this construct created two > 777mb strings to print the error message for a read query and then crashed > OOM. > {code} > if (respectTombstoneThresholds() && columnCounter.ignored() > > DatabaseDescriptor.getTombstoneWarnThreshold()) > { > StringBuilder sb = new StringBuilder(); > CellNameType type = container.metadata().comparator; > for (ColumnSlice sl : slices) > { > assert sl != null; > sb.append('['); > sb.append(type.getString(sl.start)); > sb.append('-'); > sb.append(type.getString(sl.finish)); > sb.append(']'); > } > logger.warn("Read {} live and {} tombstoned cells in {}.{} (see > tombstone_warn_threshold). {} columns was requested, slices={}, delInfo={}", > columnCounter.live(), columnCounter.ignored(), > container.metadata().ksName, container.metadata().cfName, count, sb, > container.deletionInfo()); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8566) node crash (while auto-compaction?)
[ https://issues.apache.org/jira/browse/CASSANDRA-8566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8566: --- Reproduced In: 2.1.2 Fix Version/s: 2.1.3 > node crash (while auto-compaction?) > --- > > Key: CASSANDRA-8566 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8566 > Project: Cassandra > Issue Type: Bug > Environment: Linux CentOS 6.6 64bit, Cassandra 2.1.2 (release) >Reporter: Dmitri Dmitrienko > Fix For: 2.1.3 > > Attachments: 1.log > > > As data size became 20-24GB/node this issue started happening quite > frequently. With 7GB/node I didn't notice any crashes. > HEAP size was 10GB, now increased to 16GB and it didn't help. > Log is attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8569) org.apache.cassandra.db.KeyspaceTest failing
Philip Thompson created CASSANDRA-8569: -- Summary: org.apache.cassandra.db.KeyspaceTest failing Key: CASSANDRA-8569 URL: https://issues.apache.org/jira/browse/CASSANDRA-8569 Project: Cassandra Issue Type: Bug Components: Tests Reporter: Philip Thompson Assignee: Brandon Williams Fix For: 2.1.3 org.apache.cassandra.db.KeyspaceTest began failing after the patch for CASSANDRA-8245. {code} java.lang.NullPointerException at org.apache.cassandra.db.KeyspaceTest.testGetSliceFromLarge(KeyspaceTest.java:425) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8570) org.apache.cassandra.db.compaction.CompactionsPurgeTest.testMinorCompactionPurge failing
Philip Thompson created CASSANDRA-8570: -- Summary: org.apache.cassandra.db.compaction.CompactionsPurgeTest.testMinorCompactionPurge failing Key: CASSANDRA-8570 URL: https://issues.apache.org/jira/browse/CASSANDRA-8570 Project: Cassandra Issue Type: Bug Components: Tests Reporter: Philip Thompson Assignee: Marcus Eriksson Fix For: 2.1.3 The patch for 8429 broke the test org.apache.cassandra.db.compaction.CompactionsPurgeTest.testMinorCompactionPurge {code} java.lang.NullPointerException at org.apache.cassandra.db.compaction.CompactionsPurgeTest.testMinorCompactionPurge(CompactionsPurgeTest.java:138) Standard Output ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user defined compaction ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user defined compaction ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user defined compaction ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user defined compaction {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8570) org.apache.cassandra.db.compaction.CompactionsPurgeTest.testMinorCompactionPurge failing
[ https://issues.apache.org/jira/browse/CASSANDRA-8570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8570: --- Description: The patch for CASSANDRA-8429 broke the test org.apache.cassandra.db.compaction.CompactionsPurgeTest.testMinorCompactionPurge {code} java.lang.NullPointerException at org.apache.cassandra.db.compaction.CompactionsPurgeTest.testMinorCompactionPurge(CompactionsPurgeTest.java:138) Standard Output ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user defined compaction ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user defined compaction ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user defined compaction ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user defined compaction {code} was: The patch for 8429 broke the test org.apache.cassandra.db.compaction.CompactionsPurgeTest.testMinorCompactionPurge {code} java.lang.NullPointerException at org.apache.cassandra.db.compaction.CompactionsPurgeTest.testMinorCompactionPurge(CompactionsPurgeTest.java:138) Standard Output ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user defined compaction ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user defined compaction ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user defined compaction ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user defined compaction {code} > org.apache.cassandra.db.compaction.CompactionsPurgeTest.testMinorCompactionPurge > failing > > > Key: CASSANDRA-8570 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8570 > Project: Cassandra > Issue Type: Bug > Components: Tests >Reporter: Philip Thompson >Assignee: Marcus Eriksson > Fix For: 2.1.3 > > > The patch for CASSANDRA-8429 broke the test > org.apache.cassandra.db.compaction.CompactionsPurgeTest.testMinorCompactionPurge > {code} > java.lang.NullPointerException > at > org.apache.cassandra.db.compaction.CompactionsPurgeTest.testMinorCompactionPurge(CompactionsPurgeTest.java:138) > Standard Output > ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user > defined compaction > ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user > defined compaction > ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user > defined compaction > ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user > defined compaction > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8316) "Did not get positive replies from all endpoints" error on incremental repair
[ https://issues.apache.org/jira/browse/CASSANDRA-8316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266614#comment-14266614 ] Yuki Morishita commented on CASSANDRA-8316: --- bq. 4. B finishes preparing and marks a bunch of sstables as being repaired B does not mark sstables as repaired for just receiving prepare message, doesn't it? I understand that the current issue we have is prepared repair session is left on replica nodes when preparing timed out on coordinator. (In that case, user can work around by doing "forceTerminateRepairSession" manually.) I prefer sending cancel message, though adding new message may be difficult in minor release. Also we have to make sure message won't get dropped since AntiEntropyStage may be still busy preparing when cancel message arrives. Alternatively, I think the right solution to automatically remove left sessions is to track repair status as we do in CASSANDRA-5839 and use that to determine which prepared session can be removed. Either way, I think we can move this to resolve in 3.0 if I didn't miss the severity of the issue. > "Did not get positive replies from all endpoints" error on incremental repair > -- > > Key: CASSANDRA-8316 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8316 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: cassandra 2.1.2 >Reporter: Loic Lambiel >Assignee: Marcus Eriksson > Fix For: 2.1.3 > > Attachments: 0001-patch.patch, 8316-v2.patch, > CassandraDaemon-2014-11-25-2.snapshot.tar.gz, > CassandraDaemon-2014-12-14.snapshot.tar.gz, test.sh > > > Hi, > I've got an issue with incremental repairs on our production 15 nodes 2.1.2 > (new cluster, not yet loaded, RF=3) > After having successfully performed an incremental repair (-par -inc) on 3 > nodes, I started receiving "Repair failed with error Did not get positive > replies from all endpoints." from nodetool on all remaining nodes : > [2014-11-14 09:12:36,488] Starting repair command #3, repairing 108 ranges > for keyspace (seq=false, full=false) > [2014-11-14 09:12:47,919] Repair failed with error Did not get positive > replies from all endpoints. > All the nodes are up and running and the local system log shows that the > repair commands got started and that's it. > I've also noticed that soon after the repair, several nodes started having > more cpu load indefinitely without any particular reason (no tasks / queries, > nothing in the logs). I then restarted C* on these nodes and retried the > repair on several nodes, which were successful until facing the issue again. > I tried to repro on our 3 nodes preproduction cluster without success > It looks like I'm not the only one having this issue: > http://www.mail-archive.com/user%40cassandra.apache.org/msg39145.html > Any idea? > Thanks > Loic -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8571) Free space management does not work very well
Bartłomiej Romański created CASSANDRA-8571: -- Summary: Free space management does not work very well Key: CASSANDRA-8571 URL: https://issues.apache.org/jira/browse/CASSANDRA-8571 Project: Cassandra Issue Type: Bug Reporter: Bartłomiej Romański Hi all, We've got a cluster of 2.1.2 with 18 nodes equipped with 3x 480GB SSD each (JBODs). We mostly use LCS. Recently, our nodes starts failing with 'no space left on device'. It all started with our mistake - we let our LCS accumulate too much in L0. As a result, STCS woke up and we end with some big sstables on each node (let's say 5-10 sstables, 20-50gb each). During normal operation we keep our disks about 50% full. This gives about 200 GB free space on each of them. This was too little for compacting all accumulated L0 sstables at once. Cassandra kept trying to do that and keep failing... Evantually, we managed to stabilized the situation (with some crazy code hacking, manually moving sstables etc...). However, there are a few things that would be more than helpful in recovering from such situations more automatically... First, please look at DiskAwareRunnable.runMayThrow(). This methods initiates (local) variable: writeSize. I believe we should check somewhere here if we have enough space on a chosen disk. The problem is that writeSize is never read... Am I missing something here? Btw, while in STCS we first look for the least overloaded disk, and then (if there are more than one such disks) for the one with the most free space (please note the sort order in Directories.getWriteableLocation()). That's often suboptimal (it's usually better to wait for the bigger disk than to compact fewer sstables now), but probably not crucial. Second, the strategy (used by LCS) that we first choose target disk and then use it for whole compaction is not the best one. For big compactions (eg. after some massive operations like bootstrap or repair; or after some issues with LCS like in our case) on small drives (eg. JBOD of SSDs) these will never succeed. Much better strategy would be to choose target drive for each output sstable separately, or at least round robin them. Third, it would be helpful if the default check for MAX_COMPACTING_L0 in LeveledManifest.getCandidatesFor() would be expanded to support also limit for total space. After fallback STCS in L0 you end up with very big sstables an 32 of them is just too much for one compaction on a small drives. We finally used some hack similar the last option (as it was the easiest one to implement in a hurry), but any improvents described above would save us from all this. Thanks, BR -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8571) Free space management does not work very well
[ https://issues.apache.org/jira/browse/CASSANDRA-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bartłomiej Romański updated CASSANDRA-8571: --- Description: Hi all, We've got a cluster of 2.1.2 with 18 nodes equipped with 3x 480GB SSD each (JBODs). We mostly use LCS. Recently, our nodes starts failing with 'no space left on device'. It all started with our mistake - we let our LCS accumulate too much in L0. As a result, STCS woke up and we end with some big sstables on each node (let's say 5-10 sstables, 20-50gb each). During normal operation we keep our disks about 50% full. This gives about 200 GB free space on each of them. This was too little for compacting all accumulated L0 sstables at once. Cassandra kept trying to do that and keep failing... Evantually, we managed to stabilized the situation (with some crazy code hacking, manually moving sstables etc...). However, there are a few things that would be more than helpful in recovering from such situations more automatically... First, please look at DiskAwareRunnable.runMayThrow(). This methods initiates (local) variable: writeSize. I believe we should check somewhere here if we have enough space on a chosen disk. The problem is that writeSize is never read... Am I missing something here? Btw, while in STCS we first look for the least overloaded disk, and then (if there are more than one such disks) for the one with the most free space (please note the sort order in Directories.getWriteableLocation()). That's often suboptimal (it's usually better to wait for the bigger disk than to compact fewer sstables now), but probably not crucial. Second, the strategy (used by LCS) that we first choose target disk and then use it for whole compaction is not the best one. For big compactions (eg. after some massive operations like bootstrap or repair; or after some issues with LCS like in our case) on small drives (eg. JBOD of SSDs) these will never succeed. Much better strategy would be to choose target drive for each output sstable separately, or at least round robin them. Third, it would be helpful if the default check for MAX_COMPACTING_L0 in LeveledManifest.getCandidatesFor() would be expanded to support also limit for total space. After fallback STCS in L0 you end up with very big sstables and 32 of them is just too much for one compaction on a small drives. We finally used some hack similar the last option (as it was the easiest one to implement in a hurry), but any improvents described above would save us from all this. Thanks, BR was: Hi all, We've got a cluster of 2.1.2 with 18 nodes equipped with 3x 480GB SSD each (JBODs). We mostly use LCS. Recently, our nodes starts failing with 'no space left on device'. It all started with our mistake - we let our LCS accumulate too much in L0. As a result, STCS woke up and we end with some big sstables on each node (let's say 5-10 sstables, 20-50gb each). During normal operation we keep our disks about 50% full. This gives about 200 GB free space on each of them. This was too little for compacting all accumulated L0 sstables at once. Cassandra kept trying to do that and keep failing... Evantually, we managed to stabilized the situation (with some crazy code hacking, manually moving sstables etc...). However, there are a few things that would be more than helpful in recovering from such situations more automatically... First, please look at DiskAwareRunnable.runMayThrow(). This methods initiates (local) variable: writeSize. I believe we should check somewhere here if we have enough space on a chosen disk. The problem is that writeSize is never read... Am I missing something here? Btw, while in STCS we first look for the least overloaded disk, and then (if there are more than one such disks) for the one with the most free space (please note the sort order in Directories.getWriteableLocation()). That's often suboptimal (it's usually better to wait for the bigger disk than to compact fewer sstables now), but probably not crucial. Second, the strategy (used by LCS) that we first choose target disk and then use it for whole compaction is not the best one. For big compactions (eg. after some massive operations like bootstrap or repair; or after some issues with LCS like in our case) on small drives (eg. JBOD of SSDs) these will never succeed. Much better strategy would be to choose target drive for each output sstable separately, or at least round robin them. Third, it would be helpful if the default check for MAX_COMPACTING_L0 in LeveledManifest.getCandidatesFor() would be expanded to support also limit for total space. After fallback STCS in L0 you end up with very big sstables an 32 of them is just too much for one compaction on a small drives. We finally used some hack similar the last option (as it was the easiest one to implement in a hurry), but any improvents
[jira] [Updated] (CASSANDRA-8570) org.apache.cassandra.db.compaction.CompactionsPurgeTest failing
[ https://issues.apache.org/jira/browse/CASSANDRA-8570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8570: --- Description: The patch for CASSANDRA-8429 broke the tests {{org.apache.cassandra.db.compaction.CompactionsPurgeTest.testCompactionPurgeTombstonedRow}} and {{org.apache.cassandra.db.compaction.CompactionsPurgeTest.testRowTombstoneObservedBeforePurging}} {code} junit.framework.AssertionFailedError: at org.apache.cassandra.db.compaction.CompactionsPurgeTest.testCompactionPurgeTombstonedRow(CompactionsPurgeTest.java:308) {code} {code}expected:<0> but was:<1> Stack Trace junit.framework.AssertionFailedError: expected:<0> but was:<1> at org.apache.cassandra.db.compaction.CompactionsPurgeTest.testRowTombstoneObservedBeforePurging(CompactionsPurgeTest.java:372) {code} was: The patch for CASSANDRA-8429 broke the test org.apache.cassandra.db.compaction.CompactionsPurgeTest.testMinorCompactionPurge {code} java.lang.NullPointerException at org.apache.cassandra.db.compaction.CompactionsPurgeTest.testMinorCompactionPurge(CompactionsPurgeTest.java:138) Standard Output ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user defined compaction ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user defined compaction ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user defined compaction ERROR 16:19:44 You can't mix repaired and unrepaired sstables in a user defined compaction {code} Summary: org.apache.cassandra.db.compaction.CompactionsPurgeTest failing (was: org.apache.cassandra.db.compaction.CompactionsPurgeTest.testMinorCompactionPurge failing) > org.apache.cassandra.db.compaction.CompactionsPurgeTest failing > --- > > Key: CASSANDRA-8570 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8570 > Project: Cassandra > Issue Type: Bug > Components: Tests >Reporter: Philip Thompson >Assignee: Marcus Eriksson > Fix For: 2.1.3 > > > The patch for CASSANDRA-8429 broke the tests > {{org.apache.cassandra.db.compaction.CompactionsPurgeTest.testCompactionPurgeTombstonedRow}} > and > {{org.apache.cassandra.db.compaction.CompactionsPurgeTest.testRowTombstoneObservedBeforePurging}} > {code} > junit.framework.AssertionFailedError: > at > org.apache.cassandra.db.compaction.CompactionsPurgeTest.testCompactionPurgeTombstonedRow(CompactionsPurgeTest.java:308) > {code} > {code}expected:<0> but was:<1> > Stack Trace > junit.framework.AssertionFailedError: expected:<0> but was:<1> > at > org.apache.cassandra.db.compaction.CompactionsPurgeTest.testRowTombstoneObservedBeforePurging(CompactionsPurgeTest.java:372) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8571) Free space management does not work very well
[ https://issues.apache.org/jira/browse/CASSANDRA-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266624#comment-14266624 ] Jeremy Hanna commented on CASSANDRA-8571: - See CASSANDRA-8329. That may be what you're looking for in 2.0.12. Also related is CASSANDRA-7386. > Free space management does not work very well > - > > Key: CASSANDRA-8571 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8571 > Project: Cassandra > Issue Type: Bug >Reporter: Bartłomiej Romański > > Hi all, > We've got a cluster of 2.1.2 with 18 nodes equipped with 3x 480GB SSD each > (JBODs). We mostly use LCS. > Recently, our nodes starts failing with 'no space left on device'. It all > started with our mistake - we let our LCS accumulate too much in L0. > As a result, STCS woke up and we end with some big sstables on each node > (let's say 5-10 sstables, 20-50gb each). > During normal operation we keep our disks about 50% full. This gives about > 200 GB free space on each of them. This was too little for compacting all > accumulated L0 sstables at once. Cassandra kept trying to do that and keep > failing... > Evantually, we managed to stabilized the situation (with some crazy code > hacking, manually moving sstables etc...). However, there are a few things > that would be more than helpful in recovering from such situations more > automatically... > First, please look at DiskAwareRunnable.runMayThrow(). This methods initiates > (local) variable: writeSize. I believe we should check somewhere here if we > have enough space on a chosen disk. The problem is that writeSize is never > read... Am I missing something here? > Btw, while in STCS we first look for the least overloaded disk, and then (if > there are more than one such disks) for the one with the most free space > (please note the sort order in Directories.getWriteableLocation()). That's > often suboptimal (it's usually better to wait for the bigger disk than to > compact fewer sstables now), but probably not crucial. > Second, the strategy (used by LCS) that we first choose target disk and then > use it for whole compaction is not the best one. For big compactions (eg. > after some massive operations like bootstrap or repair; or after some issues > with LCS like in our case) on small drives (eg. JBOD of SSDs) these will > never succeed. Much better strategy would be to choose target drive for each > output sstable separately, or at least round robin them. > Third, it would be helpful if the default check for MAX_COMPACTING_L0 in > LeveledManifest.getCandidatesFor() would be expanded to support also limit > for total space. After fallback STCS in L0 you end up with very big sstables > and 32 of them is just too much for one compaction on a small drives. > We finally used some hack similar the last option (as it was the easiest one > to implement in a hurry), but any improvents described above would save us > from all this. > Thanks, > BR -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8571) Free space management does not work very well
[ https://issues.apache.org/jira/browse/CASSANDRA-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266624#comment-14266624 ] Jeremy Hanna edited comment on CASSANDRA-8571 at 1/6/15 7:43 PM: - See CASSANDRA-8329. That may be what you're looking for in 2.1.3 I believe. Also related is CASSANDRA-7386. was (Author: jeromatron): See CASSANDRA-8329. That may be what you're looking for in 2.0.12. Also related is CASSANDRA-7386. > Free space management does not work very well > - > > Key: CASSANDRA-8571 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8571 > Project: Cassandra > Issue Type: Bug >Reporter: Bartłomiej Romański > > Hi all, > We've got a cluster of 2.1.2 with 18 nodes equipped with 3x 480GB SSD each > (JBODs). We mostly use LCS. > Recently, our nodes starts failing with 'no space left on device'. It all > started with our mistake - we let our LCS accumulate too much in L0. > As a result, STCS woke up and we end with some big sstables on each node > (let's say 5-10 sstables, 20-50gb each). > During normal operation we keep our disks about 50% full. This gives about > 200 GB free space on each of them. This was too little for compacting all > accumulated L0 sstables at once. Cassandra kept trying to do that and keep > failing... > Evantually, we managed to stabilized the situation (with some crazy code > hacking, manually moving sstables etc...). However, there are a few things > that would be more than helpful in recovering from such situations more > automatically... > First, please look at DiskAwareRunnable.runMayThrow(). This methods initiates > (local) variable: writeSize. I believe we should check somewhere here if we > have enough space on a chosen disk. The problem is that writeSize is never > read... Am I missing something here? > Btw, while in STCS we first look for the least overloaded disk, and then (if > there are more than one such disks) for the one with the most free space > (please note the sort order in Directories.getWriteableLocation()). That's > often suboptimal (it's usually better to wait for the bigger disk than to > compact fewer sstables now), but probably not crucial. > Second, the strategy (used by LCS) that we first choose target disk and then > use it for whole compaction is not the best one. For big compactions (eg. > after some massive operations like bootstrap or repair; or after some issues > with LCS like in our case) on small drives (eg. JBOD of SSDs) these will > never succeed. Much better strategy would be to choose target drive for each > output sstable separately, or at least round robin them. > Third, it would be helpful if the default check for MAX_COMPACTING_L0 in > LeveledManifest.getCandidatesFor() would be expanded to support also limit > for total space. After fallback STCS in L0 you end up with very big sstables > and 32 of them is just too much for one compaction on a small drives. > We finally used some hack similar the last option (as it was the easiest one > to implement in a hurry), but any improvents described above would save us > from all this. > Thanks, > BR -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8448) "Comparison method violates its general contract" in AbstractEndpointSnitch
[ https://issues.apache.org/jira/browse/CASSANDRA-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266626#comment-14266626 ] Benedict commented on CASSANDRA-8448: - compareEndpoints() doesn't snapshot, it uses the values from the shared "scores" object property, and sortByProximityWithScore ultimately delegates to this. > "Comparison method violates its general contract" in AbstractEndpointSnitch > --- > > Key: CASSANDRA-8448 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8448 > Project: Cassandra > Issue Type: Bug >Reporter: J.B. Langston >Assignee: Brandon Williams > > Seen in both 1.2 and 2.0. The error is occurring here: > https://github.com/apache/cassandra/blob/cassandra-2.0/src/java/org/apache/cassandra/locator/AbstractEndpointSnitch.java#L49 > {code} > ERROR [Thrift:9] 2014-12-04 20:12:28,732 CustomTThreadPoolServer.java (line > 219) Error occurred during processing of message. > com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.IllegalArgumentException: Comparison method violates its general > contract! > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199) > at com.google.common.cache.LocalCache.get(LocalCache.java:3932) > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936) > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806) > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:352) > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:224) > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:218) > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:202) > at > org.apache.cassandra.thrift.CassandraServer.createMutationList(CassandraServer.java:822) > at > org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:954) > at com.datastax.bdp.server.DseServer.batch_mutate(DseServer.java:576) > at > org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3922) > at > org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3906) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > at > org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:201) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.IllegalArgumentException: Comparison method violates its > general contract! > at java.util.TimSort.mergeHi(TimSort.java:868) > at java.util.TimSort.mergeAt(TimSort.java:485) > at java.util.TimSort.mergeCollapse(TimSort.java:410) > at java.util.TimSort.sort(TimSort.java:214) > at java.util.TimSort.sort(TimSort.java:173) > at java.util.Arrays.sort(Arrays.java:659) > at java.util.Collections.sort(Collections.java:217) > at > org.apache.cassandra.locator.AbstractEndpointSnitch.sortByProximity(AbstractEndpointSnitch.java:49) > at > org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithScore(DynamicEndpointSnitch.java:157) > at > org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithBadness(DynamicEndpointSnitch.java:186) > at > org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximity(DynamicEndpointSnitch.java:151) > at > org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1408) > at > org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1402) > at > org.apache.cassandra.service.AbstractReadExecutor.getReadExecutor(AbstractReadExecutor.java:148) > at > org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1223) > at > org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1165) > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:255) > at > org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:225) > at org.apache.cassandra.auth.Auth.selectUser(Auth.java:243) > at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:84) > at > org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) > at > org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:69) > at org.a
[jira] [Commented] (CASSANDRA-8403) limit disregarded when paging with IN clause under certain conditions
[ https://issues.apache.org/jira/browse/CASSANDRA-8403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266644#comment-14266644 ] Russ Hatch commented on CASSANDRA-8403: --- [~blerer] I did a bisect of sorts and it looks like that was handled by the patch for CASSANDRA-8408. > limit disregarded when paging with IN clause under certain conditions > - > > Key: CASSANDRA-8403 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8403 > Project: Cassandra > Issue Type: Bug >Reporter: Russ Hatch >Assignee: Benjamin Lerer >Priority: Minor > > This issue was originally reported on the python-driver userlist and > confirmed by [~aholmber] > When: > page_size < limit < data size, > the limit value is disregarded and all rows are paged back. > to repro: > create a table and populate it with two partitions > CREATE TABLE paging_test ( id int, value text, PRIMARY KEY (id, value) ) > Add data: in one partition create 10 rows, an in a second partition create 20 > rows > perform a query with page_size of 10 and a LIMIT of 20, like so: > SELECT * FROM paging_test where id in (1,2) LIMIT 20; > The limit is disregarded and three pages of 10 records each will be returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (CASSANDRA-7622) Implement virtual tables
[ https://issues.apache.org/jira/browse/CASSANDRA-7622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer reassigned CASSANDRA-7622: - Assignee: Benjamin Lerer > Implement virtual tables > > > Key: CASSANDRA-7622 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7622 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Tupshin Harper >Assignee: Benjamin Lerer > Fix For: 3.1 > > > There are a variety of reasons to want virtual tables, which would be any > table that would be backed by an API, rather than data explicitly managed and > stored as sstables. > One possible use case would be to expose JMX data through CQL as a > resurrection of CASSANDRA-3527. > Another is a more general framework to implement the ability to expose yaml > configuration information. So it would be an alternate approach to > CASSANDRA-7370. > A possible implementation would be in terms of CASSANDRA-7443, but I am not > presupposing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8552) Large compactions run out of off-heap RAM
[ https://issues.apache.org/jira/browse/CASSANDRA-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266643#comment-14266643 ] Brent Haines commented on CASSANDRA-8552: - Quick update - the node was cleared and has re-joined the cluster. It managed to work through the sync and resultant compactions (more than 500 jobs) without fail. This is a good sign, but it worked the first time too, failing during the first repair thereafter. I have started the repair this morning and we are part of the way through it. I will let you know if / when the issue reappears. > Large compactions run out of off-heap RAM > - > > Key: CASSANDRA-8552 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8552 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 14.4 > AWS EC2 > 12 m1.xlarge nodes [4 cores, 16GB RAM, 1TB storage (251GB Used)] > Java build 1.7.0_55-b13 and build 1.8.0_25-b17 >Reporter: Brent Haines >Assignee: Benedict >Priority: Blocker > Fix For: 2.1.3 > > Attachments: Screen Shot 2015-01-02 at 9.36.11 PM.png, fhandles.log, > freelog.log, lsof.txt, meminfo.txt, sysctl.txt, system.log > > > We have a large table of storing, effectively event logs and a pair of > denormalized tables for indexing. > When updating from 2.0 to 2.1 we saw performance improvements, but some > random and silent crashes during nightly repairs. We lost a node (totally > corrupted) and replaced it. That node has never stabilized -- it simply can't > finish the compactions. > Smaller compactions finish. Larger compactions, like these two never finish - > {code} > pending tasks: 48 >compaction type keyspace table completed total > unit progress > Compaction data stories 16532973358 75977993784 > bytes 21.76% > Compaction data stories_by_text 10593780658 38555048812 > bytes 27.48% > Active compaction remaining time : 0h10m51s > {code} > We are not getting exceptions and are not running out of heap space. The > Ubuntu OOM killer is reaping the process after all of the memory is consumed. > We watch memory in the opscenter console and it will grow. If we turn off the > OOM killer for the process, it will run until everything else is killed > instead and then the kernel panics. > We have the following settings configured: > 2G Heap > 512M New > {code} > memtable_heap_space_in_mb: 1024 > memtable_offheap_space_in_mb: 1024 > memtable_allocation_type: heap_buffers > commitlog_total_space_in_mb: 2048 > concurrent_compactors: 1 > compaction_throughput_mb_per_sec: 128 > {code} > The compaction strategy is leveled (these are read-intensive tables that are > rarely updated) > I have tried every setting, every option and I have the system where the MTBF > is about an hour now, but we never finish compacting because there are some > large compactions pending. None of the GC tools or settings help because it > is not a GC problem. It is an off-heap memory problem. > We are getting these messages in our syslog > {code} > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219527] BUG: Bad page map in > process java pte:0320 pmd:2d6fa5067 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219545] addr:7fb820be3000 > vm_flags:0870 anon_vma: (null) mapping: (null) > index:7fb820be3 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219556] CPU: 3 PID: 27344 > Comm: java Tainted: GB3.13.0-24-generic #47-Ubuntu > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219559] 880028510e40 > 88020d43da98 81715ac4 7fb820be3000 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219565] 88020d43dae0 > 81174183 0320 0007fb820be3 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219568] 8802d6fa5f18 > 0320 7fb820be3000 7fb820be4000 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219572] Call Trace: > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219584] [] > dump_stack+0x45/0x56 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219591] [] > print_bad_pte+0x1a3/0x250 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219594] [] > vm_normal_page+0x69/0x80 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219598] [] > unmap_page_range+0x3bb/0x7f0 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219602] [] > unmap_single_vma+0x81/0xf0 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219605] [] > unmap_vmas+0x49/0x90 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219610] [] > exit_mmap+0x9c/0x170 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151.219617] [] > ? __delayacct_add_tsk+0x153/0x170 > Jan 2 07:06:00 ip-10-0-2-226 kernel: [49801151
[jira] [Resolved] (CASSANDRA-8403) limit disregarded when paging with IN clause under certain conditions
[ https://issues.apache.org/jira/browse/CASSANDRA-8403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russ Hatch resolved CASSANDRA-8403. --- Resolution: Not a Problem > limit disregarded when paging with IN clause under certain conditions > - > > Key: CASSANDRA-8403 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8403 > Project: Cassandra > Issue Type: Bug >Reporter: Russ Hatch >Assignee: Benjamin Lerer >Priority: Minor > > This issue was originally reported on the python-driver userlist and > confirmed by [~aholmber] > When: > page_size < limit < data size, > the limit value is disregarded and all rows are paged back. > to repro: > create a table and populate it with two partitions > CREATE TABLE paging_test ( id int, value text, PRIMARY KEY (id, value) ) > Add data: in one partition create 10 rows, an in a second partition create 20 > rows > perform a query with page_size of 10 and a LIMIT of 20, like so: > SELECT * FROM paging_test where id in (1,2) LIMIT 20; > The limit is disregarded and three pages of 10 records each will be returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8329) LeveledCompactionStrategy should split large files across data directories when compacting
[ https://issues.apache.org/jira/browse/CASSANDRA-8329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeremy Hanna updated CASSANDRA-8329: Fix Version/s: 2.1.3 > LeveledCompactionStrategy should split large files across data directories > when compacting > -- > > Key: CASSANDRA-8329 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8329 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: J.B. Langston >Assignee: Marcus Eriksson > Fix For: 2.0.12, 2.1.3 > > Attachments: > 0001-get-new-sstable-directory-for-every-new-file-during-.patch, > test_no_patch_2.0.jpg, test_with_patch_2.0.jpg > > > Because we fall back to STCS for L0 when LCS gets behind, the sstables in L0 > can get quite large during sustained periods of heavy writes. This can > result in large imbalances between data volumes when using JBOD support. > Eventually these large files get broken up as L0 sstables are moved up into > higher levels; however, because LCS only chooses a single volume on which to > write all of the sstables created during a single compaction, the imbalance > is persisted. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-8571) Free space management does not work very well
[ https://issues.apache.org/jira/browse/CASSANDRA-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266624#comment-14266624 ] Jeremy Hanna edited comment on CASSANDRA-8571 at 1/6/15 7:54 PM: - See CASSANDRA-8329. That may be what you're looking for in the upcoming 2.1.3. Also related is CASSANDRA-7386. was (Author: jeromatron): See CASSANDRA-8329. That may be what you're looking for in 2.1.3 I believe. Also related is CASSANDRA-7386. > Free space management does not work very well > - > > Key: CASSANDRA-8571 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8571 > Project: Cassandra > Issue Type: Bug >Reporter: Bartłomiej Romański > > Hi all, > We've got a cluster of 2.1.2 with 18 nodes equipped with 3x 480GB SSD each > (JBODs). We mostly use LCS. > Recently, our nodes starts failing with 'no space left on device'. It all > started with our mistake - we let our LCS accumulate too much in L0. > As a result, STCS woke up and we end with some big sstables on each node > (let's say 5-10 sstables, 20-50gb each). > During normal operation we keep our disks about 50% full. This gives about > 200 GB free space on each of them. This was too little for compacting all > accumulated L0 sstables at once. Cassandra kept trying to do that and keep > failing... > Evantually, we managed to stabilized the situation (with some crazy code > hacking, manually moving sstables etc...). However, there are a few things > that would be more than helpful in recovering from such situations more > automatically... > First, please look at DiskAwareRunnable.runMayThrow(). This methods initiates > (local) variable: writeSize. I believe we should check somewhere here if we > have enough space on a chosen disk. The problem is that writeSize is never > read... Am I missing something here? > Btw, while in STCS we first look for the least overloaded disk, and then (if > there are more than one such disks) for the one with the most free space > (please note the sort order in Directories.getWriteableLocation()). That's > often suboptimal (it's usually better to wait for the bigger disk than to > compact fewer sstables now), but probably not crucial. > Second, the strategy (used by LCS) that we first choose target disk and then > use it for whole compaction is not the best one. For big compactions (eg. > after some massive operations like bootstrap or repair; or after some issues > with LCS like in our case) on small drives (eg. JBOD of SSDs) these will > never succeed. Much better strategy would be to choose target drive for each > output sstable separately, or at least round robin them. > Third, it would be helpful if the default check for MAX_COMPACTING_L0 in > LeveledManifest.getCandidatesFor() would be expanded to support also limit > for total space. After fallback STCS in L0 you end up with very big sstables > and 32 of them is just too much for one compaction on a small drives. > We finally used some hack similar the last option (as it was the easiest one > to implement in a hurry), but any improvents described above would save us > from all this. > Thanks, > BR -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8570) org.apache.cassandra.db.compaction.CompactionsPurgeTest failing
[ https://issues.apache.org/jira/browse/CASSANDRA-8570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266651#comment-14266651 ] Philip Thompson commented on CASSANDRA-8570: It appears to have also broken {{org.apache.cassandra.db.compaction.LeveledCompactionStrategyTest.testNewRepairedSSTable}} > org.apache.cassandra.db.compaction.CompactionsPurgeTest failing > --- > > Key: CASSANDRA-8570 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8570 > Project: Cassandra > Issue Type: Bug > Components: Tests >Reporter: Philip Thompson >Assignee: Marcus Eriksson > Fix For: 2.1.3 > > > The patch for CASSANDRA-8429 broke the tests > {{org.apache.cassandra.db.compaction.CompactionsPurgeTest.testCompactionPurgeTombstonedRow}} > and > {{org.apache.cassandra.db.compaction.CompactionsPurgeTest.testRowTombstoneObservedBeforePurging}} > {code} > junit.framework.AssertionFailedError: > at > org.apache.cassandra.db.compaction.CompactionsPurgeTest.testCompactionPurgeTombstonedRow(CompactionsPurgeTest.java:308) > {code} > {code}expected:<0> but was:<1> > Stack Trace > junit.framework.AssertionFailedError: expected:<0> but was:<1> > at > org.apache.cassandra.db.compaction.CompactionsPurgeTest.testRowTombstoneObservedBeforePurging(CompactionsPurgeTest.java:372) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8571) Free space management does not work very well
[ https://issues.apache.org/jira/browse/CASSANDRA-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8571: --- Reproduced In: 2.1.2 Fix Version/s: 2.1.3 > Free space management does not work very well > - > > Key: CASSANDRA-8571 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8571 > Project: Cassandra > Issue Type: Bug >Reporter: Bartłomiej Romański > Fix For: 2.1.3 > > > Hi all, > We've got a cluster of 2.1.2 with 18 nodes equipped with 3x 480GB SSD each > (JBODs). We mostly use LCS. > Recently, our nodes starts failing with 'no space left on device'. It all > started with our mistake - we let our LCS accumulate too much in L0. > As a result, STCS woke up and we end with some big sstables on each node > (let's say 5-10 sstables, 20-50gb each). > During normal operation we keep our disks about 50% full. This gives about > 200 GB free space on each of them. This was too little for compacting all > accumulated L0 sstables at once. Cassandra kept trying to do that and keep > failing... > Evantually, we managed to stabilized the situation (with some crazy code > hacking, manually moving sstables etc...). However, there are a few things > that would be more than helpful in recovering from such situations more > automatically... > First, please look at DiskAwareRunnable.runMayThrow(). This methods initiates > (local) variable: writeSize. I believe we should check somewhere here if we > have enough space on a chosen disk. The problem is that writeSize is never > read... Am I missing something here? > Btw, while in STCS we first look for the least overloaded disk, and then (if > there are more than one such disks) for the one with the most free space > (please note the sort order in Directories.getWriteableLocation()). That's > often suboptimal (it's usually better to wait for the bigger disk than to > compact fewer sstables now), but probably not crucial. > Second, the strategy (used by LCS) that we first choose target disk and then > use it for whole compaction is not the best one. For big compactions (eg. > after some massive operations like bootstrap or repair; or after some issues > with LCS like in our case) on small drives (eg. JBOD of SSDs) these will > never succeed. Much better strategy would be to choose target drive for each > output sstable separately, or at least round robin them. > Third, it would be helpful if the default check for MAX_COMPACTING_L0 in > LeveledManifest.getCandidatesFor() would be expanded to support also limit > for total space. After fallback STCS in L0 you end up with very big sstables > and 32 of them is just too much for one compaction on a small drives. > We finally used some hack similar the last option (as it was the easiest one > to implement in a hurry), but any improvents described above would save us > from all this. > Thanks, > BR -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8499) Ensure SSTableWriter cleans up properly after failure
[ https://issues.apache.org/jira/browse/CASSANDRA-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1427#comment-1427 ] Marcus Eriksson commented on CASSANDRA-8499: +1 (remove the unused "boolean closed" in SequentialWriter on commit) > Ensure SSTableWriter cleans up properly after failure > - > > Key: CASSANDRA-8499 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8499 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Benedict >Assignee: Benedict > Fix For: 2.0.12 > > Attachments: 8499-20.txt, 8499-20v2, 8499-21.txt, 8499-21v2, 8499-21v3 > > > In 2.0 we do not free a bloom filter, in 2.1 we do not free a small piece of > offheap memory for writing compression metadata. In both we attempt to flush > the BF despite having encountered an exception, making the exception slow to > propagate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7159) sstablemetadata command should print some more stuff
[ https://issues.apache.org/jira/browse/CASSANDRA-7159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266671#comment-14266671 ] Philip Thompson commented on CASSANDRA-7159: [~vsinjavin], any update on this? 2.1.3 is rolling out soon, and it would be nice to get this patch in. > sstablemetadata command should print some more stuff > > > Key: CASSANDRA-7159 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7159 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Jeremiah Jordan >Assignee: Vladislav Sinjavin >Priority: Trivial > Labels: lhf > Fix For: 2.1.3 > > Attachments: > CASSANDRA-7159_-_sstablemetadata_command_should_print_some_more_stuff.patch > > > It would be nice if the sstablemetadata command printed out some more of the > stuff we track. Like the Min/Max column names and the min/max token in the > file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-7269) Make CqlInputFormat and CqlRecordReader consistent with comments
[ https://issues.apache.org/jira/browse/CASSANDRA-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-7269: --- Reviewer: Brandon Williams Fix Version/s: 2.1.3 > Make CqlInputFormat and CqlRecordReader consistent with comments > > > Key: CASSANDRA-7269 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7269 > Project: Cassandra > Issue Type: Bug > Components: Hadoop >Reporter: Jeremy Hanna >Assignee: Rekha Joshi >Priority: Minor > Labels: hadoop > Fix For: 2.1.3 > > Attachments: CASSANDRA-7269.txt > > > Both the CqlInputFormat and CqlPagingInputFormat have the following comment: > {code} > /** > ... > * the number of CQL rows per page > * CQLConfigHelper.setInputCQLPageRowSize. The default page row size is > 1000. You > * should set it to "as big as possible, but no bigger." It set the LIMIT > for the CQL > * query, so you need set it big enough to minimize the network overhead, > and also > * not too big to avoid out of memory issue. > ... > **/ > {code} > The property is used in both classes, but the default is only set to 1000 in > CqlPagingRecordReader explicitly. > We should either make the default part of the CqlConfigHelper so it's set in > both places or update the comments in the CqlInputFormat to say that if it's > not set, it will default to the java driver fetch size which is 5000. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8566) node crash (while auto-compaction?)
[ https://issues.apache.org/jira/browse/CASSANDRA-8566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266696#comment-14266696 ] Philip Thompson commented on CASSANDRA-8566: This looks possibly related to CASSANDRA-8552 > node crash (while auto-compaction?) > --- > > Key: CASSANDRA-8566 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8566 > Project: Cassandra > Issue Type: Bug > Environment: Linux CentOS 6.6 64bit, Cassandra 2.1.2 (release) >Reporter: Dmitri Dmitrienko > Fix For: 2.1.3 > > Attachments: 1.log > > > As data size became 20-24GB/node this issue started happening quite > frequently. With 7GB/node I didn't notice any crashes. > HEAP size was 10GB, now increased to 16GB and it didn't help. > Log is attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8563) cqlsh broken for some thrift created tables.
[ https://issues.apache.org/jira/browse/CASSANDRA-8563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8563: --- Assignee: Tyler Hobbs > cqlsh broken for some thrift created tables. > > > Key: CASSANDRA-8563 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8563 > Project: Cassandra > Issue Type: Bug > Components: Tools >Reporter: Jeremiah Jordan >Assignee: Tyler Hobbs > Labels: cqlsh > Fix For: 2.1.3 > > > The new python driver based cqlsh is broken for some tables. This was fixed > recently in: https://datastax-oss.atlassian.net/browse/PYTHON-192 > So we should pull in a new version of the python driver. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8357) ArrayOutOfBounds in cassandra-stress with inverted exponential distribution
[ https://issues.apache.org/jira/browse/CASSANDRA-8357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8357: --- Reproduced In: 2.1.1 Fix Version/s: 2.1.3 > ArrayOutOfBounds in cassandra-stress with inverted exponential distribution > --- > > Key: CASSANDRA-8357 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8357 > Project: Cassandra > Issue Type: Bug > Components: Tools > Environment: 6-node cassandra cluster (2.1.1) on debian. >Reporter: Jens Preußner > Fix For: 2.1.3 > > > When using the CQLstress example from GitHub > (https://github.com/apache/cassandra/blob/trunk/tools/cqlstress-example.yaml) > with an inverted exponential distribution in the insert-partitions field, > generated threads fail with > Exception in thread "Thread-20" java.lang.ArrayIndexOutOfBoundsException: 20 > at > org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:307) > See the gist https://gist.github.com/jenzopr/9edde53122554729c852 for the > typetest.yaml I used. > The call was: > cassandra-stress user profile=typetest.yaml ops\(insert=1\) -node $NODES -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8310) Assertion error in 2.1.1: SSTableReader.cloneWithNewSummarySamplingLevel(SSTableReader.java:988)
[ https://issues.apache.org/jira/browse/CASSANDRA-8310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8310: --- Reproduced In: 2.1.1 Fix Version/s: 2.1.3 > Assertion error in 2.1.1: > SSTableReader.cloneWithNewSummarySamplingLevel(SSTableReader.java:988) > > > Key: CASSANDRA-8310 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8310 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Donald Smith > Fix For: 2.1.3 > > > Using C* 2.1.1 on linux Centos 6.4, we're getting this AssertionError on 5 > nodes in a 12 node cluster. Also, compactions are lagging on all nodes. > {noformat} > ERROR [IndexSummaryManager:1] 2014-11-13 09:15:16,221 CassandraDaemon.java > (line 153) Exception in thread Thread[IndexSummaryManager:1,1,main] > java.lang.AssertionError: null > at > org.apache.cassandra.io.sstable.SSTableReader.cloneWithNewSummarySamplingLevel(SSTableReader.java:988) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.IndexSummaryManager.adjustSamplingLevels(IndexSummaryManager.java:420) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:298) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.IndexSummaryManager.redistributeSummaries(IndexSummaryManager.java:238) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.IndexSummaryManager$1.runMayThrow(IndexSummaryManager.java:139) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:77) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > [na:1.7.0_60] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > [na:1.7.0_60] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > [na:1.7.0_60] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [na:1.7.0_60] > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8311) C* 2.1.1: AssertionError in AbstractionCompactionTask "not correctly marked compacting"
[ https://issues.apache.org/jira/browse/CASSANDRA-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8311: --- Reproduced In: 2.1.1 Fix Version/s: 2.1.3 > C* 2.1.1: AssertionError in AbstractionCompactionTask "not correctly marked > compacting" > > > Key: CASSANDRA-8311 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8311 > Project: Cassandra > Issue Type: Bug >Reporter: Donald Smith > Fix For: 2.1.3 > > > Using 2.1.1 on CentOS6.4, we see this AssertionError on 3 out of 12 nodes in > one DC. > {noformat} > ERROR [CompactionExecutor:7] 2014-11-12 10:15:13,980 CassandraDaemon.java > (line 153) Exception in thread Thread[CompactionExecutor:7,1,RMI Runtime] > java.lang.AssertionError: > /data/data/KEYSPACE_NAME/TABLE_NAME/KEYSPACE_NAME-TABLE_NAME-jb-308572-Data.db > is not correctly marked compacting > at > org.apache.cassandra.db.compaction.AbstractCompactionTask.(AbstractCompactionTask.java:49) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.compaction.CompactionTask.(CompactionTask.java:62) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.compaction.LeveledCompactionTask.(LeveledCompactionTask.java:33) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getCompactionTask(LeveledCompactionStrategy.java:170) > ~[apache-cassandra-2.1.1.jar:2.1.1] > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8314) C* 2.1.1: AssertionError: "stream can only read forward"
[ https://issues.apache.org/jira/browse/CASSANDRA-8314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8314: --- Reproduced In: 2.1.1 Fix Version/s: 2.1.3 > C* 2.1.1: AssertionError: "stream can only read forward" > - > > Key: CASSANDRA-8314 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8314 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Donald Smith > Fix For: 2.1.3 > > > I see this multiple nodes on a 2.1.1 cluster running on CentOS 6.4: > {noformat} > ERROR [STREAM-IN-/10.6.1.104] 2014-11-13 14:13:16,565 StreamSession.java > (line 470) [Stream #45bdfe30-6b81-11e4-a7ca-b150b4554347] Streaming error > occurred > java.io.IOException: Too many retries for Header (cfId: > aaefa7d7-9d72-3d18-b5f0-02b30cee5bd7, #29, version: jb, estimated keys: > 12672, transfer size: 130005779, compressed?: true, repairedAt: 0) > at > org.apache.cassandra.streaming.StreamSession.doRetry(StreamSession.java:594) > [apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:53) > [apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38) > [apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) > [apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245) > [apache-cassandra-2.1.1.jar:2.1.1] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60] > Caused by: java.lang.AssertionError: stream can only read forward. > at > org.apache.cassandra.streaming.compress.CompressedInputStream.position(CompressedInputStream.java:107) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:85) > ~[apache-cassandra-2.1.1.jar:2.1.1] > at > org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:48) > [apache-cassandra-2.1.1.jar:2.1.1] > ... 4 common frames omitted > {noformat} > We couldn't upgrade SStables due to exceptions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8347) 2.1.1: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException after accidental computer crash
[ https://issues.apache.org/jira/browse/CASSANDRA-8347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266717#comment-14266717 ] Philip Thompson commented on CASSANDRA-8347: If your sstable is corrupt, run nodetool scrub against it, then repair. If that doesn't fix the problem, please re-open this. > 2.1.1: org.apache.cassandra.io.sstable.CorruptSSTableException: > java.io.EOFException after accidental computer crash > > > Key: CASSANDRA-8347 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8347 > Project: Cassandra > Issue Type: Bug >Reporter: Evgeny Pasynkov > > {code}9:08:56.972 [SSTableBatchOpen:1] ERROR o.a.c.service.CassandraDaemon - > Exception in thread Thread[SSTableBatchOpen:1,5,main] > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException > at > org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:129) > ~[cassandra-all-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:83) > ~[cassandra-all-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:50) > ~[cassandra-all-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:48) > ~[cassandra-all-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:766) > ~[cassandra-all-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:725) > ~[cassandra-all-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:402) > ~[cassandra-all-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:302) > ~[cassandra-all-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:438) > ~[cassandra-all-2.1.1.jar:2.1.1] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > ~[na:1.7.0_65] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_65] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_65] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_65] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] > Caused by: java.io.EOFException: null > at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) > ~[na:1.7.0_65] > at java.io.DataInputStream.readUTF(DataInputStream.java:589) ~[na:1.7.0_65] > at java.io.DataInputStream.readUTF(DataInputStream.java:564) ~[na:1.7.0_65] > at > org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:104) > ~[cassandra-all-2.1.1.jar:2.1.1] > ... 13 common frames omitted{code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CASSANDRA-8347) 2.1.1: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException after accidental computer crash
[ https://issues.apache.org/jira/browse/CASSANDRA-8347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson resolved CASSANDRA-8347. Resolution: Not a Problem > 2.1.1: org.apache.cassandra.io.sstable.CorruptSSTableException: > java.io.EOFException after accidental computer crash > > > Key: CASSANDRA-8347 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8347 > Project: Cassandra > Issue Type: Bug >Reporter: Evgeny Pasynkov > > {code}9:08:56.972 [SSTableBatchOpen:1] ERROR o.a.c.service.CassandraDaemon - > Exception in thread Thread[SSTableBatchOpen:1,5,main] > org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException > at > org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:129) > ~[cassandra-all-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:83) > ~[cassandra-all-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:50) > ~[cassandra-all-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:48) > ~[cassandra-all-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:766) > ~[cassandra-all-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:725) > ~[cassandra-all-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:402) > ~[cassandra-all-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:302) > ~[cassandra-all-2.1.1.jar:2.1.1] > at > org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:438) > ~[cassandra-all-2.1.1.jar:2.1.1] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > ~[na:1.7.0_65] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_65] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_65] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [na:1.7.0_65] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] > Caused by: java.io.EOFException: null > at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) > ~[na:1.7.0_65] > at java.io.DataInputStream.readUTF(DataInputStream.java:589) ~[na:1.7.0_65] > at java.io.DataInputStream.readUTF(DataInputStream.java:564) ~[na:1.7.0_65] > at > org.apache.cassandra.io.compress.CompressionMetadata.(CompressionMetadata.java:104) > ~[cassandra-all-2.1.1.jar:2.1.1] > ... 13 common frames omitted{code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8041) Utility sstablesplit should prevent users from running when C* is running
[ https://issues.apache.org/jira/browse/CASSANDRA-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-8041: --- Fix Version/s: 2.0.12 > Utility sstablesplit should prevent users from running when C* is running > - > > Key: CASSANDRA-8041 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8041 > Project: Cassandra > Issue Type: Bug > Components: Documentation & website, Tools >Reporter: Erick Ramirez >Priority: Minor > Fix For: 2.0.12 > > > The sstablesplit utility is designed for use when C* is offline, but there is > nothing stopping the user from running it on a live system. There are also no > warning messages alerting the user to this effect. > The help information should also be updated to explicitly state that the > utility should only be used when C* is offline. > Finally, this utility is not included in any of the documentation. Please > update accordingly. Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7438) Serializing Row cache alternative (Fully off heap)
[ https://issues.apache.org/jira/browse/CASSANDRA-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266809#comment-14266809 ] Robert Stupp commented on CASSANDRA-7438: - OHC works in Cassandra: * unit tests pass ({{ant test}}, not difference against trunk) * get and put verified in debugger and a (simple) table * row cache saving and load working, too > Serializing Row cache alternative (Fully off heap) > -- > > Key: CASSANDRA-7438 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7438 > Project: Cassandra > Issue Type: Improvement > Components: Core > Environment: Linux >Reporter: Vijay >Assignee: Robert Stupp > Labels: performance > Fix For: 3.0 > > Attachments: 0001-CASSANDRA-7438.patch, tests.zip > > > Currently SerializingCache is partially off heap, keys are still stored in > JVM heap as BB, > * There is a higher GC costs for a reasonably big cache. > * Some users have used the row cache efficiently in production for better > results, but this requires careful tunning. > * Overhead in Memory for the cache entries are relatively high. > So the proposal for this ticket is to move the LRU cache logic completely off > heap and use JNI to interact with cache. We might want to ensure that the new > implementation match the existing API's (ICache), and the implementation > needs to have safe memory access, low overhead in memory and less memcpy's > (As much as possible). > We might also want to make this cache configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8566) node crash (while auto-compaction?)
[ https://issues.apache.org/jira/browse/CASSANDRA-8566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14266829#comment-14266829 ] Benedict commented on CASSANDRA-8566: - Could you provide the information requested off of Brent in CASSANDRA-8552 [here|https://issues.apache.org/jira/browse/CASSANDRA-8552?focusedCommentId=14263495&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14263495]? Could you also attach the contents of /var/log/kernel/? Optionally, including a thread and heap dump could also be helpful (if the latter, please reduce your heap size, so that we can explore it easily). Heap size will not affect this (except to make it more likely), as it is native memory related. > node crash (while auto-compaction?) > --- > > Key: CASSANDRA-8566 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8566 > Project: Cassandra > Issue Type: Bug > Environment: Linux CentOS 6.6 64bit, Cassandra 2.1.2 (release) >Reporter: Dmitri Dmitrienko > Fix For: 2.1.3 > > Attachments: 1.log > > > As data size became 20-24GB/node this issue started happening quite > frequently. With 7GB/node I didn't notice any crashes. > HEAP size was 10GB, now increased to 16GB and it didn't help. > Log is attached -- This message was sent by Atlassian JIRA (v6.3.4#6332)