[jira] [Commented] (CASSANDRA-13896) Improving Cassandra write performance
[ https://issues.apache.org/jira/browse/CASSANDRA-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441984#comment-16441984 ] Michael Burman commented on CASSANDRA-13896: I don't think there's notable performance difference here compared to 3.10. The scaling seems to be limited to around the same number of operations and adding hardware does not change that (I also tried to match the hardware, but there's no point in going beyond 16 cores as the limitation is at around 9 cores when using this stress-profile). Single node. > Improving Cassandra write performance > --- > > Key: CASSANDRA-13896 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13896 > Project: Cassandra > Issue Type: Improvement > Components: Local Write-Read Paths > Environment: Skylake server with 2 sockets, 192GB RAM, 3xPCIe SSDs > OS: Centos 7.3 > Java: Oracle JDK1.8.0_121 >Reporter: Prince Nana Owusu Boateng >Priority: Major > Labels: Performance > Fix For: 4.x > > Attachments: Screen Shot 2017-09-22 at 11.22.43 AM.png, Screen Shot > 2017-09-22 at 3.31.09 PM.png > > > During our Cassandra performance testing, we see high percentage of the CPU > spent in *org.apache.cassandra.utils.memory.SlabAllocator.allocate(int, > OpOrder Group) * method. Appears to be high contention of the > ** atomic Integer in write workloads. This structure is > used by the threads for keeping track of the region bytebuffer allocation. > When the contention appears, adding more clients, modifications of write > specific parameters does not change write throughput performance. Attached > are the details of Java Flight Recorder (JFR), showing hot functions and also > performance results. When we see this contention, we still have plenty of > CPU and throughput left ( *<20%* Total average CPU utilization and *<11%* > of the storage write total throughput). This occurs on Cassandra 3.10.0-src > version using the Cassandra-Stress. > Proposal: > We will like to introduce a solution which eliminates the atomic operations > on the ** atomic Integer. This implementation will allow > concurrent allocation of bytebuffers without an atomic compareAndSet and > incrementAndGet operations. The solution is expected to increase overall > write performance while improving CPU utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-12526) For LCS, single SSTable up-level is handled inefficiently
[ https://issues.apache.org/jira/browse/CASSANDRA-12526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-12526: Status: Patch Available (was: Open) https://github.com/krummas/cassandra/commits/marcuse/12526 (includes CASSANDRA-14388 to make the test work) https://circleci.com/gh/krummas/cassandra/tree/marcuse%2F12526 in my silly benchmarks (writing 100M keys to a LCS table with 10MB sstables and measuring time until LCS is fully leveled), this reduces the time spent compacting with about 20% > For LCS, single SSTable up-level is handled inefficiently > - > > Key: CASSANDRA-12526 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12526 > Project: Cassandra > Issue Type: Improvement > Components: Compaction >Reporter: Wei Deng >Assignee: Marcus Eriksson >Priority: Major > Labels: compaction, lcs, performance > Fix For: 4.x > > > I'm using the latest trunk (as of August 2016, which probably is going to be > 3.10) to run some experiments on LeveledCompactionStrategy and noticed this > inefficiency. > The test data is generated using cassandra-stress default parameters > (keyspace1.standard1), so as you can imagine, it consists of a ton of newly > inserted partitions that will never merge in compactions, which is probably > the worst kind of workload for LCS (however, I'll detail later why this > scenario should not be ignored as a corner case; for now, let's just assume > we still want to handle this scenario efficiently). > After the compaction test is done, I scrubbed debug.log for patterns that > match the "Compacted" summary so that I can see how long each individual > compaction took and how many bytes they processed. The search pattern is like > the following: > {noformat} > grep 'Compacted.*standard1' debug.log > {noformat} > Interestingly, I noticed a lot of the finished compactions are marked as > having *only one* SSTable involved. With the workload mentioned above, the > "single SSTable" compactions actually consist of the majority of all > compactions (as shown below), so its efficiency can affect the overall > compaction throughput quite a bit. > {noformat} > automaton@0ce59d338-1:~/cassandra-trunk/logs$ grep 'Compacted.*standard1' > debug.log-test1 | wc -l > 243 > automaton@0ce59d338-1:~/cassandra-trunk/logs$ grep 'Compacted.*standard1' > debug.log-test1 | grep ") 1 sstable" | wc -l > 218 > {noformat} > By looking at the code, it appears that there's a way to directly edit the > level of a particular SSTable like the following: > {code} > sstable.descriptor.getMetadataSerializer().mutateLevel(sstable.descriptor, > targetLevel); > sstable.reloadSSTableMetadata(); > {code} > To be exact, I summed up the time spent for these single-SSTable compactions > (the total data size is 60GB) and found that if each compaction only needs to > spend 100ms for only the metadata change (instead of the 10+ second they're > doing now), it can already achieve 22.75% saving on total compaction time. > Compared to what we have now (reading the whole single-SSTable from old level > and writing out the same single-SSTable at the new level), the only > difference I could think of by using this approach is that the new SSTable > will have the same file name (sequence number) as the old one's, which could > break some assumptions on some other part of the code. However, not having to > go through the full read/write IO, and not having to bear the overhead of > cleaning up the old file, creating the new file, creating more churns in heap > and file buffer, it seems the benefits outweigh the inconvenience. So I'd > argue this JIRA belongs to LHF and should be made available in 3.0.x as well. > As mentioned in the 2nd paragraph, I'm also going to address why this kind of > all-new-partition workload should not be ignored as a corner case. Basically, > for the main use case of LCS where you need to frequently merge partitions to > optimize read and eliminate tombstones and expired data sooner, LCS can be > perfectly happy and efficiently perform the partition merge and tombstone > elimination for a long time. However, as soon as the node becomes a bit > unhealthy for various reasons (could be a bad disk so it's missing a whole > bunch of mutations and need repair, could be the user chooses to ingest way > more data than it usually takes and exceeds its capability, or god-forbidden, > some DBA chooses to run offline sstablelevelreset), you will have to handle > this kind of "all-new-partition with a lot of SSTables in L0" scenario, and > once all L0 SSTables finally gets up-leveled to L1, you will likely see a lot > of such single-SSTable compactions, which is the situation this JIRA is > inten
[jira] [Updated] (CASSANDRA-14397) Stop compactions quicker when compacting wide partitions
[ https://issues.apache.org/jira/browse/CASSANDRA-14397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-14397: Status: Patch Available (was: Open) https://github.com/krummas/cassandra/commits/marcuse/14397 https://circleci.com/gh/krummas/cassandra/tree/marcuse%2F14397 > Stop compactions quicker when compacting wide partitions > > > Key: CASSANDRA-14397 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14397 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Major > Fix For: 4.x > > > We should allow compactions to be stopped when compacting wide partitions, > this will help when a user wants to run upgradesstables for example. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14397) Stop compactions quicker when compacting wide partitions
Marcus Eriksson created CASSANDRA-14397: --- Summary: Stop compactions quicker when compacting wide partitions Key: CASSANDRA-14397 URL: https://issues.apache.org/jira/browse/CASSANDRA-14397 Project: Cassandra Issue Type: Improvement Reporter: Marcus Eriksson Assignee: Marcus Eriksson Fix For: 4.x We should allow compactions to be stopped when compacting wide partitions, this will help when a user wants to run upgradesstables for example. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14396) Error about JNA on Startup
[ https://issues.apache.org/jira/browse/CASSANDRA-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441913#comment-16441913 ] Joe commented on CASSANDRA-14396: - [~jasobrown] I already gave all permissions to tmp folder but not working. even removed folder and ran. thank you for ur reply :) > Error about JNA on Startup > --- > > Key: CASSANDRA-14396 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14396 > Project: Cassandra > Issue Type: Test > Components: Build, Configuration, Core, Lifecycle > Environment: > * CentOS release 6.8 > * Cassandra 3.11.2 > * Java 1.8.0_152 > * No provide network > >Reporter: Joe >Priority: Major > Fix For: 3.11.2 > > Attachments: cassandra.log > > > > Hi, all. > I got some error on startup. > this is my own backup server which can't use network. > I just extracted 'apache-cassandra-3.11.2-bin.tar.gz' file to > /usr/local/cassandra. > and then ran like this 'cassandra -f'. > but log displayed below's error. > > I found some way to solve. but it's not working. > after JNA library download, make JNA library symbolic link - not working > can anyone advise to me about this issue?(I attached full logging file) > > ERROR [main] 2018-04-18 10:44:59,536 NativeLibraryLinux.java:62 - Failed to > link the C library against JNA. Native methods will be unavailable. > java.lang.UnsatisfiedLinkError: /tmp/jna-3506402/jna5682737284440877593.tmp: > /tmp/jna-3506402/jna5682737284440877593.tmp: failed to map segment from > shared object: Operation not permitted > at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[na:1.8.0_152] > at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[na:1.8.0_152] > at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[na:1.8.0_152] > at java.lang.Runtime.load0(Runtime.java:809) ~[na:1.8.0_152] > at java.lang.System.load(System.java:1086) ~[na:1.8.0_152] > at > com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:851) > ~[jna-4.2.2.jar:4.2.2 (b0)] > at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826) > ~[jna-4.2.2.jar:4.2.2 (b0)] > at com.sun.jna.Native.(Native.java:140) ~[jna-4.2.2.jar:4.2.2 (b0)] > at > org.apache.cassandra.utils.NativeLibraryLinux.(NativeLibraryLinux.java:53) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at org.apache.cassandra.utils.NativeLibrary.(NativeLibrary.java:93) > [apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:196) > [apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:602) > [apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) > [apache-cassandra-3.11.2.jar:3.11.2] > WARN [main] 2018-04-18 10:44:59,537 StartupChecks.java:136 - jemalloc shared > library could not be preloaded to speed up memory allocations > WARN [main] 2018-04-18 10:44:59,537 StartupChecks.java:169 - JMX is not > enabled to receive remote connections. Please see cassandra-env.sh for more > info. > ERROR [main] 2018-04-18 10:44:59,539 CassandraDaemon.java:708 - The native > library could not be initialized properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14396) Error about JNA on Startup
[ https://issues.apache.org/jira/browse/CASSANDRA-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441897#comment-16441897 ] Jason Brown commented on CASSANDRA-14396: - Some naive internet searches for "failed to map segment from shared object: Operation not permitted" suggest that {{/tmp}} needs to be executable. (Note: {{/tmp}} is where java apps often decompress/move native libraries from jar files so they can be used at runtime) > Error about JNA on Startup > --- > > Key: CASSANDRA-14396 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14396 > Project: Cassandra > Issue Type: Test > Components: Build, Configuration, Core, Lifecycle > Environment: > * CentOS release 6.8 > * Cassandra 3.11.2 > * Java 1.8.0_152 > * No provide network > >Reporter: Joe >Priority: Major > Fix For: 3.11.2 > > Attachments: cassandra.log > > > > Hi, all. > I got some error on startup. > this is my own backup server which can't use network. > I just extracted 'apache-cassandra-3.11.2-bin.tar.gz' file to > /usr/local/cassandra. > and then ran like this 'cassandra -f'. > but log displayed below's error. > > I found some way to solve. but it's not working. > after JNA library download, make JNA library symbolic link - not working > can anyone advise to me about this issue?(I attached full logging file) > > ERROR [main] 2018-04-18 10:44:59,536 NativeLibraryLinux.java:62 - Failed to > link the C library against JNA. Native methods will be unavailable. > java.lang.UnsatisfiedLinkError: /tmp/jna-3506402/jna5682737284440877593.tmp: > /tmp/jna-3506402/jna5682737284440877593.tmp: failed to map segment from > shared object: Operation not permitted > at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[na:1.8.0_152] > at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[na:1.8.0_152] > at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[na:1.8.0_152] > at java.lang.Runtime.load0(Runtime.java:809) ~[na:1.8.0_152] > at java.lang.System.load(System.java:1086) ~[na:1.8.0_152] > at > com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:851) > ~[jna-4.2.2.jar:4.2.2 (b0)] > at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826) > ~[jna-4.2.2.jar:4.2.2 (b0)] > at com.sun.jna.Native.(Native.java:140) ~[jna-4.2.2.jar:4.2.2 (b0)] > at > org.apache.cassandra.utils.NativeLibraryLinux.(NativeLibraryLinux.java:53) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at org.apache.cassandra.utils.NativeLibrary.(NativeLibrary.java:93) > [apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:196) > [apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:602) > [apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) > [apache-cassandra-3.11.2.jar:3.11.2] > WARN [main] 2018-04-18 10:44:59,537 StartupChecks.java:136 - jemalloc shared > library could not be preloaded to speed up memory allocations > WARN [main] 2018-04-18 10:44:59,537 StartupChecks.java:169 - JMX is not > enabled to receive remote connections. Please see cassandra-env.sh for more > info. > ERROR [main] 2018-04-18 10:44:59,539 CassandraDaemon.java:708 - The native > library could not be initialized properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14365) Commit log replay failure for static columns with collections in clustering keys
[ https://issues.apache.org/jira/browse/CASSANDRA-14365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kurt Greaves reassigned CASSANDRA-14365: Assignee: Vincent White > Commit log replay failure for static columns with collections in clustering > keys > > > Key: CASSANDRA-14365 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14365 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Vincent White >Assignee: Vincent White >Priority: Major > > In the old storage engine, static cells with a collection as part of the > clustering key fail to validate because a 0 byte collection (like in the cell > name of a static cell) isn't valid. > To reproduce: > 1. > {code:java} > CREATE TABLE test.x ( > id int, > id2 frozen>, > st int static, > PRIMARY KEY (id, id2) > ); > INSERT INTO test.x (id, st) VALUES (1, 2); > {code} > 2. > Kill the cassandra process > 3. > Restart cassandra to replay the commitlog > Outcome: > {noformat} > ERROR [main] 2018-04-05 04:58:23,741 JVMStabilityInspector.java:99 - Exiting > due to error while processing commit log during initialization. > org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: > Unexpected error deserializing mutation; saved to > /tmp/mutation3825739904516830950dat. This may be caused by replaying a > mutation against a table with the same name but incompatible schema. > Exception follows: org.apache.cassandra.serializers.MarshalException: Not > enough bytes to read a set > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:638) > [main/:na] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:565) > [main/:na] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:517) > [main/:na] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:397) > [main/:na] > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:143) > [main/:na] > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:181) > [main/:na] > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:161) > [main/:na] > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:284) > [main/:na] > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:533) > [main/:na] > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:642) > [main/:na] > {noformat} > I haven't investigated if there are other more subtle issues caused by these > cells failing to validate other places in the code, but I believe the fix for > this is to check for 0 byte length collections and accept them as valid as we > do with other types. > I haven't had a chance for any extensive testing but this naive patch seems > to have the desired affect. [2.2 PoC > Patch|https://github.com/vincewhite/cassandra/commits/zero_length_collection] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14304) DELETE after INSERT IF NOT EXISTS does not work
[ https://issues.apache.org/jira/browse/CASSANDRA-14304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441876#comment-16441876 ] Kurt Greaves commented on CASSANDRA-14304: -- bq. Should we make our own retry policy to prevent retry of such batch? We use batches to write large binary BLOBs chunked in smaller statements. You can, but you'd also have to ensure you don't use speculative execution either and possibly other things using client timestamps I don't know about, otherwise you risk mixing client and server timestamps (more so than LWT). bq. However using LWT everywhere seems overkill. It probably is, but can you break out the statements that absolutely need to be ordered into another table? That might be a better alternative... > DELETE after INSERT IF NOT EXISTS does not work > --- > > Key: CASSANDRA-14304 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14304 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Julien >Assignee: Vinay Chella >Priority: Major > Attachments: debug.log, system.log > > > DELETE a row immediately after INSERT IF NOT EXISTS does not work. > Can be reproduced with this CQL script: > {code:java} > CREATE KEYSPACE ks WITH REPLICATION = { 'class' : 'SimpleStrategy', > 'replication_factor' : 1 }; > CREATE TABLE ks.ta ( id text PRIMARY KEY, col text ); > INSERT INTO ks.ta (id, col) VALUES ('myId', 'myCol') IF NOT EXISTS; > DELETE FROM ks.ta WHERE id = 'myId'; > SELECT * FROM ks.ta WHERE id='myId'; > {code} > {code:java} > [cqlsh 5.0.1 | Cassandra 3.11.2 | CQL spec 3.4.4 | Native protocol v4] > Use HELP for help. > WARNING: pyreadline dependency missing. Install to enable tab completion. > cqlsh> CREATE KEYSPACE ks WITH REPLICATION = { 'class' : 'SimpleStrategy', > 'replication_factor' : 1 }; > cqlsh> CREATE TABLE ks.ta ( id text PRIMARY KEY, col text ); > cqlsh> INSERT INTO ks.ta (id, col) VALUES ('myId', 'myCol') IF NOT EXISTS; > [applied] > --- > True > cqlsh> DELETE FROM ks.ta WHERE id = 'myId'; > cqlsh> SELECT * FROM ks.ta WHERE id='myId'; > id | col > --+--- > myId | myCol > {code} > * Only happens if the client is on a different host (works as expected on > the same host) > * Works as expected without IF NOT EXISTS > * A ~500 ms delay between INSERT and DELETE fixes the issue. > Logs attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing
[ https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Lohfink updated CASSANDRA-14281: -- Attachment: benchmark.html > Improve LatencyMetrics performance by reducing write path processing > > > Key: CASSANDRA-14281 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14281 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Burman >Assignee: Michael Burman >Priority: Major > Attachments: bench.png, bench2.png, benchmark.html, benchmark2.png > > > Currently for each write/read/rangequery/CAS touching the CFS we write a > latency metric which takes a lot of processing time (up to 66% of the total > processing time if the update was empty). > The way latencies are recorded is to use both a dropwizard "Timer" as well as > "Counter". Latter is used for totalLatency and the previous is decaying > metric for rates and certain percentile metrics. We then replicate all of > these CFS writes to the KeyspaceMetrics and globalWriteLatencies. > Instead of doing this on the write phase we should merge the metrics when > they're read. This is much less common occurrence and thus we save a lot of > CPU time in total. This also speeds up the write path. > Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each > update operation, which causes a contention if there are more than one thread > updating the histogram. This impacts scalability when using larger machines. > We should make it lock-free as much as possible and also avoid a single > CAS-update from blocking all the concurrent threads from making an update. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing
[ https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441870#comment-16441870 ] Chris Lohfink commented on CASSANDRA-14281: --- fwiw tried this again on a beefier linux box instead of my laptop and saw the improvement: !benchmark2.png|width=799,height=417! > Improve LatencyMetrics performance by reducing write path processing > > > Key: CASSANDRA-14281 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14281 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Burman >Assignee: Michael Burman >Priority: Major > Attachments: bench.png, bench2.png, benchmark2.png > > > Currently for each write/read/rangequery/CAS touching the CFS we write a > latency metric which takes a lot of processing time (up to 66% of the total > processing time if the update was empty). > The way latencies are recorded is to use both a dropwizard "Timer" as well as > "Counter". Latter is used for totalLatency and the previous is decaying > metric for rates and certain percentile metrics. We then replicate all of > these CFS writes to the KeyspaceMetrics and globalWriteLatencies. > Instead of doing this on the write phase we should merge the metrics when > they're read. This is much less common occurrence and thus we save a lot of > CPU time in total. This also speeds up the write path. > Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each > update operation, which causes a contention if there are more than one thread > updating the histogram. This impacts scalability when using larger machines. > We should make it lock-free as much as possible and also avoid a single > CAS-update from blocking all the concurrent threads from making an update. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14281) Improve LatencyMetrics performance by reducing write path processing
[ https://issues.apache.org/jira/browse/CASSANDRA-14281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Lohfink updated CASSANDRA-14281: -- Attachment: benchmark2.png > Improve LatencyMetrics performance by reducing write path processing > > > Key: CASSANDRA-14281 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14281 > Project: Cassandra > Issue Type: Improvement > Components: Core >Reporter: Michael Burman >Assignee: Michael Burman >Priority: Major > Attachments: bench.png, bench2.png, benchmark2.png > > > Currently for each write/read/rangequery/CAS touching the CFS we write a > latency metric which takes a lot of processing time (up to 66% of the total > processing time if the update was empty). > The way latencies are recorded is to use both a dropwizard "Timer" as well as > "Counter". Latter is used for totalLatency and the previous is decaying > metric for rates and certain percentile metrics. We then replicate all of > these CFS writes to the KeyspaceMetrics and globalWriteLatencies. > Instead of doing this on the write phase we should merge the metrics when > they're read. This is much less common occurrence and thus we save a lot of > CPU time in total. This also speeds up the write path. > Currently, the DecayingEstimatedHistogramReservoir acquires a lock for each > update operation, which causes a contention if there are more than one thread > updating the histogram. This impacts scalability when using larger machines. > We should make it lock-free as much as possible and also avoid a single > CAS-update from blocking all the concurrent threads from making an update. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14396) Error about JNA on Startup
[ https://issues.apache.org/jira/browse/CASSANDRA-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe updated CASSANDRA-14396: Description: Hi, all. I got some error on startup. this is my own backup server which can't use network. I just extracted 'apache-cassandra-3.11.2-bin.tar.gz' file to /usr/local/cassandra. and then ran like this 'cassandra -f'. but log displayed below's error. I found some way to solve. but it's not working. after JNA library download, make JNA library symbolic link - not working can anyone advise to me about this issue?(I attached full logging file) ERROR [main] 2018-04-18 10:44:59,536 NativeLibraryLinux.java:62 - Failed to link the C library against JNA. Native methods will be unavailable. java.lang.UnsatisfiedLinkError: /tmp/jna-3506402/jna5682737284440877593.tmp: /tmp/jna-3506402/jna5682737284440877593.tmp: failed to map segment from shared object: Operation not permitted at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[na:1.8.0_152] at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[na:1.8.0_152] at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[na:1.8.0_152] at java.lang.Runtime.load0(Runtime.java:809) ~[na:1.8.0_152] at java.lang.System.load(System.java:1086) ~[na:1.8.0_152] at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:851) ~[jna-4.2.2.jar:4.2.2 (b0)] at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826) ~[jna-4.2.2.jar:4.2.2 (b0)] at com.sun.jna.Native.(Native.java:140) ~[jna-4.2.2.jar:4.2.2 (b0)] at org.apache.cassandra.utils.NativeLibraryLinux.(NativeLibraryLinux.java:53) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.utils.NativeLibrary.(NativeLibrary.java:93) [apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:196) [apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:602) [apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) [apache-cassandra-3.11.2.jar:3.11.2] WARN [main] 2018-04-18 10:44:59,537 StartupChecks.java:136 - jemalloc shared library could not be preloaded to speed up memory allocations WARN [main] 2018-04-18 10:44:59,537 StartupChecks.java:169 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info. ERROR [main] 2018-04-18 10:44:59,539 CassandraDaemon.java:708 - The native library could not be initialized properly. was: Hi, all. I got some error on startup. this is my own backup server which can't use network. I just extracted 'apache-cassandra-3.11.2-bin.tar.gz' file to /usr/local/cassandra. and then ran like this 'cassandra -f'. but log displayed below's error. I found some way to solve. but it's not working. after JNA library download, make JNA library symbolic link - not working can anyone advise to me about this issue? ERROR [main] 2018-04-18 10:44:59,536 NativeLibraryLinux.java:62 - Failed to link the C library against JNA. Native methods will be unavailable. java.lang.UnsatisfiedLinkError: /tmp/jna-3506402/jna5682737284440877593.tmp: /tmp/jna-3506402/jna5682737284440877593.tmp: failed to map segment from shared object: Operation not permitted at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[na:1.8.0_152] at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[na:1.8.0_152] at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[na:1.8.0_152] at java.lang.Runtime.load0(Runtime.java:809) ~[na:1.8.0_152] at java.lang.System.load(System.java:1086) ~[na:1.8.0_152] at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:851) ~[jna-4.2.2.jar:4.2.2 (b0)] at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826) ~[jna-4.2.2.jar:4.2.2 (b0)] at com.sun.jna.Native.(Native.java:140) ~[jna-4.2.2.jar:4.2.2 (b0)] at org.apache.cassandra.utils.NativeLibraryLinux.(NativeLibraryLinux.java:53) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.utils.NativeLibrary.(NativeLibrary.java:93) [apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:196) [apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:602) [apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) [apache-cassandra-3.11.2.jar:3.11.2] WARN [main] 2018-04-18 10:44:59,537 StartupChecks.java:136 - jemalloc shared library could not be preloaded to speed up memory allocations WARN [main] 2018-04-18 10:44:59,537 StartupChecks.java:169 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info. ERROR [main] 2018-04-18 10:44:59,539 CassandraDaemon.java:708 - The native librar
[jira] [Updated] (CASSANDRA-14396) Error about JNA on Startup
[ https://issues.apache.org/jira/browse/CASSANDRA-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe updated CASSANDRA-14396: Attachment: cassandra.log > Error about JNA on Startup > --- > > Key: CASSANDRA-14396 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14396 > Project: Cassandra > Issue Type: Test > Components: Build, Configuration, Core, Lifecycle > Environment: > * CentOS release 6.8 > * Cassandra 3.11.2 > * Java 1.8.0_152 > * No provide network > >Reporter: Joe >Priority: Major > Fix For: 3.11.2 > > Attachments: cassandra.log > > > > Hi, all. > I got some error on startup. > this is my own backup server which can't use network. > I just extracted 'apache-cassandra-3.11.2-bin.tar.gz' file to > /usr/local/cassandra. > and then ran like this 'cassandra -f'. > but log displayed below's error. > > I found some way to solve. but it's not working. > after JNA library download, make JNA library symbolic link - not working > can anyone advise to me about this issue? > > ERROR [main] 2018-04-18 10:44:59,536 NativeLibraryLinux.java:62 - Failed to > link the C library against JNA. Native methods will be unavailable. > java.lang.UnsatisfiedLinkError: /tmp/jna-3506402/jna5682737284440877593.tmp: > /tmp/jna-3506402/jna5682737284440877593.tmp: failed to map segment from > shared object: Operation not permitted > at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[na:1.8.0_152] > at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[na:1.8.0_152] > at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[na:1.8.0_152] > at java.lang.Runtime.load0(Runtime.java:809) ~[na:1.8.0_152] > at java.lang.System.load(System.java:1086) ~[na:1.8.0_152] > at > com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:851) > ~[jna-4.2.2.jar:4.2.2 (b0)] > at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826) > ~[jna-4.2.2.jar:4.2.2 (b0)] > at com.sun.jna.Native.(Native.java:140) ~[jna-4.2.2.jar:4.2.2 (b0)] > at > org.apache.cassandra.utils.NativeLibraryLinux.(NativeLibraryLinux.java:53) > ~[apache-cassandra-3.11.2.jar:3.11.2] > at org.apache.cassandra.utils.NativeLibrary.(NativeLibrary.java:93) > [apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:196) > [apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:602) > [apache-cassandra-3.11.2.jar:3.11.2] > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) > [apache-cassandra-3.11.2.jar:3.11.2] > WARN [main] 2018-04-18 10:44:59,537 StartupChecks.java:136 - jemalloc shared > library could not be preloaded to speed up memory allocations > WARN [main] 2018-04-18 10:44:59,537 StartupChecks.java:169 - JMX is not > enabled to receive remote connections. Please see cassandra-env.sh for more > info. > ERROR [main] 2018-04-18 10:44:59,539 CassandraDaemon.java:708 - The native > library could not be initialized properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14396) Error about JNA on Startup
Joe created CASSANDRA-14396: --- Summary: Error about JNA on Startup Key: CASSANDRA-14396 URL: https://issues.apache.org/jira/browse/CASSANDRA-14396 Project: Cassandra Issue Type: Test Components: Build, Configuration, Core, Lifecycle Environment: * CentOS release 6.8 * Cassandra 3.11.2 * Java 1.8.0_152 * No provide network Reporter: Joe Fix For: 3.11.2 Hi, all. I got some error on startup. this is my own backup server which can't use network. I just extracted 'apache-cassandra-3.11.2-bin.tar.gz' file to /usr/local/cassandra. and then ran like this 'cassandra -f'. but log displayed below's error. I found some way to solve. but it's not working. after JNA library download, make JNA library symbolic link - not working can anyone advise to me about this issue? ERROR [main] 2018-04-18 10:44:59,536 NativeLibraryLinux.java:62 - Failed to link the C library against JNA. Native methods will be unavailable. java.lang.UnsatisfiedLinkError: /tmp/jna-3506402/jna5682737284440877593.tmp: /tmp/jna-3506402/jna5682737284440877593.tmp: failed to map segment from shared object: Operation not permitted at java.lang.ClassLoader$NativeLibrary.load(Native Method) ~[na:1.8.0_152] at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941) ~[na:1.8.0_152] at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824) ~[na:1.8.0_152] at java.lang.Runtime.load0(Runtime.java:809) ~[na:1.8.0_152] at java.lang.System.load(System.java:1086) ~[na:1.8.0_152] at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:851) ~[jna-4.2.2.jar:4.2.2 (b0)] at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:826) ~[jna-4.2.2.jar:4.2.2 (b0)] at com.sun.jna.Native.(Native.java:140) ~[jna-4.2.2.jar:4.2.2 (b0)] at org.apache.cassandra.utils.NativeLibraryLinux.(NativeLibraryLinux.java:53) ~[apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.utils.NativeLibrary.(NativeLibrary.java:93) [apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:196) [apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:602) [apache-cassandra-3.11.2.jar:3.11.2] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) [apache-cassandra-3.11.2.jar:3.11.2] WARN [main] 2018-04-18 10:44:59,537 StartupChecks.java:136 - jemalloc shared library could not be preloaded to speed up memory allocations WARN [main] 2018-04-18 10:44:59,537 StartupChecks.java:169 - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info. ERROR [main] 2018-04-18 10:44:59,539 CassandraDaemon.java:708 - The native library could not be initialized properly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14298) cqlshlib tests broken on b.a.o
[ https://issues.apache.org/jira/browse/CASSANDRA-14298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440276#comment-16440276 ] Patrick Bannister edited comment on CASSANDRA-14298 at 4/18/18 3:05 AM: All tests in cqlsh_tests/cqlsh_tests.py pass on trunk, cassandra-3.11, and cassandra-3.0 on Ubuntu 16.04 LTS. was (Author: ptbannister): All tests in cqlsh_tests/cqlsh_tests.py pass on trunk on Ubuntu 16.04 LTS (except one test that is skipped by pytest - probably version related). > cqlshlib tests broken on b.a.o > -- > > Key: CASSANDRA-14298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14298 > Project: Cassandra > Issue Type: Bug > Components: Build, Testing >Reporter: Stefan Podkowinski >Assignee: Patrick Bannister >Priority: Major > Attachments: cqlsh_tests_notes.md > > > It appears that cqlsh-tests on builds.apache.org on all branches stopped > working since we removed nosetests from the system environment. See e.g. > [here|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-cqlsh-tests/458/cython=no,jdk=JDK%201.8%20(latest),label=cassandra/console]. > Looks like we either have to make nosetests available again or migrate to > pytest as we did with dtests. Giving pytest a quick try resulted in many > errors locally, but I haven't inspected them in detail yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Issue Comment Deleted] (CASSANDRA-14298) cqlshlib tests broken on b.a.o
[ https://issues.apache.org/jira/browse/CASSANDRA-14298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Bannister updated CASSANDRA-14298: -- Comment: was deleted (was: I just updated cqlsh_tests_nodes.md - my initial analysis of a ccm usage problem was wrong.) > cqlshlib tests broken on b.a.o > -- > > Key: CASSANDRA-14298 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14298 > Project: Cassandra > Issue Type: Bug > Components: Build, Testing >Reporter: Stefan Podkowinski >Assignee: Patrick Bannister >Priority: Major > Attachments: cqlsh_tests_notes.md > > > It appears that cqlsh-tests on builds.apache.org on all branches stopped > working since we removed nosetests from the system environment. See e.g. > [here|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-cqlsh-tests/458/cython=no,jdk=JDK%201.8%20(latest),label=cassandra/console]. > Looks like we either have to make nosetests available again or migrate to > pytest as we did with dtests. Giving pytest a quick try resulted in many > errors locally, but I haven't inspected them in detail yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14394) Streaming fails with AssertionError
[ https://issues.apache.org/jira/browse/CASSANDRA-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441707#comment-16441707 ] Paulo Motta edited comment on CASSANDRA-14394 at 4/18/18 2:58 AM: -- {quote}Ah, looks like 13938. Now I feel less bad about breaking something in what was meant to be a minor refactor. Encourage either of you to commit the dtest with/without proving it's a dupe. {quote} -The funny thing is that during bissect I noticed that this starts failing consistently after CASSANDRA-14260, but after investigating I haven't found anything there that broken this, so I think this is some timing/synchronization bug that became more likely after 14260. o- edit: I confirmed this reproduces consistently after CASSANDRA-12229 (and not CASSANDRA-14260), I probably screwed something up in the previous bisect. {quote}I think it might be a dupe of CASSANDRA-13938, introduced by CASANDRA-12229 {quote} Good call, I did a quick search on google of " stream can only read forward" and found only CASSANDRA-8314 so I thought it was a new issue, next time I will use the JIRA search bar. :) I will close this as dupe of CASSANDRA-13938 for now. Feel free to reopen if this is not the case and to include the dtest on that Thanks for the quick feedback! was (Author: pauloricardomg): bq. Ah, looks like 13938. Now I feel less bad about breaking something in what was meant to be a minor refactor. Encourage either of you to commit the dtest with/without proving it's a dupe. The funny thing is that during bissect I noticed that this starts failing consistently after CASSANDRA-14260, but after investigating I haven't found anything there that broken this, so I think this is some timing/synchronization bug that became more likely after 14260. bq. I think it might be a dupe of CASSANDRA-13938, introduced by CASANDRA-12229 Good call, I did a quick search on google of " stream can only read forward" and found only CASSANDRA-8314 so I thought it was a new issue, next time I will use the JIRA search bar. :) I will close this as dupe of CASSANDRA-13938 for now. Feel free to reopen if this is not the case and to include the dtest on that Thanks for the quick feedback! > Streaming fails with AssertionError > --- > > Key: CASSANDRA-14394 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14394 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Paulo Motta >Priority: Major > Fix For: 4.x > > > The bootstrap from the attached dtest fails with the following error on trunk: > {code:none} > ERROR [Stream-Deserializer-127.0.0.1:38369-af884494] 2018-04-17 12:36:53,420 > StreamingInboundHandler.java:210 - [Stream channel: af884494] stream > operation from 127.0.0.1:38369 failedjava.lang.AssertionError: stream can > only read forward. at > org.apache.cassandra.db.streaming.CompressedInputStream.position(CompressedInputStream.java:108) > at > org.apache.cassandra.db.streaming.CompressedCassandraStreamReader.read(CompressedCassandraStreamReader.java:93) > at > org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:63) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:49) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36) > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) > at > org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:177) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13971) Automatic certificate management using Vault
[ https://issues.apache.org/jira/browse/CASSANDRA-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441812#comment-16441812 ] Nate McCall commented on CASSANDRA-13971: - [~jasobrown] MPLv2.0 is on the Category-B list as you pointed out, so we are good to go there. [~spo...@gmail.com] Just make sure you put a reference to the version and license following the pattern in [https://github.com/apache/cassandra/tree/trunk/lib/licenses|https://github.com/apache/cassandra/tree/trunk/lib/licenses.] > Automatic certificate management using Vault > > > Key: CASSANDRA-13971 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13971 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > Labels: security > Fix For: 4.x > > > We've been adding security features during the last years to enable users to > secure their clusters, if they are willing to use them and do so correctly. > Some features are powerful and easy to work with, such as role based > authorization. Other features that require to manage a local keystore are > rather painful to deal with. Think about setting up SSL.. > To be fair, keystore related issues and certificate handling hasn't been > invented by us. We're just following Java standards there. But that doesn't > mean that we absolutely have to, if there are better options. I'd like to > give it a shoot and find out if we can automate certificate/key handling > (PKI) by using external APIs. In this case, the implementation will be based > on [Vault|https://vaultproject.io]. But certificate management services > offered by cloud providers may also be able to handle the use-case and I > intend to create a generic, pluggable API for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13971) Automatic certificate management using Vault
[ https://issues.apache.org/jira/browse/CASSANDRA-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441788#comment-16441788 ] Jason Brown edited comment on CASSANDRA-13971 at 4/18/18 2:32 AM: -- Getting to it this week. Can you just check with ASF legal about the MPL? It should be legit, but let's be sure. Also, looks like Vault 0.10.0 has been released. Can you upgrade to that? Also, let me know if there's any significant changes because of that. was (Author: jasobrown): Getting to it this week. Can you just check with ASF legal about the MPL? It should be legit, but let's be sure. > Automatic certificate management using Vault > > > Key: CASSANDRA-13971 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13971 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > Labels: security > Fix For: 4.x > > > We've been adding security features during the last years to enable users to > secure their clusters, if they are willing to use them and do so correctly. > Some features are powerful and easy to work with, such as role based > authorization. Other features that require to manage a local keystore are > rather painful to deal with. Think about setting up SSL.. > To be fair, keystore related issues and certificate handling hasn't been > invented by us. We're just following Java standards there. But that doesn't > mean that we absolutely have to, if there are better options. I'd like to > give it a shoot and find out if we can automate certificate/key handling > (PKI) by using external APIs. In this case, the implementation will be based > on [Vault|https://vaultproject.io]. But certificate management services > offered by cloud providers may also be able to handle the use-case and I > intend to create a generic, pluggable API for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13971) Automatic certificate management using Vault
[ https://issues.apache.org/jira/browse/CASSANDRA-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441788#comment-16441788 ] Jason Brown commented on CASSANDRA-13971: - Getting to it this week. Can you just check with ASF legal about the MPL? It should be legit, but let's be sure. > Automatic certificate management using Vault > > > Key: CASSANDRA-13971 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13971 > Project: Cassandra > Issue Type: Improvement > Components: Streaming and Messaging >Reporter: Stefan Podkowinski >Assignee: Stefan Podkowinski >Priority: Major > Labels: security > Fix For: 4.x > > > We've been adding security features during the last years to enable users to > secure their clusters, if they are willing to use them and do so correctly. > Some features are powerful and easy to work with, such as role based > authorization. Other features that require to manage a local keystore are > rather painful to deal with. Think about setting up SSL.. > To be fair, keystore related issues and certificate handling hasn't been > invented by us. We're just following Java standards there. But that doesn't > mean that we absolutely have to, if there are better options. I'd like to > give it a shoot and find out if we can automate certificate/key handling > (PKI) by using external APIs. In this case, the implementation will be based > on [Vault|https://vaultproject.io]. But certificate management services > offered by cloud providers may also be able to handle the use-case and I > intend to create a generic, pluggable API for that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14394) Streaming fails with AssertionError
[ https://issues.apache.org/jira/browse/CASSANDRA-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441765#comment-16441765 ] Jason Brown commented on CASSANDRA-14394: - [~pauloricardomg] cool, thanks for the research. I may end up stealing your dtest if it's handy :) > Streaming fails with AssertionError > --- > > Key: CASSANDRA-14394 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14394 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Paulo Motta >Priority: Major > Fix For: 4.x > > > The bootstrap from the attached dtest fails with the following error on trunk: > {code:none} > ERROR [Stream-Deserializer-127.0.0.1:38369-af884494] 2018-04-17 12:36:53,420 > StreamingInboundHandler.java:210 - [Stream channel: af884494] stream > operation from 127.0.0.1:38369 failedjava.lang.AssertionError: stream can > only read forward. at > org.apache.cassandra.db.streaming.CompressedInputStream.position(CompressedInputStream.java:108) > at > org.apache.cassandra.db.streaming.CompressedCassandraStreamReader.read(CompressedCassandraStreamReader.java:93) > at > org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:63) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:49) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36) > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) > at > org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:177) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13047) Point cqlsh help to the new doc
[ https://issues.apache.org/jira/browse/CASSANDRA-13047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441758#comment-16441758 ] Patrick Bannister commented on CASSANDRA-13047: --- Concur with recommendation by [~spo...@gmail.com] that cqlsh should link to versioned documentation for the corresponding CQL spec. > Point cqlsh help to the new doc > --- > > Key: CASSANDRA-13047 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13047 > Project: Cassandra > Issue Type: Bug > Components: CQL >Reporter: Sylvain Lebresne >Assignee: Patrick Bannister >Priority: Major > Fix For: 4.0 > > Attachments: 13047-3.11.txt > > > Cqlsh still points to the "old" textile CQL doc, but that's not really > maintain anymore now that we have the new doc (which include everything the > old doc had and more). We should update cqlsh to point to the new doc so we > can remove the old one completely. > Any takers? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13047) Point cqlsh help to the new doc
[ https://issues.apache.org/jira/browse/CASSANDRA-13047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441752#comment-16441752 ] Patrick Bannister commented on CASSANDRA-13047: --- I would be happy to revise my work after 13907, I think it would provide much better documentation! > Point cqlsh help to the new doc > --- > > Key: CASSANDRA-13047 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13047 > Project: Cassandra > Issue Type: Bug > Components: CQL >Reporter: Sylvain Lebresne >Assignee: Patrick Bannister >Priority: Major > Fix For: 4.0 > > Attachments: 13047-3.11.txt > > > Cqlsh still points to the "old" textile CQL doc, but that's not really > maintain anymore now that we have the new doc (which include everything the > old doc had and more). We should update cqlsh to point to the new doc so we > can remove the old one completely. > Any takers? -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14394) Streaming fails with AssertionError
[ https://issues.apache.org/jira/browse/CASSANDRA-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441707#comment-16441707 ] Paulo Motta edited comment on CASSANDRA-14394 at 4/18/18 12:40 AM: --- bq. Ah, looks like 13938. Now I feel less bad about breaking something in what was meant to be a minor refactor. Encourage either of you to commit the dtest with/without proving it's a dupe. The funny thing is that during bissect I noticed that this starts failing consistently after CASSANDRA-14260, but after investigating I haven't found anything there that broken this, so I think this is some timing/synchronization bug that became more likely after 14260. bq. I think it might be a dupe of CASSANDRA-13938, introduced by CASANDRA-12229 Good call, I did a quick search on google of " stream can only read forward" and found only CASSANDRA-8314 so I thought it was a new issue, next time I will use the JIRA search bar. :) I will close this as dupe of CASSANDRA-13938 for now. Feel free to reopen if this is not the case and to include the dtest on that Thanks for the quick feedback! was (Author: pauloricardomg): bq. Ah, looks like 13938. Now I feel less bad about breaking something in what was meant to be a minor refactor. Encourage either of you to commit the dtest with/without proving it's a dupe. The funny thing is that during bissect I noticed that this starts failing more often after CASSANDRA-14260, but after investigating I haven't found anything there that broken this, so I think this is some timing/synchronization bug which became more likely after 14260. bq. I think it might be a dupe of CASSANDRA-13938, introduced by CASANDRA-12229 Good call, I did a quick search on google of " stream can only read forward" and found only CASSANDRA-8314 so I thought it was a new issue, next time I will use the JIRA search bar. :) I will close this as dupe of CASSANDRA-13938 for now. Feel free to reopen if this is not the case and to include the dtest on that Thanks for the quick feedback! > Streaming fails with AssertionError > --- > > Key: CASSANDRA-14394 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14394 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Paulo Motta >Priority: Major > Fix For: 4.x > > > The bootstrap from the attached dtest fails with the following error on trunk: > {code:none} > ERROR [Stream-Deserializer-127.0.0.1:38369-af884494] 2018-04-17 12:36:53,420 > StreamingInboundHandler.java:210 - [Stream channel: af884494] stream > operation from 127.0.0.1:38369 failedjava.lang.AssertionError: stream can > only read forward. at > org.apache.cassandra.db.streaming.CompressedInputStream.position(CompressedInputStream.java:108) > at > org.apache.cassandra.db.streaming.CompressedCassandraStreamReader.read(CompressedCassandraStreamReader.java:93) > at > org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:63) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:49) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36) > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) > at > org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:177) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14394) Streaming fails with AssertionError
[ https://issues.apache.org/jira/browse/CASSANDRA-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441707#comment-16441707 ] Paulo Motta edited comment on CASSANDRA-14394 at 4/18/18 12:37 AM: --- bq. Ah, looks like 13938. Now I feel less bad about breaking something in what was meant to be a minor refactor. Encourage either of you to commit the dtest with/without proving it's a dupe. The funny thing is that during bissect I noticed that this starts failing more often after CASSANDRA-14260, but after investigating I haven't found anything there that broken this, so I think this is some timing/synchronization bug which became more likely after 14260. bq. I think it might be a dupe of CASSANDRA-13938, introduced by CASANDRA-12229 Good call, I did a quick search on google of " stream can only read forward" and found only CASSANDRA-8314 so I thought it was a new issue, next time I will use the JIRA search bar. :) I will close this as dupe of CASSANDRA-13938 for now. Feel free to reopen if this is not the case and to include the dtest on that Thanks for the quick feedback! was (Author: pauloricardomg): bq. Ah, looks like 13938. Now I feel less bad about breaking something in what was meant to be a minor refactor. Encourage either of you to commit the dtest with/without proving it's a dupe. The funny thing is that during bissect I noticed that this starts failing more often after CASSANDRA-14260, but after investigating I haven't found anything there that broken this, so I think this is some timing/synchronization bug which became more likely after 14260. bq. I think it might be a dupe of CASSANDRA-13938, introduced by CASANDRA-12229 Good call, I did a quick search on google of " stream can only read forward" and found only CASSANDRA-8314 so I thought it was a new issue, next time I will use the JIRA search bar. :) I will close this as dupe of CASSANDRA-13938 for now. Feel free to reopen if this is not the case and to include the dtest on that Thanks for the quick feedback! > Streaming fails with AssertionError > --- > > Key: CASSANDRA-14394 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14394 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Paulo Motta >Priority: Major > Fix For: 4.x > > > The bootstrap from the attached dtest fails with the following error on trunk: > {code:none} > ERROR [Stream-Deserializer-127.0.0.1:38369-af884494] 2018-04-17 12:36:53,420 > StreamingInboundHandler.java:210 - [Stream channel: af884494] stream > operation from 127.0.0.1:38369 failedjava.lang.AssertionError: stream can > only read forward. at > org.apache.cassandra.db.streaming.CompressedInputStream.position(CompressedInputStream.java:108) > at > org.apache.cassandra.db.streaming.CompressedCassandraStreamReader.read(CompressedCassandraStreamReader.java:93) > at > org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:63) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:49) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36) > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) > at > org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:177) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Resolved] (CASSANDRA-14394) Streaming fails with AssertionError
[ https://issues.apache.org/jira/browse/CASSANDRA-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paulo Motta resolved CASSANDRA-14394. - Resolution: Duplicate bq. Ah, looks like 13938. Now I feel less bad about breaking something in what was meant to be a minor refactor. Encourage either of you to commit the dtest with/without proving it's a dupe. The funny thing is that during bissect I noticed that this starts failing more often after CASSANDRA-14260, but after investigating I haven't found anything there that broken this, so I think this is some timing/synchronization bug which became more likely after 14260. bq. I think it might be a dupe of CASSANDRA-13938, introduced by CASANDRA-12229 Good call, I did a quick search on google of " stream can only read forward" and found only CASSANDRA-8314 so I thought it was a new issue, next time I will use the JIRA search bar. :) I will close this as dupe of CASSANDRA-13938 for now. Feel free to reopen if this is not the case and to include the dtest on that Thanks for the quick feedback! > Streaming fails with AssertionError > --- > > Key: CASSANDRA-14394 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14394 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Paulo Motta >Priority: Major > Fix For: 4.x > > > The bootstrap from the attached dtest fails with the following error on trunk: > {code:none} > ERROR [Stream-Deserializer-127.0.0.1:38369-af884494] 2018-04-17 12:36:53,420 > StreamingInboundHandler.java:210 - [Stream channel: af884494] stream > operation from 127.0.0.1:38369 failedjava.lang.AssertionError: stream can > only read forward. at > org.apache.cassandra.db.streaming.CompressedInputStream.position(CompressedInputStream.java:108) > at > org.apache.cassandra.db.streaming.CompressedCassandraStreamReader.read(CompressedCassandraStreamReader.java:93) > at > org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:63) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:49) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36) > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) > at > org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:177) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13851) Allow existing nodes to use all peers in shadow round
[ https://issues.apache.org/jira/browse/CASSANDRA-13851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441687#comment-16441687 ] Kurt Greaves commented on CASSANDRA-13851: -- Thanks heaps [~beobal]! > Allow existing nodes to use all peers in shadow round > - > > Key: CASSANDRA-13851 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13851 > Project: Cassandra > Issue Type: Bug > Components: Lifecycle >Reporter: Kurt Greaves >Assignee: Kurt Greaves >Priority: Major > Fix For: 4.0, 3.11.3 > > > In CASSANDRA-10134 we made collision checks necessary on every startup. A > side-effect was introduced that then requires a nodes seeds to be contacted > on every startup. Prior to this change an existing node could start up > regardless whether it could contact a seed node or not (because > checkForEndpointCollision() was only called for bootstrapping nodes). > Now if a nodes seeds are removed/deleted/fail it will no longer be able to > start up until live seeds are configured (or itself is made a seed), even > though it already knows about the rest of the ring. This is inconvenient for > operators and has the potential to cause some nasty surprises and increase > downtime. > One solution would be to use all a nodes existing peers as seeds in the > shadow round. Not a Gossip guru though so not sure of implications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14395) C* Management process
Dinesh Joshi created CASSANDRA-14395: Summary: C* Management process Key: CASSANDRA-14395 URL: https://issues.apache.org/jira/browse/CASSANDRA-14395 Project: Cassandra Issue Type: New Feature Reporter: Dinesh Joshi Assignee: Dinesh Joshi I would like to propose amending Cassandra's architecture to include a management process. The detailed description is here: https://docs.google.com/document/d/1UV9pE81NaIUF3g4L1wxq09nT11AkSQcMijgLFwGsY3s/edit I'd like to propose seeding this with a few simple use-cases such as Health Checks, Bulk Commands with a simple REST API interface. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14394) Streaming fails with AssertionError
[ https://issues.apache.org/jira/browse/CASSANDRA-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441561#comment-16441561 ] Jason Brown edited comment on CASSANDRA-14394 at 4/17/18 9:59 PM: -- I can't get to fixing this issue this week, but I can take a at the dtest and compare it with the instructions [~zznate] added on CASSANDRA-13938 - to confirm if the tickets are indeed the same problem was (Author: jasobrown): I can't get to fixing this issue this week, but I can take a at the dtest and compare it with the instructions [~zznate] added on CASSANDRA-13938. > Streaming fails with AssertionError > --- > > Key: CASSANDRA-14394 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14394 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Paulo Motta >Priority: Major > Fix For: 4.x > > > The bootstrap from the attached dtest fails with the following error on trunk: > {code:none} > ERROR [Stream-Deserializer-127.0.0.1:38369-af884494] 2018-04-17 12:36:53,420 > StreamingInboundHandler.java:210 - [Stream channel: af884494] stream > operation from 127.0.0.1:38369 failedjava.lang.AssertionError: stream can > only read forward. at > org.apache.cassandra.db.streaming.CompressedInputStream.position(CompressedInputStream.java:108) > at > org.apache.cassandra.db.streaming.CompressedCassandraStreamReader.read(CompressedCassandraStreamReader.java:93) > at > org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:63) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:49) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36) > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) > at > org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:177) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14394) Streaming fails with AssertionError
[ https://issues.apache.org/jira/browse/CASSANDRA-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441561#comment-16441561 ] Jason Brown commented on CASSANDRA-14394: - I can't get to fixing this issue this week, but I can take a at the dtest and compare it with the instructions [~zznate] added on CASSANDRA-13938. > Streaming fails with AssertionError > --- > > Key: CASSANDRA-14394 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14394 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Paulo Motta >Priority: Major > Fix For: 4.x > > > The bootstrap from the attached dtest fails with the following error on trunk: > {code:none} > ERROR [Stream-Deserializer-127.0.0.1:38369-af884494] 2018-04-17 12:36:53,420 > StreamingInboundHandler.java:210 - [Stream channel: af884494] stream > operation from 127.0.0.1:38369 failedjava.lang.AssertionError: stream can > only read forward. at > org.apache.cassandra.db.streaming.CompressedInputStream.position(CompressedInputStream.java:108) > at > org.apache.cassandra.db.streaming.CompressedCassandraStreamReader.read(CompressedCassandraStreamReader.java:93) > at > org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:63) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:49) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36) > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) > at > org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:177) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14394) Streaming fails with AssertionError
[ https://issues.apache.org/jira/browse/CASSANDRA-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-14394: --- Fix Version/s: 4.x > Streaming fails with AssertionError > --- > > Key: CASSANDRA-14394 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14394 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Paulo Motta >Priority: Major > Fix For: 4.x > > > The bootstrap from the attached dtest fails with the following error on trunk: > {code:none} > ERROR [Stream-Deserializer-127.0.0.1:38369-af884494] 2018-04-17 12:36:53,420 > StreamingInboundHandler.java:210 - [Stream channel: af884494] stream > operation from 127.0.0.1:38369 failedjava.lang.AssertionError: stream can > only read forward. at > org.apache.cassandra.db.streaming.CompressedInputStream.position(CompressedInputStream.java:108) > at > org.apache.cassandra.db.streaming.CompressedCassandraStreamReader.read(CompressedCassandraStreamReader.java:93) > at > org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:63) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:49) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36) > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) > at > org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:177) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14394) Streaming fails with AssertionError
[ https://issues.apache.org/jira/browse/CASSANDRA-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-14394: --- Component/s: Streaming and Messaging > Streaming fails with AssertionError > --- > > Key: CASSANDRA-14394 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14394 > Project: Cassandra > Issue Type: Bug > Components: Streaming and Messaging >Reporter: Paulo Motta >Priority: Major > Fix For: 4.x > > > The bootstrap from the attached dtest fails with the following error on trunk: > {code:none} > ERROR [Stream-Deserializer-127.0.0.1:38369-af884494] 2018-04-17 12:36:53,420 > StreamingInboundHandler.java:210 - [Stream channel: af884494] stream > operation from 127.0.0.1:38369 failedjava.lang.AssertionError: stream can > only read forward. at > org.apache.cassandra.db.streaming.CompressedInputStream.position(CompressedInputStream.java:108) > at > org.apache.cassandra.db.streaming.CompressedCassandraStreamReader.read(CompressedCassandraStreamReader.java:93) > at > org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:63) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:49) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36) > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) > at > org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:177) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13896) Improving Cassandra write performance
[ https://issues.apache.org/jira/browse/CASSANDRA-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441532#comment-16441532 ] Ariel Weisberg commented on CASSANDRA-13896: The ticket was filed against 3.10 so I wonder if there is a performance difference here. Also are we comparing a cluster to a single node to determine scaling issues or Michael did you also test with a single node? > Improving Cassandra write performance > --- > > Key: CASSANDRA-13896 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13896 > Project: Cassandra > Issue Type: Improvement > Components: Local Write-Read Paths > Environment: Skylake server with 2 sockets, 192GB RAM, 3xPCIe SSDs > OS: Centos 7.3 > Java: Oracle JDK1.8.0_121 >Reporter: Prince Nana Owusu Boateng >Priority: Major > Labels: Performance > Fix For: 4.x > > Attachments: Screen Shot 2017-09-22 at 11.22.43 AM.png, Screen Shot > 2017-09-22 at 3.31.09 PM.png > > > During our Cassandra performance testing, we see high percentage of the CPU > spent in *org.apache.cassandra.utils.memory.SlabAllocator.allocate(int, > OpOrder Group) * method. Appears to be high contention of the > ** atomic Integer in write workloads. This structure is > used by the threads for keeping track of the region bytebuffer allocation. > When the contention appears, adding more clients, modifications of write > specific parameters does not change write throughput performance. Attached > are the details of Java Flight Recorder (JFR), showing hot functions and also > performance results. When we see this contention, we still have plenty of > CPU and throughput left ( *<20%* Total average CPU utilization and *<11%* > of the storage write total throughput). This occurs on Cassandra 3.10.0-src > version using the Cassandra-Stress. > Proposal: > We will like to introduce a solution which eliminates the atomic operations > on the ** atomic Integer. This implementation will allow > concurrent allocation of bytebuffers without an atomic compareAndSet and > incrementAndGet operations. The solution is expected to increase overall > write performance while improving CPU utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14394) Streaming fails with AssertionError
[ https://issues.apache.org/jira/browse/CASSANDRA-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa reassigned CASSANDRA-14394: -- Assignee: (was: Jeff Jirsa) > Streaming fails with AssertionError > --- > > Key: CASSANDRA-14394 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14394 > Project: Cassandra > Issue Type: Bug >Reporter: Paulo Motta >Priority: Major > > The bootstrap from the attached dtest fails with the following error on trunk: > {code:none} > ERROR [Stream-Deserializer-127.0.0.1:38369-af884494] 2018-04-17 12:36:53,420 > StreamingInboundHandler.java:210 - [Stream channel: af884494] stream > operation from 127.0.0.1:38369 failedjava.lang.AssertionError: stream can > only read forward. at > org.apache.cassandra.db.streaming.CompressedInputStream.position(CompressedInputStream.java:108) > at > org.apache.cassandra.db.streaming.CompressedCassandraStreamReader.read(CompressedCassandraStreamReader.java:93) > at > org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:63) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:49) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36) > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) > at > org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:177) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14394) Streaming fails with AssertionError
[ https://issues.apache.org/jira/browse/CASSANDRA-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441533#comment-16441533 ] Jeff Jirsa commented on CASSANDRA-14394: Ah, looks like 13938. Now I feel less bad about breaking something in what was meant to be a minor refactor. Encourage either of you to commit the dtest with/without proving it's a dupe. > Streaming fails with AssertionError > --- > > Key: CASSANDRA-14394 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14394 > Project: Cassandra > Issue Type: Bug >Reporter: Paulo Motta >Assignee: Jeff Jirsa >Priority: Major > > The bootstrap from the attached dtest fails with the following error on trunk: > {code:none} > ERROR [Stream-Deserializer-127.0.0.1:38369-af884494] 2018-04-17 12:36:53,420 > StreamingInboundHandler.java:210 - [Stream channel: af884494] stream > operation from 127.0.0.1:38369 failedjava.lang.AssertionError: stream can > only read forward. at > org.apache.cassandra.db.streaming.CompressedInputStream.position(CompressedInputStream.java:108) > at > org.apache.cassandra.db.streaming.CompressedCassandraStreamReader.read(CompressedCassandraStreamReader.java:93) > at > org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:63) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:49) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36) > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) > at > org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:177) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14394) Streaming fails with AssertionError
[ https://issues.apache.org/jira/browse/CASSANDRA-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441525#comment-16441525 ] Jason Brown commented on CASSANDRA-14394: - I think it might be a dupe of CASSANDRA-13938, introduced by CASANDRA-12229 > Streaming fails with AssertionError > --- > > Key: CASSANDRA-14394 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14394 > Project: Cassandra > Issue Type: Bug >Reporter: Paulo Motta >Assignee: Jeff Jirsa >Priority: Major > > The bootstrap from the attached dtest fails with the following error on trunk: > {code:none} > ERROR [Stream-Deserializer-127.0.0.1:38369-af884494] 2018-04-17 12:36:53,420 > StreamingInboundHandler.java:210 - [Stream channel: af884494] stream > operation from 127.0.0.1:38369 failedjava.lang.AssertionError: stream can > only read forward. at > org.apache.cassandra.db.streaming.CompressedInputStream.position(CompressedInputStream.java:108) > at > org.apache.cassandra.db.streaming.CompressedCassandraStreamReader.read(CompressedCassandraStreamReader.java:93) > at > org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:63) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:49) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36) > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) > at > org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:177) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14332) Fix unbounded validation compactions on repair
[ https://issues.apache.org/jira/browse/CASSANDRA-14332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Eggleston updated CASSANDRA-14332: Resolution: Fixed Fix Version/s: 3.11.3 3.0.17 Status: Resolved (was: Patch Available) committed as {{00e5a3d508eb41944ce01c6cc96ae18cb16dad8c}} thanks! > Fix unbounded validation compactions on repair > -- > > Key: CASSANDRA-14332 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14332 > Project: Cassandra > Issue Type: Bug >Reporter: Kurt Greaves >Assignee: Kurt Greaves >Priority: Major > Fix For: 3.0.17, 3.11.3 > > > After CASSANDRA-13797 it's possible to cause unbounded, simultaneous > validation compactions as we no longer wait for validations to finish. > Potential fix is to have a sane default for the # of concurrent validation > compactions by backporting CASSANDRA-13521 and setting a sane default. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk
Merge branch 'cassandra-3.11' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/703db6ff Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/703db6ff Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/703db6ff Branch: refs/heads/trunk Commit: 703db6fff5fdf30f98e61d175a2b268846f7b692 Parents: f165e72 e9462b9 Author: Blake Eggleston Authored: Tue Apr 17 14:18:59 2018 -0700 Committer: Blake Eggleston Committed: Tue Apr 17 14:19:29 2018 -0700 -- CHANGES.txt | 1 + 1 file changed, 1 insertion(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/703db6ff/CHANGES.txt -- - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[1/6] cassandra git commit: Fix unbounded validation compactions on repair / revert CASSANDRA-13797
Repository: cassandra Updated Branches: refs/heads/cassandra-3.0 598008d43 -> 00e5a3d50 refs/heads/cassandra-3.11 28ccf3fe3 -> e9462b9af refs/heads/trunk f165e72bf -> 703db6fff Fix unbounded validation compactions on repair / revert CASSANDRA-13797 This reverts commit e7299c08f940057e8fd4dfa3f24dcc6e0cb5f78d. Patch by Kurt Greaves; Reviewed by Blake Eggleston for CASSANDRA-14332 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/00e5a3d5 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/00e5a3d5 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/00e5a3d5 Branch: refs/heads/cassandra-3.0 Commit: 00e5a3d508eb41944ce01c6cc96ae18cb16dad8c Parents: 598008d Author: kurt Authored: Tue Feb 27 05:24:25 2018 + Committer: Blake Eggleston Committed: Tue Apr 17 14:04:09 2018 -0700 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/repair/RepairJob.java | 3 +++ 2 files changed, 4 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/00e5a3d5/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index d3d8036..1aa291f 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.17 + * Fix unbounded validation compactions on repair / revert CASSANDRA-13797 (CASSANDRA-14332) * Avoid deadlock when running nodetool refresh before node is fully up (CASSANDRA-14310) * Handle all exceptions when opening sstables (CASSANDRA-14202) * Handle incompletely written hint descriptors during startup (CASSANDRA-14080) http://git-wip-us.apache.org/repos/asf/cassandra/blob/00e5a3d5/src/java/org/apache/cassandra/repair/RepairJob.java -- diff --git a/src/java/org/apache/cassandra/repair/RepairJob.java b/src/java/org/apache/cassandra/repair/RepairJob.java index 0711a64..cba176c 100644 --- a/src/java/org/apache/cassandra/repair/RepairJob.java +++ b/src/java/org/apache/cassandra/repair/RepairJob.java @@ -155,6 +155,9 @@ public class RepairJob extends AbstractFuture implements Runnable setException(t); } }, taskExecutor); + +// Wait for validation to complete +Futures.getUnchecked(validations); } /** - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[3/6] cassandra git commit: Fix unbounded validation compactions on repair / revert CASSANDRA-13797
Fix unbounded validation compactions on repair / revert CASSANDRA-13797 This reverts commit e7299c08f940057e8fd4dfa3f24dcc6e0cb5f78d. Patch by Kurt Greaves; Reviewed by Blake Eggleston for CASSANDRA-14332 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/00e5a3d5 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/00e5a3d5 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/00e5a3d5 Branch: refs/heads/trunk Commit: 00e5a3d508eb41944ce01c6cc96ae18cb16dad8c Parents: 598008d Author: kurt Authored: Tue Feb 27 05:24:25 2018 + Committer: Blake Eggleston Committed: Tue Apr 17 14:04:09 2018 -0700 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/repair/RepairJob.java | 3 +++ 2 files changed, 4 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/00e5a3d5/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index d3d8036..1aa291f 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.17 + * Fix unbounded validation compactions on repair / revert CASSANDRA-13797 (CASSANDRA-14332) * Avoid deadlock when running nodetool refresh before node is fully up (CASSANDRA-14310) * Handle all exceptions when opening sstables (CASSANDRA-14202) * Handle incompletely written hint descriptors during startup (CASSANDRA-14080) http://git-wip-us.apache.org/repos/asf/cassandra/blob/00e5a3d5/src/java/org/apache/cassandra/repair/RepairJob.java -- diff --git a/src/java/org/apache/cassandra/repair/RepairJob.java b/src/java/org/apache/cassandra/repair/RepairJob.java index 0711a64..cba176c 100644 --- a/src/java/org/apache/cassandra/repair/RepairJob.java +++ b/src/java/org/apache/cassandra/repair/RepairJob.java @@ -155,6 +155,9 @@ public class RepairJob extends AbstractFuture implements Runnable setException(t); } }, taskExecutor); + +// Wait for validation to complete +Futures.getUnchecked(validations); } /** - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e9462b9a Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e9462b9a Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e9462b9a Branch: refs/heads/trunk Commit: e9462b9af7c85bc5e1de6bc1662768da71c1061b Parents: 28ccf3f 00e5a3d Author: Blake Eggleston Authored: Tue Apr 17 14:15:12 2018 -0700 Committer: Blake Eggleston Committed: Tue Apr 17 14:15:12 2018 -0700 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/repair/RepairJob.java | 3 +++ 2 files changed, 4 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/e9462b9a/CHANGES.txt -- diff --cc CHANGES.txt index 39213a1,1aa291f..7253ec6 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,15 -1,5 +1,16 @@@ -3.0.17 +3.11.3 + * Allow existing nodes to use all peers in shadow round (CASSANDRA-13851) + * Fix cqlsh to read connection.ssl cqlshrc option again (CASSANDRA-14299) + * Downgrade log level to trace for CommitLogSegmentManager (CASSANDRA-14370) + * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891) + * Serialize empty buffer as empty string for json output format (CASSANDRA-14245) + * Allow logging implementation to be interchanged for embedded testing (CASSANDRA-13396) + * SASI tokenizer for simple delimiter based entries (CASSANDRA-14247) + * Fix Loss of digits when doing CAST from varint/bigint to decimal (CASSANDRA-14170) + * RateBasedBackPressure unnecessarily invokes a lock on the Guava RateLimiter (CASSANDRA-14163) + * Fix wildcard GROUP BY queries (CASSANDRA-14209) +Merged from 3.0: + * Fix unbounded validation compactions on repair / revert CASSANDRA-13797 (CASSANDRA-14332) * Avoid deadlock when running nodetool refresh before node is fully up (CASSANDRA-14310) * Handle all exceptions when opening sstables (CASSANDRA-14202) * Handle incompletely written hint descriptors during startup (CASSANDRA-14080) http://git-wip-us.apache.org/repos/asf/cassandra/blob/e9462b9a/src/java/org/apache/cassandra/repair/RepairJob.java -- - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11
Merge branch 'cassandra-3.0' into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/e9462b9a Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/e9462b9a Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/e9462b9a Branch: refs/heads/cassandra-3.11 Commit: e9462b9af7c85bc5e1de6bc1662768da71c1061b Parents: 28ccf3f 00e5a3d Author: Blake Eggleston Authored: Tue Apr 17 14:15:12 2018 -0700 Committer: Blake Eggleston Committed: Tue Apr 17 14:15:12 2018 -0700 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/repair/RepairJob.java | 3 +++ 2 files changed, 4 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/e9462b9a/CHANGES.txt -- diff --cc CHANGES.txt index 39213a1,1aa291f..7253ec6 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,15 -1,5 +1,16 @@@ -3.0.17 +3.11.3 + * Allow existing nodes to use all peers in shadow round (CASSANDRA-13851) + * Fix cqlsh to read connection.ssl cqlshrc option again (CASSANDRA-14299) + * Downgrade log level to trace for CommitLogSegmentManager (CASSANDRA-14370) + * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891) + * Serialize empty buffer as empty string for json output format (CASSANDRA-14245) + * Allow logging implementation to be interchanged for embedded testing (CASSANDRA-13396) + * SASI tokenizer for simple delimiter based entries (CASSANDRA-14247) + * Fix Loss of digits when doing CAST from varint/bigint to decimal (CASSANDRA-14170) + * RateBasedBackPressure unnecessarily invokes a lock on the Guava RateLimiter (CASSANDRA-14163) + * Fix wildcard GROUP BY queries (CASSANDRA-14209) +Merged from 3.0: + * Fix unbounded validation compactions on repair / revert CASSANDRA-13797 (CASSANDRA-14332) * Avoid deadlock when running nodetool refresh before node is fully up (CASSANDRA-14310) * Handle all exceptions when opening sstables (CASSANDRA-14202) * Handle incompletely written hint descriptors during startup (CASSANDRA-14080) http://git-wip-us.apache.org/repos/asf/cassandra/blob/e9462b9a/src/java/org/apache/cassandra/repair/RepairJob.java -- - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[2/6] cassandra git commit: Fix unbounded validation compactions on repair / revert CASSANDRA-13797
Fix unbounded validation compactions on repair / revert CASSANDRA-13797 This reverts commit e7299c08f940057e8fd4dfa3f24dcc6e0cb5f78d. Patch by Kurt Greaves; Reviewed by Blake Eggleston for CASSANDRA-14332 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/00e5a3d5 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/00e5a3d5 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/00e5a3d5 Branch: refs/heads/cassandra-3.11 Commit: 00e5a3d508eb41944ce01c6cc96ae18cb16dad8c Parents: 598008d Author: kurt Authored: Tue Feb 27 05:24:25 2018 + Committer: Blake Eggleston Committed: Tue Apr 17 14:04:09 2018 -0700 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/repair/RepairJob.java | 3 +++ 2 files changed, 4 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/00e5a3d5/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index d3d8036..1aa291f 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.0.17 + * Fix unbounded validation compactions on repair / revert CASSANDRA-13797 (CASSANDRA-14332) * Avoid deadlock when running nodetool refresh before node is fully up (CASSANDRA-14310) * Handle all exceptions when opening sstables (CASSANDRA-14202) * Handle incompletely written hint descriptors during startup (CASSANDRA-14080) http://git-wip-us.apache.org/repos/asf/cassandra/blob/00e5a3d5/src/java/org/apache/cassandra/repair/RepairJob.java -- diff --git a/src/java/org/apache/cassandra/repair/RepairJob.java b/src/java/org/apache/cassandra/repair/RepairJob.java index 0711a64..cba176c 100644 --- a/src/java/org/apache/cassandra/repair/RepairJob.java +++ b/src/java/org/apache/cassandra/repair/RepairJob.java @@ -155,6 +155,9 @@ public class RepairJob extends AbstractFuture implements Runnable setException(t); } }, taskExecutor); + +// Wait for validation to complete +Futures.getUnchecked(validations); } /** - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14394) Streaming fails with AssertionError
[ https://issues.apache.org/jira/browse/CASSANDRA-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa reassigned CASSANDRA-14394: -- Assignee: Jeff Jirsa > Streaming fails with AssertionError > --- > > Key: CASSANDRA-14394 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14394 > Project: Cassandra > Issue Type: Bug >Reporter: Paulo Motta >Assignee: Jeff Jirsa >Priority: Major > > The bootstrap from the attached dtest fails with the following error on trunk: > {code:none} > ERROR [Stream-Deserializer-127.0.0.1:38369-af884494] 2018-04-17 12:36:53,420 > StreamingInboundHandler.java:210 - [Stream channel: af884494] stream > operation from 127.0.0.1:38369 failedjava.lang.AssertionError: stream can > only read forward. at > org.apache.cassandra.db.streaming.CompressedInputStream.position(CompressedInputStream.java:108) > at > org.apache.cassandra.db.streaming.CompressedCassandraStreamReader.read(CompressedCassandraStreamReader.java:93) > at > org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:63) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:49) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36) > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) > at > org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:177) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14394) Streaming fails with AssertionError
[ https://issues.apache.org/jira/browse/CASSANDRA-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441506#comment-16441506 ] Jeff Jirsa commented on CASSANDRA-14394: Assuming without evidence that this is probably a bug from CASSANDRA-14260 - I'll check it (though I'm really surprised we didn't have another test that caught it) > Streaming fails with AssertionError > --- > > Key: CASSANDRA-14394 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14394 > Project: Cassandra > Issue Type: Bug >Reporter: Paulo Motta >Priority: Major > > The bootstrap from the attached dtest fails with the following error on trunk: > {code:none} > ERROR [Stream-Deserializer-127.0.0.1:38369-af884494] 2018-04-17 12:36:53,420 > StreamingInboundHandler.java:210 - [Stream channel: af884494] stream > operation from 127.0.0.1:38369 failedjava.lang.AssertionError: stream can > only read forward. at > org.apache.cassandra.db.streaming.CompressedInputStream.position(CompressedInputStream.java:108) > at > org.apache.cassandra.db.streaming.CompressedCassandraStreamReader.read(CompressedCassandraStreamReader.java:93) > at > org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:63) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:49) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36) > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) > at > org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:177) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13910) Remove read_repair_chance/dclocal_read_repair_chance
[ https://issues.apache.org/jira/browse/CASSANDRA-13910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441403#comment-16441403 ] Blake Eggleston commented on CASSANDRA-13910: - +1, although please remove the unused ClientWarn imports in AlterTableStatement, AlterViewStatement, and CreateTableStatement on commit > Remove read_repair_chance/dclocal_read_repair_chance > > > Key: CASSANDRA-13910 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13910 > Project: Cassandra > Issue Type: Improvement >Reporter: Sylvain Lebresne >Assignee: Aleksey Yeschenko >Priority: Minor > Fix For: 4.0 > > > First, let me clarify so this is not misunderstood that I'm not *at all* > suggesting to remove the read-repair mechanism of detecting and repairing > inconsistencies between read responses: that mechanism is imo fine and > useful. But the {{read_repair_chance}} and {{dclocal_read_repair_chance}} > have never been about _enabling_ that mechanism, they are about querying all > replicas (even when this is not required by the consistency level) for the > sole purpose of maybe read-repairing some of the replica that wouldn't have > been queried otherwise. Which btw, bring me to reason 1 for considering their > removal: their naming/behavior is super confusing. Over the years, I've seen > countless users (and not only newbies) misunderstanding what those options > do, and as a consequence misunderstand when read-repair itself was happening. > But my 2nd reason for suggesting this is that I suspect > {{read_repair_chance}}/{{dclocal_read_repair_chance}} are, especially > nowadays, more harmful than anything else when enabled. When those option > kick in, what you trade-off is additional resources consumption (all nodes > have to execute the read) for a _fairly remote chance_ of having some > inconsistencies repaired on _some_ replica _a bit faster_ than they would > otherwise be. To justify that last part, let's recall that: > # most inconsistencies are actually fixed by hints in practice; and in the > case where a node stay dead for a long time so that hints ends up timing-out, > you really should repair the node when it comes back (if not simply > re-bootstrapping it). Read-repair probably don't fix _that_ much stuff in > the first place. > # again, read-repair do happen without those options kicking in. If you do > reads at {{QUORUM}}, inconsistencies will eventually get read-repaired all > the same. Just a tiny bit less quickly. > # I suspect almost everyone use a low "chance" for those options at best > (because the extra resources consumption is real), so at the end of the day, > it's up to chance how much faster this fixes inconsistencies. > Overall, I'm having a hard time imagining real cases where that trade-off > really make sense. Don't get me wrong, those options had their places a long > time ago when hints weren't working all that well, but I think they bring > more confusion than benefits now. > And I think it's sane to reconsider stuffs every once in a while, and to > clean up anything that may not make all that much sense anymore, which I > think is the case here. > Tl;dr, I feel the benefits brought by those options are very slim at best and > well overshadowed by the confusion they bring, and not worth maintaining the > code that supports them (which, to be fair, isn't huge, but getting rid of > {{ReadCallback.AsyncRepairRunner}} wouldn't hurt for instance). > Lastly, if the consensus here ends up being that they can have their use in > weird case and that we fill supporting those cases is worth confusing > everyone else and maintaining that code, I would still suggest disabling them > totally by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13910) Remove read_repair_chance/dclocal_read_repair_chance
[ https://issues.apache.org/jira/browse/CASSANDRA-13910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Eggleston updated CASSANDRA-13910: Status: Ready to Commit (was: Patch Available) > Remove read_repair_chance/dclocal_read_repair_chance > > > Key: CASSANDRA-13910 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13910 > Project: Cassandra > Issue Type: Improvement >Reporter: Sylvain Lebresne >Assignee: Aleksey Yeschenko >Priority: Minor > Fix For: 4.0 > > > First, let me clarify so this is not misunderstood that I'm not *at all* > suggesting to remove the read-repair mechanism of detecting and repairing > inconsistencies between read responses: that mechanism is imo fine and > useful. But the {{read_repair_chance}} and {{dclocal_read_repair_chance}} > have never been about _enabling_ that mechanism, they are about querying all > replicas (even when this is not required by the consistency level) for the > sole purpose of maybe read-repairing some of the replica that wouldn't have > been queried otherwise. Which btw, bring me to reason 1 for considering their > removal: their naming/behavior is super confusing. Over the years, I've seen > countless users (and not only newbies) misunderstanding what those options > do, and as a consequence misunderstand when read-repair itself was happening. > But my 2nd reason for suggesting this is that I suspect > {{read_repair_chance}}/{{dclocal_read_repair_chance}} are, especially > nowadays, more harmful than anything else when enabled. When those option > kick in, what you trade-off is additional resources consumption (all nodes > have to execute the read) for a _fairly remote chance_ of having some > inconsistencies repaired on _some_ replica _a bit faster_ than they would > otherwise be. To justify that last part, let's recall that: > # most inconsistencies are actually fixed by hints in practice; and in the > case where a node stay dead for a long time so that hints ends up timing-out, > you really should repair the node when it comes back (if not simply > re-bootstrapping it). Read-repair probably don't fix _that_ much stuff in > the first place. > # again, read-repair do happen without those options kicking in. If you do > reads at {{QUORUM}}, inconsistencies will eventually get read-repaired all > the same. Just a tiny bit less quickly. > # I suspect almost everyone use a low "chance" for those options at best > (because the extra resources consumption is real), so at the end of the day, > it's up to chance how much faster this fixes inconsistencies. > Overall, I'm having a hard time imagining real cases where that trade-off > really make sense. Don't get me wrong, those options had their places a long > time ago when hints weren't working all that well, but I think they bring > more confusion than benefits now. > And I think it's sane to reconsider stuffs every once in a while, and to > clean up anything that may not make all that much sense anymore, which I > think is the case here. > Tl;dr, I feel the benefits brought by those options are very slim at best and > well overshadowed by the confusion they bring, and not worth maintaining the > code that supports them (which, to be fair, isn't huge, but getting rid of > {{ReadCallback.AsyncRepairRunner}} wouldn't hurt for instance). > Lastly, if the consensus here ends up being that they can have their use in > weird case and that we fill supporting those cases is worth confusing > everyone else and maintaining that code, I would still suggest disabling them > totally by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14394) Streaming fails with AssertionError
Paulo Motta created CASSANDRA-14394: --- Summary: Streaming fails with AssertionError Key: CASSANDRA-14394 URL: https://issues.apache.org/jira/browse/CASSANDRA-14394 Project: Cassandra Issue Type: Bug Reporter: Paulo Motta The bootstrap from the attached dtest fails with the following error on trunk: {noformat} ERROR [Stream-Deserializer-127.0.0.1:38369-af884494] 2018-04-17 12:36:53,420 StreamingInboundHandler.java:210 - [Stream channel: af884494] stream operation from 127.0.0.1:38369 failedjava.lang.AssertionError: stream can only read forward. at org.apache.cassandra.db.streaming.CompressedInputStream.position(CompressedInputStream.java:108) at org.apache.cassandra.db.streaming.CompressedCassandraStreamReader.read(CompressedCassandraStreamReader.java:93) at org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:63) at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:49) at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36) at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) at org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:177) at java.lang.Thread.run(Thread.java:748) {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14394) Streaming fails with AssertionError
[ https://issues.apache.org/jira/browse/CASSANDRA-14394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paulo Motta updated CASSANDRA-14394: Description: The bootstrap from the attached dtest fails with the following error on trunk: {code:none} ERROR [Stream-Deserializer-127.0.0.1:38369-af884494] 2018-04-17 12:36:53,420 StreamingInboundHandler.java:210 - [Stream channel: af884494] stream operation from 127.0.0.1:38369 failedjava.lang.AssertionError: stream can only read forward. at org.apache.cassandra.db.streaming.CompressedInputStream.position(CompressedInputStream.java:108) at org.apache.cassandra.db.streaming.CompressedCassandraStreamReader.read(CompressedCassandraStreamReader.java:93) at org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:63) at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:49) at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36) at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) at org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:177) at java.lang.Thread.run(Thread.java:748) {code} was: The bootstrap from the attached dtest fails with the following error on trunk: {noformat} ERROR [Stream-Deserializer-127.0.0.1:38369-af884494] 2018-04-17 12:36:53,420 StreamingInboundHandler.java:210 - [Stream channel: af884494] stream operation from 127.0.0.1:38369 failedjava.lang.AssertionError: stream can only read forward. at org.apache.cassandra.db.streaming.CompressedInputStream.position(CompressedInputStream.java:108) at org.apache.cassandra.db.streaming.CompressedCassandraStreamReader.read(CompressedCassandraStreamReader.java:93) at org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:63) at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:49) at org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36) at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) at org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:177) at java.lang.Thread.run(Thread.java:748) {noformat} > Streaming fails with AssertionError > --- > > Key: CASSANDRA-14394 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14394 > Project: Cassandra > Issue Type: Bug >Reporter: Paulo Motta >Priority: Major > > The bootstrap from the attached dtest fails with the following error on trunk: > {code:none} > ERROR [Stream-Deserializer-127.0.0.1:38369-af884494] 2018-04-17 12:36:53,420 > StreamingInboundHandler.java:210 - [Stream channel: af884494] stream > operation from 127.0.0.1:38369 failedjava.lang.AssertionError: stream can > only read forward. at > org.apache.cassandra.db.streaming.CompressedInputStream.position(CompressedInputStream.java:108) > at > org.apache.cassandra.db.streaming.CompressedCassandraStreamReader.read(CompressedCassandraStreamReader.java:93) > at > org.apache.cassandra.db.streaming.CassandraIncomingFile.read(CassandraIncomingFile.java:63) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:49) > at > org.apache.cassandra.streaming.messages.IncomingStreamMessage$1.deserialize(IncomingStreamMessage.java:36) > at > org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:55) > at > org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:177) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14374) Speculative retry parsing breaks on non-english locale
[ https://issues.apache.org/jira/browse/CASSANDRA-14374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paulo Motta updated CASSANDRA-14374: Resolution: Fixed Status: Resolved (was: Ready to Commit) Committed as {{f165e72bf19e8d12457b8f569517012628513d24}} to trunk. Thanks for the review! > Speculative retry parsing breaks on non-english locale > -- > > Key: CASSANDRA-14374 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14374 > Project: Cassandra > Issue Type: Bug >Reporter: Paulo Motta >Assignee: Paulo Motta >Priority: Minor > Fix For: 4.0 > > Attachments: > 0001-Use-Locale.US-on-PercentileSpeculativeRetryPolicy.to.patch, > 0002-Use-Locale.US-on-PercentileSpeculativeRetryPolicy.to.patch, > 0003-Use-Locale.US-on-PercentileSpeculativeRetryPolicy.to.patch, > 0004-Use-Locale.US-on-PercentileSpeculativeRetryPolicy.to.patch > > > I was getting the following error when running unit tests on my machine: > {code:none} > Error setting schema for test (query was: CREATE TABLE > cql_test_keyspace.table_32 (a int, b int, c text, primary key (a, b))) > java.lang.RuntimeException: Error setting schema for test (query was: CREATE > TABLE cql_test_keyspace.table_32 (a int, b int, c text, primary key (a, b))) > at org.apache.cassandra.cql3.CQLTester.schemaChange(CQLTester.java:819) > at org.apache.cassandra.cql3.CQLTester.createTable(CQLTester.java:632) > at org.apache.cassandra.cql3.CQLTester.createTable(CQLTester.java:624) > at > org.apache.cassandra.cql3.validation.operations.DeleteTest.testDeleteWithNonoverlappingRange(DeleteTest.java:663) > Caused by: org.apache.cassandra.exceptions.ConfigurationException: Specified > Speculative Retry Policy [99,00p] is not supported > at > org.apache.cassandra.service.reads.SpeculativeRetryPolicy.fromString(SpeculativeRetryPolicy.java:135) > at > org.apache.cassandra.schema.SchemaKeyspace.createTableParamsFromRow(SchemaKeyspace.java:1006) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTable(SchemaKeyspace.java:981) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchTables(SchemaKeyspace.java:941) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspace(SchemaKeyspace.java:900) > at > org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspaces(SchemaKeyspace.java:1301) > at org.apache.cassandra.schema.Schema.merge(Schema.java:608) > at > org.apache.cassandra.schema.MigrationManager.announce(MigrationManager.java:425) > at > org.apache.cassandra.schema.MigrationManager.announceNewTable(MigrationManager.java:239) > at > org.apache.cassandra.schema.MigrationManager.announceNewTable(MigrationManager.java:224) > at > org.apache.cassandra.schema.MigrationManager.announceNewTable(MigrationManager.java:204) > at > org.apache.cassandra.cql3.statements.CreateTableStatement.announceMigration(CreateTableStatement.java:88) > at > org.apache.cassandra.cql3.statements.SchemaAlteringStatement.executeInternal(SchemaAlteringStatement.java:120) > at org.apache.cassandra.cql3.CQLTester.schemaChange(CQLTester.java:814) > {code} > It turns out that my machine is configured with {{pt_BR}} locale, which uses > comma instead of dot for decimal separator, so the speculative retry option > parsing introduced by CASSANDRA-14293, which assumed {{en_US}} locale was not > working. > To reproduce on Linux: > {code:none} > export LC_CTYPE=pt_BR.UTF-8 > ant test -Dtest.name="DeleteTest" > ant test -Dtest.name="SpeculativeRetryParseTest" > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13896) Improving Cassandra write performance
[ https://issues.apache.org/jira/browse/CASSANDRA-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441330#comment-16441330 ] Michael Burman commented on CASSANDRA-13896: I compared 4.0-trunk vs patched version. > Improving Cassandra write performance > --- > > Key: CASSANDRA-13896 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13896 > Project: Cassandra > Issue Type: Improvement > Components: Local Write-Read Paths > Environment: Skylake server with 2 sockets, 192GB RAM, 3xPCIe SSDs > OS: Centos 7.3 > Java: Oracle JDK1.8.0_121 >Reporter: Prince Nana Owusu Boateng >Priority: Major > Labels: Performance > Fix For: 4.x > > Attachments: Screen Shot 2017-09-22 at 11.22.43 AM.png, Screen Shot > 2017-09-22 at 3.31.09 PM.png > > > During our Cassandra performance testing, we see high percentage of the CPU > spent in *org.apache.cassandra.utils.memory.SlabAllocator.allocate(int, > OpOrder Group) * method. Appears to be high contention of the > ** atomic Integer in write workloads. This structure is > used by the threads for keeping track of the region bytebuffer allocation. > When the contention appears, adding more clients, modifications of write > specific parameters does not change write throughput performance. Attached > are the details of Java Flight Recorder (JFR), showing hot functions and also > performance results. When we see this contention, we still have plenty of > CPU and throughput left ( *<20%* Total average CPU utilization and *<11%* > of the storage write total throughput). This occurs on Cassandra 3.10.0-src > version using the Cassandra-Stress. > Proposal: > We will like to introduce a solution which eliminates the atomic operations > on the ** atomic Integer. This implementation will allow > concurrent allocation of bytebuffers without an atomic compareAndSet and > incrementAndGet operations. The solution is expected to increase overall > write performance while improving CPU utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
cassandra git commit: Fix speculative retry parsing on non-english default Locale
Repository: cassandra Updated Branches: refs/heads/trunk 88b01c866 -> f165e72bf Fix speculative retry parsing on non-english default Locale Patch by Paulo Motta; Reviewed by Aleksey Yeschenko for CASSANDRA-14374 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/f165e72b Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/f165e72b Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/f165e72b Branch: refs/heads/trunk Commit: f165e72bf19e8d12457b8f569517012628513d24 Parents: 88b01c8 Author: Paulo Motta Authored: Tue Apr 17 15:47:29 2018 -0300 Committer: Paulo Motta Committed: Tue Apr 17 15:47:29 2018 -0300 -- .../reads/PercentileSpeculativeRetryPolicy.java | 7 ++- .../service/reads/SpeculativeRetryParseTest.java | 14 -- 2 files changed, 18 insertions(+), 3 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/f165e72b/src/java/org/apache/cassandra/service/reads/PercentileSpeculativeRetryPolicy.java -- diff --git a/src/java/org/apache/cassandra/service/reads/PercentileSpeculativeRetryPolicy.java b/src/java/org/apache/cassandra/service/reads/PercentileSpeculativeRetryPolicy.java index 9bf5d95..42f90fe 100644 --- a/src/java/org/apache/cassandra/service/reads/PercentileSpeculativeRetryPolicy.java +++ b/src/java/org/apache/cassandra/service/reads/PercentileSpeculativeRetryPolicy.java @@ -18,6 +18,8 @@ package org.apache.cassandra.service.reads; import java.text.DecimalFormat; +import java.text.DecimalFormatSymbols; +import java.util.Locale; import java.util.regex.Matcher; import java.util.regex.Pattern; @@ -32,7 +34,10 @@ public class PercentileSpeculativeRetryPolicy implements SpeculativeRetryPolicy public static final PercentileSpeculativeRetryPolicy NINETY_NINE_P = new PercentileSpeculativeRetryPolicy(99.0); private static final Pattern PATTERN = Pattern.compile("^(?[0-9.]+)p(ercentile)?$", Pattern.CASE_INSENSITIVE); -private static final DecimalFormat FORMATTER = new DecimalFormat("#."); +/** + * The pattern above uses dot as decimal separator, so we use {@link Locale#ENGLISH} to enforce that. (CASSANDRA-14374) + */ +private static final DecimalFormat FORMATTER = new DecimalFormat("#.", new DecimalFormatSymbols(Locale.ENGLISH)); private final double percentile; http://git-wip-us.apache.org/repos/asf/cassandra/blob/f165e72b/test/unit/org/apache/cassandra/service/reads/SpeculativeRetryParseTest.java -- diff --git a/test/unit/org/apache/cassandra/service/reads/SpeculativeRetryParseTest.java b/test/unit/org/apache/cassandra/service/reads/SpeculativeRetryParseTest.java index 8c742be..86b307e 100644 --- a/test/unit/org/apache/cassandra/service/reads/SpeculativeRetryParseTest.java +++ b/test/unit/org/apache/cassandra/service/reads/SpeculativeRetryParseTest.java @@ -20,6 +20,7 @@ package org.apache.cassandra.service.reads; import java.util.Arrays; import java.util.Collection; +import java.util.Locale; import org.junit.Test; import org.junit.experimental.runners.Enclosed; @@ -36,10 +37,11 @@ import static org.apache.cassandra.service.reads.HybridSpeculativeRetryPolicy.Fu @RunWith(Enclosed.class) public class SpeculativeRetryParseTest { - @RunWith(Parameterized.class) public static class SuccessfulParseTest { +static final Locale defaultLocale = Locale.getDefault(); + private final String string; private final SpeculativeRetryPolicy expectedValue; @@ -104,9 +106,17 @@ public class SpeculativeRetryParseTest } @Test -public void testToStringRoundTrip() +public void testToStringRoundTripDefaultLocale() +{ +assertEquals(expectedValue, SpeculativeRetryPolicy.fromString(expectedValue.toString())); +} + +@Test +public void testToStringRoundTripCommaDecimalSeparatorLocale() { +Locale.setDefault(new Locale("pt","BR")); // CASSANDRA-14374: Brazil uses comma instead of dot as decimal separator assertEquals(expectedValue, SpeculativeRetryPolicy.fromString(expectedValue.toString())); +Locale.setDefault(defaultLocale); } } - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13896) Improving Cassandra write performance
[ https://issues.apache.org/jira/browse/CASSANDRA-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441324#comment-16441324 ] Ariel Weisberg commented on CASSANDRA-13896: Did we compare different versions of Casasndra here? 3.10 vs ??? > Improving Cassandra write performance > --- > > Key: CASSANDRA-13896 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13896 > Project: Cassandra > Issue Type: Improvement > Components: Local Write-Read Paths > Environment: Skylake server with 2 sockets, 192GB RAM, 3xPCIe SSDs > OS: Centos 7.3 > Java: Oracle JDK1.8.0_121 >Reporter: Prince Nana Owusu Boateng >Priority: Major > Labels: Performance > Fix For: 4.x > > Attachments: Screen Shot 2017-09-22 at 11.22.43 AM.png, Screen Shot > 2017-09-22 at 3.31.09 PM.png > > > During our Cassandra performance testing, we see high percentage of the CPU > spent in *org.apache.cassandra.utils.memory.SlabAllocator.allocate(int, > OpOrder Group) * method. Appears to be high contention of the > ** atomic Integer in write workloads. This structure is > used by the threads for keeping track of the region bytebuffer allocation. > When the contention appears, adding more clients, modifications of write > specific parameters does not change write throughput performance. Attached > are the details of Java Flight Recorder (JFR), showing hot functions and also > performance results. When we see this contention, we still have plenty of > CPU and throughput left ( *<20%* Total average CPU utilization and *<11%* > of the storage write total throughput). This occurs on Cassandra 3.10.0-src > version using the Cassandra-Stress. > Proposal: > We will like to introduce a solution which eliminates the atomic operations > on the ** atomic Integer. This implementation will allow > concurrent allocation of bytebuffers without an atomic compareAndSet and > incrementAndGet operations. The solution is expected to increase overall > write performance while improving CPU utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13896) Improving Cassandra write performance
[ https://issues.apache.org/jira/browse/CASSANDRA-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441314#comment-16441314 ] Michael Burman commented on CASSANDRA-13896: [~aweisberg] I think my patch was misunderstood. It's meant to rather prove that this isn't the contention point that Cassandra is hitting at this point. If it were, the patch would at least increase performance somewhat, since it reduces the contention on that single CAS. But there's no effect at all to performance. The most probable reason of scaling bottleneck is long before this place in the pipeline and it's most likely somewhere in our threading model executors, we can't process enough data from the network. There is a very large change in CPI (and reduce in front-end bound) when batching multiple data points. This reduces the hits to do_futex for example considerably and perf-map-agent shows reduced hits to our SharedExecutorPool's CSLM. We probably do either too much in the Netty pipeline before sending off the work or the pipeline for mutations is simply too long to be effective (instead of smaller parts where we could reuse the same thread and potentially same L1/L2/L3 cache hits for better performance). But I have no patch for that yet, profilers are not exactly brilliant in showing hotpoint when it's about "not doing anything". And you're right about contended annotation, I only tried several places to make sure no false sharing was responsible for no added performance. That place just happened to be in the pushed commit as I wasn't intending this to be the fix but instead to prove this isn't the correct scaling limitation - yet. > Improving Cassandra write performance > --- > > Key: CASSANDRA-13896 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13896 > Project: Cassandra > Issue Type: Improvement > Components: Local Write-Read Paths > Environment: Skylake server with 2 sockets, 192GB RAM, 3xPCIe SSDs > OS: Centos 7.3 > Java: Oracle JDK1.8.0_121 >Reporter: Prince Nana Owusu Boateng >Priority: Major > Labels: Performance > Fix For: 4.x > > Attachments: Screen Shot 2017-09-22 at 11.22.43 AM.png, Screen Shot > 2017-09-22 at 3.31.09 PM.png > > > During our Cassandra performance testing, we see high percentage of the CPU > spent in *org.apache.cassandra.utils.memory.SlabAllocator.allocate(int, > OpOrder Group) * method. Appears to be high contention of the > ** atomic Integer in write workloads. This structure is > used by the threads for keeping track of the region bytebuffer allocation. > When the contention appears, adding more clients, modifications of write > specific parameters does not change write throughput performance. Attached > are the details of Java Flight Recorder (JFR), showing hot functions and also > performance results. When we see this contention, we still have plenty of > CPU and throughput left ( *<20%* Total average CPU utilization and *<11%* > of the storage write total throughput). This occurs on Cassandra 3.10.0-src > version using the Cassandra-Stress. > Proposal: > We will like to introduce a solution which eliminates the atomic operations > on the ** atomic Integer. This implementation will allow > concurrent allocation of bytebuffers without an atomic compareAndSet and > incrementAndGet operations. The solution is expected to increase overall > write performance while improving CPU utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14393) Incorrect view updates
Duarte Nunes created CASSANDRA-14393: Summary: Incorrect view updates Key: CASSANDRA-14393 URL: https://issues.apache.org/jira/browse/CASSANDRA-14393 Project: Cassandra Issue Type: Bug Components: Materialized Views Reporter: Duarte Nunes Consider the following: {noformat} create table t (p int, c int, v1 int, v2 int, primary key(p, c)); create materialized view mv as select p, c, v1 from t where p is not null and c is not null primary key (c, p); insert into t (p, c, v1, v2) VALUES(1, 1, 1, 1) using ttl 5; update t using ttl 1000 set v2 = 1 where p = 1 and c = 1; delete v2 from t where p = 1 and c = 1; // Wait 5 seconds select * from mv; c | p | v1 ---+---+-- 1 | 1 | null{noformat} The view row should be dead after 5 seconds, but it is not. This is because the liveness info calculated when deleting v2 is based on the base table update liveness info, which has the timestamp of the first insert statement. That liveness info is shadowed by the liveness info created in the update, which has a higher timestamp. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14381) nodetool listsnapshots is missing snapshots
[ https://issues.apache.org/jira/browse/CASSANDRA-14381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441224#comment-16441224 ] Ariel Weisberg commented on CASSANDRA-14381: The cicleci change is not part of the change it's just necessary to get CircleCI to run the dtests for me. > nodetool listsnapshots is missing snapshots > --- > > Key: CASSANDRA-14381 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14381 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: MacOs 10.12.5 > Java 1.8.0_144 > Cassandra 3.11.2 (brew install) >Reporter: Cyril Scetbon >Assignee: Ariel Weisberg >Priority: Major > > The output of *nodetool listsnapshots* is inconsistent with the snapshots > created : > {code:java} > $ nodetool listsnapshots > Snapshot Details: > There are no snapshots > $ nodetool snapshot -t tag1 --table local system > Requested creating snapshot(s) for [system] with snapshot name [tag1] and > options {skipFlush=false} > Snapshot directory: tag1 > $ nodetool snapshot -t tag2 --table local system > Requested creating snapshot(s) for [system] with snapshot name [tag2] and > options {skipFlush=false} > Snapshot directory: tag2 > $ nodetool listsnapshots > Snapshot Details: > There are no snapshots > $ ls > /usr/local/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/snapshots/ > tag1 tag2{code} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13896) Improving Cassandra write performance
[ https://issues.apache.org/jira/browse/CASSANDRA-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441223#comment-16441223 ] Ariel Weisberg commented on CASSANDRA-13896: I'm not sure what [~PrinceNana] was proposing to eliminate the contention entirely. Unless we have each thread allocate from its own memory region they will be shared and it will have to be an atomic operation. If we could {{incrementAndGet}} instead of {{compareAndSet}} it might also be faster under contention. I think splitting the contended resource is a classic solution to scaling contended access to shared state. The way Cassandra currently works with large numbers of threads it's the right kind of optimization. It's not totally crazy to optimize this based on microbenchmarks of the allocator rather then end to end benchmarks. Having headroom in the allocator is worth the risk of this particular change IMO, but we should know what we are getting. If we attempted to prove the correctness of the allocator we could be more confident about changing it. Like have multiple threads do a bunch of allocations and then compare the allocations across threads for correctness. They don't overlap, they are one after another, maybe other checks. The contended annotation is not used correctly (I think) as we don't technically know where the contended atomic integer is stored and it would probably be stored away from the object just due to the annotation. The annotation only applies to the memory around the class or field being annotated (I think). We also have two contended fields, the allocation and the count. I think maybe count can just go away or alternatively be CASed into a single atomic long along with the offset. Assuming we can't manage to make that incrementAndGet. Once we have consensus on this approach vs some other I can look closer at is this correct. And the answer for that is usually going to be well is there a test that convinces me it's correct? > Improving Cassandra write performance > --- > > Key: CASSANDRA-13896 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13896 > Project: Cassandra > Issue Type: Improvement > Components: Local Write-Read Paths > Environment: Skylake server with 2 sockets, 192GB RAM, 3xPCIe SSDs > OS: Centos 7.3 > Java: Oracle JDK1.8.0_121 >Reporter: Prince Nana Owusu Boateng >Priority: Major > Labels: Performance > Fix For: 4.x > > Attachments: Screen Shot 2017-09-22 at 11.22.43 AM.png, Screen Shot > 2017-09-22 at 3.31.09 PM.png > > > During our Cassandra performance testing, we see high percentage of the CPU > spent in *org.apache.cassandra.utils.memory.SlabAllocator.allocate(int, > OpOrder Group) * method. Appears to be high contention of the > ** atomic Integer in write workloads. This structure is > used by the threads for keeping track of the region bytebuffer allocation. > When the contention appears, adding more clients, modifications of write > specific parameters does not change write throughput performance. Attached > are the details of Java Flight Recorder (JFR), showing hot functions and also > performance results. When we see this contention, we still have plenty of > CPU and throughput left ( *<20%* Total average CPU utilization and *<11%* > of the storage write total throughput). This occurs on Cassandra 3.10.0-src > version using the Cassandra-Stress. > Proposal: > We will like to introduce a solution which eliminates the atomic operations > on the ** atomic Integer. This implementation will allow > concurrent allocation of bytebuffers without an atomic compareAndSet and > incrementAndGet operations. The solution is expected to increase overall > write performance while improving CPU utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
cassandra-dtest git commit: Add tests for running shadow round using seeds and/or peers
Repository: cassandra-dtest Updated Branches: refs/heads/master 6edfe7fb5 -> 2ee611a6d Add tests for running shadow round using seeds and/or peers Patch by Kurt Greaves; reveiewd by Sam Tunnicliffe for CASSANDRA-13851 Project: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/commit/2ee611a6 Tree: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/tree/2ee611a6 Diff: http://git-wip-us.apache.org/repos/asf/cassandra-dtest/diff/2ee611a6 Branch: refs/heads/master Commit: 2ee611a6d2bf5090d52856c7bc2368a2dc37f153 Parents: 6edfe7f Author: kurt Authored: Wed Nov 1 09:57:10 2017 + Committer: Sam Tunnicliffe Committed: Tue Apr 17 18:22:04 2018 +0100 -- dtest.py | 13 +++ seed_test.py | 109 ++ 2 files changed, 122 insertions(+) -- http://git-wip-us.apache.org/repos/asf/cassandra-dtest/blob/2ee611a6/dtest.py -- diff --git a/dtest.py b/dtest.py index 914f2f7..2aa30b8 100644 --- a/dtest.py +++ b/dtest.py @@ -276,6 +276,19 @@ class Tester: runner.start() return runner +def assert_log_had_msg(self, node, msg, timeout=600, **kwargs): +""" +Wrapper for ccmlib.node.Node#watch_log_for to cause an assertion failure when a log message isn't found +within the timeout. +:param node: Node which logs we should watch +:param msg: String message we expect to see in the logs. +:param timeout: Seconds to wait for msg to appear +""" +try: +node.watch_log_for(msg, timeout=timeout, **kwargs) +except TimeoutError: +pytest.fail("Log message was not seen within timeout:\n{0}".format(msg)) + def get_eager_protocol_version(cassandra_version): """ Returns the highest protocol version accepted http://git-wip-us.apache.org/repos/asf/cassandra-dtest/blob/2ee611a6/seed_test.py -- diff --git a/seed_test.py b/seed_test.py new file mode 100644 index 000..1f00d8b --- /dev/null +++ b/seed_test.py @@ -0,0 +1,109 @@ +from ccmlib import node +from dtest import Tester +from time import sleep + +import pytest + +since = pytest.mark.since + + +class TestGossiper(Tester): +""" +Test gossip states +""" + +@since('3.11.2') +def test_startup_no_live_seeds(self): +""" +Test that a node won't start with no live seeds. +@jira_ticket CASSANDRA-13851 +""" + +self.fixture_dtest_setup.allow_log_errors = True +n1 = self.cluster.create_node('node1', True, None, ('127.0.0.1', 7000), '7100', + None, None, binary_interface=('127.0.0.1', 9042)) +self.cluster.add(n1, False) +node1 = self.cluster.nodelist()[0] +self.cluster.set_configuration_options({ +'seed_provider': [{'class_name': 'org.apache.cassandra.locator.SimpleSeedProvider', + 'parameters': [{'seeds': '127.0.0.2'}] # dummy node doesn't exist + }] +}) + +try: +STARTUP_TIMEOUT = 15 # seconds +RING_DELAY = 1 # ms +# set startup timeout > ring delay so that startup failure happens before the call to start returns +node1.start(wait_for_binary_proto=STARTUP_TIMEOUT, jvm_args=['-Dcassandra.ring_delay_ms={}'.format(RING_DELAY)]) +except node.TimeoutError: +self.assert_log_had_msg(node1, "Unable to gossip with any peers") +except Exception as e: +raise e +else: +pytest.fail("Expecting startup to raise a TimeoutError, but nothing was raised.") + +@since('3.11.2') +def test_startup_non_seed_with_peers(self): +""" +Test that a node can start if peers are alive, or if a node has been bootstrapped +but there are no live seeds or peers +@jira_ticket CASSANDRA-13851 +""" + +self.fixture_dtest_setup.allow_log_errors = True + +n1 = self.cluster.create_node('node1', True, None, ('127.0.0.1', 7000), '7100', + None, None, binary_interface=('127.0.0.1', 9042)) +n2 = self.cluster.create_node('node2', True, None, ('127.0.0.2', 7000), '7101', + None, None, binary_interface=('127.0.0.2', 9042)) +n3 = self.cluster.create_node('node3', True, None, ('127.0.0.3', 7000), '7102', + None, None, binary_interface=('127.0.0.3', 9042)) +self.cluster.add(n1, True) +self.cluster.add(n2, True) +self.cluster.add(n3, True) + +node1, node2,
[jira] [Updated] (CASSANDRA-13851) Allow existing nodes to use all peers in shadow round
[ https://issues.apache.org/jira/browse/CASSANDRA-13851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Tunnicliffe updated CASSANDRA-13851: Resolution: Fixed Fix Version/s: (was: 3.11.x) (was: 4.x) 3.11.3 4.0 Status: Resolved (was: Patch Available) Thanks [~KurtG], latest CI (after rebases) looks good committed so I've to cassandra-3.11 in {{28ccf3fe3989d9d80063fe4d4bb048efe471936b}} and merged to trunk. I'll get the dtest committed directly. > Allow existing nodes to use all peers in shadow round > - > > Key: CASSANDRA-13851 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13851 > Project: Cassandra > Issue Type: Bug > Components: Lifecycle >Reporter: Kurt Greaves >Assignee: Kurt Greaves >Priority: Major > Fix For: 4.0, 3.11.3 > > > In CASSANDRA-10134 we made collision checks necessary on every startup. A > side-effect was introduced that then requires a nodes seeds to be contacted > on every startup. Prior to this change an existing node could start up > regardless whether it could contact a seed node or not (because > checkForEndpointCollision() was only called for bootstrapping nodes). > Now if a nodes seeds are removed/deleted/fail it will no longer be able to > start up until live seeds are configured (or itself is made a seed), even > though it already knows about the rest of the ring. This is inconvenient for > operators and has the potential to cause some nasty surprises and increase > downtime. > One solution would be to use all a nodes existing peers as seeds in the > shadow round. Not a Gossip guru though so not sure of implications. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[3/3] cassandra git commit: Merge branch 'cassandra-3.11' into trunk
Merge branch 'cassandra-3.11' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/88b01c86 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/88b01c86 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/88b01c86 Branch: refs/heads/trunk Commit: 88b01c866d420c58e104640ddb78e3ebeef4d915 Parents: 70d9535 28ccf3f Author: Sam Tunnicliffe Authored: Tue Apr 17 18:05:00 2018 +0100 Committer: Sam Tunnicliffe Committed: Tue Apr 17 18:13:58 2018 +0100 -- CHANGES.txt | 3 ++ src/java/org/apache/cassandra/gms/Gossiper.java | 38 +++- .../cassandra/service/StorageService.java | 10 -- 3 files changed, 40 insertions(+), 11 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/88b01c86/CHANGES.txt -- diff --cc CHANGES.txt index 5e36674,39213a1..c9f6685 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,229 -1,8 +1,232 @@@ +4.0 + * Bind to correct local address in 4.0 streaming (CASSANDRA-14362) + * Use standard Amazon naming for datacenter and rack in Ec2Snitch (CASSANDRA-7839) + * Fix junit failure for SSTableReaderTest (CASSANDRA-14387) + * Abstract write path for pluggable storage (CASSANDRA-14118) + * nodetool describecluster should be more informative (CASSANDRA-13853) + * Compaction performance improvements (CASSANDRA-14261) + * Refactor Pair usage to avoid boxing ints/longs (CASSANDRA-14260) + * Add options to nodetool tablestats to sort and limit output (CASSANDRA-13889) + * Rename internals to reflect CQL vocabulary (CASSANDRA-14354) + * Add support for hybrid MIN(), MAX() speculative retry policies + (CASSANDRA-14293, CASSANDRA-14338, CASSANDRA-14352) + * Fix some regressions caused by 14058 (CASSANDRA-14353) + * Abstract repair for pluggable storage (CASSANDRA-14116) + * Add meaningful toString() impls (CASSANDRA-13653) + * Add sstableloader option to accept target keyspace name (CASSANDRA-13884) + * Move processing of EchoMessage response to gossip stage (CASSANDRA-13713) + * Add coordinator write metric per CF (CASSANDRA-14232) + * Correct and clarify SSLFactory.getSslContext method and call sites (CASSANDRA-14314) + * Handle static and partition deletion properly on ThrottledUnfilteredIterator (CASSANDRA-14315) + * NodeTool clientstats should show SSL Cipher (CASSANDRA-14322) + * Add ability to specify driver name and version (CASSANDRA-14275) + * Abstract streaming for pluggable storage (CASSANDRA-14115) + * Forced incremental repairs should promote sstables if they can (CASSANDRA-14294) + * Use Murmur3 for validation compactions (CASSANDRA-14002) + * Comma at the end of the seed list is interpretated as localhost (CASSANDRA-14285) + * Refactor read executor and response resolver, abstract read repair (CASSANDRA-14058) + * Add optional startup delay to wait until peers are ready (CASSANDRA-13993) + * Add a few options to nodetool verify (CASSANDRA-14201) + * CVE-2017-5929 Security vulnerability and redefine default log rotation policy (CASSANDRA-14183) + * Use JVM default SSL validation algorithm instead of custom default (CASSANDRA-13259) + * Better document in code InetAddressAndPort usage post 7544, incorporate port into UUIDGen node (CASSANDRA-14226) + * Fix sstablemetadata date string for minLocalDeletionTime (CASSANDRA-14132) + * Make it possible to change neverPurgeTombstones during runtime (CASSANDRA-14214) + * Remove GossipDigestSynVerbHandler#doSort() (CASSANDRA-14174) + * Add nodetool clientlist (CASSANDRA-13665) + * Revert ProtocolVersion changes from CASSANDRA-7544 (CASSANDRA-14211) + * Non-disruptive seed node list reload (CASSANDRA-14190) + * Nodetool tablehistograms to print statics for all the tables (CASSANDRA-14185) + * Migrate dtests to use pytest and python3 (CASSANDRA-14134) + * Allow storage port to be configurable per node (CASSANDRA-7544) + * Make sub-range selection for non-frozen collections return null instead of empty (CASSANDRA-14182) + * BloomFilter serialization format should not change byte ordering (CASSANDRA-9067) + * Remove unused on-heap BloomFilter implementation (CASSANDRA-14152) + * Delete temp test files on exit (CASSANDRA-14153) + * Make PartitionUpdate and Mutation immutable (CASSANDRA-13867) + * Fix CommitLogReplayer exception for CDC data (CASSANDRA-14066) + * Fix cassandra-stress startup failure (CASSANDRA-14106) + * Remove initialDirectories from CFS (CASSANDRA-13928) + * Fix trivial log format error (CASSANDRA-14015) + * Allow sstabledump to do a json object per partition (CASSANDRA-13848) + * Add option to optimise merkle tree comparison across replicas (CASSANDRA-3200) + * Remove unused and deprecated methods
[1/3] cassandra git commit: Don't fail startup if peers aren't live
Repository: cassandra Updated Branches: refs/heads/cassandra-3.11 95cfee623 -> 28ccf3fe3 refs/heads/trunk 70d95359d -> 88b01c866 Don't fail startup if peers aren't live Patch by Kurt Greaves; reviewed by Sam Tunnicliffe for CASSANDRA-13851 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/28ccf3fe Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/28ccf3fe Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/28ccf3fe Branch: refs/heads/cassandra-3.11 Commit: 28ccf3fe3989d9d80063fe4d4bb048efe471936b Parents: 95cfee6 Author: kurt Authored: Wed Nov 1 08:58:54 2017 + Committer: Sam Tunnicliffe Committed: Tue Apr 17 17:59:48 2018 +0100 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/gms/Gossiper.java | 38 +++- .../cassandra/service/StorageService.java | 10 -- 3 files changed, 38 insertions(+), 11 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/28ccf3fe/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 42ea3b4..39213a1 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.11.3 + * Allow existing nodes to use all peers in shadow round (CASSANDRA-13851) * Fix cqlsh to read connection.ssl cqlshrc option again (CASSANDRA-14299) * Downgrade log level to trace for CommitLogSegmentManager (CASSANDRA-14370) * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891) http://git-wip-us.apache.org/repos/asf/cassandra/blob/28ccf3fe/src/java/org/apache/cassandra/gms/Gossiper.java -- diff --git a/src/java/org/apache/cassandra/gms/Gossiper.java b/src/java/org/apache/cassandra/gms/Gossiper.java index 2dac5c2..ea05525 100644 --- a/src/java/org/apache/cassandra/gms/Gossiper.java +++ b/src/java/org/apache/cassandra/gms/Gossiper.java @@ -1361,6 +1361,11 @@ public class Gossiper implements IFailureDetectionEventListener, GossiperMBean TimeUnit.MILLISECONDS); } +public synchronized Map doShadowRound() +{ +return doShadowRound(Collections.EMPTY_SET); +} + /** * Do a single 'shadow' round of gossip by retrieving endpoint states that will be stored exclusively in the * map return value, instead of endpointStateMap. @@ -1376,16 +1381,21 @@ public class Gossiper implements IFailureDetectionEventListener, GossiperMBean * caller of {@link Gossiper#doShadowRound()}. Therefor only a single shadow round execution is permitted at * the same time. * + * @param peers Additional peers to try gossiping with. * @return endpoint states gathered during shadow round or empty map */ -public synchronized Map doShadowRound() +public synchronized Map doShadowRound(Set peers) { buildSeedsList(); -// it may be that the local address is the only entry in the seed +// it may be that the local address is the only entry in the seed + peers // list in which case, attempting a shadow round is pointless -if (seeds.isEmpty()) +if (seeds.isEmpty() && peers.isEmpty()) return endpointShadowStateMap; +boolean isSeed = DatabaseDescriptor.getSeeds().contains(FBUtilities.getBroadcastAddress()); +// We double RING_DELAY if we're not a seed to increase chance of successful startup during a full cluster bounce, +// giving the seeds a chance to startup before we fail the shadow round +int shadowRoundDelay = isSeed ? StorageService.RING_DELAY : StorageService.RING_DELAY * 2; seedsInShadowRound.clear(); endpointShadowStateMap.clear(); // send a completely empty syn @@ -1398,6 +1408,7 @@ public class Gossiper implements IFailureDetectionEventListener, GossiperMBean GossipDigestSyn.serializer); inShadowRound = true; +boolean includePeers = false; int slept = 0; try { @@ -1409,6 +1420,15 @@ public class Gossiper implements IFailureDetectionEventListener, GossiperMBean for (InetAddress seed : seeds) MessagingService.instance().sendOneWay(message, seed); + +// Send to any peers we already know about, but only if a seed didn't respond. +if (includePeers) +{ +logger.trace("Sending shadow round GOSSIP DIGEST SYN to known peers {}", peers); +for (InetAddress peer : peers) +MessagingService.instance().sendOneWay(message, peer); +} +
[2/3] cassandra git commit: Don't fail startup if peers aren't live
Don't fail startup if peers aren't live Patch by Kurt Greaves; reviewed by Sam Tunnicliffe for CASSANDRA-13851 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/28ccf3fe Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/28ccf3fe Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/28ccf3fe Branch: refs/heads/trunk Commit: 28ccf3fe3989d9d80063fe4d4bb048efe471936b Parents: 95cfee6 Author: kurt Authored: Wed Nov 1 08:58:54 2017 + Committer: Sam Tunnicliffe Committed: Tue Apr 17 17:59:48 2018 +0100 -- CHANGES.txt | 1 + src/java/org/apache/cassandra/gms/Gossiper.java | 38 +++- .../cassandra/service/StorageService.java | 10 -- 3 files changed, 38 insertions(+), 11 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/28ccf3fe/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 42ea3b4..39213a1 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 3.11.3 + * Allow existing nodes to use all peers in shadow round (CASSANDRA-13851) * Fix cqlsh to read connection.ssl cqlshrc option again (CASSANDRA-14299) * Downgrade log level to trace for CommitLogSegmentManager (CASSANDRA-14370) * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891) http://git-wip-us.apache.org/repos/asf/cassandra/blob/28ccf3fe/src/java/org/apache/cassandra/gms/Gossiper.java -- diff --git a/src/java/org/apache/cassandra/gms/Gossiper.java b/src/java/org/apache/cassandra/gms/Gossiper.java index 2dac5c2..ea05525 100644 --- a/src/java/org/apache/cassandra/gms/Gossiper.java +++ b/src/java/org/apache/cassandra/gms/Gossiper.java @@ -1361,6 +1361,11 @@ public class Gossiper implements IFailureDetectionEventListener, GossiperMBean TimeUnit.MILLISECONDS); } +public synchronized Map doShadowRound() +{ +return doShadowRound(Collections.EMPTY_SET); +} + /** * Do a single 'shadow' round of gossip by retrieving endpoint states that will be stored exclusively in the * map return value, instead of endpointStateMap. @@ -1376,16 +1381,21 @@ public class Gossiper implements IFailureDetectionEventListener, GossiperMBean * caller of {@link Gossiper#doShadowRound()}. Therefor only a single shadow round execution is permitted at * the same time. * + * @param peers Additional peers to try gossiping with. * @return endpoint states gathered during shadow round or empty map */ -public synchronized Map doShadowRound() +public synchronized Map doShadowRound(Set peers) { buildSeedsList(); -// it may be that the local address is the only entry in the seed +// it may be that the local address is the only entry in the seed + peers // list in which case, attempting a shadow round is pointless -if (seeds.isEmpty()) +if (seeds.isEmpty() && peers.isEmpty()) return endpointShadowStateMap; +boolean isSeed = DatabaseDescriptor.getSeeds().contains(FBUtilities.getBroadcastAddress()); +// We double RING_DELAY if we're not a seed to increase chance of successful startup during a full cluster bounce, +// giving the seeds a chance to startup before we fail the shadow round +int shadowRoundDelay = isSeed ? StorageService.RING_DELAY : StorageService.RING_DELAY * 2; seedsInShadowRound.clear(); endpointShadowStateMap.clear(); // send a completely empty syn @@ -1398,6 +1408,7 @@ public class Gossiper implements IFailureDetectionEventListener, GossiperMBean GossipDigestSyn.serializer); inShadowRound = true; +boolean includePeers = false; int slept = 0; try { @@ -1409,6 +1420,15 @@ public class Gossiper implements IFailureDetectionEventListener, GossiperMBean for (InetAddress seed : seeds) MessagingService.instance().sendOneWay(message, seed); + +// Send to any peers we already know about, but only if a seed didn't respond. +if (includePeers) +{ +logger.trace("Sending shadow round GOSSIP DIGEST SYN to known peers {}", peers); +for (InetAddress peer : peers) +MessagingService.instance().sendOneWay(message, peer); +} +includePeers = true; } Thread.sleep(1000); @@ -1416,13 +1436,12 @@ public class Gossiper implem
[jira] [Commented] (CASSANDRA-14381) nodetool listsnapshots is missing snapshots
[ https://issues.apache.org/jira/browse/CASSANDRA-14381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441199#comment-16441199 ] Cyril Scetbon commented on CASSANDRA-14381: --- Thanks [~aweisberg] > nodetool listsnapshots is missing snapshots > --- > > Key: CASSANDRA-14381 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14381 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: MacOs 10.12.5 > Java 1.8.0_144 > Cassandra 3.11.2 (brew install) >Reporter: Cyril Scetbon >Assignee: Ariel Weisberg >Priority: Major > > The output of *nodetool listsnapshots* is inconsistent with the snapshots > created : > {code:java} > $ nodetool listsnapshots > Snapshot Details: > There are no snapshots > $ nodetool snapshot -t tag1 --table local system > Requested creating snapshot(s) for [system] with snapshot name [tag1] and > options {skipFlush=false} > Snapshot directory: tag1 > $ nodetool snapshot -t tag2 --table local system > Requested creating snapshot(s) for [system] with snapshot name [tag2] and > options {skipFlush=false} > Snapshot directory: tag2 > $ nodetool listsnapshots > Snapshot Details: > There are no snapshots > $ ls > /usr/local/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/snapshots/ > tag1 tag2{code} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14381) nodetool listsnapshots is missing snapshots
[ https://issues.apache.org/jira/browse/CASSANDRA-14381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441171#comment-16441171 ] Jay Zhuang commented on CASSANDRA-14381: +1 LGTM. Is the circleci configuration change needed? > nodetool listsnapshots is missing snapshots > --- > > Key: CASSANDRA-14381 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14381 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: MacOs 10.12.5 > Java 1.8.0_144 > Cassandra 3.11.2 (brew install) >Reporter: Cyril Scetbon >Assignee: Ariel Weisberg >Priority: Major > > The output of *nodetool listsnapshots* is inconsistent with the snapshots > created : > {code:java} > $ nodetool listsnapshots > Snapshot Details: > There are no snapshots > $ nodetool snapshot -t tag1 --table local system > Requested creating snapshot(s) for [system] with snapshot name [tag1] and > options {skipFlush=false} > Snapshot directory: tag1 > $ nodetool snapshot -t tag2 --table local system > Requested creating snapshot(s) for [system] with snapshot name [tag2] and > options {skipFlush=false} > Snapshot directory: tag2 > $ nodetool listsnapshots > Snapshot Details: > There are no snapshots > $ ls > /usr/local/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/snapshots/ > tag1 tag2{code} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13896) Improving Cassandra write performance
[ https://issues.apache.org/jira/browse/CASSANDRA-13896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441167#comment-16441167 ] Jeff Jirsa commented on CASSANDRA-13896: [~PrinceNana] - do you have an opinion on [~burmanm]'s patch? cc [~aweisberg] for visibility. > Improving Cassandra write performance > --- > > Key: CASSANDRA-13896 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13896 > Project: Cassandra > Issue Type: Improvement > Components: Local Write-Read Paths > Environment: Skylake server with 2 sockets, 192GB RAM, 3xPCIe SSDs > OS: Centos 7.3 > Java: Oracle JDK1.8.0_121 >Reporter: Prince Nana Owusu Boateng >Priority: Major > Labels: Performance > Fix For: 4.x > > Attachments: Screen Shot 2017-09-22 at 11.22.43 AM.png, Screen Shot > 2017-09-22 at 3.31.09 PM.png > > > During our Cassandra performance testing, we see high percentage of the CPU > spent in *org.apache.cassandra.utils.memory.SlabAllocator.allocate(int, > OpOrder Group) * method. Appears to be high contention of the > ** atomic Integer in write workloads. This structure is > used by the threads for keeping track of the region bytebuffer allocation. > When the contention appears, adding more clients, modifications of write > specific parameters does not change write throughput performance. Attached > are the details of Java Flight Recorder (JFR), showing hot functions and also > performance results. When we see this contention, we still have plenty of > CPU and throughput left ( *<20%* Total average CPU utilization and *<11%* > of the storage write total throughput). This occurs on Cassandra 3.10.0-src > version using the Cassandra-Stress. > Proposal: > We will like to introduce a solution which eliminates the atomic operations > on the ** atomic Integer. This implementation will allow > concurrent allocation of bytebuffers without an atomic compareAndSet and > incrementAndGet operations. The solution is expected to increase overall > write performance while improving CPU utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-6719) redesign loadnewsstables
[ https://issues.apache.org/jira/browse/CASSANDRA-6719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441160#comment-16441160 ] Jordan West commented on CASSANDRA-6719: Changes look good [~krummas]. +1. > redesign loadnewsstables > > > Key: CASSANDRA-6719 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6719 > Project: Cassandra > Issue Type: New Feature > Components: Tools >Reporter: Jonathan Ellis >Assignee: Marcus Eriksson >Priority: Minor > Labels: lhf > Fix For: 4.x > > Attachments: 6719.patch > > > CFSMBean.loadNewSSTables scans data directories for new sstables dropped > there by an external agent. This is dangerous because of possible filename > conflicts with existing or newly generated sstables. > Instead, we should support leaving the new sstables in a separate directory > (specified by a parameter, or configured as a new location in yaml) and take > care of renaming as necessary automagically. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14381) nodetool listsnapshots is missing snapshots
[ https://issues.apache.org/jira/browse/CASSANDRA-14381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ariel Weisberg updated CASSANDRA-14381: --- Assignee: Ariel Weisberg Status: Patch Available (was: Open) I had a free minute so put up a patch for this. [Trunk code|https://github.com/apache/cassandra/compare/trunk...aweisberg:cassandra-14381-trunk?expand=1] [CircleCI|https://circleci.com/gh/aweisberg/cassandra/tree/cassandra-14381-trunk] > nodetool listsnapshots is missing snapshots > --- > > Key: CASSANDRA-14381 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14381 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: MacOs 10.12.5 > Java 1.8.0_144 > Cassandra 3.11.2 (brew install) >Reporter: Cyril Scetbon >Assignee: Ariel Weisberg >Priority: Major > > The output of *nodetool listsnapshots* is inconsistent with the snapshots > created : > {code:java} > $ nodetool listsnapshots > Snapshot Details: > There are no snapshots > $ nodetool snapshot -t tag1 --table local system > Requested creating snapshot(s) for [system] with snapshot name [tag1] and > options {skipFlush=false} > Snapshot directory: tag1 > $ nodetool snapshot -t tag2 --table local system > Requested creating snapshot(s) for [system] with snapshot name [tag2] and > options {skipFlush=false} > Snapshot directory: tag2 > $ nodetool listsnapshots > Snapshot Details: > There are no snapshots > $ ls > /usr/local/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/snapshots/ > tag1 tag2{code} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14392) Rename nodetool --with-port to --print-port to disambiguate from --port
[ https://issues.apache.org/jira/browse/CASSANDRA-14392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ariel Weisberg updated CASSANDRA-14392: --- Status: Patch Available (was: Open) [Trunk code|https://github.com/apache/cassandra/compare/trunk...aweisberg:cassandra-14392-trunk?expand=1] [CircleCI|https://circleci.com/gh/aweisberg/cassandra/tree/cassandra-14392-trunk] > Rename nodetool --with-port to --print-port to disambiguate from --port > --- > > Key: CASSANDRA-14392 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14392 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Ariel Weisberg >Assignee: Ariel Weisberg >Priority: Major > Fix For: 4.0 > > > Right now it looks kind of like a third way to set the port number. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Resolved] (CASSANDRA-14301) Error running cqlsh: unexpected keyword argument 'no_compact'
[ https://issues.apache.org/jira/browse/CASSANDRA-14301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa resolved CASSANDRA-14301. Resolution: Not A Problem > Error running cqlsh: unexpected keyword argument 'no_compact' > - > > Key: CASSANDRA-14301 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14301 > Project: Cassandra > Issue Type: Bug >Reporter: M. Justin >Priority: Major > Labels: cqlsh, proposed-wontfix > > I recently installed Cassandra 3.11.2 on my Mac using Homebrew. When I run > the "cqlsh" command, I get the following error: > {code:none} > $ cqlsh > Traceback (most recent call last): > File "/usr/local/Cellar/cassandra/3.11.2/libexec/bin/cqlsh.py", line 2443, > in > main(*read_options(sys.argv[1:], os.environ)) > File "/usr/local/Cellar/cassandra/3.11.2/libexec/bin/cqlsh.py", line 2421, > in main > encoding=options.encoding) > File "/usr/local/Cellar/cassandra/3.11.2/libexec/bin/cqlsh.py", line 488, > in __init__ > **kwargs) > File "cassandra/cluster.py", line 735, in > cassandra.cluster.Cluster.__init__ (cassandra/cluster.c:10935) > TypeError: __init__() got an unexpected keyword argument 'no_compact' > {code} > Commenting out [line 483 of > cqlsh.py|https://github.com/apache/cassandra/blob/cassandra-3.11.2/bin/cqlsh.py#L483] > works around the issue: > {code} > # no_compact=no_compact > {code} > I am not the only person impacted, as evidenced by [this existing Stack > Overflow > post|https://stackoverflow.com/questions/48885984/was-cqlsh-5-0-1-broken-in-cassandra-3-11-2-release] > from 2/20/2018. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14160) maxPurgeableTimestamp should traverse tables in order of minTimestamp
[ https://issues.apache.org/jira/browse/CASSANDRA-14160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441124#comment-16441124 ] Jeff Jirsa commented on CASSANDRA-14160: cc [~cnlwsu] as well (we've seen some meaningful perf improvements with NEVER_PURGE_TOMBSTONES=true, tagging Chris in case he remembers if the benefit was in the purge evaluator or getFullyExpiredSSTables() ) > maxPurgeableTimestamp should traverse tables in order of minTimestamp > - > > Key: CASSANDRA-14160 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14160 > Project: Cassandra > Issue Type: Bug > Components: Compaction >Reporter: Josh Snyder >Assignee: Josh Snyder >Priority: Major > Labels: performance > Fix For: 4.x > > > In maxPurgeableTimestamp, we iterate over the bloom filters of each > overlapping SSTable. Of the bloom filter hits, we take the SSTable with the > lowest minTimestamp. If we kept the SSTables in sorted order of minTimestamp, > then we could short-circuit the operation at the first bloom filter hit, > reducing cache pressure (or worse, I/O) and CPU time. > I've written (but not yet benchmarked) [some > code|https://github.com/hashbrowncipher/cassandra/commit/29859a4a2e617f6775be49448858bc59fdafab44] > to demonstrate this possibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14332) Fix unbounded validation compactions on repair
[ https://issues.apache.org/jira/browse/CASSANDRA-14332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Eggleston updated CASSANDRA-14332: Reviewer: Blake Eggleston > Fix unbounded validation compactions on repair > -- > > Key: CASSANDRA-14332 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14332 > Project: Cassandra > Issue Type: Bug >Reporter: Kurt Greaves >Assignee: Kurt Greaves >Priority: Major > > After CASSANDRA-13797 it's possible to cause unbounded, simultaneous > validation compactions as we no longer wait for validations to finish. > Potential fix is to have a sane default for the # of concurrent validation > compactions by backporting CASSANDRA-13521 and setting a sane default. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14392) Rename nodetool --with-port to --print-port to disambiguate from --port
Ariel Weisberg created CASSANDRA-14392: -- Summary: Rename nodetool --with-port to --print-port to disambiguate from --port Key: CASSANDRA-14392 URL: https://issues.apache.org/jira/browse/CASSANDRA-14392 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Ariel Weisberg Assignee: Ariel Weisberg Fix For: 4.0 Right now it looks kind of like a third way to set the port number. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14301) Error running cqlsh: unexpected keyword argument 'no_compact'
[ https://issues.apache.org/jira/browse/CASSANDRA-14301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441081#comment-16441081 ] M. Justin commented on CASSANDRA-14301: --- Homebrew ([has reported|https://github.com/Homebrew/homebrew-core/issues/24977#issuecomment-372047111]) that they've fixed the bug on their end. I believe this can be closed as a (now-fixed) issue with Homebrew and not an issue with Cassandra. > Error running cqlsh: unexpected keyword argument 'no_compact' > - > > Key: CASSANDRA-14301 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14301 > Project: Cassandra > Issue Type: Bug >Reporter: M. Justin >Priority: Major > Labels: cqlsh, proposed-wontfix > > I recently installed Cassandra 3.11.2 on my Mac using Homebrew. When I run > the "cqlsh" command, I get the following error: > {code:none} > $ cqlsh > Traceback (most recent call last): > File "/usr/local/Cellar/cassandra/3.11.2/libexec/bin/cqlsh.py", line 2443, > in > main(*read_options(sys.argv[1:], os.environ)) > File "/usr/local/Cellar/cassandra/3.11.2/libexec/bin/cqlsh.py", line 2421, > in main > encoding=options.encoding) > File "/usr/local/Cellar/cassandra/3.11.2/libexec/bin/cqlsh.py", line 488, > in __init__ > **kwargs) > File "cassandra/cluster.py", line 735, in > cassandra.cluster.Cluster.__init__ (cassandra/cluster.c:10935) > TypeError: __init__() got an unexpected keyword argument 'no_compact' > {code} > Commenting out [line 483 of > cqlsh.py|https://github.com/apache/cassandra/blob/cassandra-3.11.2/bin/cqlsh.py#L483] > works around the issue: > {code} > # no_compact=no_compact > {code} > I am not the only person impacted, as evidenced by [this existing Stack > Overflow > post|https://stackoverflow.com/questions/48885984/was-cqlsh-5-0-1-broken-in-cassandra-3-11-2-release] > from 2/20/2018. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14301) Error running cqlsh: unexpected keyword argument 'no_compact'
[ https://issues.apache.org/jira/browse/CASSANDRA-14301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441081#comment-16441081 ] M. Justin edited comment on CASSANDRA-14301 at 4/17/18 4:06 PM: Homebrew [has reported|https://github.com/Homebrew/homebrew-core/issues/24977#issuecomment-372047111] that they've fixed the bug on their end. I believe this can be closed as a (now-fixed) issue with Homebrew and not an issue with Cassandra. was (Author: mjjustin): Homebrew ([has reported|https://github.com/Homebrew/homebrew-core/issues/24977#issuecomment-372047111]) that they've fixed the bug on their end. I believe this can be closed as a (now-fixed) issue with Homebrew and not an issue with Cassandra. > Error running cqlsh: unexpected keyword argument 'no_compact' > - > > Key: CASSANDRA-14301 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14301 > Project: Cassandra > Issue Type: Bug >Reporter: M. Justin >Priority: Major > Labels: cqlsh, proposed-wontfix > > I recently installed Cassandra 3.11.2 on my Mac using Homebrew. When I run > the "cqlsh" command, I get the following error: > {code:none} > $ cqlsh > Traceback (most recent call last): > File "/usr/local/Cellar/cassandra/3.11.2/libexec/bin/cqlsh.py", line 2443, > in > main(*read_options(sys.argv[1:], os.environ)) > File "/usr/local/Cellar/cassandra/3.11.2/libexec/bin/cqlsh.py", line 2421, > in main > encoding=options.encoding) > File "/usr/local/Cellar/cassandra/3.11.2/libexec/bin/cqlsh.py", line 488, > in __init__ > **kwargs) > File "cassandra/cluster.py", line 735, in > cassandra.cluster.Cluster.__init__ (cassandra/cluster.c:10935) > TypeError: __init__() got an unexpected keyword argument 'no_compact' > {code} > Commenting out [line 483 of > cqlsh.py|https://github.com/apache/cassandra/blob/cassandra-3.11.2/bin/cqlsh.py#L483] > works around the issue: > {code} > # no_compact=no_compact > {code} > I am not the only person impacted, as evidenced by [this existing Stack > Overflow > post|https://stackoverflow.com/questions/48885984/was-cqlsh-5-0-1-broken-in-cassandra-3-11-2-release] > from 2/20/2018. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14391) COPY FROM ignores headers
M. Justin created CASSANDRA-14391: - Summary: COPY FROM ignores headers Key: CASSANDRA-14391 URL: https://issues.apache.org/jira/browse/CASSANDRA-14391 Project: Cassandra Issue Type: Bug Components: CQL Environment: cqlsh 5.0.1 and Cassandra 3.11.2 on macOS 10.13.2. Reporter: M. Justin COPY FROM appears to ignore the headers value, even when "headers = true" is specified. This means that if the columns are reordered, the import process will save values in the wrong columns. h2. Example {noformat:title=temp.csv} col2,col1,col3 column value 1,key2,3 column value 2,key4,3 column value 3,key3,3 column value 4,key1,3 {noformat} {code:sql} create keyspace copy_to_from_test WITH replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 }; use copy_to_from_test; create table test_table (col1 text primary key, col2 text, col3 bigint); copy test_table from 'temp.csv' with header = true; {code} The above code will incorrectly swap the "col2" and "col1" values, since it expects the first column to be "col1". If I had instead swapped the order of "col3", I would have received an error on input, as it would have attempted to store text in a numerical column. h2. Expected Behavior I would expect specifying "with header = true" on a COPY FROM statement to use the headers as column names for insertion, rather than merely skipping the header row. A question is whether missing columns should be an error, or just not imported. h2. Other I ran across this issue when copying between two of my environments. One of the environments had changed the columns in the primary key, but the other had not yet. This caused the order of the columns to vary between the environments. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14381) nodetool listsnapshots is missing snapshots
[ https://issues.apache.org/jira/browse/CASSANDRA-14381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440995#comment-16440995 ] Cyril Scetbon commented on CASSANDRA-14381: --- Okay that's what I thought. Whenever I find some time, I should be able to remove that piece of code that skips the system keyspace > nodetool listsnapshots is missing snapshots > --- > > Key: CASSANDRA-14381 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14381 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: MacOs 10.12.5 > Java 1.8.0_144 > Cassandra 3.11.2 (brew install) >Reporter: Cyril Scetbon >Priority: Major > > The output of *nodetool listsnapshots* is inconsistent with the snapshots > created : > {code:java} > $ nodetool listsnapshots > Snapshot Details: > There are no snapshots > $ nodetool snapshot -t tag1 --table local system > Requested creating snapshot(s) for [system] with snapshot name [tag1] and > options {skipFlush=false} > Snapshot directory: tag1 > $ nodetool snapshot -t tag2 --table local system > Requested creating snapshot(s) for [system] with snapshot name [tag2] and > options {skipFlush=false} > Snapshot directory: tag2 > $ nodetool listsnapshots > Snapshot Details: > There are no snapshots > $ ls > /usr/local/var/lib/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/snapshots/ > tag1 tag2{code} > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-7544) Allow storage port to be configurable per node
[ https://issues.apache.org/jira/browse/CASSANDRA-7544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440983#comment-16440983 ] Ariel Weisberg commented on CASSANDRA-7544: --- I think it's fine to tweak what that is called. You are right --with-port seems wrong as you would expect it to accept a port number as an argument with the Java builder idiom and maybe also CLI idioms. I'll create a separate ticket for it. > Allow storage port to be configurable per node > -- > > Key: CASSANDRA-7544 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7544 > Project: Cassandra > Issue Type: Improvement >Reporter: Sam Overton >Assignee: Ariel Weisberg >Priority: Major > Fix For: 4.0 > > > Currently storage_port must be configured identically on all nodes in a > cluster and it is assumed that this is the case when connecting to a remote > node. > This prevents running in any environment that requires multiple nodes to be > able to bind to the same network interface, such as with many automatic > provisioning/deployment frameworks. > The current solutions seems to be > * use a separate network interface for each node deployed to the same box. > This puts a big requirement on IP allocation at large scale. > * allow multiple clusters to be provisioned from the same resource pool, but > restrict allocation to a maximum of one node per host from each cluster, > assuming each cluster is running on a different storage port. > It would make operations much simpler in these kind of environments if the > environment provisioning the resources could assign the ports to be used when > bringing up a new node on shared hardware. > The changes required would be at least the following: > 1. configure seeds as IP:port instead of just IP > 2. gossip the storage port as part of a node's ApplicationState > 3. refer internally to nodes by hostID instead of IP, since there will be > multiple nodes with the same IP > (1) & (2) are mostly trivial and I already have a patch for these. The bulk > of the work to enable this is (3), and I would structure this as a separate > pre-requisite patch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Resolved] (CASSANDRA-14371) dtest failure: sstablesplit_test.TestSSTableSplit.test_single_file_split
[ https://issues.apache.org/jira/browse/CASSANDRA-14371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko resolved CASSANDRA-14371. --- Resolution: Fixed Looks like we are good now after the ccm PR got merged. Thanks. > dtest failure: sstablesplit_test.TestSSTableSplit.test_single_file_split > > > Key: CASSANDRA-14371 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14371 > Project: Cassandra > Issue Type: Improvement >Reporter: Marcus Eriksson >Assignee: Patrick Bannister >Priority: Major > > https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-dtest/489/testReport/sstablesplit_test/TestSSTableSplit/test_single_file_split/ > {code} > for (stdout, stderr, rc) in result: > logger.debug(stderr) > > failure = stderr.find("java.lang.AssertionError: Data component > > is missing") > E TypeError: a bytes-like object is required, not 'str' > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14388) Fix setting min/max compaction threshold with LCS
[ https://issues.apache.org/jira/browse/CASSANDRA-14388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440896#comment-16440896 ] Chris Lohfink commented on CASSANDRA-14388: --- I like the alternative personally {quote}An alternative would be to check if there is more than 32 ({{MAX_COMPACTING_L0}}) sstables in L0, if so, grab {{max_threshold}} sstables and run STCS on them {quote} Thinking mostly for cases when its critically far behind (ie 50k in L0) and a lot are tiny (from bad repairs), just want to quickly reduce the number of them. The {{MAX_COMPACTING_L0}} I think makes sense for normal use, but when theres a huge backlog just want STCS to chew up backlog faster than 32 at a time. But if increase L0 max compacting to 1000 it may not kick off STCS until already negatively impacting things. Might be nice to have {{MAX_COMPACTING_L0}} as another table option but that could be done another ticket. > Fix setting min/max compaction threshold with LCS > - > > Key: CASSANDRA-14388 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14388 > Project: Cassandra > Issue Type: Bug >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Major > Fix For: 4.x > > > To be able to actually set max/min_threshold in compaction options we need to > remove it from the options map when validating. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14160) maxPurgeableTimestamp should traverse tables in order of minTimestamp
[ https://issues.apache.org/jira/browse/CASSANDRA-14160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440885#comment-16440885 ] Marcus Eriksson commented on CASSANDRA-14160: - hey [~josnyder] the code looks good to me, but, as [~jjirsa] mentioned above, a unit test that makes sure the sstables are returned in the correct order should be added. You also mentioned a that you had not yet benchmarked the change, have you done that? If not, that would also be nice. > maxPurgeableTimestamp should traverse tables in order of minTimestamp > - > > Key: CASSANDRA-14160 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14160 > Project: Cassandra > Issue Type: Bug > Components: Compaction >Reporter: Josh Snyder >Assignee: Josh Snyder >Priority: Major > Labels: performance > Fix For: 4.x > > > In maxPurgeableTimestamp, we iterate over the bloom filters of each > overlapping SSTable. Of the bloom filter hits, we take the SSTable with the > lowest minTimestamp. If we kept the SSTables in sorted order of minTimestamp, > then we could short-circuit the operation at the first bloom filter hit, > reducing cache pressure (or worse, I/O) and CPU time. > I've written (but not yet benchmarked) [some > code|https://github.com/hashbrowncipher/cassandra/commit/29859a4a2e617f6775be49448858bc59fdafab44] > to demonstrate this possibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14390) Cassandra's Debian package depends on java-X-jre which requires GUI components
Richard Whitehouse created CASSANDRA-14390: -- Summary: Cassandra's Debian package depends on java-X-jre which requires GUI components Key: CASSANDRA-14390 URL: https://issues.apache.org/jira/browse/CASSANDRA-14390 Project: Cassandra Issue Type: Bug Components: Packaging Reporter: Richard Whitehouse Cassandra can't be installed against a later version of the JRE without also installing a bunch of GUI packages which aren't relevant on a server installation. e.g. Cassandra 3 can't be installed against a Java 9 JRE without also installing GUI components. This is because of the Debian package dependencies. Cassandra 3+ depends on `openjdk-8-jre-headless | java-8-jre`. Cassandra 2.x depends on `opendjk-7-jre-headless | java-8-jre` `java-X-jre` is a metapackage which specifies a Java version compatible with the given Java version that includes GUI components. It's supplied by `openjdk-X-jre` - e.g. `java-8-jre` is supplied by `openjdk-8-jre` and `openjdk-9-jre`. In comparison, `java-X-jre-headless` is a metapackage which specifies a Java version compatible with the given Java version but doesn't require GUI components.It's supplied by `openjdk-X-jre-headless` - e.g. `java-8-jre-headless` is supplied by `openjdk-8-jre-headless` and `openjdk-9-jre-headless`. Can Cassandra be changed to depend on `java-8-jre-headless` instead of `java-8-jre`? This affects all releases since Debian packaging was introduced according to the commit logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14362) Bind to correct local address in 4.0 streaming
[ https://issues.apache.org/jira/browse/CASSANDRA-14362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440818#comment-16440818 ] Jason Brown edited comment on CASSANDRA-14362 at 4/17/18 12:54 PM: --- [~Lerh Low]'s patch is correct and fixes a bug I introduced with CASSANDRA-12229. Further, there is something wrong with my handling of CASSANDRA-12673, but I don't want to hold up Lerh or anyone else looking at 4.0 in ec2. Thus, I've decided to commit Lerh's patch and will open a followup ticket to address the local address binding issue. Committed as sha {{70d95359d2dca1c35f573776d11ed87bb9b4b441}}. Nice find and fix, Lehr. was (Author: jasobrown): [~Lerh Low]'s patch is correct and fixes a bug I introduced with CASSANDRA-12229. Further, there is something wrong with my handling of CASSANDRA-12673, but I don't want to hold up Lerh or anyone else looking at 4.0 in ec2. Thus, I've decided to commit Lerh's patch and will open a followup ticket to address the local address binding issue. Committed as sha {{stF4fFtT9pMYpabtyomEEFKARMeB8A7H}}. Nice find and fix, Lehr. > Bind to correct local address in 4.0 streaming > -- > > Key: CASSANDRA-14362 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14362 > Project: Cassandra > Issue Type: Bug >Reporter: Lerh Chuan Low >Assignee: Lerh Chuan Low >Priority: Major > Fix For: 4.0 > > > I'm not sure if anybody has tried using {{trunk}} and starting an actual EC2 > cluster, but with a 3 node setup that works on 3.11, it fails to work on > {{trunk}}. I mistakenly stumbled into it while testing my own code changes. > My setup is as follows: > broadcast_rpc_address: publicIP > broadcast_address: publicIP > listen_address: omitted. Ends up as privateIP. > Works on 3.11 just fine. > On {{trunk}} though, it never works. My node is never able to join the > cluster: > {code} > Apr 03 05:57:06 ip-10-0-47-122 cassandra[13914]: INFO [main] 2018-04-03 > 05:57:05,895 RangeStreamer.java:195 - Bootstrap: range > (-128373781239966537,-122439194129870521] exists on 52.88.241.181:7000 for > keyspace system_traces > Apr 03 05:57:06 ip-10-0-47-122 cassandra[13914]: INFO [main] 2018-04-03 > 05:57:05,895 RangeStreamer.java:195 - Bootstrap: range > (6968670424536541270,6973888347502882935] exists on 52.88.241.181:7000 for > keyspace system_traces > Apr 03 05:57:42 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:57:42,298 FailureDetector.java:324 - Not marking nodes down due > to local pause of 26215173446ns > 50ns > Apr 03 05:57:53 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:57:53,035 FailureDetector.java:324 - Not marking nodes down due > to local pause of 10736485907ns > 50ns > Apr 03 05:58:30 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:58:30,790 Gossiper.java:814 - Gossip stage has 28 pending > tasks; skipping status check (no nodes will be marked down) > Apr 03 05:58:33 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:58:33,060 Gossiper.java:814 - Gossip stage has 20 pending > tasks; skipping status check (no nodes will be marked down) > Apr 03 06:04:33 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 06:04:33,826 FailureDetector.java:324 - Not marking nodes down due > to local pause of 400790432954ns > 50ns > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 06:04:49,133 Gossiper.java:814 - Gossip stage has 2 pending tasks; > skipping status check (no nodes will be marked down) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: ERROR > [StreamConnectionEstablisher:1] 2018-04-03 06:04:49,138 > StreamSession.java:524 - [Stream #d4cd6420-3703-11e8-a6a5-e51ddc10cfe6] > Streaming error occurred on session with peer 52.88.241.181:7000 > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: java.io.IOException: failed > to connect to 52.88.241.181:7000 (STREAM) for streaming data > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:98) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:57) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.createChannel(NettyStreamingMessageSender.java:183) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.setupControlMessageChannel(NettyStreamingMessageSender.java:165) > Apr 03 06:04:49 ip-10-0-47-122 cassandr
[jira] [Commented] (CASSANDRA-14362) Bind to correct local address in 4.0 streaming
[ https://issues.apache.org/jira/browse/CASSANDRA-14362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16440820#comment-16440820 ] Jason Brown commented on CASSANDRA-14362: - Opened CASSANDRA-14389 for the local address binding regression. Thanks, [~djoshi3] for also looking at this ticket. > Bind to correct local address in 4.0 streaming > -- > > Key: CASSANDRA-14362 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14362 > Project: Cassandra > Issue Type: Bug >Reporter: Lerh Chuan Low >Assignee: Lerh Chuan Low >Priority: Major > Fix For: 4.0 > > > I'm not sure if anybody has tried using {{trunk}} and starting an actual EC2 > cluster, but with a 3 node setup that works on 3.11, it fails to work on > {{trunk}}. I mistakenly stumbled into it while testing my own code changes. > My setup is as follows: > broadcast_rpc_address: publicIP > broadcast_address: publicIP > listen_address: omitted. Ends up as privateIP. > Works on 3.11 just fine. > On {{trunk}} though, it never works. My node is never able to join the > cluster: > {code} > Apr 03 05:57:06 ip-10-0-47-122 cassandra[13914]: INFO [main] 2018-04-03 > 05:57:05,895 RangeStreamer.java:195 - Bootstrap: range > (-128373781239966537,-122439194129870521] exists on 52.88.241.181:7000 for > keyspace system_traces > Apr 03 05:57:06 ip-10-0-47-122 cassandra[13914]: INFO [main] 2018-04-03 > 05:57:05,895 RangeStreamer.java:195 - Bootstrap: range > (6968670424536541270,6973888347502882935] exists on 52.88.241.181:7000 for > keyspace system_traces > Apr 03 05:57:42 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:57:42,298 FailureDetector.java:324 - Not marking nodes down due > to local pause of 26215173446ns > 50ns > Apr 03 05:57:53 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:57:53,035 FailureDetector.java:324 - Not marking nodes down due > to local pause of 10736485907ns > 50ns > Apr 03 05:58:30 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:58:30,790 Gossiper.java:814 - Gossip stage has 28 pending > tasks; skipping status check (no nodes will be marked down) > Apr 03 05:58:33 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:58:33,060 Gossiper.java:814 - Gossip stage has 20 pending > tasks; skipping status check (no nodes will be marked down) > Apr 03 06:04:33 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 06:04:33,826 FailureDetector.java:324 - Not marking nodes down due > to local pause of 400790432954ns > 50ns > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 06:04:49,133 Gossiper.java:814 - Gossip stage has 2 pending tasks; > skipping status check (no nodes will be marked down) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: ERROR > [StreamConnectionEstablisher:1] 2018-04-03 06:04:49,138 > StreamSession.java:524 - [Stream #d4cd6420-3703-11e8-a6a5-e51ddc10cfe6] > Streaming error occurred on session with peer 52.88.241.181:7000 > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: java.io.IOException: failed > to connect to 52.88.241.181:7000 (STREAM) for streaming data > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:98) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:57) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.createChannel(NettyStreamingMessageSender.java:183) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.setupControlMessageChannel(NettyStreamingMessageSender.java:165) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.sendMessage(NettyStreamingMessageSender.java:222) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.initialize(NettyStreamingMessageSender.java:146) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:271) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:273) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]:
[jira] [Created] (CASSANDRA-14389) Resolve local address binding in 4.0
Jason Brown created CASSANDRA-14389: --- Summary: Resolve local address binding in 4.0 Key: CASSANDRA-14389 URL: https://issues.apache.org/jira/browse/CASSANDRA-14389 Project: Cassandra Issue Type: Bug Reporter: Jason Brown Assignee: Jason Brown Fix For: 4.x CASSANDRA-8457/CASSANDRA-12229 introduced a regression against CASSANDRA-12673. This was discovered with CASSANDRA-14362 and moved here for resolution independent of that ticket. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14362) Bind to correct local address in 4.0 streaming
[ https://issues.apache.org/jira/browse/CASSANDRA-14362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown updated CASSANDRA-14362: Reviewer: Jason Brown > Bind to correct local address in 4.0 streaming > -- > > Key: CASSANDRA-14362 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14362 > Project: Cassandra > Issue Type: Bug >Reporter: Lerh Chuan Low >Assignee: Lerh Chuan Low >Priority: Major > Fix For: 4.0 > > > I'm not sure if anybody has tried using {{trunk}} and starting an actual EC2 > cluster, but with a 3 node setup that works on 3.11, it fails to work on > {{trunk}}. I mistakenly stumbled into it while testing my own code changes. > My setup is as follows: > broadcast_rpc_address: publicIP > broadcast_address: publicIP > listen_address: omitted. Ends up as privateIP. > Works on 3.11 just fine. > On {{trunk}} though, it never works. My node is never able to join the > cluster: > {code} > Apr 03 05:57:06 ip-10-0-47-122 cassandra[13914]: INFO [main] 2018-04-03 > 05:57:05,895 RangeStreamer.java:195 - Bootstrap: range > (-128373781239966537,-122439194129870521] exists on 52.88.241.181:7000 for > keyspace system_traces > Apr 03 05:57:06 ip-10-0-47-122 cassandra[13914]: INFO [main] 2018-04-03 > 05:57:05,895 RangeStreamer.java:195 - Bootstrap: range > (6968670424536541270,6973888347502882935] exists on 52.88.241.181:7000 for > keyspace system_traces > Apr 03 05:57:42 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:57:42,298 FailureDetector.java:324 - Not marking nodes down due > to local pause of 26215173446ns > 50ns > Apr 03 05:57:53 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:57:53,035 FailureDetector.java:324 - Not marking nodes down due > to local pause of 10736485907ns > 50ns > Apr 03 05:58:30 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:58:30,790 Gossiper.java:814 - Gossip stage has 28 pending > tasks; skipping status check (no nodes will be marked down) > Apr 03 05:58:33 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:58:33,060 Gossiper.java:814 - Gossip stage has 20 pending > tasks; skipping status check (no nodes will be marked down) > Apr 03 06:04:33 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 06:04:33,826 FailureDetector.java:324 - Not marking nodes down due > to local pause of 400790432954ns > 50ns > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 06:04:49,133 Gossiper.java:814 - Gossip stage has 2 pending tasks; > skipping status check (no nodes will be marked down) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: ERROR > [StreamConnectionEstablisher:1] 2018-04-03 06:04:49,138 > StreamSession.java:524 - [Stream #d4cd6420-3703-11e8-a6a5-e51ddc10cfe6] > Streaming error occurred on session with peer 52.88.241.181:7000 > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: java.io.IOException: failed > to connect to 52.88.241.181:7000 (STREAM) for streaming data > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:98) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:57) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.createChannel(NettyStreamingMessageSender.java:183) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.setupControlMessageChannel(NettyStreamingMessageSender.java:165) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.sendMessage(NettyStreamingMessageSender.java:222) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.initialize(NettyStreamingMessageSender.java:146) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:271) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:273) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]:
[jira] [Assigned] (CASSANDRA-14362) Bind to correct local address in 4.0 streaming
[ https://issues.apache.org/jira/browse/CASSANDRA-14362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown reassigned CASSANDRA-14362: --- Assignee: Lerh Chuan Low (was: Jason Brown) > Bind to correct local address in 4.0 streaming > -- > > Key: CASSANDRA-14362 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14362 > Project: Cassandra > Issue Type: Bug >Reporter: Lerh Chuan Low >Assignee: Lerh Chuan Low >Priority: Major > Fix For: 4.0 > > > I'm not sure if anybody has tried using {{trunk}} and starting an actual EC2 > cluster, but with a 3 node setup that works on 3.11, it fails to work on > {{trunk}}. I mistakenly stumbled into it while testing my own code changes. > My setup is as follows: > broadcast_rpc_address: publicIP > broadcast_address: publicIP > listen_address: omitted. Ends up as privateIP. > Works on 3.11 just fine. > On {{trunk}} though, it never works. My node is never able to join the > cluster: > {code} > Apr 03 05:57:06 ip-10-0-47-122 cassandra[13914]: INFO [main] 2018-04-03 > 05:57:05,895 RangeStreamer.java:195 - Bootstrap: range > (-128373781239966537,-122439194129870521] exists on 52.88.241.181:7000 for > keyspace system_traces > Apr 03 05:57:06 ip-10-0-47-122 cassandra[13914]: INFO [main] 2018-04-03 > 05:57:05,895 RangeStreamer.java:195 - Bootstrap: range > (6968670424536541270,6973888347502882935] exists on 52.88.241.181:7000 for > keyspace system_traces > Apr 03 05:57:42 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:57:42,298 FailureDetector.java:324 - Not marking nodes down due > to local pause of 26215173446ns > 50ns > Apr 03 05:57:53 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:57:53,035 FailureDetector.java:324 - Not marking nodes down due > to local pause of 10736485907ns > 50ns > Apr 03 05:58:30 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:58:30,790 Gossiper.java:814 - Gossip stage has 28 pending > tasks; skipping status check (no nodes will be marked down) > Apr 03 05:58:33 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:58:33,060 Gossiper.java:814 - Gossip stage has 20 pending > tasks; skipping status check (no nodes will be marked down) > Apr 03 06:04:33 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 06:04:33,826 FailureDetector.java:324 - Not marking nodes down due > to local pause of 400790432954ns > 50ns > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 06:04:49,133 Gossiper.java:814 - Gossip stage has 2 pending tasks; > skipping status check (no nodes will be marked down) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: ERROR > [StreamConnectionEstablisher:1] 2018-04-03 06:04:49,138 > StreamSession.java:524 - [Stream #d4cd6420-3703-11e8-a6a5-e51ddc10cfe6] > Streaming error occurred on session with peer 52.88.241.181:7000 > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: java.io.IOException: failed > to connect to 52.88.241.181:7000 (STREAM) for streaming data > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:98) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:57) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.createChannel(NettyStreamingMessageSender.java:183) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.setupControlMessageChannel(NettyStreamingMessageSender.java:165) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.sendMessage(NettyStreamingMessageSender.java:222) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.initialize(NettyStreamingMessageSender.java:146) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:271) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:273) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > Apr 03 06:04:49 ip-10-0-
[jira] [Updated] (CASSANDRA-14362) Bind to correct local address in 4.0 streaming
[ https://issues.apache.org/jira/browse/CASSANDRA-14362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown updated CASSANDRA-14362: Resolution: Fixed Status: Resolved (was: Patch Available) [~Lerh Low]'s patch is correct and fixes a bug I introduced with CASSANDRA-12229. Further, there is something wrong with my handling of CASSANDRA-12673, but I don't want to hold up Lerh or anyone else looking at 4.0 in ec2. Thus, I've decided to commit Lerh's patch and will open a followup ticket to address the local address binding issue. Committed as sha {{stF4fFtT9pMYpabtyomEEFKARMeB8A7H}}. Nice find and fix, Lehr. > Bind to correct local address in 4.0 streaming > -- > > Key: CASSANDRA-14362 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14362 > Project: Cassandra > Issue Type: Bug >Reporter: Lerh Chuan Low >Assignee: Jason Brown >Priority: Major > Fix For: 4.0 > > > I'm not sure if anybody has tried using {{trunk}} and starting an actual EC2 > cluster, but with a 3 node setup that works on 3.11, it fails to work on > {{trunk}}. I mistakenly stumbled into it while testing my own code changes. > My setup is as follows: > broadcast_rpc_address: publicIP > broadcast_address: publicIP > listen_address: omitted. Ends up as privateIP. > Works on 3.11 just fine. > On {{trunk}} though, it never works. My node is never able to join the > cluster: > {code} > Apr 03 05:57:06 ip-10-0-47-122 cassandra[13914]: INFO [main] 2018-04-03 > 05:57:05,895 RangeStreamer.java:195 - Bootstrap: range > (-128373781239966537,-122439194129870521] exists on 52.88.241.181:7000 for > keyspace system_traces > Apr 03 05:57:06 ip-10-0-47-122 cassandra[13914]: INFO [main] 2018-04-03 > 05:57:05,895 RangeStreamer.java:195 - Bootstrap: range > (6968670424536541270,6973888347502882935] exists on 52.88.241.181:7000 for > keyspace system_traces > Apr 03 05:57:42 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:57:42,298 FailureDetector.java:324 - Not marking nodes down due > to local pause of 26215173446ns > 50ns > Apr 03 05:57:53 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:57:53,035 FailureDetector.java:324 - Not marking nodes down due > to local pause of 10736485907ns > 50ns > Apr 03 05:58:30 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:58:30,790 Gossiper.java:814 - Gossip stage has 28 pending > tasks; skipping status check (no nodes will be marked down) > Apr 03 05:58:33 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:58:33,060 Gossiper.java:814 - Gossip stage has 20 pending > tasks; skipping status check (no nodes will be marked down) > Apr 03 06:04:33 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 06:04:33,826 FailureDetector.java:324 - Not marking nodes down due > to local pause of 400790432954ns > 50ns > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 06:04:49,133 Gossiper.java:814 - Gossip stage has 2 pending tasks; > skipping status check (no nodes will be marked down) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: ERROR > [StreamConnectionEstablisher:1] 2018-04-03 06:04:49,138 > StreamSession.java:524 - [Stream #d4cd6420-3703-11e8-a6a5-e51ddc10cfe6] > Streaming error occurred on session with peer 52.88.241.181:7000 > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: java.io.IOException: failed > to connect to 52.88.241.181:7000 (STREAM) for streaming data > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:98) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:57) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.createChannel(NettyStreamingMessageSender.java:183) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.setupControlMessageChannel(NettyStreamingMessageSender.java:165) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.sendMessage(NettyStreamingMessageSender.java:222) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.initialize(NettyStreamingMessageSender.java:146) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:271) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13
cassandra git commit: Bind to correct local address in 4.0 streaming
Repository: cassandra Updated Branches: refs/heads/trunk 05d7661d0 -> 70d95359d Bind to correct local address in 4.0 streaming patch by Lerh Chaun Low; reviewed by jasobrown for CASSANDRA-14362 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/70d95359 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/70d95359 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/70d95359 Branch: refs/heads/trunk Commit: 70d95359d2dca1c35f573776d11ed87bb9b4b441 Parents: 05d7661 Author: Lerh Chuan Low Authored: Tue Apr 3 16:39:11 2018 +1000 Committer: Jason Brown Committed: Tue Apr 17 05:44:03 2018 -0700 -- CHANGES.txt| 1 + src/java/org/apache/cassandra/streaming/StreamSession.java | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/70d95359/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 4c557b4..5e36674 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 4.0 + * Bind to correct local address in 4.0 streaming (CASSANDRA-14362) * Use standard Amazon naming for datacenter and rack in Ec2Snitch (CASSANDRA-7839) * Fix junit failure for SSTableReaderTest (CASSANDRA-14387) * Abstract write path for pluggable storage (CASSANDRA-14118) http://git-wip-us.apache.org/repos/asf/cassandra/blob/70d95359/src/java/org/apache/cassandra/streaming/StreamSession.java -- diff --git a/src/java/org/apache/cassandra/streaming/StreamSession.java b/src/java/org/apache/cassandra/streaming/StreamSession.java index adf5d76..42d1d97 100644 --- a/src/java/org/apache/cassandra/streaming/StreamSession.java +++ b/src/java/org/apache/cassandra/streaming/StreamSession.java @@ -183,7 +183,7 @@ public class StreamSession implements IEndpointStateChangeSubscriber this.connecting = connecting; this.index = index; -OutboundConnectionIdentifier id = OutboundConnectionIdentifier.stream(InetAddressAndPort.getByAddressOverrideDefaults(FBUtilities.getBroadcastAddressAndPort().address, 0), +OutboundConnectionIdentifier id = OutboundConnectionIdentifier.stream(InetAddressAndPort.getByAddressOverrideDefaults(FBUtilities.getJustLocalAddress(), 0), InetAddressAndPort.getByAddressOverrideDefaults(connecting.address, MessagingService.instance().portFor(connecting))); this.messageSender = new NettyStreamingMessageSender(this, id, factory, StreamMessage.CURRENT_VERSION, previewKind.isPreview()); this.metrics = StreamingMetrics.get(connecting); - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14362) Bind to correct local address in 4.0 streaming
[ https://issues.apache.org/jira/browse/CASSANDRA-14362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Brown updated CASSANDRA-14362: Summary: Bind to correct local address in 4.0 streaming (was: Node unable to join cluster in trunk) > Bind to correct local address in 4.0 streaming > -- > > Key: CASSANDRA-14362 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14362 > Project: Cassandra > Issue Type: Bug >Reporter: Lerh Chuan Low >Assignee: Jason Brown >Priority: Major > Fix For: 4.0 > > > I'm not sure if anybody has tried using {{trunk}} and starting an actual EC2 > cluster, but with a 3 node setup that works on 3.11, it fails to work on > {{trunk}}. I mistakenly stumbled into it while testing my own code changes. > My setup is as follows: > broadcast_rpc_address: publicIP > broadcast_address: publicIP > listen_address: omitted. Ends up as privateIP. > Works on 3.11 just fine. > On {{trunk}} though, it never works. My node is never able to join the > cluster: > {code} > Apr 03 05:57:06 ip-10-0-47-122 cassandra[13914]: INFO [main] 2018-04-03 > 05:57:05,895 RangeStreamer.java:195 - Bootstrap: range > (-128373781239966537,-122439194129870521] exists on 52.88.241.181:7000 for > keyspace system_traces > Apr 03 05:57:06 ip-10-0-47-122 cassandra[13914]: INFO [main] 2018-04-03 > 05:57:05,895 RangeStreamer.java:195 - Bootstrap: range > (6968670424536541270,6973888347502882935] exists on 52.88.241.181:7000 for > keyspace system_traces > Apr 03 05:57:42 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:57:42,298 FailureDetector.java:324 - Not marking nodes down due > to local pause of 26215173446ns > 50ns > Apr 03 05:57:53 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:57:53,035 FailureDetector.java:324 - Not marking nodes down due > to local pause of 10736485907ns > 50ns > Apr 03 05:58:30 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:58:30,790 Gossiper.java:814 - Gossip stage has 28 pending > tasks; skipping status check (no nodes will be marked down) > Apr 03 05:58:33 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 05:58:33,060 Gossiper.java:814 - Gossip stage has 20 pending > tasks; skipping status check (no nodes will be marked down) > Apr 03 06:04:33 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 06:04:33,826 FailureDetector.java:324 - Not marking nodes down due > to local pause of 400790432954ns > 50ns > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: WARN [GossipTasks:1] > 2018-04-03 06:04:49,133 Gossiper.java:814 - Gossip stage has 2 pending tasks; > skipping status check (no nodes will be marked down) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: ERROR > [StreamConnectionEstablisher:1] 2018-04-03 06:04:49,138 > StreamSession.java:524 - [Stream #d4cd6420-3703-11e8-a6a5-e51ddc10cfe6] > Streaming error occurred on session with peer 52.88.241.181:7000 > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: java.io.IOException: failed > to connect to 52.88.241.181:7000 (STREAM) for streaming data > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:98) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:57) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.createChannel(NettyStreamingMessageSender.java:183) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.setupControlMessageChannel(NettyStreamingMessageSender.java:165) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.sendMessage(NettyStreamingMessageSender.java:222) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.async.NettyStreamingMessageSender.initialize(NettyStreamingMessageSender.java:146) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.StreamSession.start(StreamSession.java:271) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > org.apache.cassandra.streaming.StreamCoordinator$StreamSessionConnector.run(StreamCoordinator.java:273) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > Apr 03 06:04:49 ip-10-0-47-122 cassandra[13914]: at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPo
[jira] [Resolved] (CASSANDRA-14286) IndexOutOfBoundsException with SELECT JSON using IN and ORDER BY
[ https://issues.apache.org/jira/browse/CASSANDRA-14286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Lerer resolved CASSANDRA-14286. Resolution: Fixed Committed into 2.2 at 594cde7937de6f848998bac35d35591f8a18aa10 and merged into 3.0, 3.11 and trunk. > IndexOutOfBoundsException with SELECT JSON using IN and ORDER BY > > > Key: CASSANDRA-14286 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14286 > Project: Cassandra > Issue Type: Bug > Components: CQL > Environment: Kubernetes cluster using cassandra:3.11.1 Docker image. >Reporter: Szymon Acedański >Assignee: Francisco Fernandez >Priority: Major > Attachments: orderbug-traceback.txt > > > When running the following code: > {code} > public class CassandraJsonOrderingBug { > public static void main(String[] args) { > Session session = CassandraFactory.getSession(); > session.execute("CREATE TABLE thebug ( PRIMARY KEY (a, b), a INT, b > INT)"); > try { > session.execute("INSERT INTO thebug (a, b) VALUES (20, 30)"); > session.execute("INSERT INTO thebug (a, b) VALUES (100, 200)"); > Statement statement = new SimpleStatement("SELECT JSON a, b FROM > thebug WHERE a IN (20, 100) ORDER BY b"); > statement.setFetchSize(Integer.MAX_VALUE); > for (Row w: session.execute(statement)) { > System.out.println(w.toString()); > } > } finally { > session.execute("DROP TABLE thebug"); > } > } > } > {code} > The following exception is thrown server-side: > {noformat} > java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 > at java.util.Collections$SingletonList.get(Collections.java:4815) > ~[na:1.8.0_151] > at > org.apache.cassandra.cql3.statements.SelectStatement$SingleColumnComparator.compare(SelectStatement.java:1297) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at > org.apache.cassandra.cql3.statements.SelectStatement$SingleColumnComparator.compare(SelectStatement.java:1284) > ~[apache-cassandra-3.11.1.jar:3.11.1] > at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355) > ~[na:1.8.0_151] > at java.util.TimSort.sort(TimSort.java:220) ~[na:1.8.0_151] > at java.util.Arrays.sort(Arrays.java:1512) ~[na:1.8.0_151] > at java.util.ArrayList.sort(ArrayList.java:1460) ~[na:1.8.0_151] > at java.util.Collections.sort(Collections.java:175) ~[na:1.8.0_151] > {noformat} > (full traceback attached) > The accessed index is the index of the sorted column in the SELECT JSON > fields list. > Similarly, if the select clause is changed to > SELECT JSON b, a FROM thebug WHERE a IN (20, 100) ORDER BY b > then the query finishes, but the output is sorted incorrectly (by textual > JSON representation): > {noformat} > Row[{"b": 200, "a": 100}] > Row[{"b": 30, "a": 20}] > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[05/10] cassandra git commit: Merge branch cassandra-2.2 into cassandra-3.0
Merge branch cassandra-2.2 into cassandra-3.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/598008d4 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/598008d4 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/598008d4 Branch: refs/heads/cassandra-3.11 Commit: 598008d43ce88ea276974ec0f247faa0d79c Parents: 22bb413 594cde7 Author: Benjamin Lerer Authored: Tue Apr 17 12:12:48 2018 +0200 Committer: Benjamin Lerer Committed: Tue Apr 17 12:12:48 2018 +0200 -- CHANGES.txt | 2 + .../org/apache/cassandra/cql3/ResultSet.java| 5 ++ .../cassandra/cql3/selection/Selection.java | 69 +--- .../cql3/statements/SelectStatement.java| 19 +++--- .../cql3/validation/entities/JsonTest.java | 21 ++ 5 files changed, 97 insertions(+), 19 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/598008d4/CHANGES.txt -- diff --cc CHANGES.txt index 9012f8c,967ee05..d3d8036 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,20 -1,7 +1,22 @@@ -2.2.13 +3.0.17 + * Avoid deadlock when running nodetool refresh before node is fully up (CASSANDRA-14310) + * Handle all exceptions when opening sstables (CASSANDRA-14202) + * Handle incompletely written hint descriptors during startup (CASSANDRA-14080) + * Handle repeat open bound from SRP in read repair (CASSANDRA-14330) + * Use zero as default score in DynamicEndpointSnitch (CASSANDRA-14252) + * Respect max hint window when hinting for LWT (CASSANDRA-14215) + * Adding missing WriteType enum values to v3, v4, and v5 spec (CASSANDRA-13697) + * Don't regenerate bloomfilter and summaries on startup (CASSANDRA-11163) + * Fix NPE when performing comparison against a null frozen in LWT (CASSANDRA-14087) + * Log when SSTables are deleted (CASSANDRA-14302) + * Fix batch commitlog sync regression (CASSANDRA-14292) + * Write to pending endpoint when view replica is also base replica (CASSANDRA-14251) + * Chain commit log marker potential performance regression in batch commit mode (CASSANDRA-14194) + * Fully utilise specified compaction threads (CASSANDRA-14210) + * Pre-create deletion log records to finish compactions quicker (CASSANDRA-12763) +Merged from 2.2: + * Fix JSON queries with IN restrictions and ORDER BY clause (CASSANDRA-14286) + * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891) - * Fix query pager DEBUG log leak causing hit in paged reads throughput (CASSANDRA-14318) * Backport circleci yaml (CASSANDRA-14240) Merged from 2.1: * Check checksum before decompressing data (CASSANDRA-14284) http://git-wip-us.apache.org/repos/asf/cassandra/blob/598008d4/src/java/org/apache/cassandra/cql3/ResultSet.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/598008d4/src/java/org/apache/cassandra/cql3/selection/Selection.java -- diff --cc src/java/org/apache/cassandra/cql3/selection/Selection.java index 406f849,5385fc6..6227158 --- a/src/java/org/apache/cassandra/cql3/selection/Selection.java +++ b/src/java/org/apache/cassandra/cql3/selection/Selection.java @@@ -399,8 -434,17 +450,10 @@@ public abstract class Selectio sb.append(spec.type.toJSONString(buffer, protocolVersion)); } sb.append("}"); - return Collections.singletonList(UTF8Type.instance.getSerializer().serialize(sb.toString())); + List jsonRow = new ArrayList<>(); + jsonRow.add(UTF8Type.instance.getSerializer().serialize(sb.toString())); + return jsonRow; } - -private ByteBuffer value(Cell c) -{ -return (c instanceof CounterCell) -? ByteBufferUtil.bytes(CounterContext.instance().total(c.value())) -: c.value(); -} } private static interface Selectors http://git-wip-us.apache.org/repos/asf/cassandra/blob/598008d4/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java -- diff --cc src/java/org/apache/cassandra/cql3/statements/SelectStatement.java index 1e867bc,729cf83..a5e6254 --- a/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java +++ b/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java @@@ -911,14 -841,14 +911,14 @@@ public class SelectStatement implement if (!parameters.orderings.isEmpty()) { +assert !forView; verifyOrderingIsAllowed(restrictions); - orderingCo
[08/10] cassandra git commit: Merge branch cassandra-3.0 into cassandra-3.11
Merge branch cassandra-3.0 into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/95cfee62 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/95cfee62 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/95cfee62 Branch: refs/heads/cassandra-3.11 Commit: 95cfee623456f1df51ce548cc6cf42fe5a78050c Parents: 845243d 598008d Author: Benjamin Lerer Authored: Tue Apr 17 12:14:58 2018 +0200 Committer: Benjamin Lerer Committed: Tue Apr 17 12:27:25 2018 +0200 -- CHANGES.txt | 1 + .../org/apache/cassandra/cql3/ResultSet.java| 5 ++ .../cassandra/cql3/selection/Selection.java | 70 +--- .../cql3/statements/SelectStatement.java| 20 +++--- .../cql3/validation/entities/JsonTest.java | 41 +++- 5 files changed, 116 insertions(+), 21 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/95cfee62/CHANGES.txt -- diff --cc CHANGES.txt index 4c513d7,d3d8036..42ea3b4 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -25,6 -15,8 +25,7 @@@ Merged from 3.0 * Fully utilise specified compaction threads (CASSANDRA-14210) * Pre-create deletion log records to finish compactions quicker (CASSANDRA-12763) Merged from 2.2: + * Fix JSON queries with IN restrictions and ORDER BY clause (CASSANDRA-14286) - * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891) * Backport circleci yaml (CASSANDRA-14240) Merged from 2.1: * Check checksum before decompressing data (CASSANDRA-14284) http://git-wip-us.apache.org/repos/asf/cassandra/blob/95cfee62/src/java/org/apache/cassandra/cql3/ResultSet.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/95cfee62/src/java/org/apache/cassandra/cql3/selection/Selection.java -- diff --cc src/java/org/apache/cassandra/cql3/selection/Selection.java index 7b4e80c,6227158..fb0b60c --- a/src/java/org/apache/cassandra/cql3/selection/Selection.java +++ b/src/java/org/apache/cassandra/cql3/selection/Selection.java @@@ -271,41 -288,13 +311,43 @@@ public abstract class Selectio @Override public String toString() { -return Objects.toStringHelper(this) -.add("columns", columns) -.add("columnMapping", columnMapping) -.add("metadata", metadata) -.add("collectTimestamps", collectTimestamps) -.add("collectTTLs", collectTTLs) -.toString(); +return MoreObjects.toStringHelper(this) + .add("columns", columns) + .add("columnMapping", columnMapping) + .add("metadata", metadata) + .add("collectTimestamps", collectTimestamps) + .add("collectTTLs", collectTTLs) + .toString(); +} + +public static List rowToJson(List row, ProtocolVersion protocolVersion, ResultSet.ResultMetadata metadata) +{ +StringBuilder sb = new StringBuilder("{"); - for (int i = 0; i < metadata.names.size(); i++) ++for (int i = 0; i < metadata.getColumnCount(); i++) +{ +if (i > 0) +sb.append(", "); + +ColumnSpecification spec = metadata.names.get(i); +String columnName = spec.name.toString(); +if (!columnName.equals(columnName.toLowerCase(Locale.US))) +columnName = "\"" + columnName + "\""; + +ByteBuffer buffer = row.get(i); +sb.append('"'); +sb.append(Json.quoteAsJsonString(columnName)); +sb.append("\": "); +if (buffer == null) +sb.append("null"); +else if (!buffer.hasRemaining()) +sb.append("\"\""); +else +sb.append(spec.type.toJSONString(buffer, protocolVersion)); +} +sb.append("}"); - return Collections.singletonList(UTF8Type.instance.getSerializer().serialize(sb.toString())); ++List jsonRow = new ArrayList<>(); ++ jsonRow.add(UTF8Type.instance.getSerializer().serialize(sb.toString())); ++return jsonRow; } public class ResultSetBuilder @@@ -445,12 -409,51 +487,24 @@@ return resultSet; } -private List getOutputRow(int protocolVersion) +private List getOutputRow() { List outputRow = selectors.getOutputRow(protocolVersion); - return isJson ? rowToJson(outputRow, protocolVersion, m
[06/10] cassandra git commit: Merge branch cassandra-2.2 into cassandra-3.0
Merge branch cassandra-2.2 into cassandra-3.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/598008d4 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/598008d4 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/598008d4 Branch: refs/heads/cassandra-3.0 Commit: 598008d43ce88ea276974ec0f247faa0d79c Parents: 22bb413 594cde7 Author: Benjamin Lerer Authored: Tue Apr 17 12:12:48 2018 +0200 Committer: Benjamin Lerer Committed: Tue Apr 17 12:12:48 2018 +0200 -- CHANGES.txt | 2 + .../org/apache/cassandra/cql3/ResultSet.java| 5 ++ .../cassandra/cql3/selection/Selection.java | 69 +--- .../cql3/statements/SelectStatement.java| 19 +++--- .../cql3/validation/entities/JsonTest.java | 21 ++ 5 files changed, 97 insertions(+), 19 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/598008d4/CHANGES.txt -- diff --cc CHANGES.txt index 9012f8c,967ee05..d3d8036 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,20 -1,7 +1,22 @@@ -2.2.13 +3.0.17 + * Avoid deadlock when running nodetool refresh before node is fully up (CASSANDRA-14310) + * Handle all exceptions when opening sstables (CASSANDRA-14202) + * Handle incompletely written hint descriptors during startup (CASSANDRA-14080) + * Handle repeat open bound from SRP in read repair (CASSANDRA-14330) + * Use zero as default score in DynamicEndpointSnitch (CASSANDRA-14252) + * Respect max hint window when hinting for LWT (CASSANDRA-14215) + * Adding missing WriteType enum values to v3, v4, and v5 spec (CASSANDRA-13697) + * Don't regenerate bloomfilter and summaries on startup (CASSANDRA-11163) + * Fix NPE when performing comparison against a null frozen in LWT (CASSANDRA-14087) + * Log when SSTables are deleted (CASSANDRA-14302) + * Fix batch commitlog sync regression (CASSANDRA-14292) + * Write to pending endpoint when view replica is also base replica (CASSANDRA-14251) + * Chain commit log marker potential performance regression in batch commit mode (CASSANDRA-14194) + * Fully utilise specified compaction threads (CASSANDRA-14210) + * Pre-create deletion log records to finish compactions quicker (CASSANDRA-12763) +Merged from 2.2: + * Fix JSON queries with IN restrictions and ORDER BY clause (CASSANDRA-14286) + * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891) - * Fix query pager DEBUG log leak causing hit in paged reads throughput (CASSANDRA-14318) * Backport circleci yaml (CASSANDRA-14240) Merged from 2.1: * Check checksum before decompressing data (CASSANDRA-14284) http://git-wip-us.apache.org/repos/asf/cassandra/blob/598008d4/src/java/org/apache/cassandra/cql3/ResultSet.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/598008d4/src/java/org/apache/cassandra/cql3/selection/Selection.java -- diff --cc src/java/org/apache/cassandra/cql3/selection/Selection.java index 406f849,5385fc6..6227158 --- a/src/java/org/apache/cassandra/cql3/selection/Selection.java +++ b/src/java/org/apache/cassandra/cql3/selection/Selection.java @@@ -399,8 -434,17 +450,10 @@@ public abstract class Selectio sb.append(spec.type.toJSONString(buffer, protocolVersion)); } sb.append("}"); - return Collections.singletonList(UTF8Type.instance.getSerializer().serialize(sb.toString())); + List jsonRow = new ArrayList<>(); + jsonRow.add(UTF8Type.instance.getSerializer().serialize(sb.toString())); + return jsonRow; } - -private ByteBuffer value(Cell c) -{ -return (c instanceof CounterCell) -? ByteBufferUtil.bytes(CounterContext.instance().total(c.value())) -: c.value(); -} } private static interface Selectors http://git-wip-us.apache.org/repos/asf/cassandra/blob/598008d4/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java -- diff --cc src/java/org/apache/cassandra/cql3/statements/SelectStatement.java index 1e867bc,729cf83..a5e6254 --- a/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java +++ b/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java @@@ -911,14 -841,14 +911,14 @@@ public class SelectStatement implement if (!parameters.orderings.isEmpty()) { +assert !forView; verifyOrderingIsAllowed(restrictions); - orderingCom
[02/10] cassandra git commit: Fix JSON queries with IN restrictions and ORDER BY clause
Fix JSON queries with IN restrictions and ORDER BY clause patch by Francisco Fernandez; reviewed by Benjamin Lerer for CASSANDRA-14286 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/594cde79 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/594cde79 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/594cde79 Branch: refs/heads/cassandra-3.0 Commit: 594cde7937de6f848998bac35d35591f8a18aa10 Parents: b3ac793 Author: Francisco Fernandez Authored: Tue Apr 17 12:02:25 2018 +0200 Committer: Benjamin Lerer Committed: Tue Apr 17 12:04:33 2018 +0200 -- CHANGES.txt | 1 + .../org/apache/cassandra/cql3/ResultSet.java| 5 ++ .../cassandra/cql3/selection/Selection.java | 69 +--- .../cql3/statements/SelectStatement.java| 19 +++--- .../cql3/validation/entities/JsonTest.java | 21 ++ 5 files changed, 96 insertions(+), 19 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/594cde79/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 5221b1e..967ee05 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.2.13 + * Fix JSON queries with IN restrictions and ORDER BY clause (CASSANDRA-14286) * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891) * Fix query pager DEBUG log leak causing hit in paged reads throughput (CASSANDRA-14318) * Backport circleci yaml (CASSANDRA-14240) http://git-wip-us.apache.org/repos/asf/cassandra/blob/594cde79/src/java/org/apache/cassandra/cql3/ResultSet.java -- diff --git a/src/java/org/apache/cassandra/cql3/ResultSet.java b/src/java/org/apache/cassandra/cql3/ResultSet.java index 028691f..16f0d1b 100644 --- a/src/java/org/apache/cassandra/cql3/ResultSet.java +++ b/src/java/org/apache/cassandra/cql3/ResultSet.java @@ -254,6 +254,11 @@ public class ResultSet return new ResultMetadata(EnumSet.copyOf(flags), names, columnCount, pagingState); } +public int getColumnCount() +{ +return columnCount; +} + /** * Return only the column names requested by the user, excluding those added for post-query re-orderings, * see definition of names and columnCount. http://git-wip-us.apache.org/repos/asf/cassandra/blob/594cde79/src/java/org/apache/cassandra/cql3/selection/Selection.java -- diff --git a/src/java/org/apache/cassandra/cql3/selection/Selection.java b/src/java/org/apache/cassandra/cql3/selection/Selection.java index dc4e240..5385fc6 100644 --- a/src/java/org/apache/cassandra/cql3/selection/Selection.java +++ b/src/java/org/apache/cassandra/cql3/selection/Selection.java @@ -24,6 +24,7 @@ import com.google.common.base.Objects; import com.google.common.base.Predicate; import com.google.common.collect.Iterables; import com.google.common.collect.Iterators; +import com.google.common.collect.Lists; import org.apache.cassandra.config.CFMetaData; import org.apache.cassandra.config.ColumnDefinition; @@ -56,6 +57,8 @@ public abstract class Selection private final ResultSet.ResultMetadata metadata; private final boolean collectTimestamps; private final boolean collectTTLs; +// Columns used to order the result set for multi-partition queries +private Map orderingIndex; protected Selection(CFMetaData cfm, List columns, @@ -130,6 +133,24 @@ public abstract class Selection return false; } +public Map getOrderingIndex(boolean isJson) +{ +if (!isJson) +return orderingIndex; + +// If we order post-query in json, the first and only column that we ship to the client is the json column. +// In that case, we should keep ordering columns around to perform the ordering, then these columns will +// be placed after the json column. As a consequence of where the colums are placed, we should give the +// ordering index a value based on their position in the json encoding and discard the original index. +// (CASSANDRA-14286) +int columnIndex = 1; +Map jsonOrderingIndex = new LinkedHashMap<>(orderingIndex.size()); +for (ColumnDefinition column : orderingIndex.keySet()) +jsonOrderingIndex.put(column, columnIndex++); + +return jsonOrderingIndex; +} + public ResultSet.ResultMetadata getResultMetadata(boolean isJson) { if (!isJson) @@ -137,7 +158,13 @@ public abstract class Selection ColumnSpecification firstColumn = metadata.names.get(0); ColumnS
[09/10] cassandra git commit: Merge branch cassandra-3.0 into cassandra-3.11
Merge branch cassandra-3.0 into cassandra-3.11 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/95cfee62 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/95cfee62 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/95cfee62 Branch: refs/heads/trunk Commit: 95cfee623456f1df51ce548cc6cf42fe5a78050c Parents: 845243d 598008d Author: Benjamin Lerer Authored: Tue Apr 17 12:14:58 2018 +0200 Committer: Benjamin Lerer Committed: Tue Apr 17 12:27:25 2018 +0200 -- CHANGES.txt | 1 + .../org/apache/cassandra/cql3/ResultSet.java| 5 ++ .../cassandra/cql3/selection/Selection.java | 70 +--- .../cql3/statements/SelectStatement.java| 20 +++--- .../cql3/validation/entities/JsonTest.java | 41 +++- 5 files changed, 116 insertions(+), 21 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/95cfee62/CHANGES.txt -- diff --cc CHANGES.txt index 4c513d7,d3d8036..42ea3b4 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -25,6 -15,8 +25,7 @@@ Merged from 3.0 * Fully utilise specified compaction threads (CASSANDRA-14210) * Pre-create deletion log records to finish compactions quicker (CASSANDRA-12763) Merged from 2.2: + * Fix JSON queries with IN restrictions and ORDER BY clause (CASSANDRA-14286) - * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891) * Backport circleci yaml (CASSANDRA-14240) Merged from 2.1: * Check checksum before decompressing data (CASSANDRA-14284) http://git-wip-us.apache.org/repos/asf/cassandra/blob/95cfee62/src/java/org/apache/cassandra/cql3/ResultSet.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/95cfee62/src/java/org/apache/cassandra/cql3/selection/Selection.java -- diff --cc src/java/org/apache/cassandra/cql3/selection/Selection.java index 7b4e80c,6227158..fb0b60c --- a/src/java/org/apache/cassandra/cql3/selection/Selection.java +++ b/src/java/org/apache/cassandra/cql3/selection/Selection.java @@@ -271,41 -288,13 +311,43 @@@ public abstract class Selectio @Override public String toString() { -return Objects.toStringHelper(this) -.add("columns", columns) -.add("columnMapping", columnMapping) -.add("metadata", metadata) -.add("collectTimestamps", collectTimestamps) -.add("collectTTLs", collectTTLs) -.toString(); +return MoreObjects.toStringHelper(this) + .add("columns", columns) + .add("columnMapping", columnMapping) + .add("metadata", metadata) + .add("collectTimestamps", collectTimestamps) + .add("collectTTLs", collectTTLs) + .toString(); +} + +public static List rowToJson(List row, ProtocolVersion protocolVersion, ResultSet.ResultMetadata metadata) +{ +StringBuilder sb = new StringBuilder("{"); - for (int i = 0; i < metadata.names.size(); i++) ++for (int i = 0; i < metadata.getColumnCount(); i++) +{ +if (i > 0) +sb.append(", "); + +ColumnSpecification spec = metadata.names.get(i); +String columnName = spec.name.toString(); +if (!columnName.equals(columnName.toLowerCase(Locale.US))) +columnName = "\"" + columnName + "\""; + +ByteBuffer buffer = row.get(i); +sb.append('"'); +sb.append(Json.quoteAsJsonString(columnName)); +sb.append("\": "); +if (buffer == null) +sb.append("null"); +else if (!buffer.hasRemaining()) +sb.append("\"\""); +else +sb.append(spec.type.toJSONString(buffer, protocolVersion)); +} +sb.append("}"); - return Collections.singletonList(UTF8Type.instance.getSerializer().serialize(sb.toString())); ++List jsonRow = new ArrayList<>(); ++ jsonRow.add(UTF8Type.instance.getSerializer().serialize(sb.toString())); ++return jsonRow; } public class ResultSetBuilder @@@ -445,12 -409,51 +487,24 @@@ return resultSet; } -private List getOutputRow(int protocolVersion) +private List getOutputRow() { List outputRow = selectors.getOutputRow(protocolVersion); - return isJson ? rowToJson(outputRow, protocolVersion, metadata)
[07/10] cassandra git commit: Merge branch cassandra-2.2 into cassandra-3.0
Merge branch cassandra-2.2 into cassandra-3.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/598008d4 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/598008d4 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/598008d4 Branch: refs/heads/trunk Commit: 598008d43ce88ea276974ec0f247faa0d79c Parents: 22bb413 594cde7 Author: Benjamin Lerer Authored: Tue Apr 17 12:12:48 2018 +0200 Committer: Benjamin Lerer Committed: Tue Apr 17 12:12:48 2018 +0200 -- CHANGES.txt | 2 + .../org/apache/cassandra/cql3/ResultSet.java| 5 ++ .../cassandra/cql3/selection/Selection.java | 69 +--- .../cql3/statements/SelectStatement.java| 19 +++--- .../cql3/validation/entities/JsonTest.java | 21 ++ 5 files changed, 97 insertions(+), 19 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/598008d4/CHANGES.txt -- diff --cc CHANGES.txt index 9012f8c,967ee05..d3d8036 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,20 -1,7 +1,22 @@@ -2.2.13 +3.0.17 + * Avoid deadlock when running nodetool refresh before node is fully up (CASSANDRA-14310) + * Handle all exceptions when opening sstables (CASSANDRA-14202) + * Handle incompletely written hint descriptors during startup (CASSANDRA-14080) + * Handle repeat open bound from SRP in read repair (CASSANDRA-14330) + * Use zero as default score in DynamicEndpointSnitch (CASSANDRA-14252) + * Respect max hint window when hinting for LWT (CASSANDRA-14215) + * Adding missing WriteType enum values to v3, v4, and v5 spec (CASSANDRA-13697) + * Don't regenerate bloomfilter and summaries on startup (CASSANDRA-11163) + * Fix NPE when performing comparison against a null frozen in LWT (CASSANDRA-14087) + * Log when SSTables are deleted (CASSANDRA-14302) + * Fix batch commitlog sync regression (CASSANDRA-14292) + * Write to pending endpoint when view replica is also base replica (CASSANDRA-14251) + * Chain commit log marker potential performance regression in batch commit mode (CASSANDRA-14194) + * Fully utilise specified compaction threads (CASSANDRA-14210) + * Pre-create deletion log records to finish compactions quicker (CASSANDRA-12763) +Merged from 2.2: + * Fix JSON queries with IN restrictions and ORDER BY clause (CASSANDRA-14286) + * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891) - * Fix query pager DEBUG log leak causing hit in paged reads throughput (CASSANDRA-14318) * Backport circleci yaml (CASSANDRA-14240) Merged from 2.1: * Check checksum before decompressing data (CASSANDRA-14284) http://git-wip-us.apache.org/repos/asf/cassandra/blob/598008d4/src/java/org/apache/cassandra/cql3/ResultSet.java -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/598008d4/src/java/org/apache/cassandra/cql3/selection/Selection.java -- diff --cc src/java/org/apache/cassandra/cql3/selection/Selection.java index 406f849,5385fc6..6227158 --- a/src/java/org/apache/cassandra/cql3/selection/Selection.java +++ b/src/java/org/apache/cassandra/cql3/selection/Selection.java @@@ -399,8 -434,17 +450,10 @@@ public abstract class Selectio sb.append(spec.type.toJSONString(buffer, protocolVersion)); } sb.append("}"); - return Collections.singletonList(UTF8Type.instance.getSerializer().serialize(sb.toString())); + List jsonRow = new ArrayList<>(); + jsonRow.add(UTF8Type.instance.getSerializer().serialize(sb.toString())); + return jsonRow; } - -private ByteBuffer value(Cell c) -{ -return (c instanceof CounterCell) -? ByteBufferUtil.bytes(CounterContext.instance().total(c.value())) -: c.value(); -} } private static interface Selectors http://git-wip-us.apache.org/repos/asf/cassandra/blob/598008d4/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java -- diff --cc src/java/org/apache/cassandra/cql3/statements/SelectStatement.java index 1e867bc,729cf83..a5e6254 --- a/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java +++ b/src/java/org/apache/cassandra/cql3/statements/SelectStatement.java @@@ -911,14 -841,14 +911,14 @@@ public class SelectStatement implement if (!parameters.orderings.isEmpty()) { +assert !forView; verifyOrderingIsAllowed(restrictions); - orderingComparator
[03/10] cassandra git commit: Fix JSON queries with IN restrictions and ORDER BY clause
Fix JSON queries with IN restrictions and ORDER BY clause patch by Francisco Fernandez; reviewed by Benjamin Lerer for CASSANDRA-14286 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/594cde79 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/594cde79 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/594cde79 Branch: refs/heads/cassandra-3.11 Commit: 594cde7937de6f848998bac35d35591f8a18aa10 Parents: b3ac793 Author: Francisco Fernandez Authored: Tue Apr 17 12:02:25 2018 +0200 Committer: Benjamin Lerer Committed: Tue Apr 17 12:04:33 2018 +0200 -- CHANGES.txt | 1 + .../org/apache/cassandra/cql3/ResultSet.java| 5 ++ .../cassandra/cql3/selection/Selection.java | 69 +--- .../cql3/statements/SelectStatement.java| 19 +++--- .../cql3/validation/entities/JsonTest.java | 21 ++ 5 files changed, 96 insertions(+), 19 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/594cde79/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 5221b1e..967ee05 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.2.13 + * Fix JSON queries with IN restrictions and ORDER BY clause (CASSANDRA-14286) * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891) * Fix query pager DEBUG log leak causing hit in paged reads throughput (CASSANDRA-14318) * Backport circleci yaml (CASSANDRA-14240) http://git-wip-us.apache.org/repos/asf/cassandra/blob/594cde79/src/java/org/apache/cassandra/cql3/ResultSet.java -- diff --git a/src/java/org/apache/cassandra/cql3/ResultSet.java b/src/java/org/apache/cassandra/cql3/ResultSet.java index 028691f..16f0d1b 100644 --- a/src/java/org/apache/cassandra/cql3/ResultSet.java +++ b/src/java/org/apache/cassandra/cql3/ResultSet.java @@ -254,6 +254,11 @@ public class ResultSet return new ResultMetadata(EnumSet.copyOf(flags), names, columnCount, pagingState); } +public int getColumnCount() +{ +return columnCount; +} + /** * Return only the column names requested by the user, excluding those added for post-query re-orderings, * see definition of names and columnCount. http://git-wip-us.apache.org/repos/asf/cassandra/blob/594cde79/src/java/org/apache/cassandra/cql3/selection/Selection.java -- diff --git a/src/java/org/apache/cassandra/cql3/selection/Selection.java b/src/java/org/apache/cassandra/cql3/selection/Selection.java index dc4e240..5385fc6 100644 --- a/src/java/org/apache/cassandra/cql3/selection/Selection.java +++ b/src/java/org/apache/cassandra/cql3/selection/Selection.java @@ -24,6 +24,7 @@ import com.google.common.base.Objects; import com.google.common.base.Predicate; import com.google.common.collect.Iterables; import com.google.common.collect.Iterators; +import com.google.common.collect.Lists; import org.apache.cassandra.config.CFMetaData; import org.apache.cassandra.config.ColumnDefinition; @@ -56,6 +57,8 @@ public abstract class Selection private final ResultSet.ResultMetadata metadata; private final boolean collectTimestamps; private final boolean collectTTLs; +// Columns used to order the result set for multi-partition queries +private Map orderingIndex; protected Selection(CFMetaData cfm, List columns, @@ -130,6 +133,24 @@ public abstract class Selection return false; } +public Map getOrderingIndex(boolean isJson) +{ +if (!isJson) +return orderingIndex; + +// If we order post-query in json, the first and only column that we ship to the client is the json column. +// In that case, we should keep ordering columns around to perform the ordering, then these columns will +// be placed after the json column. As a consequence of where the colums are placed, we should give the +// ordering index a value based on their position in the json encoding and discard the original index. +// (CASSANDRA-14286) +int columnIndex = 1; +Map jsonOrderingIndex = new LinkedHashMap<>(orderingIndex.size()); +for (ColumnDefinition column : orderingIndex.keySet()) +jsonOrderingIndex.put(column, columnIndex++); + +return jsonOrderingIndex; +} + public ResultSet.ResultMetadata getResultMetadata(boolean isJson) { if (!isJson) @@ -137,7 +158,13 @@ public abstract class Selection ColumnSpecification firstColumn = metadata.names.get(0); Column
[04/10] cassandra git commit: Fix JSON queries with IN restrictions and ORDER BY clause
Fix JSON queries with IN restrictions and ORDER BY clause patch by Francisco Fernandez; reviewed by Benjamin Lerer for CASSANDRA-14286 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/594cde79 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/594cde79 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/594cde79 Branch: refs/heads/trunk Commit: 594cde7937de6f848998bac35d35591f8a18aa10 Parents: b3ac793 Author: Francisco Fernandez Authored: Tue Apr 17 12:02:25 2018 +0200 Committer: Benjamin Lerer Committed: Tue Apr 17 12:04:33 2018 +0200 -- CHANGES.txt | 1 + .../org/apache/cassandra/cql3/ResultSet.java| 5 ++ .../cassandra/cql3/selection/Selection.java | 69 +--- .../cql3/statements/SelectStatement.java| 19 +++--- .../cql3/validation/entities/JsonTest.java | 21 ++ 5 files changed, 96 insertions(+), 19 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/594cde79/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 5221b1e..967ee05 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.2.13 + * Fix JSON queries with IN restrictions and ORDER BY clause (CASSANDRA-14286) * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891) * Fix query pager DEBUG log leak causing hit in paged reads throughput (CASSANDRA-14318) * Backport circleci yaml (CASSANDRA-14240) http://git-wip-us.apache.org/repos/asf/cassandra/blob/594cde79/src/java/org/apache/cassandra/cql3/ResultSet.java -- diff --git a/src/java/org/apache/cassandra/cql3/ResultSet.java b/src/java/org/apache/cassandra/cql3/ResultSet.java index 028691f..16f0d1b 100644 --- a/src/java/org/apache/cassandra/cql3/ResultSet.java +++ b/src/java/org/apache/cassandra/cql3/ResultSet.java @@ -254,6 +254,11 @@ public class ResultSet return new ResultMetadata(EnumSet.copyOf(flags), names, columnCount, pagingState); } +public int getColumnCount() +{ +return columnCount; +} + /** * Return only the column names requested by the user, excluding those added for post-query re-orderings, * see definition of names and columnCount. http://git-wip-us.apache.org/repos/asf/cassandra/blob/594cde79/src/java/org/apache/cassandra/cql3/selection/Selection.java -- diff --git a/src/java/org/apache/cassandra/cql3/selection/Selection.java b/src/java/org/apache/cassandra/cql3/selection/Selection.java index dc4e240..5385fc6 100644 --- a/src/java/org/apache/cassandra/cql3/selection/Selection.java +++ b/src/java/org/apache/cassandra/cql3/selection/Selection.java @@ -24,6 +24,7 @@ import com.google.common.base.Objects; import com.google.common.base.Predicate; import com.google.common.collect.Iterables; import com.google.common.collect.Iterators; +import com.google.common.collect.Lists; import org.apache.cassandra.config.CFMetaData; import org.apache.cassandra.config.ColumnDefinition; @@ -56,6 +57,8 @@ public abstract class Selection private final ResultSet.ResultMetadata metadata; private final boolean collectTimestamps; private final boolean collectTTLs; +// Columns used to order the result set for multi-partition queries +private Map orderingIndex; protected Selection(CFMetaData cfm, List columns, @@ -130,6 +133,24 @@ public abstract class Selection return false; } +public Map getOrderingIndex(boolean isJson) +{ +if (!isJson) +return orderingIndex; + +// If we order post-query in json, the first and only column that we ship to the client is the json column. +// In that case, we should keep ordering columns around to perform the ordering, then these columns will +// be placed after the json column. As a consequence of where the colums are placed, we should give the +// ordering index a value based on their position in the json encoding and discard the original index. +// (CASSANDRA-14286) +int columnIndex = 1; +Map jsonOrderingIndex = new LinkedHashMap<>(orderingIndex.size()); +for (ColumnDefinition column : orderingIndex.keySet()) +jsonOrderingIndex.put(column, columnIndex++); + +return jsonOrderingIndex; +} + public ResultSet.ResultMetadata getResultMetadata(boolean isJson) { if (!isJson) @@ -137,7 +158,13 @@ public abstract class Selection ColumnSpecification firstColumn = metadata.names.get(0); ColumnSpecifica
[10/10] cassandra git commit: Merge branch cassandra-3.11 into trunk
Merge branch cassandra-3.11 into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/05d7661d Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/05d7661d Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/05d7661d Branch: refs/heads/trunk Commit: 05d7661d04b2cd47b29a767986d4b6d283419db2 Parents: 09f3c96 95cfee6 Author: Benjamin Lerer Authored: Tue Apr 17 12:28:16 2018 +0200 Committer: Benjamin Lerer Committed: Tue Apr 17 12:30:53 2018 +0200 -- CHANGES.txt | 6 +- .../cassandra/cql3/selection/Selection.java | 66 +--- .../cql3/statements/SelectStatement.java| 23 +-- .../cql3/validation/entities/JsonTest.java | 39 4 files changed, 103 insertions(+), 31 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/05d7661d/CHANGES.txt -- diff --cc CHANGES.txt index 71d947a,42ea3b4..4c557b4 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,229 -1,7 +1,228 @@@ +4.0 + * Use standard Amazon naming for datacenter and rack in Ec2Snitch (CASSANDRA-7839) + * Fix junit failure for SSTableReaderTest (CASSANDRA-14387) + * Abstract write path for pluggable storage (CASSANDRA-14118) + * nodetool describecluster should be more informative (CASSANDRA-13853) + * Compaction performance improvements (CASSANDRA-14261) + * Refactor Pair usage to avoid boxing ints/longs (CASSANDRA-14260) + * Add options to nodetool tablestats to sort and limit output (CASSANDRA-13889) + * Rename internals to reflect CQL vocabulary (CASSANDRA-14354) + * Add support for hybrid MIN(), MAX() speculative retry policies + (CASSANDRA-14293, CASSANDRA-14338, CASSANDRA-14352) + * Fix some regressions caused by 14058 (CASSANDRA-14353) + * Abstract repair for pluggable storage (CASSANDRA-14116) + * Add meaningful toString() impls (CASSANDRA-13653) + * Add sstableloader option to accept target keyspace name (CASSANDRA-13884) + * Move processing of EchoMessage response to gossip stage (CASSANDRA-13713) + * Add coordinator write metric per CF (CASSANDRA-14232) + * Correct and clarify SSLFactory.getSslContext method and call sites (CASSANDRA-14314) + * Handle static and partition deletion properly on ThrottledUnfilteredIterator (CASSANDRA-14315) + * NodeTool clientstats should show SSL Cipher (CASSANDRA-14322) + * Add ability to specify driver name and version (CASSANDRA-14275) + * Abstract streaming for pluggable storage (CASSANDRA-14115) + * Forced incremental repairs should promote sstables if they can (CASSANDRA-14294) + * Use Murmur3 for validation compactions (CASSANDRA-14002) + * Comma at the end of the seed list is interpretated as localhost (CASSANDRA-14285) + * Refactor read executor and response resolver, abstract read repair (CASSANDRA-14058) + * Add optional startup delay to wait until peers are ready (CASSANDRA-13993) + * Add a few options to nodetool verify (CASSANDRA-14201) + * CVE-2017-5929 Security vulnerability and redefine default log rotation policy (CASSANDRA-14183) + * Use JVM default SSL validation algorithm instead of custom default (CASSANDRA-13259) + * Better document in code InetAddressAndPort usage post 7544, incorporate port into UUIDGen node (CASSANDRA-14226) + * Fix sstablemetadata date string for minLocalDeletionTime (CASSANDRA-14132) + * Make it possible to change neverPurgeTombstones during runtime (CASSANDRA-14214) + * Remove GossipDigestSynVerbHandler#doSort() (CASSANDRA-14174) + * Add nodetool clientlist (CASSANDRA-13665) + * Revert ProtocolVersion changes from CASSANDRA-7544 (CASSANDRA-14211) + * Non-disruptive seed node list reload (CASSANDRA-14190) + * Nodetool tablehistograms to print statics for all the tables (CASSANDRA-14185) + * Migrate dtests to use pytest and python3 (CASSANDRA-14134) + * Allow storage port to be configurable per node (CASSANDRA-7544) + * Make sub-range selection for non-frozen collections return null instead of empty (CASSANDRA-14182) + * BloomFilter serialization format should not change byte ordering (CASSANDRA-9067) + * Remove unused on-heap BloomFilter implementation (CASSANDRA-14152) + * Delete temp test files on exit (CASSANDRA-14153) + * Make PartitionUpdate and Mutation immutable (CASSANDRA-13867) + * Fix CommitLogReplayer exception for CDC data (CASSANDRA-14066) + * Fix cassandra-stress startup failure (CASSANDRA-14106) + * Remove initialDirectories from CFS (CASSANDRA-13928) + * Fix trivial log format error (CASSANDRA-14015) + * Allow sstabledump to do a json object per partition (CASSANDRA-13848) + * Add option to optimise merkle tree comparison across replicas (CASSANDRA-3200) + * Remove unused and deprecated methods from
[01/10] cassandra git commit: Fix JSON queries with IN restrictions and ORDER BY clause
Repository: cassandra Updated Branches: refs/heads/cassandra-2.2 b3ac7937e -> 594cde793 refs/heads/cassandra-3.0 22bb413ba -> 598008d43 refs/heads/cassandra-3.11 845243db9 -> 95cfee623 refs/heads/trunk 09f3c969b -> 05d7661d0 Fix JSON queries with IN restrictions and ORDER BY clause patch by Francisco Fernandez; reviewed by Benjamin Lerer for CASSANDRA-14286 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/594cde79 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/594cde79 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/594cde79 Branch: refs/heads/cassandra-2.2 Commit: 594cde7937de6f848998bac35d35591f8a18aa10 Parents: b3ac793 Author: Francisco Fernandez Authored: Tue Apr 17 12:02:25 2018 +0200 Committer: Benjamin Lerer Committed: Tue Apr 17 12:04:33 2018 +0200 -- CHANGES.txt | 1 + .../org/apache/cassandra/cql3/ResultSet.java| 5 ++ .../cassandra/cql3/selection/Selection.java | 69 +--- .../cql3/statements/SelectStatement.java| 19 +++--- .../cql3/validation/entities/JsonTest.java | 21 ++ 5 files changed, 96 insertions(+), 19 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/594cde79/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 5221b1e..967ee05 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,4 +1,5 @@ 2.2.13 + * Fix JSON queries with IN restrictions and ORDER BY clause (CASSANDRA-14286) * CQL fromJson(null) throws NullPointerException (CASSANDRA-13891) * Fix query pager DEBUG log leak causing hit in paged reads throughput (CASSANDRA-14318) * Backport circleci yaml (CASSANDRA-14240) http://git-wip-us.apache.org/repos/asf/cassandra/blob/594cde79/src/java/org/apache/cassandra/cql3/ResultSet.java -- diff --git a/src/java/org/apache/cassandra/cql3/ResultSet.java b/src/java/org/apache/cassandra/cql3/ResultSet.java index 028691f..16f0d1b 100644 --- a/src/java/org/apache/cassandra/cql3/ResultSet.java +++ b/src/java/org/apache/cassandra/cql3/ResultSet.java @@ -254,6 +254,11 @@ public class ResultSet return new ResultMetadata(EnumSet.copyOf(flags), names, columnCount, pagingState); } +public int getColumnCount() +{ +return columnCount; +} + /** * Return only the column names requested by the user, excluding those added for post-query re-orderings, * see definition of names and columnCount. http://git-wip-us.apache.org/repos/asf/cassandra/blob/594cde79/src/java/org/apache/cassandra/cql3/selection/Selection.java -- diff --git a/src/java/org/apache/cassandra/cql3/selection/Selection.java b/src/java/org/apache/cassandra/cql3/selection/Selection.java index dc4e240..5385fc6 100644 --- a/src/java/org/apache/cassandra/cql3/selection/Selection.java +++ b/src/java/org/apache/cassandra/cql3/selection/Selection.java @@ -24,6 +24,7 @@ import com.google.common.base.Objects; import com.google.common.base.Predicate; import com.google.common.collect.Iterables; import com.google.common.collect.Iterators; +import com.google.common.collect.Lists; import org.apache.cassandra.config.CFMetaData; import org.apache.cassandra.config.ColumnDefinition; @@ -56,6 +57,8 @@ public abstract class Selection private final ResultSet.ResultMetadata metadata; private final boolean collectTimestamps; private final boolean collectTTLs; +// Columns used to order the result set for multi-partition queries +private Map orderingIndex; protected Selection(CFMetaData cfm, List columns, @@ -130,6 +133,24 @@ public abstract class Selection return false; } +public Map getOrderingIndex(boolean isJson) +{ +if (!isJson) +return orderingIndex; + +// If we order post-query in json, the first and only column that we ship to the client is the json column. +// In that case, we should keep ordering columns around to perform the ordering, then these columns will +// be placed after the json column. As a consequence of where the colums are placed, we should give the +// ordering index a value based on their position in the json encoding and discard the original index. +// (CASSANDRA-14286) +int columnIndex = 1; +Map jsonOrderingIndex = new LinkedHashMap<>(orderingIndex.size()); +for (ColumnDefinition column : orderingIndex.keySet()) +jsonOrderingIndex.put(column, columnIndex++); + +return jsonOrderingIndex; +} +