date:20170206

[jira] [Updated] (CASSANDRA-11105) cassandra-stress tool - InvalidQueryException: Batch too large

2017-02-06 Thread Alexander Dejanovski (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Dejanovski updated CASSANDRA-11105:
-
Attachment: (was: 11105-trunk.txt)

> cassandra-stress tool - InvalidQueryException: Batch too large
> --
>
> Key: CASSANDRA-11105
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11105
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
> Environment: Cassandra 2.2.4, Java 8, CentOS 6.5
>Reporter: Ralf Steppacher
> Fix For: 4.0
>
> Attachments: 11105-trunk.txt, batch_too_large.yaml
>
>
> I am using Cassandra 2.2.4 and I am struggling to get the cassandra-stress 
> tool to work for my test scenario. I have followed the example on 
> http://www.datastax.com/dev/blog/improved-cassandra-2-1-stress-tool-benchmark-any-schema
>  to create a yaml file describing my test (attached).
> I am collecting events per user id (text, partition key). Events have a 
> session type (text), event type (text), and creation time (timestamp) 
> (clustering keys, in that order). Plus some more attributes required for 
> rendering the events in a UI. For testing purposes I ended up with the 
> following column spec and insert distribution:
> {noformat}
> columnspec:
>   - name: created_at
> cluster: uniform(10..1)
>   - name: event_type
> size: uniform(5..10)
> population: uniform(1..30)
> cluster: uniform(1..30)
>   - name: session_type
> size: fixed(5)
> population: uniform(1..4)
> cluster: uniform(1..4)
>   - name: user_id
> size: fixed(15)
> population: uniform(1..100)
>   - name: message
> size: uniform(10..100)
> population: uniform(1..100B)
> insert:
>   partitions: fixed(1)
>   batchtype: UNLOGGED
>   select: fixed(1)/120
> {noformat}
> Running stress tool for just the insert prints 
> {noformat}
> Generating batches with [1..1] partitions and [0..1] rows (of [10..120] 
> total rows in the partitions)
> {noformat}
> and then immediately starts flooding me with 
> {{com.datastax.driver.core.exceptions.InvalidQueryException: Batch too 
> large}}. 
> Why I should be exceeding the {{batch_size_fail_threshold_in_kb: 50}} in the 
> {{cassandra.yaml}} I do not understand. My understanding is that the stress 
> tool should generate one row per batch. The size of a single row should not 
> exceed {{8+10*3+5*3+15*3+100*3 = 398 bytes}}. Assuming a worst case of all 
> text characters being 3 byte unicode characters. 
> This is how I start the attached user scenario:
> {noformat}
> [rsteppac@centos bin]$ ./cassandra-stress user 
> profile=../batch_too_large.yaml ops\(insert=1\) -log level=verbose 
> file=~/centos_event_by_patient_session_event_timestamp_insert_only.log -node 
> 10.211.55.8
> INFO  08:00:07 Did not find Netty's native epoll transport in the classpath, 
> defaulting to NIO.
> INFO  08:00:08 Using data-center name 'datacenter1' for 
> DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct 
> datacenter name with DCAwareRoundRobinPolicy constructor)
> INFO  08:00:08 New Cassandra host /10.211.55.8:9042 added
> Connected to cluster: Titan_DEV
> Datatacenter: datacenter1; Host: /10.211.55.8; Rack: rack1
> Created schema. Sleeping 1s for propagation.
> Generating batches with [1..1] partitions and [0..1] rows (of [10..120] 
> total rows in the partitions)
> com.datastax.driver.core.exceptions.InvalidQueryException: Batch too large
>   at 
> com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:35)
>   at 
> com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:271)
>   at 
> com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:185)
>   at 
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:55)
>   at 
> org.apache.cassandra.stress.operations.userdefined.SchemaInsert$JavaDriverRun.run(SchemaInsert.java:87)
>   at 
> org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:159)
>   at 
> org.apache.cassandra.stress.operations.userdefined.SchemaInsert.run(SchemaInsert.java:119)
>   at 
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:309)
> Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Batch 
> too large
>   at 
> com.datastax.driver.core.Responses$Error.asException(Responses.java:125)
>   at 
> com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:120)
>   at 
> com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:186)
>   at 
> com.datastax.driver.core.RequestHandler.access$2300(RequestHandler.java:45)
>   at 
>

[jira] [Updated] (CASSANDRA-11105) cassandra-stress tool - InvalidQueryException: Batch too large

2017-02-06 Thread Alexander Dejanovski (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Dejanovski updated CASSANDRA-11105:
-
Attachment: 11105-trunk.txt

> cassandra-stress tool - InvalidQueryException: Batch too large
> --
>
> Key: CASSANDRA-11105
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11105
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
> Environment: Cassandra 2.2.4, Java 8, CentOS 6.5
>Reporter: Ralf Steppacher
> Fix For: 4.0
>
> Attachments: 11105-trunk.txt, batch_too_large.yaml
>
>
> I am using Cassandra 2.2.4 and I am struggling to get the cassandra-stress 
> tool to work for my test scenario. I have followed the example on 
> http://www.datastax.com/dev/blog/improved-cassandra-2-1-stress-tool-benchmark-any-schema
>  to create a yaml file describing my test (attached).
> I am collecting events per user id (text, partition key). Events have a 
> session type (text), event type (text), and creation time (timestamp) 
> (clustering keys, in that order). Plus some more attributes required for 
> rendering the events in a UI. For testing purposes I ended up with the 
> following column spec and insert distribution:
> {noformat}
> columnspec:
>   - name: created_at
> cluster: uniform(10..1)
>   - name: event_type
> size: uniform(5..10)
> population: uniform(1..30)
> cluster: uniform(1..30)
>   - name: session_type
> size: fixed(5)
> population: uniform(1..4)
> cluster: uniform(1..4)
>   - name: user_id
> size: fixed(15)
> population: uniform(1..100)
>   - name: message
> size: uniform(10..100)
> population: uniform(1..100B)
> insert:
>   partitions: fixed(1)
>   batchtype: UNLOGGED
>   select: fixed(1)/120
> {noformat}
> Running stress tool for just the insert prints 
> {noformat}
> Generating batches with [1..1] partitions and [0..1] rows (of [10..120] 
> total rows in the partitions)
> {noformat}
> and then immediately starts flooding me with 
> {{com.datastax.driver.core.exceptions.InvalidQueryException: Batch too 
> large}}. 
> Why I should be exceeding the {{batch_size_fail_threshold_in_kb: 50}} in the 
> {{cassandra.yaml}} I do not understand. My understanding is that the stress 
> tool should generate one row per batch. The size of a single row should not 
> exceed {{8+10*3+5*3+15*3+100*3 = 398 bytes}}. Assuming a worst case of all 
> text characters being 3 byte unicode characters. 
> This is how I start the attached user scenario:
> {noformat}
> [rsteppac@centos bin]$ ./cassandra-stress user 
> profile=../batch_too_large.yaml ops\(insert=1\) -log level=verbose 
> file=~/centos_event_by_patient_session_event_timestamp_insert_only.log -node 
> 10.211.55.8
> INFO  08:00:07 Did not find Netty's native epoll transport in the classpath, 
> defaulting to NIO.
> INFO  08:00:08 Using data-center name 'datacenter1' for 
> DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct 
> datacenter name with DCAwareRoundRobinPolicy constructor)
> INFO  08:00:08 New Cassandra host /10.211.55.8:9042 added
> Connected to cluster: Titan_DEV
> Datatacenter: datacenter1; Host: /10.211.55.8; Rack: rack1
> Created schema. Sleeping 1s for propagation.
> Generating batches with [1..1] partitions and [0..1] rows (of [10..120] 
> total rows in the partitions)
> com.datastax.driver.core.exceptions.InvalidQueryException: Batch too large
>   at 
> com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:35)
>   at 
> com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:271)
>   at 
> com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:185)
>   at 
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:55)
>   at 
> org.apache.cassandra.stress.operations.userdefined.SchemaInsert$JavaDriverRun.run(SchemaInsert.java:87)
>   at 
> org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:159)
>   at 
> org.apache.cassandra.stress.operations.userdefined.SchemaInsert.run(SchemaInsert.java:119)
>   at 
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:309)
> Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Batch 
> too large
>   at 
> com.datastax.driver.core.Responses$Error.asException(Responses.java:125)
>   at 
> com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:120)
>   at 
> com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:186)
>   at 
> com.datastax.driver.core.RequestHandler.access$2300(RequestHandler.java:45)
>   at 
>

[jira] [Updated] (CASSANDRA-12121) CommitLogReplayException on Start Up

2017-02-06 Thread Anonymous (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anonymous updated CASSANDRA-12121:
--
Reproduced In: 3.9, 3.7  (was: 3.7)
   Status: Awaiting Feedback  (was: Open)

> CommitLogReplayException on Start Up
> 
>
> Key: CASSANDRA-12121
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12121
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Tom Burdick
> Attachments: 000_epoch.cql, mutation7038154871517187161dat, 
> sane_distribution.cql
>
>
> Using cassandra 3.7 and executing the attached .cql schema change files, then 
> restarting one of the cassandra nodes in a cluster, I get this traceback
> I had to change the 000_epoch.cql file slightly to remove some important 
> pieces, apologies if that makes this more difficult to verify.
> {noformat}
> ERROR [main] 2016-06-30 09:25:23,089 JVMStabilityInspector.java:82 - Exiting 
> due to error while processing commit log during initialization.
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
> Unexpected error deserializing mutation; saved to 
> /tmp/mutation7038154871517187161dat.  This may be caused by replaying a 
> mutation against a table with the same name but incompatible schema.  
> Exception follows: org.apache.cassandra.serializers.MarshalException: Not 
> enough bytes to read 0th field java.nio.HeapByteBuffer[pos=0 lim=4 cap=4]
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:616)
>  [apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replayMutation(CommitLogReplayer.java:573)
>  [apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:526)
>  [apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:412)
>  [apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:228)
>  [apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:185) 
> [apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:165) 
> [apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:314) 
> [apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:585)
>  [apache-cassandra-3.7.0.jar:3.7.0]
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:714) 
> [apache-cassandra-3.7.0.jar:3.7.0]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13004) Corruption while adding a column to a table

2017-02-06 Thread Nimi Wariboko Jr. (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855257#comment-15855257
 ] 

Nimi Wariboko Jr. commented on CASSANDRA-13004:
---

Hi,

I think I'm hitting the same issue on 3.5 - for us it prevents us from 
bootstrapping any new nodes. Oddly we can sometimes do reads just fine (SELECT 
* FROM table), but other times we get a timeout exception with the "Corrupt 
length" in our logs. We tried scrub (no effect), and we tried dumping the 
entire table to JSON, TRUNCATING it, and reinserting all the data - and even 
the new fresh sstables hit this issue.

(Node is running 3.5, but we are using 3.10's sstabledump).

{code}
db@cass2:~/apache-cassandra-3.10/bin$ sudo ../tools/bin/sstabledump 
/mnt/dsk2/var/lib/cassandra/data/cmuser/users-68b04ac07c5011e5a85daf9a47bddf3d/ma-9473-big-Data.db
 
WARN  03:41:14,442 Only 30.140GiB free across all data volumes. Consider adding 
more capacity to your cluster or removing obsolete snapshots
[
  {
"partition" : {
  "key" : [ "560c3b78-adf2-11e6-8ef2-f45c89b76d37" ],
  "position" : 0
},
"rows" : [ ]
  }
]Exception in thread "main" 
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: 
/mnt/dsk2/var/lib/cassandra/data/cmuser/users-68b04ac07c5011e5a85daf9a47bddf3d/ma-9473-big-Data.db
at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:134)
at 
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:100)
at 
org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at 
org.apache.cassandra.tools.JsonTransformer.serializePartition(JsonTransformer.java:206)
at 
java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at 
java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
at java.util.Iterator.forEachRemaining(Iterator.java:116)
at 
java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at 
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at 
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at 
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at 
java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at 
org.apache.cassandra.tools.JsonTransformer.toJson(JsonTransformer.java:99)
at org.apache.cassandra.tools.SSTableExport.main(SSTableExport.java:240)
Caused by: java.io.IOException: Corrupt (negative) value length encountered
at 
org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:425)
at 
org.apache.cassandra.db.rows.Cell$Serializer.deserialize(Cell.java:245)
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.readComplexColumn(UnfilteredSerializer.java:636)
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.lambda$deserializeRowBody$1(UnfilteredSerializer.java:577)
at org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1222)
at org.apache.cassandra.utils.btree.BTree.applyForwards(BTree.java:1226)
at org.apache.cassandra.utils.btree.BTree.apply(BTree.java:1177)
at org.apache.cassandra.db.Columns.apply(Columns.java:377)
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:571)
at 
org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:440)
at 
org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:95)
at 
org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:73)
at 
org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
at 
org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:122)
... 16 more
db@cass2:~/apache-cassandra-3.10/bin$ 
{code}

We have done ALTER TABLE's on this table, but not for a couple months. 
Similarly, I notice we both are using collection types with custom types - 
could that have something to do with the issue?

{code}
CREATE TABLE cmuser.users (
id timeuuid PRIMARY KEY,
account_id text,
api_key text,
avatar text,
channel text,
city text,
company text,
country text,
created timestamp,
custom_id text,
customer text,
date_of_birth

[jira] [Updated] (CASSANDRA-9143) Improving consistency of repairAt field across replicas

2017-02-06 Thread Marcus Eriksson (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Marcus Eriksson updated CASSANDRA-9143:
---
   Resolution: Fixed
Fix Version/s: (was: 4.x)
   4.0
   Status: Resolved  (was: Patch Available)

+1, committed!

> Improving consistency of repairAt field across replicas 
> 
>
> Key: CASSANDRA-9143
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9143
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: sankalp kohli
>Assignee: Blake Eggleston
> Fix For: 4.0
>
>
> We currently send an anticompaction request to all replicas. During this, a 
> node will split stables and mark the appropriate ones repaired. 
> The problem is that this could fail on some replicas due to many reasons 
> leading to problems in the next repair. 
> This is what I am suggesting to improve it. 
> 1) Send anticompaction request to all replicas. This can be done at session 
> level. 
> 2) During anticompaction, stables are split but not marked repaired. 
> 3) When we get positive ack from all replicas, coordinator will send another 
> message called markRepaired. 
> 4) On getting this message, replicas will mark the appropriate stables as 
> repaired. 
> This will reduce the window of failure. We can also think of "hinting" 
> markRepaired message if required. 
> Also the stables which are streaming can be marked as repaired like it is 
> done now. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[5/7] cassandra git commit: Fix consistency of incrementally repaired data

2017-02-06 Thread marcuse

http://git-wip-us.apache.org/repos/asf/cassandra/blob/98d74ed9/src/java/org/apache/cassandra/repair/RepairRunnable.java
--
diff --git a/src/java/org/apache/cassandra/repair/RepairRunnable.java 
b/src/java/org/apache/cassandra/repair/RepairRunnable.java
index 6f7297b..422dbdb 100644
--- a/src/java/org/apache/cassandra/repair/RepairRunnable.java
+++ b/src/java/org/apache/cassandra/repair/RepairRunnable.java
@@ -20,6 +20,7 @@ package org.apache.cassandra.repair;
 import java.net.InetAddress;
 import java.nio.ByteBuffer;
 import java.util.*;
+import java.util.concurrent.ExecutorService;
 import java.util.concurrent.LinkedBlockingQueue;
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.atomic.AtomicBoolean;
@@ -44,6 +45,7 @@ import org.apache.cassandra.db.ColumnFamilyStore;
 import org.apache.cassandra.db.ConsistencyLevel;
 import org.apache.cassandra.dht.Range;
 import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.repair.consistent.CoordinatorSession;
 import org.apache.cassandra.repair.messages.RepairOption;
 import org.apache.cassandra.service.ActiveRepairService;
 import org.apache.cassandra.service.QueryState;
@@ -66,11 +68,15 @@ public class RepairRunnable extends WrappedRunnable 
implements ProgressEventNoti
 {
 private static final Logger logger = 
LoggerFactory.getLogger(RepairRunnable.class);
 
-private StorageService storageService;
+private final StorageService storageService;
 private final int cmd;
 private final RepairOption options;
 private final String keyspace;
 
+private final String tag;
+private final AtomicInteger progress = new AtomicInteger();
+private final int totalProgress;
+
 private final List listeners = new ArrayList<>();
 
 private static final AtomicInteger threadCounter = new AtomicInteger(1);
@@ -81,6 +87,10 @@ public class RepairRunnable extends WrappedRunnable 
implements ProgressEventNoti
 this.cmd = cmd;
 this.options = options;
 this.keyspace = keyspace;
+
+this.tag = "repair:" + cmd;
+// get valid column families, calculate neighbors, validation, prepare 
for repair + number of ranges to repair
+this.totalProgress = 4 + options.getRanges().size();
 }
 
 @Override
@@ -223,72 +233,35 @@ public class RepairRunnable extends WrappedRunnable 
implements ProgressEventNoti
 return;
 }
 
-// Set up RepairJob executor for this repair command.
-final ListeningExecutorService executor = 
MoreExecutors.listeningDecorator(new 
JMXConfigurableThreadPoolExecutor(options.getJobThreads(),
-   
  Integer.MAX_VALUE,
-   
  TimeUnit.SECONDS,
-   
  new LinkedBlockingQueue(),
-   
  new NamedThreadFactory("Repair#" + 
cmd),
-   
  "internal"));
-
-List futures = new 
ArrayList<>(options.getRanges().size());
-for (Pair> p : 
commonRanges)
+if (options.isIncremental())
 {
-final RepairSession session = 
ActiveRepairService.instance.submitRepairSession(parentSession,
-  p.right,
-  keyspace,
-  
options.getParallelism(),
-  p.left,
-  repairedAt,
-  
options.isPullRepair(),
-  executor,
-  cfnames);
-if (session == null)
-continue;
-// After repair session completes, notify client its result
-Futures.addCallback(session, new 
FutureCallback()
-{
-public void onSuccess(RepairSessionResult result)
-{
-/**
- * If the success message below is modified, it must also 
be updated on
- * {@link 
org.apache.cassandra.utils.progress.jmx.LegacyJMXProgressSupport}
- * for backward-compatibility support.
- */
-String message =

[3/7] cassandra git commit: Fix consistency of incrementally repaired data

2017-02-06 Thread marcuse

http://git-wip-us.apache.org/repos/asf/cassandra/blob/98d74ed9/src/java/org/apache/cassandra/service/ActiveRepairService.java
--
diff --git a/src/java/org/apache/cassandra/service/ActiveRepairService.java 
b/src/java/org/apache/cassandra/service/ActiveRepairService.java
index e7c6640..89e1954 100644
--- a/src/java/org/apache/cassandra/service/ActiveRepairService.java
+++ b/src/java/org/apache/cassandra/service/ActiveRepairService.java
@@ -18,29 +18,31 @@
 package org.apache.cassandra.service;
 
 import java.io.IOException;
+import java.lang.management.ManagementFactory;
 import java.net.InetAddress;
 import java.net.UnknownHostException;
 import java.util.*;
 import java.util.concurrent.*;
 import java.util.concurrent.atomic.AtomicBoolean;
 
+import javax.management.MBeanServer;
+import javax.management.ObjectName;
+
 import com.google.common.base.Predicate;
-import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
 import com.google.common.collect.Multimap;
 import com.google.common.collect.Sets;
 import com.google.common.util.concurrent.AbstractFuture;
-import com.google.common.util.concurrent.Futures;
-import com.google.common.util.concurrent.ListenableFuture;
+
 import com.google.common.util.concurrent.ListeningExecutorService;
 import com.google.common.util.concurrent.MoreExecutors;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
+import org.apache.cassandra.concurrent.ScheduledExecutors;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.db.ColumnFamilyStore;
-import org.apache.cassandra.db.compaction.CompactionManager;
-import org.apache.cassandra.db.lifecycle.SSTableSet;
-import org.apache.cassandra.db.lifecycle.View;
 import org.apache.cassandra.dht.Bounds;
 import org.apache.cassandra.dht.Range;
 import org.apache.cassandra.dht.Token;
@@ -59,18 +61,17 @@ import org.apache.cassandra.net.IAsyncCallbackWithFailure;
 import org.apache.cassandra.net.MessageIn;
 import org.apache.cassandra.net.MessageOut;
 import org.apache.cassandra.net.MessagingService;
-import org.apache.cassandra.repair.AnticompactionTask;
 import org.apache.cassandra.repair.RepairJobDesc;
 import org.apache.cassandra.repair.RepairParallelism;
 import org.apache.cassandra.repair.RepairSession;
+import org.apache.cassandra.repair.consistent.CoordinatorSessions;
+import org.apache.cassandra.repair.consistent.LocalSessions;
 import org.apache.cassandra.repair.messages.*;
 import org.apache.cassandra.schema.TableId;
 import org.apache.cassandra.utils.CassandraVersion;
 import org.apache.cassandra.utils.Clock;
 import org.apache.cassandra.utils.FBUtilities;
 import org.apache.cassandra.utils.UUIDGen;
-import org.apache.cassandra.utils.concurrent.Ref;
-import org.apache.cassandra.utils.concurrent.Refs;
 
 /**
  * ActiveRepairService is the starting point for manual "active" repairs.
@@ -86,7 +87,7 @@ import org.apache.cassandra.utils.concurrent.Refs;
  * The creation of a repair session is done through the submitRepairSession 
that
  * returns a future on the completion of that session.
  */
-public class ActiveRepairService implements IEndpointStateChangeSubscriber, 
IFailureDetectionEventListener
+public class ActiveRepairService implements IEndpointStateChangeSubscriber, 
IFailureDetectionEventListener, ActiveRepairServiceMBean
 {
 /**
  * @deprecated this statuses are from the previous JMX notification 
service,
@@ -98,6 +99,15 @@ public class ActiveRepairService implements 
IEndpointStateChangeSubscriber, IFai
 {
 STARTED, SESSION_SUCCESS, SESSION_FAILED, FINISHED
 }
+
+public class ConsistentSessions
+{
+public final LocalSessions local = new LocalSessions();
+public final CoordinatorSessions coordinated = new 
CoordinatorSessions();
+}
+
+public final ConsistentSessions consistent = new ConsistentSessions();
+
 private boolean registeredForEndpointChanges = false;
 
 public static CassandraVersion SUPPORTS_GLOBAL_PREPARE_FLAG_VERSION = new 
CassandraVersion("2.2.1");
@@ -107,6 +117,7 @@ public class ActiveRepairService implements 
IEndpointStateChangeSubscriber, IFai
 public static final ActiveRepairService instance = new 
ActiveRepairService(FailureDetector.instance, Gossiper.instance);
 
 public static final long UNREPAIRED_SSTABLE = 0;
+public static final UUID NO_PENDING_REPAIR = null;
 
 /**
  * A map of active coordinator session.
@@ -122,6 +133,37 @@ public class ActiveRepairService implements 
IEndpointStateChangeSubscriber, IFai
 {
 this.failureDetector = failureDetector;
 this.gossiper = gossiper;
+
+MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
+try
+{
+mbs.registerMBean(this, new ObjectName(MBEAN_NAME));
+}
+catch (Exception e)
+{
+

[6/7] cassandra git commit: Fix consistency of incrementally repaired data

2017-02-06 Thread marcuse

http://git-wip-us.apache.org/repos/asf/cassandra/blob/98d74ed9/src/java/org/apache/cassandra/db/compaction/Upgrader.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/Upgrader.java 
b/src/java/org/apache/cassandra/db/compaction/Upgrader.java
index 34ec1dd..5a60ddd 100644
--- a/src/java/org/apache/cassandra/db/compaction/Upgrader.java
+++ b/src/java/org/apache/cassandra/db/compaction/Upgrader.java
@@ -32,6 +32,7 @@ import org.apache.cassandra.io.sstable.*;
 import org.apache.cassandra.io.sstable.format.SSTableReader;
 import org.apache.cassandra.io.sstable.format.SSTableWriter;
 import org.apache.cassandra.io.sstable.metadata.MetadataCollector;
+import org.apache.cassandra.io.sstable.metadata.StatsMetadata;
 import org.apache.cassandra.utils.FBUtilities;
 import org.apache.cassandra.utils.OutputHandler;
 import org.apache.cassandra.utils.UUIDGen;
@@ -66,13 +67,14 @@ public class Upgrader
 this.estimatedRows = (long) Math.ceil((double) estimatedTotalKeys / 
estimatedSSTables);
 }
 
-private SSTableWriter createCompactionWriter(long repairedAt)
+private SSTableWriter createCompactionWriter(long repairedAt, UUID 
parentRepair)
 {
 MetadataCollector sstableMetadataCollector = new 
MetadataCollector(cfs.getComparator());
 sstableMetadataCollector.sstableLevel(sstable.getSSTableLevel());
 return SSTableWriter.create(cfs.newSSTableDescriptor(directory),
 estimatedRows,
 repairedAt,
+parentRepair,
 cfs.metadata,
 sstableMetadataCollector,
 SerializationHeader.make(cfs.metadata(), 
Sets.newHashSet(sstable)),
@@ -88,7 +90,8 @@ public class Upgrader
  AbstractCompactionStrategy.ScannerList scanners = 
strategyManager.getScanners(transaction.originals());
  CompactionIterator iter = new 
CompactionIterator(transaction.opType(), scanners.scanners, controller, 
nowInSec, UUIDGen.getTimeUUID()))
 {
-
writer.switchWriter(createCompactionWriter(sstable.getSSTableMetadata().repairedAt));
+StatsMetadata metadata = sstable.getSSTableMetadata();
+writer.switchWriter(createCompactionWriter(metadata.repairedAt, 
metadata.pendingRepair));
 while (iter.hasNext())
 writer.append(iter.next());
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/98d74ed9/src/java/org/apache/cassandra/db/compaction/Verifier.java
--
diff --git a/src/java/org/apache/cassandra/db/compaction/Verifier.java 
b/src/java/org/apache/cassandra/db/compaction/Verifier.java
index 467d50d..bca6e79 100644
--- a/src/java/org/apache/cassandra/db/compaction/Verifier.java
+++ b/src/java/org/apache/cassandra/db/compaction/Verifier.java
@@ -234,7 +234,7 @@ public class Verifier implements Closeable
 
 private void markAndThrow() throws IOException
 {
-
sstable.descriptor.getMetadataSerializer().mutateRepairedAt(sstable.descriptor, 
ActiveRepairService.UNREPAIRED_SSTABLE);
+
sstable.descriptor.getMetadataSerializer().mutateRepaired(sstable.descriptor, 
ActiveRepairService.UNREPAIRED_SSTABLE, 
sstable.getSSTableMetadata().pendingRepair);
 throw new CorruptSSTableException(new Exception(String.format("Invalid 
SSTable %s, please force repair", sstable.getFilename())), 
sstable.getFilename());
 }
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/98d74ed9/src/java/org/apache/cassandra/db/compaction/writers/CompactionAwareWriter.java
--
diff --git 
a/src/java/org/apache/cassandra/db/compaction/writers/CompactionAwareWriter.java
 
b/src/java/org/apache/cassandra/db/compaction/writers/CompactionAwareWriter.java
index 205aebe..e8f7d72 100644
--- 
a/src/java/org/apache/cassandra/db/compaction/writers/CompactionAwareWriter.java
+++ 
b/src/java/org/apache/cassandra/db/compaction/writers/CompactionAwareWriter.java
@@ -22,6 +22,7 @@ import java.io.File;
 import java.util.Collection;
 import java.util.List;
 import java.util.Set;
+import java.util.UUID;
 
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -55,6 +56,7 @@ public abstract class CompactionAwareWriter extends 
Transactional.AbstractTransa
 protected final long estimatedTotalKeys;
 protected final long maxAge;
 protected final long minRepairedAt;
+protected final UUID pendingRepair;
 
 protected final SSTableRewriter sstableWriter;
 protected final LifecycleTransaction txn;
@@ -88,6 +90,7 @@ public abstract class CompactionAwareWriter extends 
Transactional.AbstractTransa
 maxAge = CompactionTask.getMaxDataAge(nonExpiredSSTables);
 sstableWriter =

[7/7] cassandra git commit: Fix consistency of incrementally repaired data

2017-02-06 Thread marcuse

Fix consistency of incrementally repaired data

patch by Blake Eggleston, reviewed by Marcus Eriksson for CASSANDRA-9143


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/98d74ed9
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/98d74ed9
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/98d74ed9

Branch: refs/heads/trunk
Commit: 98d74ed998706e9e047dc0f7886a1e9b18df3ce9
Parents: 1757e13
Author: Blake Eggleston 
Authored: Wed Aug 31 15:48:43 2016 -0700
Committer: Marcus Eriksson 
Committed: Mon Feb 6 19:11:18 2017 -0800

--
 CHANGES.txt |   1 +
 NEWS.txt|   7 +-
 .../apache/cassandra/db/ColumnFamilyStore.java  |  10 +-
 src/java/org/apache/cassandra/db/Memtable.java  |   1 +
 .../org/apache/cassandra/db/SystemKeyspace.java |  19 +-
 .../compaction/AbstractCompactionStrategy.java  |  10 +-
 .../db/compaction/CompactionManager.java|  96 +-
 .../compaction/CompactionStrategyManager.java   | 258 +-
 .../cassandra/db/compaction/CompactionTask.java |  17 +
 .../DateTieredCompactionStrategy.java   |   6 +
 .../compaction/LeveledCompactionStrategy.java   |   6 +
 .../db/compaction/LeveledManifest.java  |  11 +
 .../db/compaction/PendingRepairManager.java | 432 +
 .../cassandra/db/compaction/Scrubber.java   |   8 +-
 .../SizeTieredCompactionStrategy.java   |   7 +
 .../TimeWindowCompactionStrategy.java   |   6 +
 .../cassandra/db/compaction/Upgrader.java   |   7 +-
 .../cassandra/db/compaction/Verifier.java   |   2 +-
 .../writers/CompactionAwareWriter.java  |   3 +
 .../writers/DefaultCompactionWriter.java|   1 +
 .../writers/MajorLeveledCompactionWriter.java   |   2 +-
 .../writers/MaxSSTableSizeWriter.java   |   1 +
 .../SplittingSizeTieredCompactionWriter.java|   1 +
 .../org/apache/cassandra/dht/RangeStreamer.java |   2 +-
 .../io/sstable/AbstractSSTableSimpleWriter.java |   3 +-
 .../cassandra/io/sstable/SSTableLoader.java |   2 +-
 .../cassandra/io/sstable/SSTableTxnWriter.java  |  15 +-
 .../io/sstable/SimpleSSTableMultiWriter.java|   4 +-
 .../sstable/format/RangeAwareSSTableWriter.java |   9 +-
 .../io/sstable/format/SSTableReader.java|  11 +
 .../io/sstable/format/SSTableWriter.java|  17 +-
 .../cassandra/io/sstable/format/Version.java|   2 +
 .../io/sstable/format/big/BigFormat.java|  14 +-
 .../io/sstable/format/big/BigTableWriter.java   |  11 +-
 .../sstable/metadata/IMetadataSerializer.java   |   5 +-
 .../io/sstable/metadata/MetadataCollector.java  |   9 +-
 .../io/sstable/metadata/MetadataSerializer.java |   9 +-
 .../io/sstable/metadata/StatsMetadata.java  |  67 +-
 .../net/IncomingStreamingConnection.java|   2 +-
 .../cassandra/repair/AnticompactionTask.java| 174 
 .../apache/cassandra/repair/LocalSyncTask.java  |   7 +-
 .../org/apache/cassandra/repair/RepairJob.java  |  30 +-
 .../repair/RepairMessageVerbHandler.java|  61 +-
 .../apache/cassandra/repair/RepairRunnable.java | 306 ---
 .../apache/cassandra/repair/RepairSession.java  |   5 +-
 .../cassandra/repair/StreamingRepairTask.java   |   6 +-
 .../repair/SystemDistributedKeyspace.java   |   3 +-
 .../org/apache/cassandra/repair/Validator.java  |  11 +-
 .../repair/consistent/ConsistentSession.java| 325 +++
 .../repair/consistent/CoordinatorSession.java   | 312 +++
 .../repair/consistent/CoordinatorSessions.java  |  95 ++
 .../repair/consistent/LocalSession.java | 129 +++
 .../repair/consistent/LocalSessionInfo.java |  67 ++
 .../repair/consistent/LocalSessions.java| 703 +++
 .../consistent/PendingAntiCompaction.java   | 195 
 .../repair/messages/AnticompactionRequest.java  | 107 ---
 .../cassandra/repair/messages/FailSession.java  |  71 ++
 .../repair/messages/FinalizeCommit.java |  78 ++
 .../repair/messages/FinalizePromise.java|  95 ++
 .../repair/messages/FinalizePropose.java|  78 ++
 .../messages/PrepareConsistentRequest.java  | 124 +++
 .../messages/PrepareConsistentResponse.java |  94 ++
 .../repair/messages/RepairMessage.java  |  14 +-
 .../cassandra/repair/messages/RepairOption.java |   5 +
 .../repair/messages/StatusRequest.java  |  77 ++
 .../repair/messages/StatusResponse.java |  90 ++
 .../cassandra/service/ActiveRepairService.java  | 329 ++-
 .../service/ActiveRepairServiceMBean.java   |  30 +
 .../cassandra/service/CassandraDaemon.java  |   1 +
 .../cassandra/streaming/ConnectionHandler.java  |   3 +-
 .../cassandra/streaming/StreamCoordinator.java  |   8 +-
 .../apache/cassandra/streaming/StreamPlan.java  |   8 +-
 .../cassandra/streaming/StreamReader.java

[4/7] cassandra git commit: Fix consistency of incrementally repaired data

2017-02-06 Thread marcuse

http://git-wip-us.apache.org/repos/asf/cassandra/blob/98d74ed9/src/java/org/apache/cassandra/repair/consistent/LocalSessions.java
--
diff --git a/src/java/org/apache/cassandra/repair/consistent/LocalSessions.java 
b/src/java/org/apache/cassandra/repair/consistent/LocalSessions.java
new file mode 100644
index 000..d10b9c5
--- /dev/null
+++ b/src/java/org/apache/cassandra/repair/consistent/LocalSessions.java
@@ -0,0 +1,703 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.consistent;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.nio.ByteBuffer;
+import java.time.Instant;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Iterables;
+import com.google.common.collect.Lists;
+import com.google.common.primitives.Ints;
+import com.google.common.util.concurrent.FutureCallback;
+import com.google.common.util.concurrent.Futures;
+import com.google.common.util.concurrent.ListenableFuture;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.schema.SchemaConstants;
+import org.apache.cassandra.cql3.QueryProcessor;
+import org.apache.cassandra.cql3.UntypedResultSet;
+import org.apache.cassandra.db.SystemKeyspace;
+import org.apache.cassandra.db.marshal.BytesType;
+import org.apache.cassandra.db.marshal.InetAddressType;
+import org.apache.cassandra.db.marshal.UUIDType;
+import org.apache.cassandra.dht.IPartitioner;
+import org.apache.cassandra.dht.Range;
+import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.gms.FailureDetector;
+import org.apache.cassandra.io.util.DataInputBuffer;
+import org.apache.cassandra.io.util.DataOutputBuffer;
+import org.apache.cassandra.net.MessageOut;
+import org.apache.cassandra.net.MessagingService;
+import org.apache.cassandra.repair.messages.FailSession;
+import org.apache.cassandra.repair.messages.FinalizeCommit;
+import org.apache.cassandra.repair.messages.FinalizePromise;
+import org.apache.cassandra.repair.messages.FinalizePropose;
+import org.apache.cassandra.repair.messages.PrepareConsistentRequest;
+import org.apache.cassandra.repair.messages.PrepareConsistentResponse;
+import org.apache.cassandra.repair.messages.RepairMessage;
+import org.apache.cassandra.repair.messages.StatusRequest;
+import org.apache.cassandra.repair.messages.StatusResponse;
+import org.apache.cassandra.schema.TableId;
+import org.apache.cassandra.service.ActiveRepairService;
+import org.apache.cassandra.service.StorageService;
+import org.apache.cassandra.utils.FBUtilities;
+
+import static org.apache.cassandra.repair.consistent.ConsistentSession.State.*;
+
+/**
+ * Manages all consistent repair sessions a node is participating in.
+ * 
+ * Since sessions need to be loaded, and since we need to handle cases where 
sessions might not exist, most of the logic
+ * around local sessions is implemented in this class, with the LocalSession 
class being treated more like a simple struct,
+ * in contrast with {@link CoordinatorSession}
+ */
+public class LocalSessions
+{
+private static final Logger logger = 
LoggerFactory.getLogger(LocalSessions.class);
+
+/**
+ * Amount of time a session can go without any activity before we start 
checking the status of other
+ * participants to see if we've missed a message
+ */
+static final int CHECK_STATUS_TIMEOUT = 
Integer.getInteger("cassandra.repair_status_check_timeout_seconds",
+   
Ints.checkedCast(TimeUnit.HOURS.toSeconds(1)));
+
+/**
+ * Amount of time a session

[2/7] cassandra git commit: Fix consistency of incrementally repaired data

2017-02-06 Thread marcuse

http://git-wip-us.apache.org/repos/asf/cassandra/blob/98d74ed9/test/unit/org/apache/cassandra/db/compaction/CompactionStrategyManagerPendingRepairTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/db/compaction/CompactionStrategyManagerPendingRepairTest.java
 
b/test/unit/org/apache/cassandra/db/compaction/CompactionStrategyManagerPendingRepairTest.java
new file mode 100644
index 000..8f53781
--- /dev/null
+++ 
b/test/unit/org/apache/cassandra/db/compaction/CompactionStrategyManagerPendingRepairTest.java
@@ -0,0 +1,291 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.db.compaction;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.UUID;
+
+import com.google.common.collect.Iterables;
+import org.junit.Assert;
+import org.junit.Test;
+
+import org.apache.cassandra.io.sstable.format.SSTableReader;
+import org.apache.cassandra.notifications.SSTableAddedNotification;
+import org.apache.cassandra.notifications.SSTableDeletingNotification;
+import org.apache.cassandra.notifications.SSTableListChangedNotification;
+import org.apache.cassandra.notifications.SSTableRepairStatusChanged;
+import org.apache.cassandra.repair.consistent.LocalSessionAccessor;
+import org.apache.cassandra.service.ActiveRepairService;
+import org.apache.cassandra.utils.FBUtilities;
+
+/**
+ * Tests CompactionStrategyManager's handling of pending repair sstables
+ */
+public class CompactionStrategyManagerPendingRepairTest extends 
AbstractPendingRepairTest
+{
+
+private static boolean 
strategiesContain(Collection strategies, 
SSTableReader sstable)
+{
+return Iterables.any(strategies, strategy -> 
strategy.getSSTables().contains(sstable));
+}
+
+private boolean pendingContains(UUID id, SSTableReader sstable)
+{
+return Iterables.any(csm.getPendingRepairManagers(), p -> p.get(id) != 
null && p.get(id).getSSTables().contains(sstable));
+}
+
+private boolean pendingContains(SSTableReader sstable)
+{
+return Iterables.any(csm.getPendingRepairManagers(), p -> 
strategiesContain(p.getStrategies(), sstable));
+}
+
+private boolean repairedContains(SSTableReader sstable)
+{
+return strategiesContain(csm.getRepaired(), sstable);
+}
+
+private boolean unrepairedContains(SSTableReader sstable)
+{
+return strategiesContain(csm.getUnrepaired(), sstable);
+}
+
+/**
+ * Pending repair strategy should be created when we encounter a new 
pending id
+ */
+@Test
+public void sstableAdded()
+{
+UUID repairID = registerSession(cfs);
+LocalSessionAccessor.prepareUnsafe(repairID, COORDINATOR, 
PARTICIPANTS);
+Assert.assertTrue(csm.pendingRepairs().isEmpty());
+
+SSTableReader sstable = makeSSTable(true);
+Assert.assertFalse(sstable.isRepaired());
+Assert.assertFalse(sstable.isPendingRepair());
+
+mutateRepaired(sstable, repairID);
+Assert.assertFalse(sstable.isRepaired());
+Assert.assertTrue(sstable.isPendingRepair());
+csm.getForPendingRepair(repairID).forEach(Assert::assertNull);
+
+// add the sstable
+csm.handleNotification(new 
SSTableAddedNotification(Collections.singleton(sstable)), cfs.getTracker());
+Assert.assertFalse(repairedContains(sstable));
+Assert.assertFalse(unrepairedContains(sstable));
+csm.getForPendingRepair(repairID).forEach(Assert::assertNotNull);
+Assert.assertTrue(pendingContains(repairID, sstable));
+}
+
+@Test
+public void sstableListChangedAddAndRemove()
+{
+UUID repairID = registerSession(cfs);
+LocalSessionAccessor.prepareUnsafe(repairID, COORDINATOR, 
PARTICIPANTS);
+
+SSTableReader sstable1 = makeSSTable(true);
+mutateRepaired(sstable1, repairID);
+
+SSTableReader sstable2 = makeSSTable(true);
+mutateRepaired(sstable2, repairID);
+
+Assert.assertFalse(repairedContains(sstable1));
+Assert.assertFalse(unrepairedContains(sstable1));
+

[1/7] cassandra git commit: Fix consistency of incrementally repaired data

2017-02-06 Thread marcuse

Repository: cassandra
Updated Branches:
  refs/heads/trunk 1757e1330 -> 98d74ed99


http://git-wip-us.apache.org/repos/asf/cassandra/blob/98d74ed9/test/unit/org/apache/cassandra/repair/consistent/LocalSessionTest.java
--
diff --git 
a/test/unit/org/apache/cassandra/repair/consistent/LocalSessionTest.java 
b/test/unit/org/apache/cassandra/repair/consistent/LocalSessionTest.java
new file mode 100644
index 000..a85e01b
--- /dev/null
+++ b/test/unit/org/apache/cassandra/repair/consistent/LocalSessionTest.java
@@ -0,0 +1,885 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.repair.consistent;
+
+import java.net.InetAddress;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.atomic.AtomicReference;
+import java.util.function.Consumer;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+import com.google.common.util.concurrent.ListenableFuture;
+import com.google.common.util.concurrent.SettableFuture;
+
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import org.apache.cassandra.SchemaLoader;
+import org.apache.cassandra.cql3.statements.CreateTableStatement;
+import org.apache.cassandra.schema.TableMetadata;
+import org.apache.cassandra.schema.Schema;
+import org.apache.cassandra.schema.SchemaConstants;
+import org.apache.cassandra.cql3.QueryProcessor;
+import org.apache.cassandra.db.ColumnFamilyStore;
+import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.db.SystemKeyspace;
+import org.apache.cassandra.repair.messages.FailSession;
+import org.apache.cassandra.repair.messages.FinalizeCommit;
+import org.apache.cassandra.repair.messages.FinalizePromise;
+import org.apache.cassandra.repair.messages.FinalizePropose;
+import org.apache.cassandra.repair.messages.PrepareConsistentRequest;
+import org.apache.cassandra.repair.messages.PrepareConsistentResponse;
+import org.apache.cassandra.repair.messages.RepairMessage;
+import org.apache.cassandra.repair.messages.StatusRequest;
+import org.apache.cassandra.repair.messages.StatusResponse;
+import org.apache.cassandra.schema.KeyspaceParams;
+import org.apache.cassandra.service.ActiveRepairService;
+import org.apache.cassandra.utils.FBUtilities;
+import org.apache.cassandra.utils.UUIDGen;
+
+import static org.apache.cassandra.repair.consistent.ConsistentSession.State.*;
+
+public class LocalSessionTest extends AbstractConsistentSessionTest
+{
+
+static LocalSession.Builder createBuilder()
+{
+LocalSession.Builder builder = LocalSession.builder();
+builder.withState(PREPARING);
+builder.withSessionID(UUIDGen.getTimeUUID());
+builder.withCoordinator(COORDINATOR);
+builder.withUUIDTableIds(Sets.newHashSet(UUIDGen.getTimeUUID(), 
UUIDGen.getTimeUUID()));
+builder.withRepairedAt(System.currentTimeMillis());
+builder.withRanges(Sets.newHashSet(RANGE1, RANGE2, RANGE3));
+builder.withParticipants(Sets.newHashSet(PARTICIPANT1, PARTICIPANT2, 
PARTICIPANT3));
+
+int now = FBUtilities.nowInSeconds();
+builder.withStartedAt(now);
+builder.withLastUpdate(now);
+
+return builder;
+}
+
+static LocalSession createSession()
+{
+return createBuilder().build();
+}
+
+private static void assertValidationFailure(Consumer 
consumer)
+{
+try
+{
+LocalSession.Builder builder = createBuilder();
+consumer.accept(builder);
+builder.build();
+Assert.fail("Expected assertion error");
+}
+catch (IllegalArgumentException e)
+{
+// expected
+}
+}
+
+private static void assertNoMessagesSent(InstrumentedLocalSessions 
sessions, InetAddress to)
+{
+Assert.assertNull(sessions.sentMessages.get(to));
+}
+
+private static void assertMessagesSent(InstrumentedLocalSessions sessions, 
InetAddress to,

[jira] [Created] (CASSANDRA-13188) compaction-stress AssertionError from getMemtableFor()

2017-02-06 Thread Jay Zhuang (JIRA)

Jay Zhuang created CASSANDRA-13188:
--

 Summary: compaction-stress AssertionError from getMemtableFor()
 Key: CASSANDRA-13188
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13188
 Project: Cassandra
  Issue Type: Bug
  Components: Testing, Tools
Reporter: Jay Zhuang
Assignee: Jay Zhuang


Exception:
{noformat}
./compaction-stress compact -d /tmp/compaction -p 
https://gist.githubusercontent.com/tjake/8995058fed11d9921e31/raw/a9334d1090017bf546d003e271747351a40692ea/blogpost.yaml
 -t 4
WARN  18:45:04,854 JNA link failure, one or more native method will be 
unavailable.
java.lang.AssertionError: []
at 
org.apache.cassandra.db.lifecycle.Tracker.getMemtableFor(Tracker.java:312)
at 
org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1315)
at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:618)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:462)
at org.apache.cassandra.db.Mutation.apply(Mutation.java:227)
at org.apache.cassandra.db.Mutation.apply(Mutation.java:232)
at org.apache.cassandra.db.Mutation.apply(Mutation.java:241)
at 
org.apache.cassandra.cql3.statements.ModificationStatement.executeInternalWithoutCondition(ModificationStatement.java:570)
at 
org.apache.cassandra.cql3.statements.ModificationStatement.executeInternal(ModificationStatement.java:564)
at 
org.apache.cassandra.cql3.QueryProcessor.executeOnceInternal(QueryProcessor.java:356)
at 
org.apache.cassandra.schema.SchemaKeyspace.saveSystemKeyspacesSchema(SchemaKeyspace.java:265)
at 
org.apache.cassandra.db.SystemKeyspace.finishStartup(SystemKeyspace.java:495)
at 
org.apache.cassandra.stress.CompactionStress$Compaction.run(CompactionStress.java:209)
at 
org.apache.cassandra.stress.CompactionStress.main(CompactionStress.java:349)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13179) Remove offheap_buffer as option for memtable_allocation_type in cassandra.yaml

2017-02-06 Thread Brad Vernon (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855174#comment-15855174
 ] 

Brad Vernon commented on CASSANDRA-13179:
-

How about:

{code}
# Note: offheap_buffers are not supported in Cassandra 3.0-3.3.
# They have been re-introduced in Cassandra 3.4. For details see
# https://issues.apache.org/jira/browse/CASSANDRA-9472 and
# https://issues.apache.org/jira/browse/CASSANDRA-11039
{code}

> Remove offheap_buffer as option for memtable_allocation_type in cassandra.yaml
> --
>
> Key: CASSANDRA-13179
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13179
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Brad Vernon
> Attachments: 3.0-cass_yaml_offheap_doc.patch
>
>
> With [CASSANDRA-11039] disallowing offheap_buffers as option for 
> memtable_allocation_type the cassandra.yaml included documentation should be 
> updated to match.
> Patch included.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13187) Allow Filtering on Cluster Key columns while Partition Key is given

2017-02-06 Thread Srini (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Srini updated CASSANDRA-13187:
--
Priority: Minor  (was: Major)

> Allow Filtering on Cluster Key columns while Partition Key is given
> ---
>
> Key: CASSANDRA-13187
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13187
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Srini
>Priority: Minor
> Fix For: 3.10
>
>
> create table table1 (
>   k_part_one text,
>   k_part_two int,
>   k_clust_one text,
>   k_clust_two int,
>   k_clust_three uuid,
>   data1 text,
>   data2 text,
>   PRIMARY KEY((k_part_one,k_part_two), k_clust_one, k_clust_two, 
> k_clust_three)  
>   );
> select k_clust_one, k_clust_three , data1 
> from table1  where k_part_one = ‘x’ and k_part_two =   and 
> k_clust_two =  allow filtering;
> Allow filtering works on non primary key columns but it doesn't work on 
> clustering key columns for a given partition key. The above example  skipped 
> k_clust_one and thus gives an error.
> InvalidRequest: Error from server: code=2200 [Invalid query] message="PRIMARY 
> KEY column "k_clust_two" cannot be restricted as preceding column 
> "k_clust_one" is not restricted"
> This is very similar to CASSANDRA-6377, CASSANDRA-11310, CASSANDRA -10715. 
> Yet, this is still unresolved.
> This is very critical use case, where we want to avoid sending entire wide 
> row to the client and do filtering locally. It is an unnecessary network hog 
> that needs be avoided by doing the filtering on the actual Cassandra node.
> Please note that there is no secondary index on this column family.. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-13187) Allow Filtering on Cluster Key columns while Partition Key is given

2017-02-06 Thread Srini (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855146#comment-15855146
 ] 

Srini edited comment on CASSANDRA-13187 at 2/7/17 2:03 AM:
---

Apologize. Wrong version. It is working in 3.10. Can be closed.


was (Author: sdasar):
Apologize. Wrong version. It is working in 3.10.

> Allow Filtering on Cluster Key columns while Partition Key is given
> ---
>
> Key: CASSANDRA-13187
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13187
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Srini
>Priority: Minor
> Fix For: 3.10
>
>
> create table table1 (
>   k_part_one text,
>   k_part_two int,
>   k_clust_one text,
>   k_clust_two int,
>   k_clust_three uuid,
>   data1 text,
>   data2 text,
>   PRIMARY KEY((k_part_one,k_part_two), k_clust_one, k_clust_two, 
> k_clust_three)  
>   );
> select k_clust_one, k_clust_three , data1 
> from table1  where k_part_one = ‘x’ and k_part_two =   and 
> k_clust_two =  allow filtering;
> Allow filtering works on non primary key columns but it doesn't work on 
> clustering key columns for a given partition key. The above example  skipped 
> k_clust_one and thus gives an error.
> InvalidRequest: Error from server: code=2200 [Invalid query] message="PRIMARY 
> KEY column "k_clust_two" cannot be restricted as preceding column 
> "k_clust_one" is not restricted"
> This is very similar to CASSANDRA-6377, CASSANDRA-11310, CASSANDRA -10715. 
> Yet, this is still unresolved.
> This is very critical use case, where we want to avoid sending entire wide 
> row to the client and do filtering locally. It is an unnecessary network hog 
> that needs be avoided by doing the filtering on the actual Cassandra node.
> Please note that there is no secondary index on this column family.. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13187) Allow Filtering on Cluster Key columns while Partition Key is given

2017-02-06 Thread Srini (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855146#comment-15855146
 ] 

Srini commented on CASSANDRA-13187:
---

Apologize. Wrong version. It is working in 3.10.

> Allow Filtering on Cluster Key columns while Partition Key is given
> ---
>
> Key: CASSANDRA-13187
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13187
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Srini
> Fix For: 3.10
>
>
> create table table1 (
>   k_part_one text,
>   k_part_two int,
>   k_clust_one text,
>   k_clust_two int,
>   k_clust_three uuid,
>   data1 text,
>   data2 text,
>   PRIMARY KEY((k_part_one,k_part_two), k_clust_one, k_clust_two, 
> k_clust_three)  
>   );
> select k_clust_one, k_clust_three , data1 
> from table1  where k_part_one = ‘x’ and k_part_two =   and 
> k_clust_two =  allow filtering;
> Allow filtering works on non primary key columns but it doesn't work on 
> clustering key columns for a given partition key. The above example  skipped 
> k_clust_one and thus gives an error.
> InvalidRequest: Error from server: code=2200 [Invalid query] message="PRIMARY 
> KEY column "k_clust_two" cannot be restricted as preceding column 
> "k_clust_one" is not restricted"
> This is very similar to CASSANDRA-6377, CASSANDRA-11310, CASSANDRA -10715. 
> Yet, this is still unresolved.
> This is very critical use case, where we want to avoid sending entire wide 
> row to the client and do filtering locally. It is an unnecessary network hog 
> that needs be avoided by doing the filtering on the actual Cassandra node.
> Please note that there is no secondary index on this column family.. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13187) Allow Filtering on Cluster Key columns while Partition Key is given

2017-02-06 Thread Srini (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Srini updated CASSANDRA-13187:
--
Description: 
create table table1 (
  k_part_one text,
  k_part_two int,
  k_clust_one text,
  k_clust_two int,
  k_clust_three uuid,
  data1 text,
  data2 text,
  PRIMARY KEY((k_part_one,k_part_two), k_clust_one, k_clust_two, 
k_clust_three)  
  );

select k_clust_one, k_clust_three , data1 
from table1  where k_part_one = ‘x’ and k_part_two =   and k_clust_two 
=  allow filtering;

Allow filtering works on non primary key columns but it doesn't work on 
clustering key columns for a given partition key. The above example  skipped 
k_clust_one and thus gives an error.

InvalidRequest: Error from server: code=2200 [Invalid query] message="PRIMARY 
KEY column "k_clust_two" cannot be restricted as preceding column "k_clust_one" 
is not restricted"

This is very similar to CASSANDRA-6377, CASSANDRA-11310, CASSANDRA -10715. Yet, 
this is still unresolved.

This is very critical use case, where we want to avoid sending entire wide row 
to the client and do filtering locally. It is an unnecessary network hog that 
needs be avoided by doing the filtering on the actual Cassandra node.

Please note that there is no secondary index on this column family.. 



  was:
create table table1 (
  k_part_one text,
  k_part_two int,
  k_clust_one text,
  k_clust_two int,
  k_clust_three uuid,
  data1 text,
  data2 text,
  PRIMARY KEY((k_part_one,k_part_two), k_clust_one, k_clust_two, 
k_clust_three)  
  );

select k_clust_one, k_clust_three , data1 
from table1  where k_part_one = ‘x’ and k_part_two = ‘’  and 
k_clust_two = ‘’ allow filtering;

Allow filtering works on non primary key columns but it doesn't work on 
clustering key columns for a given partition key. The above example  skipped 
k_clust_one and thus gives an error.

This is very similar to CASSANDRA-6377, CASSANDRA-11310, CASSANDRA -10715. Yet, 
this is still unresolved.

This is very critical use case, where we want to avoid sending entire wide row 
to the client and do filtering locally. It is an unnecessary network hog that 
needs be avoided by doing the filtering on the actual Cassandra node.

Please note that there is no secondary index on this column family.. 




> Allow Filtering on Cluster Key columns while Partition Key is given
> ---
>
> Key: CASSANDRA-13187
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13187
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Srini
> Fix For: 3.10
>
>
> create table table1 (
>   k_part_one text,
>   k_part_two int,
>   k_clust_one text,
>   k_clust_two int,
>   k_clust_three uuid,
>   data1 text,
>   data2 text,
>   PRIMARY KEY((k_part_one,k_part_two), k_clust_one, k_clust_two, 
> k_clust_three)  
>   );
> select k_clust_one, k_clust_three , data1 
> from table1  where k_part_one = ‘x’ and k_part_two =   and 
> k_clust_two =  allow filtering;
> Allow filtering works on non primary key columns but it doesn't work on 
> clustering key columns for a given partition key. The above example  skipped 
> k_clust_one and thus gives an error.
> InvalidRequest: Error from server: code=2200 [Invalid query] message="PRIMARY 
> KEY column "k_clust_two" cannot be restricted as preceding column 
> "k_clust_one" is not restricted"
> This is very similar to CASSANDRA-6377, CASSANDRA-11310, CASSANDRA -10715. 
> Yet, this is still unresolved.
> This is very critical use case, where we want to avoid sending entire wide 
> row to the client and do filtering locally. It is an unnecessary network hog 
> that needs be avoided by doing the filtering on the actual Cassandra node.
> Please note that there is no secondary index on this column family.. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-8460) Make it possible to move non-compacting sstables to slow/big storage in DTCS

2017-02-06 Thread Nate McCall (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nate McCall updated CASSANDRA-8460:
---
Fix Version/s: (was: 3.x)
   4.x

> Make it possible to move non-compacting sstables to slow/big storage in DTCS
> 
>
> Key: CASSANDRA-8460
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8460
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Marcus Eriksson
>  Labels: doc-impacting, dtcs
> Fix For: 4.x
>
>
> It would be nice if we could configure DTCS to have a set of extra data 
> directories where we move the sstables once they are older than 
> max_sstable_age_days. 
> This would enable users to have a quick, small SSD for hot, new data, and big 
> spinning disks for data that is rarely read and never compacted.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13187) Allow Filtering on Cluster Key columns while Partition Key is given

2017-02-06 Thread Srini (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Srini updated CASSANDRA-13187:
--
Fix Version/s: (was: 3.0.8)
   3.10

> Allow Filtering on Cluster Key columns while Partition Key is given
> ---
>
> Key: CASSANDRA-13187
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13187
> Project: Cassandra
>  Issue Type: Bug
>  Components: CQL
>Reporter: Srini
> Fix For: 3.10
>
>
> create table table1 (
>   k_part_one text,
>   k_part_two int,
>   k_clust_one text,
>   k_clust_two int,
>   k_clust_three uuid,
>   data1 text,
>   data2 text,
>   PRIMARY KEY((k_part_one,k_part_two), k_clust_one, k_clust_two, 
> k_clust_three)  
>   );
> select k_clust_one, k_clust_three , data1 
> from table1  where k_part_one = ‘x’ and k_part_two = ‘’  and 
> k_clust_two = ‘’ allow filtering;
> Allow filtering works on non primary key columns but it doesn't work on 
> clustering key columns for a given partition key. The above example  skipped 
> k_clust_one and thus gives an error.
> This is very similar to CASSANDRA-6377, CASSANDRA-11310, CASSANDRA -10715. 
> Yet, this is still unresolved.
> This is very critical use case, where we want to avoid sending entire wide 
> row to the client and do filtering locally. It is an unnecessary network hog 
> that needs be avoided by doing the filtering on the actual Cassandra node.
> Please note that there is no secondary index on this column family.. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13041) Do not allow removal of a DC from system_auth replication settings if the DC has active Cassandra instances

2017-02-06 Thread Jeff Jirsa (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854900#comment-15854900
 ] 

Jeff Jirsa commented on CASSANDRA-13041:


Is there a chicken-and-egg problem with CASSANDRA-12510 ?  

> Do not allow removal of a DC from system_auth replication settings if the DC 
> has active Cassandra instances
> ---
>
> Key: CASSANDRA-13041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13041
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Nachiket Patil
>Assignee: Nachiket Patil
>Priority: Minor
> Fix For: 4.x
>
> Attachments: trunk.diff
>
>
> I don’t believe it is ever correct to remove a DC from the system_auth 
> replication settings while there are nodes up in that DC. Cassandra should 
> not allow this change if there are hosts which are currently members of the 
> cluster in that DC, as any request which is routed to these hosts will meet 
> an unavailable. Also dropping the keyspace system_auth should not be allowed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (CASSANDRA-13187) Allow Filtering on Cluster Key columns while Partition Key is given

2017-02-06 Thread Srini (JIRA)

Srini created CASSANDRA-13187:
-

 Summary: Allow Filtering on Cluster Key columns while Partition 
Key is given
 Key: CASSANDRA-13187
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13187
 Project: Cassandra
  Issue Type: Bug
  Components: CQL
Reporter: Srini
 Fix For: 3.0.8


create table table1 (
  k_part_one text,
  k_part_two int,
  k_clust_one text,
  k_clust_two int,
  k_clust_three uuid,
  data1 text,
  data2 text,
  PRIMARY KEY((k_part_one,k_part_two), k_clust_one, k_clust_two, 
k_clust_three)  
  );

select k_clust_one, k_clust_three , data1 
from table1  where k_part_one = ‘x’ and k_part_two = ‘’  and 
k_clust_two = ‘’ allow filtering;

Allow filtering works on non primary key columns but it doesn't work on 
clustering key columns for a given partition key. The above example  skipped 
k_clust_one and thus gives an error.

This is very similar to CASSANDRA-6377, CASSANDRA-11310, CASSANDRA -10715. Yet, 
this is still unresolved.

This is very critical use case, where we want to avoid sending entire wide row 
to the client and do filtering locally. It is an unnecessary network hog that 
needs be avoided by doing the filtering on the actual Cassandra node.

Please note that there is no secondary index on this column family.. 





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-11105) cassandra-stress tool - InvalidQueryException: Batch too large

2017-02-06 Thread Nate McCall (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nate McCall updated CASSANDRA-11105:

Fix Version/s: 4.0

> cassandra-stress tool - InvalidQueryException: Batch too large
> --
>
> Key: CASSANDRA-11105
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11105
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
> Environment: Cassandra 2.2.4, Java 8, CentOS 6.5
>Reporter: Ralf Steppacher
> Fix For: 4.0
>
> Attachments: 11105-trunk.txt, batch_too_large.yaml
>
>
> I am using Cassandra 2.2.4 and I am struggling to get the cassandra-stress 
> tool to work for my test scenario. I have followed the example on 
> http://www.datastax.com/dev/blog/improved-cassandra-2-1-stress-tool-benchmark-any-schema
>  to create a yaml file describing my test (attached).
> I am collecting events per user id (text, partition key). Events have a 
> session type (text), event type (text), and creation time (timestamp) 
> (clustering keys, in that order). Plus some more attributes required for 
> rendering the events in a UI. For testing purposes I ended up with the 
> following column spec and insert distribution:
> {noformat}
> columnspec:
>   - name: created_at
> cluster: uniform(10..1)
>   - name: event_type
> size: uniform(5..10)
> population: uniform(1..30)
> cluster: uniform(1..30)
>   - name: session_type
> size: fixed(5)
> population: uniform(1..4)
> cluster: uniform(1..4)
>   - name: user_id
> size: fixed(15)
> population: uniform(1..100)
>   - name: message
> size: uniform(10..100)
> population: uniform(1..100B)
> insert:
>   partitions: fixed(1)
>   batchtype: UNLOGGED
>   select: fixed(1)/120
> {noformat}
> Running stress tool for just the insert prints 
> {noformat}
> Generating batches with [1..1] partitions and [0..1] rows (of [10..120] 
> total rows in the partitions)
> {noformat}
> and then immediately starts flooding me with 
> {{com.datastax.driver.core.exceptions.InvalidQueryException: Batch too 
> large}}. 
> Why I should be exceeding the {{batch_size_fail_threshold_in_kb: 50}} in the 
> {{cassandra.yaml}} I do not understand. My understanding is that the stress 
> tool should generate one row per batch. The size of a single row should not 
> exceed {{8+10*3+5*3+15*3+100*3 = 398 bytes}}. Assuming a worst case of all 
> text characters being 3 byte unicode characters. 
> This is how I start the attached user scenario:
> {noformat}
> [rsteppac@centos bin]$ ./cassandra-stress user 
> profile=../batch_too_large.yaml ops\(insert=1\) -log level=verbose 
> file=~/centos_event_by_patient_session_event_timestamp_insert_only.log -node 
> 10.211.55.8
> INFO  08:00:07 Did not find Netty's native epoll transport in the classpath, 
> defaulting to NIO.
> INFO  08:00:08 Using data-center name 'datacenter1' for 
> DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct 
> datacenter name with DCAwareRoundRobinPolicy constructor)
> INFO  08:00:08 New Cassandra host /10.211.55.8:9042 added
> Connected to cluster: Titan_DEV
> Datatacenter: datacenter1; Host: /10.211.55.8; Rack: rack1
> Created schema. Sleeping 1s for propagation.
> Generating batches with [1..1] partitions and [0..1] rows (of [10..120] 
> total rows in the partitions)
> com.datastax.driver.core.exceptions.InvalidQueryException: Batch too large
>   at 
> com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:35)
>   at 
> com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:271)
>   at 
> com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:185)
>   at 
> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:55)
>   at 
> org.apache.cassandra.stress.operations.userdefined.SchemaInsert$JavaDriverRun.run(SchemaInsert.java:87)
>   at 
> org.apache.cassandra.stress.Operation.timeWithRetry(Operation.java:159)
>   at 
> org.apache.cassandra.stress.operations.userdefined.SchemaInsert.run(SchemaInsert.java:119)
>   at 
> org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:309)
> Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Batch 
> too large
>   at 
> com.datastax.driver.core.Responses$Error.asException(Responses.java:125)
>   at 
> com.datastax.driver.core.DefaultResultSetFuture.onSet(DefaultResultSetFuture.java:120)
>   at 
> com.datastax.driver.core.RequestHandler.setFinalResult(RequestHandler.java:186)
>   at 
> com.datastax.driver.core.RequestHandler.access$2300(RequestHandler.java:45)
>   at 
>

[jira] [Commented] (CASSANDRA-13041) Do not allow removal of a DC from system_auth replication settings if the DC has active Cassandra instances

2017-02-06 Thread Nachiket Patil (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854739#comment-15854739
 ] 

Nachiket Patil commented on CASSANDRA-13041:


Thanks [~beobal] for the explanation. In this case I dont mind removing the 
drop-keyspace changes. I dont see any problem with the changes you have made 
either. 

> Do not allow removal of a DC from system_auth replication settings if the DC 
> has active Cassandra instances
> ---
>
> Key: CASSANDRA-13041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13041
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Nachiket Patil
>Assignee: Nachiket Patil
>Priority: Minor
> Fix For: 4.x
>
> Attachments: trunk.diff
>
>
> I don’t believe it is ever correct to remove a DC from the system_auth 
> replication settings while there are nodes up in that DC. Cassandra should 
> not allow this change if there are hosts which are currently members of the 
> cluster in that DC, as any request which is routed to these hosts will meet 
> an unavailable. Also dropping the keyspace system_auth should not be allowed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-11596) Add native transport port to system.peers

2017-02-06 Thread Ariel Weisberg (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854718#comment-15854718
 ] 

Ariel Weisberg commented on CASSANDRA-11596:


As part of this ticket I don't believe it has to change since system.peers is 
read using the existing wire protocol.

> Add native transport port to system.peers
> -
>
> Key: CASSANDRA-11596
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11596
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Nico Haller
>Assignee: Ariel Weisberg
>Priority: Minor
>  Labels: lhf
>
> Is there any reason why the native transport port is not being stored in 
> system.peers along the rpc broadcast address and transmitted to the connected 
> drivers?
> I would love to have that feature, that would allow me to "hide" my cluster 
> behind a reverse NAT or LB and only consume one external IP address and 
> forward packets based on the port the client is connecting to.
> I guess it makes sense to provide the complete socket information instead of 
> just the address and using a default port setting on the client to complete 
> the connection information.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-11596) Add native transport port to system.peers

2017-02-06 Thread Alex P (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854690#comment-15854690
 ] 

Alex P commented on CASSANDRA-11596:


Does this ticket require changes to the native protocol? 

> Add native transport port to system.peers
> -
>
> Key: CASSANDRA-11596
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11596
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Nico Haller
>Assignee: Ariel Weisberg
>Priority: Minor
>  Labels: lhf
>
> Is there any reason why the native transport port is not being stored in 
> system.peers along the rpc broadcast address and transmitted to the connected 
> drivers?
> I would love to have that feature, that would allow me to "hide" my cluster 
> behind a reverse NAT or LB and only consume one external IP address and 
> forward packets based on the port the client is connecting to.
> I guess it makes sense to provide the complete socket information instead of 
> just the address and using a default port setting on the client to complete 
> the connection information.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13008) Add vm.max_map_count StartupCheck

2017-02-06 Thread Jay Zhuang (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854684#comment-15854684
 ] 

Jay Zhuang commented on CASSANDRA-13008:


Thanks.

> Add vm.max_map_count StartupCheck
> -
>
> Key: CASSANDRA-13008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13008
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 3.0.x
>
> Attachments: 13008-3.0.txt
>
>
> It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563)
> When the max_map_count is low, it throws OOM exception, which is hard to link 
> to the real issue of vm.max_map_count.
> The problem happened when we tried to remove one node, all the other nodes in 
> cluster crashed. As each node was trying to load more local SSTable files for 
> streaming.
> I would suggest to add a StartupCheck for max_map_count, at least it could 
> give a warning message to help the debug.
> {code}
> ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to 
> be unstable.  Exiting forcefully due to:
> java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) 
> ~[na:1.8.0_112]
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13124) Abort or retry on failed hints delivery

2017-02-06 Thread Paulo Motta (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-13124:

   Resolution: Fixed
Fix Version/s: 3.0.11
   Status: Resolved  (was: Patch Available)

LGTM, thanks! Committed as dab0e31ba0294ccc5e8af35cbbe0a6733f35794e.

> Abort or retry on failed hints delivery
> ---
>
> Key: CASSANDRA-13124
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13124
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: hintedhandoff
> Fix For: 3.0.11
>
>
> CASSANDRA-12905 changed the hints path to be asynchronous in the normal case, 
> but when the hint is non-local and should be stored (ie. on decommission) an 
> ack is not sent so the operation does not complete successfully.
> CASSANDRA-13058 fixed this for 3.0+, but for 3.11+ but there is an additional 
> concern which on 3.0 is that non-acked hints are being completed successfully 
> so we should probably look into this as well here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13124) Abort or retry on failed hints delivery

2017-02-06 Thread Paulo Motta (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-13124:

Summary: Abort or retry on failed hints delivery  (was: Send ack when 
received hint is non-local)

> Abort or retry on failed hints delivery
> ---
>
> Key: CASSANDRA-13124
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13124
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: hintedhandoff
>
> CASSANDRA-12905 changed the hints path to be asynchronous in the normal case, 
> but when the hint is non-local and should be stored (ie. on decommission) an 
> ack is not sent so the operation does not complete successfully.
> CASSANDRA-13058 fixed this for 3.0+, but for 3.11+ but there is an additional 
> concern which on 3.0 is that non-acked hints are being completed successfully 
> so we should probably look into this as well here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-02-06 Thread paulo

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/97861e68
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/97861e68
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/97861e68

Branch: refs/heads/cassandra-3.11
Commit: 97861e68c67e1cdfdf01358cb50a8b102a11c4dd
Parents: 8fc72a5 dab0e31
Author: Paulo Motta 
Authored: Mon Feb 6 18:00:16 2017 -0200
Committer: Paulo Motta 
Committed: Mon Feb 6 18:00:16 2017 -0200

--

--

[3/6] cassandra git commit: Abort or retry on failed hints delivery

2017-02-06 Thread paulo

Abort or retry on failed hints delivery

patch by Stefan Podkowinski; reviewed by Paulo Motta for CASSANDRA-13124


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dab0e31b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dab0e31b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dab0e31b

Branch: refs/heads/trunk
Commit: dab0e31ba0294ccc5e8af35cbbe0a6733f35794e
Parents: cc384a5
Author: Stefan Podkowinski 
Authored: Tue Jan 17 11:00:05 2017 +0100
Committer: Paulo Motta 
Committed: Mon Feb 6 17:59:07 2017 -0200

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/hints/HintVerbHandler.java | 2 ++
 src/java/org/apache/cassandra/hints/HintsDispatcher.java | 9 +++--
 3 files changed, 6 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/dab0e31b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1d6b53c..1a90b1f 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Abort or retry on failed hints delivery (CASSANDRA-13124)
  * Fix handling of partition with partition-level deletion plus
live rows in sstabledump (CASSANDRA-13177)
  * Provide user workaround when system_schema.columns does not contain entries

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dab0e31b/src/java/org/apache/cassandra/hints/HintVerbHandler.java
--
diff --git a/src/java/org/apache/cassandra/hints/HintVerbHandler.java 
b/src/java/org/apache/cassandra/hints/HintVerbHandler.java
index abcd1f9..1fa479c 100644
--- a/src/java/org/apache/cassandra/hints/HintVerbHandler.java
+++ b/src/java/org/apache/cassandra/hints/HintVerbHandler.java
@@ -78,12 +78,14 @@ public final class HintVerbHandler implements 
IVerbHandler
 // the node is not the final destination of the hint (must have 
gotten it from a decommissioning node),
 // so just store it locally, to be delivered later.
 HintsService.instance.write(hostId, hint);
+reply(id, message.from);
 }
 else if (!StorageProxy.instance.appliesLocally(hint.mutation))
 {
 // the topology has changed, and we are no longer a replica of the 
mutation - since we don't know which node(s)
 // it has been handed over to, re-address the hint to all 
replicas; see CASSANDRA-5902.
 HintsService.instance.writeForAllReplicas(hint);
+reply(id, message.from);
 }
 else
 {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dab0e31b/src/java/org/apache/cassandra/hints/HintsDispatcher.java
--
diff --git a/src/java/org/apache/cassandra/hints/HintsDispatcher.java 
b/src/java/org/apache/cassandra/hints/HintsDispatcher.java
index e582d88..d7a3515 100644
--- a/src/java/org/apache/cassandra/hints/HintsDispatcher.java
+++ b/src/java/org/apache/cassandra/hints/HintsDispatcher.java
@@ -42,7 +42,7 @@ import org.apache.cassandra.utils.concurrent.SimpleCondition;
  */
 final class HintsDispatcher implements AutoCloseable
 {
-private enum Action { CONTINUE, ABORT, RETRY }
+private enum Action { CONTINUE, ABORT }
 
 private final HintsReader reader;
 private final UUID hostId;
@@ -116,10 +116,7 @@ final class HintsDispatcher implements AutoCloseable
 // retry in case of a timeout; stop in case of a failure, host going down, 
or delivery paused
 private Action dispatch(HintsReader.Page page)
 {
-Action action = sendHintsAndAwait(page);
-return action == Action.RETRY
- ? dispatch(page)
- : action;
+return sendHintsAndAwait(page);
 }
 
 private Action sendHintsAndAwait(HintsReader.Page page)
@@ -142,7 +139,7 @@ final class HintsDispatcher implements AutoCloseable
 
 for (Callback cb : callbacks)
 if (cb.await() != Callback.Outcome.SUCCESS)
-return Action.RETRY;
+return Action.ABORT;
 
 return Action.CONTINUE;
 }

[1/6] cassandra git commit: Abort or retry on failed hints delivery

2017-02-06 Thread paulo

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 cc384a5d7 -> dab0e31ba
  refs/heads/cassandra-3.11 8fc72a5eb -> 97861e68c
  refs/heads/trunk 217306669 -> 1757e1330


Abort or retry on failed hints delivery

patch by Stefan Podkowinski; reviewed by Paulo Motta for CASSANDRA-13124


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dab0e31b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dab0e31b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dab0e31b

Branch: refs/heads/cassandra-3.0
Commit: dab0e31ba0294ccc5e8af35cbbe0a6733f35794e
Parents: cc384a5
Author: Stefan Podkowinski 
Authored: Tue Jan 17 11:00:05 2017 +0100
Committer: Paulo Motta 
Committed: Mon Feb 6 17:59:07 2017 -0200

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/hints/HintVerbHandler.java | 2 ++
 src/java/org/apache/cassandra/hints/HintsDispatcher.java | 9 +++--
 3 files changed, 6 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/dab0e31b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1d6b53c..1a90b1f 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Abort or retry on failed hints delivery (CASSANDRA-13124)
  * Fix handling of partition with partition-level deletion plus
live rows in sstabledump (CASSANDRA-13177)
  * Provide user workaround when system_schema.columns does not contain entries

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dab0e31b/src/java/org/apache/cassandra/hints/HintVerbHandler.java
--
diff --git a/src/java/org/apache/cassandra/hints/HintVerbHandler.java 
b/src/java/org/apache/cassandra/hints/HintVerbHandler.java
index abcd1f9..1fa479c 100644
--- a/src/java/org/apache/cassandra/hints/HintVerbHandler.java
+++ b/src/java/org/apache/cassandra/hints/HintVerbHandler.java
@@ -78,12 +78,14 @@ public final class HintVerbHandler implements 
IVerbHandler
 // the node is not the final destination of the hint (must have 
gotten it from a decommissioning node),
 // so just store it locally, to be delivered later.
 HintsService.instance.write(hostId, hint);
+reply(id, message.from);
 }
 else if (!StorageProxy.instance.appliesLocally(hint.mutation))
 {
 // the topology has changed, and we are no longer a replica of the 
mutation - since we don't know which node(s)
 // it has been handed over to, re-address the hint to all 
replicas; see CASSANDRA-5902.
 HintsService.instance.writeForAllReplicas(hint);
+reply(id, message.from);
 }
 else
 {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dab0e31b/src/java/org/apache/cassandra/hints/HintsDispatcher.java
--
diff --git a/src/java/org/apache/cassandra/hints/HintsDispatcher.java 
b/src/java/org/apache/cassandra/hints/HintsDispatcher.java
index e582d88..d7a3515 100644
--- a/src/java/org/apache/cassandra/hints/HintsDispatcher.java
+++ b/src/java/org/apache/cassandra/hints/HintsDispatcher.java
@@ -42,7 +42,7 @@ import org.apache.cassandra.utils.concurrent.SimpleCondition;
  */
 final class HintsDispatcher implements AutoCloseable
 {
-private enum Action { CONTINUE, ABORT, RETRY }
+private enum Action { CONTINUE, ABORT }
 
 private final HintsReader reader;
 private final UUID hostId;
@@ -116,10 +116,7 @@ final class HintsDispatcher implements AutoCloseable
 // retry in case of a timeout; stop in case of a failure, host going down, 
or delivery paused
 private Action dispatch(HintsReader.Page page)
 {
-Action action = sendHintsAndAwait(page);
-return action == Action.RETRY
- ? dispatch(page)
- : action;
+return sendHintsAndAwait(page);
 }
 
 private Action sendHintsAndAwait(HintsReader.Page page)
@@ -142,7 +139,7 @@ final class HintsDispatcher implements AutoCloseable
 
 for (Callback cb : callbacks)
 if (cb.await() != Callback.Outcome.SUCCESS)
-return Action.RETRY;
+return Action.ABORT;
 
 return Action.CONTINUE;
 }

[2/6] cassandra git commit: Abort or retry on failed hints delivery

2017-02-06 Thread paulo

Abort or retry on failed hints delivery

patch by Stefan Podkowinski; reviewed by Paulo Motta for CASSANDRA-13124


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/dab0e31b
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/dab0e31b
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/dab0e31b

Branch: refs/heads/cassandra-3.11
Commit: dab0e31ba0294ccc5e8af35cbbe0a6733f35794e
Parents: cc384a5
Author: Stefan Podkowinski 
Authored: Tue Jan 17 11:00:05 2017 +0100
Committer: Paulo Motta 
Committed: Mon Feb 6 17:59:07 2017 -0200

--
 CHANGES.txt  | 1 +
 src/java/org/apache/cassandra/hints/HintVerbHandler.java | 2 ++
 src/java/org/apache/cassandra/hints/HintsDispatcher.java | 9 +++--
 3 files changed, 6 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/dab0e31b/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 1d6b53c..1a90b1f 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.11
+ * Abort or retry on failed hints delivery (CASSANDRA-13124)
  * Fix handling of partition with partition-level deletion plus
live rows in sstabledump (CASSANDRA-13177)
  * Provide user workaround when system_schema.columns does not contain entries

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dab0e31b/src/java/org/apache/cassandra/hints/HintVerbHandler.java
--
diff --git a/src/java/org/apache/cassandra/hints/HintVerbHandler.java 
b/src/java/org/apache/cassandra/hints/HintVerbHandler.java
index abcd1f9..1fa479c 100644
--- a/src/java/org/apache/cassandra/hints/HintVerbHandler.java
+++ b/src/java/org/apache/cassandra/hints/HintVerbHandler.java
@@ -78,12 +78,14 @@ public final class HintVerbHandler implements 
IVerbHandler
 // the node is not the final destination of the hint (must have 
gotten it from a decommissioning node),
 // so just store it locally, to be delivered later.
 HintsService.instance.write(hostId, hint);
+reply(id, message.from);
 }
 else if (!StorageProxy.instance.appliesLocally(hint.mutation))
 {
 // the topology has changed, and we are no longer a replica of the 
mutation - since we don't know which node(s)
 // it has been handed over to, re-address the hint to all 
replicas; see CASSANDRA-5902.
 HintsService.instance.writeForAllReplicas(hint);
+reply(id, message.from);
 }
 else
 {

http://git-wip-us.apache.org/repos/asf/cassandra/blob/dab0e31b/src/java/org/apache/cassandra/hints/HintsDispatcher.java
--
diff --git a/src/java/org/apache/cassandra/hints/HintsDispatcher.java 
b/src/java/org/apache/cassandra/hints/HintsDispatcher.java
index e582d88..d7a3515 100644
--- a/src/java/org/apache/cassandra/hints/HintsDispatcher.java
+++ b/src/java/org/apache/cassandra/hints/HintsDispatcher.java
@@ -42,7 +42,7 @@ import org.apache.cassandra.utils.concurrent.SimpleCondition;
  */
 final class HintsDispatcher implements AutoCloseable
 {
-private enum Action { CONTINUE, ABORT, RETRY }
+private enum Action { CONTINUE, ABORT }
 
 private final HintsReader reader;
 private final UUID hostId;
@@ -116,10 +116,7 @@ final class HintsDispatcher implements AutoCloseable
 // retry in case of a timeout; stop in case of a failure, host going down, 
or delivery paused
 private Action dispatch(HintsReader.Page page)
 {
-Action action = sendHintsAndAwait(page);
-return action == Action.RETRY
- ? dispatch(page)
- : action;
+return sendHintsAndAwait(page);
 }
 
 private Action sendHintsAndAwait(HintsReader.Page page)
@@ -142,7 +139,7 @@ final class HintsDispatcher implements AutoCloseable
 
 for (Callback cb : callbacks)
 if (cb.await() != Callback.Outcome.SUCCESS)
-return Action.RETRY;
+return Action.ABORT;
 
 return Action.CONTINUE;
 }

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

2017-02-06 Thread paulo

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/97861e68
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/97861e68
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/97861e68

Branch: refs/heads/trunk
Commit: 97861e68c67e1cdfdf01358cb50a8b102a11c4dd
Parents: 8fc72a5 dab0e31
Author: Paulo Motta 
Authored: Mon Feb 6 18:00:16 2017 -0200
Committer: Paulo Motta 
Committed: Mon Feb 6 18:00:16 2017 -0200

--

--

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

2017-02-06 Thread paulo

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/1757e133
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/1757e133
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/1757e133

Branch: refs/heads/trunk
Commit: 1757e1330eefd6b0173fec96b4956e4749bd3289
Parents: 2173066 97861e6
Author: Paulo Motta 
Authored: Mon Feb 6 18:01:09 2017 -0200
Committer: Paulo Motta 
Committed: Mon Feb 6 18:01:09 2017 -0200

--

--

[jira] [Commented] (CASSANDRA-9143) Improving consistency of repairAt field across replicas

2017-02-06 Thread Blake Eggleston (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-9143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854579#comment-15854579
 ] 

Blake Eggleston commented on CASSANDRA-9143:


Created dtest PR: https://github.com/riptano/cassandra-dtest/pull/1436

> Improving consistency of repairAt field across replicas 
> 
>
> Key: CASSANDRA-9143
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9143
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: sankalp kohli
>Assignee: Blake Eggleston
> Fix For: 4.x
>
>
> We currently send an anticompaction request to all replicas. During this, a 
> node will split stables and mark the appropriate ones repaired. 
> The problem is that this could fail on some replicas due to many reasons 
> leading to problems in the next repair. 
> This is what I am suggesting to improve it. 
> 1) Send anticompaction request to all replicas. This can be done at session 
> level. 
> 2) During anticompaction, stables are split but not marked repaired. 
> 3) When we get positive ack from all replicas, coordinator will send another 
> message called markRepaired. 
> 4) On getting this message, replicas will mark the appropriate stables as 
> repaired. 
> This will reduce the window of failure. We can also think of "hinting" 
> markRepaired message if required. 
> Also the stables which are streaming can be marked as repaired like it is 
> done now. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (CASSANDRA-5901) Bootstrap should also make the data consistent on the new node

2017-02-06 Thread Blake Eggleston (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-5901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Blake Eggleston reassigned CASSANDRA-5901:
--

Assignee: Blake Eggleston

> Bootstrap should also make the data consistent on the new node
> --
>
> Key: CASSANDRA-5901
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5901
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: sankalp kohli
>Assignee: Blake Eggleston
>Priority: Minor
>
> Currently when we are bootstrapping a new node, it might bootstrap from a 
> node which does not have most upto date data. Because of this, we need to run 
> a repair after that.
> Most people will always run the repair so it would help if we can provide a 
> parameter to bootstrap to run the repair once the bootstrap has finished. 
> It can also stop the node from responding to reads till repair has finished. 
> This could be another param as well. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13090) Coalescing strategy sleeps too much

2017-02-06 Thread Ariel Weisberg (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854389#comment-15854389
 ] 

Ariel Weisberg commented on CASSANDRA-13090:


Yes can you please try creating them. I did the merge and it doesn't do 
anything wonky and the compare view in GitHub shows what I expect. I'm not sure 
why git diff shows additional stuff that doesn't end up in the merge.

Generally the assignee is supposed to produce the branches ready to merge with 
the tests passing in Cassci. Talk to ptnapoleon in IRC to see if you can get 
set up to run your branches in cassci.

> Coalescing strategy sleeps too much
> ---
>
> Key: CASSANDRA-13090
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13090
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Corentin Chary
>Assignee: Corentin Chary
> Fix For: 3.x
>
> Attachments: 0001-Fix-wait-time-coalescing-CASSANDRA-13090-2.patch, 
> 0001-Fix-wait-time-coalescing-CASSANDRA-13090.patch
>
>
> With the current code maybeSleep is called even if we managed to take 
> maxItems out of the backlog. In this case we should really avoid sleeping 
> because it means that backlog is building up.
> I'll send a patch shortly.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-9289) Recover from unknown table when deserializing internode messages

2017-02-06 Thread Joshua McKenzie (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-9289:
---
Reviewer: Tyler Hobbs

> Recover from unknown table when deserializing internode messages
> 
>
> Key: CASSANDRA-9289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9289
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Tyler Hobbs
>Assignee: ZhaoYang
>Priority: Minor
>  Labels: lhf
> Fix For: 3.x
>
>
> As discussed in CASSANDRA-9136, it would be nice to gracefully recover from 
> seeing an unknown table in a message from another node.  Currently, we close 
> the connection and reconnect, which can cause other concurrent queries to 
> fail.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (CASSANDRA-13109) Lightweight transactions temporarily fail after upgrade from 2.1 to 3.0

2017-02-06 Thread Sylvain Lebresne (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sylvain Lebresne reassigned CASSANDRA-13109:


Assignee: Samuel Klock

> Lightweight transactions temporarily fail after upgrade from 2.1 to 3.0
> ---
>
> Key: CASSANDRA-13109
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13109
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Samuel Klock
>Assignee: Samuel Klock
> Attachments: 13109-3.0.txt
>
>
> We've observed this upgrading from 2.1.15 to 3.0.8 and from 2.1.16 to 3.0.10: 
> some lightweight transactions executed on upgraded nodes fail with a read 
> failure.  The following conditions seem relevant to this occurring:
> * The transaction must be conditioned on the current value of at least one 
> column, e.g., {{IF NOT EXISTS}} transactions don't seem to be affected.
> * There should be a collection column (in our case, a map) defined on the 
> table on which the transaction is executed.
> * The transaction should be executed before sstables on the node are 
> upgraded.  The failure does not occur after the sstables have been upgraded 
> (whether via {{nodetool upgradesstables}} or effectively via compaction).
> * Upgraded nodes seem to be able to participate in lightweight transactions 
> as long as they're not the coordinator.
> * The values in the row being manipulated by the transaction must have been 
> consistently manipulated by lightweight transactions (perhaps the existence 
> of Paxos state for the partition is somehow relevant?).
> * In 3.0.10, it _seems_ to be necessary to have the partition split across 
> multiple legacy sstables.  This was not necessary to reproduce the bug in 
> 3.0.8 or .9.
> For applications affected by this bug, a possible workaround is to prevent 
> nodes being upgraded from coordinating requests until sstables have been 
> upgraded.
> We're able to reproduce this when upgrading from 2.1.16 to 3.0.10 with the 
> following steps on a single-node cluster using a mostly pristine 
> {{cassandra.yaml}} from the source distribution.
> # Start Cassandra-2.1.16 on the node.
> # Create a table with a collection column and insert some data into it.
> {code:sql}
> CREATE KEYSPACE test WITH REPLICATION = {'class': 'SimpleStrategy', 
> 'replication_factor': 1};
> CREATE TABLE test.test (key TEXT PRIMARY KEY, cas_target TEXT, 
> some_collection MAP);
> INSERT INTO test.test (key, cas_target, some_collection) VALUES ('key', 
> 'value', {}) IF NOT EXISTS;
> {code}
> # Flush the row to an sstable: {{nodetool flush}}.
> # Update the row:
> {code:sql}
> UPDATE test.test SET cas_target = 'newvalue', some_collection = {} WHERE key 
> = 'key' IF cas_target = 'value';
> {code}
> # Drain the node: {{nodetool drain}}
> # Stop the node, upgrade to 3.0.10, and start the node.
> # Attempt to update the row again:
> {code:sql}
> UPDATE test.test SET cas_target = 'lastvalue' WHERE key = 'key' IF cas_target 
> = 'newvalue';
> {code}
> Using {{cqlsh}}, if the error is reproduced, the following output will be 
> returned:
> {code:sql}
> $ ./cqlsh <<< "UPDATE test.test SET cas_target = 'newvalue', some_collection 
> = {} WHERE key = 'key' IF cas_target = 'value';"
> (start: 2016-12-22 10:14:27 EST)
> :2:ReadFailure: Error from server: code=1300 [Replica(s) failed to 
> execute read] message="Operation failed - received 0 responses and 1 
> failures" info={'failures': 1, 'received_responses': 0, 'required_responses': 
> 1, 'consistency': 'QUORUM'}
> {code}
> and the following stack trace will be present in the system log:
> {noformat}
> WARN  15:14:28 Uncaught exception on thread 
> Thread[SharedPool-Worker-10,10,main]: {}
> java.lang.RuntimeException: java.lang.NullPointerException
>   at 
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2476)
>  ~[main/:na]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_101]
>   at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  ~[main/:na]
>   at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
>  [main/:na]
>   at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) 
> [main/:na]
>   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> Caused by: java.lang.NullPointerException: null
>   at 
> org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:617)
>  ~[main/:na]
>   at 
> org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:569)
>  ~[main/:na]
>   at 
>

[jira] [Updated] (CASSANDRA-13173) Reloading logback.xml does not work

2017-02-06 Thread Joshua McKenzie (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-13173:

Reviewer: Yuki Morishita

> Reloading logback.xml does not work
> ---
>
> Key: CASSANDRA-13173
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13173
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Minor
> Fix For: 3.0.x
>
>
> Regression of CASSANDRA-12535
> Reloading of logback.xml is broken by CASSANDRA-12535 because the delegate 
> {{ReconfigureOnChangeFilter}} is not properly initialized.
> (Broken in 3.0.11 + 3.10)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (CASSANDRA-13124) Send ack when received hint is non-local

2017-02-06 Thread Joshua McKenzie (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie reassigned CASSANDRA-13124:
---

Assignee: Stefan Podkowinski
Reviewer: Paulo Motta

> Send ack when received hint is non-local
> 
>
> Key: CASSANDRA-13124
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13124
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Stefan Podkowinski
>Priority: Minor
>  Labels: hintedhandoff
>
> CASSANDRA-12905 changed the hints path to be asynchronous in the normal case, 
> but when the hint is non-local and should be stored (ie. on decommission) an 
> ack is not sent so the operation does not complete successfully.
> CASSANDRA-13058 fixed this for 3.0+, but for 3.11+ but there is an additional 
> concern which on 3.0 is that non-acked hints are being completed successfully 
> so we should probably look into this as well here.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13108) Uncaught exeption stack traces not logged

2017-02-06 Thread Joshua McKenzie (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-13108:

Reviewer: Tyler Hobbs

> Uncaught exeption stack traces not logged
> -
>
> Key: CASSANDRA-13108
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13108
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
>Reporter: Christopher Batey
>Assignee: Christopher Batey
>Priority: Trivial
>
> In a refactoring to parameterized logging we lost the stack traces of 
> uncaught exceptions. This means, apart from the thread, I have no idea where 
> they come from e.g.
> {code}
> ERROR [OptionalTasks:1] 2017-01-06 12:53:14,204 CassandraDaemon.java:231 - 
> Exception in thread Thread[OptionalTasks:1,5,main]
> java.lang.NullPointerException: null
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13109) Lightweight transactions temporarily fail after upgrade from 2.1 to 3.0

2017-02-06 Thread Joshua McKenzie (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-13109:

Reproduced In: 3.0.10, 3.0.9, 3.0.8  (was: 3.0.8, 3.0.9, 3.0.10)
 Reviewer: Sylvain Lebresne

> Lightweight transactions temporarily fail after upgrade from 2.1 to 3.0
> ---
>
> Key: CASSANDRA-13109
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13109
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Samuel Klock
> Attachments: 13109-3.0.txt
>
>
> We've observed this upgrading from 2.1.15 to 3.0.8 and from 2.1.16 to 3.0.10: 
> some lightweight transactions executed on upgraded nodes fail with a read 
> failure.  The following conditions seem relevant to this occurring:
> * The transaction must be conditioned on the current value of at least one 
> column, e.g., {{IF NOT EXISTS}} transactions don't seem to be affected.
> * There should be a collection column (in our case, a map) defined on the 
> table on which the transaction is executed.
> * The transaction should be executed before sstables on the node are 
> upgraded.  The failure does not occur after the sstables have been upgraded 
> (whether via {{nodetool upgradesstables}} or effectively via compaction).
> * Upgraded nodes seem to be able to participate in lightweight transactions 
> as long as they're not the coordinator.
> * The values in the row being manipulated by the transaction must have been 
> consistently manipulated by lightweight transactions (perhaps the existence 
> of Paxos state for the partition is somehow relevant?).
> * In 3.0.10, it _seems_ to be necessary to have the partition split across 
> multiple legacy sstables.  This was not necessary to reproduce the bug in 
> 3.0.8 or .9.
> For applications affected by this bug, a possible workaround is to prevent 
> nodes being upgraded from coordinating requests until sstables have been 
> upgraded.
> We're able to reproduce this when upgrading from 2.1.16 to 3.0.10 with the 
> following steps on a single-node cluster using a mostly pristine 
> {{cassandra.yaml}} from the source distribution.
> # Start Cassandra-2.1.16 on the node.
> # Create a table with a collection column and insert some data into it.
> {code:sql}
> CREATE KEYSPACE test WITH REPLICATION = {'class': 'SimpleStrategy', 
> 'replication_factor': 1};
> CREATE TABLE test.test (key TEXT PRIMARY KEY, cas_target TEXT, 
> some_collection MAP);
> INSERT INTO test.test (key, cas_target, some_collection) VALUES ('key', 
> 'value', {}) IF NOT EXISTS;
> {code}
> # Flush the row to an sstable: {{nodetool flush}}.
> # Update the row:
> {code:sql}
> UPDATE test.test SET cas_target = 'newvalue', some_collection = {} WHERE key 
> = 'key' IF cas_target = 'value';
> {code}
> # Drain the node: {{nodetool drain}}
> # Stop the node, upgrade to 3.0.10, and start the node.
> # Attempt to update the row again:
> {code:sql}
> UPDATE test.test SET cas_target = 'lastvalue' WHERE key = 'key' IF cas_target 
> = 'newvalue';
> {code}
> Using {{cqlsh}}, if the error is reproduced, the following output will be 
> returned:
> {code:sql}
> $ ./cqlsh <<< "UPDATE test.test SET cas_target = 'newvalue', some_collection 
> = {} WHERE key = 'key' IF cas_target = 'value';"
> (start: 2016-12-22 10:14:27 EST)
> :2:ReadFailure: Error from server: code=1300 [Replica(s) failed to 
> execute read] message="Operation failed - received 0 responses and 1 
> failures" info={'failures': 1, 'received_responses': 0, 'required_responses': 
> 1, 'consistency': 'QUORUM'}
> {code}
> and the following stack trace will be present in the system log:
> {noformat}
> WARN  15:14:28 Uncaught exception on thread 
> Thread[SharedPool-Worker-10,10,main]: {}
> java.lang.RuntimeException: java.lang.NullPointerException
>   at 
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2476)
>  ~[main/:na]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_101]
>   at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  ~[main/:na]
>   at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
>  [main/:na]
>   at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) 
> [main/:na]
>   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> Caused by: java.lang.NullPointerException: null
>   at 
> org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:617)
>  ~[main/:na]
>   at 
> org.apache.cassandra.db.rows.Row$Merger$ColumnDataReducer.getReduced(Row.java:569)
>  ~[main/:na]
>   at 
>

[jira] [Updated] (CASSANDRA-13070) unittest antiCompactionSizeTest is unstable

2017-02-06 Thread Joshua McKenzie (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-13070:

Reviewer: Yuki Morishita

> unittest antiCompactionSizeTest is unstable
> ---
>
> Key: CASSANDRA-13070
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13070
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
> Fix For: 3.0.x
>
> Attachments: 13070-3.0.txt
>
>
> Unittest 
> {{org.apache.cassandra.db.compaction.AntiCompactionTest.antiCompactionSizeTest}}
>  is failing from time to time on branch 3.0 
> ([13e9396|https://github.com/apache/cassandra/commit/13e939624d21eaf6e16d60b28636125e817ab286]):
> {code}
> [junit] Testcase: 
> antiCompactionSizeTest(org.apache.cassandra.db.compaction.AntiCompactionTest):
> FAILED
> [junit] expected:<1.29786604E8> but was:<1.13623354E8>
> [junit] junit.framework.AssertionFailedError: expected:<1.29786604E8> but 
> was:<1.13623354E8>
> [junit] at 
> org.apache.cassandra.db.compaction.AntiCompactionTest.antiCompactionSizeTest(AntiCompactionTest.java:164)
> {code}
> Run the single test 21 times:
> {code}
> for i in {0..20}; do echo ==$i; ant testsome 
> -Dtest.name=org.apache.cassandra.db.compaction.AntiCompactionTest 
> -Dtest.methods=antiCompactionSizeTest; echo $i $? >> ~/testresult; done
> {code}
> On mac, it passed for 5 times, on linux it passed for 6 times.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13006) Disable automatic heap dumps on OOM error

2017-02-06 Thread Joshua McKenzie (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-13006:

Status: Open  (was: Patch Available)

> Disable automatic heap dumps on OOM error
> -
>
> Key: CASSANDRA-13006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13006
> Project: Cassandra
>  Issue Type: Bug
>  Components: Configuration
>Reporter: anmols
>Assignee: Benjamin Lerer
>Priority: Minor
> Fix For: 3.0.9
>
> Attachments: 13006-3.0.9.txt
>
>
> With CASSANDRA-9861, a change was added to enable collecting heap dumps by 
> default if the process encountered an OOM error. These heap dumps are stored 
> in the Apache Cassandra home directory unless configured otherwise (see 
> [Cassandra Support 
> Document|https://support.datastax.com/hc/en-us/articles/204225959-Generating-and-Analyzing-Heap-Dumps]
>  for this feature).
>  
> The creation and storage of heap dumps aides debugging and investigative 
> workflows, but is not be desirable for a production environment where these 
> heap dumps may occupy a large amount of disk space and require manual 
> intervention for cleanups. 
>  
> Managing heap dumps on out of memory errors and configuring the paths for 
> these heap dumps are available as JVM options in JVM. The current behavior 
> conflicts with the Boolean JVM flag HeapDumpOnOutOfMemoryError. 
>  
> A patch can be proposed here that would make the heap dump on OOM error honor 
> the HeapDumpOnOutOfMemoryError flag. Users who would want to still generate 
> heap dumps on OOM errors can set the -XX:+HeapDumpOnOutOfMemoryError JVM 
> option.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-12513) IOException (No such file or directory) closing MessagingService's server socket (locally)

2017-02-06 Thread Robert Stupp (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp updated CASSANDRA-12513:
-
Status: Open  (was: Awaiting Feedback)

> IOException (No such file or directory) closing MessagingService's server 
> socket (locally)
> --
>
> Key: CASSANDRA-12513
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12513
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Minor
>
> _Sometimes_ the {{RemoveTest}} fails with the following exception. It's not 
> related to the test itself.
> The exception is risen in 
> {{ServerSocketChannelImpl.implCloseSelectableChannel}} where it checks that a 
> thread ID is non-zero. The {{thread}} instance field is set inside its accept 
> and poll methods. It looks like this is caused by some race condition - i.e. 
> stopping in debugger at certain points prevents it from being triggered.
> I could not find any misuse in the code base - but want to document this 
> issue.
> No difference between 8u92 and 8u102
> {code}
> INFO  [ACCEPT-/127.0.0.1] 2016-08-22 08:35:16,606 ?:? - MessagingService has 
> terminated the accept() thread
> java.io.IOError: java.io.IOException: No such file or directory
>   at 
> org.apache.cassandra.net.MessagingService.shutdown(MessagingService.java:914)
>   at org.apache.cassandra.service.RemoveTest.tearDown(RemoveTest.java:103)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
>   at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:159)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:262)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
> Caused by: java.io.IOException: No such file or directory
>   at sun.nio.ch.NativeThread.signal(Native Method)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.implCloseSelectableChannel(ServerSocketChannelImpl.java:292)
>   at 
> java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:234)
>   at 
> java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
>   at sun.nio.ch.ServerSocketAdaptor.close(ServerSocketAdaptor.java:137)
>   at 
> org.apache.cassandra.net.MessagingService$SocketThread.close(MessagingService.java:1249)
>   at 
> org.apache.cassandra.net.MessagingService.shutdown(MessagingService.java:904)
>   ... 22 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-12928) Assert error, 3.9 mutation stage

2017-02-06 Thread Joshua McKenzie (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-12928:

Reviewer: Jeff Jirsa

> Assert error, 3.9 mutation stage
> 
>
> Key: CASSANDRA-12928
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12928
> Project: Cassandra
>  Issue Type: Bug
> Environment: 3.9 
>Reporter: Jeff Jirsa
> Attachments: 0001-Bump-version-of-netty-all-to-4.1.6.patch
>
>
> {code}
> WARN  [MutationStage-341] 2016-11-17 18:39:18,781 
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread 
> Thread[MutationStage-341,5,main]: {}
> java.lang.AssertionError: null
>   at 
> io.netty.util.Recycler$WeakOrderQueue.(Recycler.java:225) 
> ~[netty-all-4.0.39.Final.jar:4.0.39.Final]
>   at 
> io.netty.util.Recycler$DefaultHandle.recycle(Recycler.java:180) 
> ~[netty-all-4.0.39.Final.jar:4.0.39.Final]
>   at io.netty.util.Recycler.recycle(Recycler.java:141) 
> ~[netty-all-4.0.39.Final.jar:4.0.39.Final]
>   at 
> org.apache.cassandra.utils.btree.BTree$Builder.recycle(BTree.java:836) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.utils.btree.BTree$Builder.build(BTree.java:1089) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.partitions.PartitionUpdate.build(PartitionUpdate.java:587)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.partitions.PartitionUpdate.maybeBuild(PartitionUpdate.java:577)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.partitions.PartitionUpdate.holder(PartitionUpdate.java:388)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.partitions.AbstractBTreePartition.unfilteredIterator(AbstractBTreePartition.java:177)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.partitions.AbstractBTreePartition.unfilteredIterator(AbstractBTreePartition.java:172)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.serializedSize(PartitionUpdate.java:868)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.Mutation$MutationSerializer.serializedSize(Mutation.java:456)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:257) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at org.apache.cassandra.db.Mutation.apply(Mutation.java:227) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at org.apache.cassandra.db.Mutation.apply(Mutation.java:241) 
> ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.service.StorageProxy$8.runMayThrow(StorageProxy.java:1347)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2539)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> ~[na:1.8.0_91]
>   at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
>  ~[apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
>  [apache-cassandra-3.9.jar:3.9]
>   at 
> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) 
> [apache-cassandra-3.9.jar:3.9]
>   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12513) IOException (No such file or directory) closing MessagingService's server socket (locally)

2017-02-06 Thread Robert Stupp (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-12513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854308#comment-15854308
 ] 

Robert Stupp commented on CASSANDRA-12513:
--

Oops. Must have been my second me...

> IOException (No such file or directory) closing MessagingService's server 
> socket (locally)
> --
>
> Key: CASSANDRA-12513
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12513
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Minor
>
> _Sometimes_ the {{RemoveTest}} fails with the following exception. It's not 
> related to the test itself.
> The exception is risen in 
> {{ServerSocketChannelImpl.implCloseSelectableChannel}} where it checks that a 
> thread ID is non-zero. The {{thread}} instance field is set inside its accept 
> and poll methods. It looks like this is caused by some race condition - i.e. 
> stopping in debugger at certain points prevents it from being triggered.
> I could not find any misuse in the code base - but want to document this 
> issue.
> No difference between 8u92 and 8u102
> {code}
> INFO  [ACCEPT-/127.0.0.1] 2016-08-22 08:35:16,606 ?:? - MessagingService has 
> terminated the accept() thread
> java.io.IOError: java.io.IOException: No such file or directory
>   at 
> org.apache.cassandra.net.MessagingService.shutdown(MessagingService.java:914)
>   at org.apache.cassandra.service.RemoveTest.tearDown(RemoveTest.java:103)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
>   at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:159)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:262)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
> Caused by: java.io.IOException: No such file or directory
>   at sun.nio.ch.NativeThread.signal(Native Method)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.implCloseSelectableChannel(ServerSocketChannelImpl.java:292)
>   at 
> java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:234)
>   at 
> java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
>   at sun.nio.ch.ServerSocketAdaptor.close(ServerSocketAdaptor.java:137)
>   at 
> org.apache.cassandra.net.MessagingService$SocketThread.close(MessagingService.java:1249)
>   at 
> org.apache.cassandra.net.MessagingService.shutdown(MessagingService.java:904)
>   ... 22 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (CASSANDRA-13186) Create index fails if the primary key is included, but docs claim it is supported

2017-02-06 Thread Ariel Weisberg (JIRA)

Ariel Weisberg created CASSANDRA-13186:
--

 Summary: Create index fails if the primary key is included, but 
docs claim it is supported
 Key: CASSANDRA-13186
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13186
 Project: Cassandra
  Issue Type: Bug
  Components: CQL
Reporter: Ariel Weisberg


{noformat}
cqlsh:foo> CREATE TABLE users (
   ...   userid text PRIMARY KEY,
   ...   first_name text,
   ...   last_name text,
   ...   emails set,
   ...   top_scores list,
   ...   todo map
   ... );
cqlsh:foo> create index bar on foo.users (userid, last_name);
InvalidRequest: Error from server: code=2200 [Invalid query] message="Cannot 
create secondary index on partition key column userid"
{noformat}

{quote}
 yes, it's a bug in CreateIndexStatement. The check to enforce the PK 
has only a single component is wrong
it considers each target in isolation, so it doesn't take into account that you 
might be creating a custom index on a PK component + something else
{quote}

http://docs.datastax.com/en/cql/3.3/cql/cql_reference/cqlCreateIndex.html
{quote}
Cassandra supports creating an index on most columns, excluding counter columns 
but including a clustering column of a compound primary key or on the partition 
(primary) key itself. 
{quote}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-12917) Increase error margin in SplitterTest

2017-02-06 Thread Joshua McKenzie (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-12917:

Reviewer: Branimir Lambov

> Increase error margin in SplitterTest
> -
>
> Key: CASSANDRA-12917
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12917
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>  Labels: lhf
> Fix For: 3.x
>
>
> SplitterTest is a randomized test - it generates random tokens and splits the 
> ranges in equal parts. Since it is random we sometimes get very big vnodes 
> right where we want a split and that makes the split unbalanced
> Bumping the error margin a bit will avoid these false positives.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12513) IOException (No such file or directory) closing MessagingService's server socket (locally)

2017-02-06 Thread Joshua McKenzie (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-12513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854301#comment-15854301
 ] 

Joshua McKenzie commented on CASSANDRA-12513:
-

... [~snazy]: Are you reporter, assignee, and asking yourself questions on this 
ticket?

> IOException (No such file or directory) closing MessagingService's server 
> socket (locally)
> --
>
> Key: CASSANDRA-12513
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12513
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Minor
>
> _Sometimes_ the {{RemoveTest}} fails with the following exception. It's not 
> related to the test itself.
> The exception is risen in 
> {{ServerSocketChannelImpl.implCloseSelectableChannel}} where it checks that a 
> thread ID is non-zero. The {{thread}} instance field is set inside its accept 
> and poll methods. It looks like this is caused by some race condition - i.e. 
> stopping in debugger at certain points prevents it from being triggered.
> I could not find any misuse in the code base - but want to document this 
> issue.
> No difference between 8u92 and 8u102
> {code}
> INFO  [ACCEPT-/127.0.0.1] 2016-08-22 08:35:16,606 ?:? - MessagingService has 
> terminated the accept() thread
> java.io.IOError: java.io.IOException: No such file or directory
>   at 
> org.apache.cassandra.net.MessagingService.shutdown(MessagingService.java:914)
>   at org.apache.cassandra.service.RemoveTest.tearDown(RemoveTest.java:103)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
>   at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:159)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
>   at 
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:262)
>   at 
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
> Caused by: java.io.IOException: No such file or directory
>   at sun.nio.ch.NativeThread.signal(Native Method)
>   at 
> sun.nio.ch.ServerSocketChannelImpl.implCloseSelectableChannel(ServerSocketChannelImpl.java:292)
>   at 
> java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel(AbstractSelectableChannel.java:234)
>   at 
> java.nio.channels.spi.AbstractInterruptibleChannel.close(AbstractInterruptibleChannel.java:115)
>   at sun.nio.ch.ServerSocketAdaptor.close(ServerSocketAdaptor.java:137)
>   at 
> org.apache.cassandra.net.MessagingService$SocketThread.close(MessagingService.java:1249)
>   at 
> org.apache.cassandra.net.MessagingService.shutdown(MessagingService.java:904)
>   ... 22 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-11723) Cassandra upgrade from 2.1.11 to 3.0.5 leads to unstable nodes (jemalloc to blame)

2017-02-06 Thread Joshua McKenzie (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-11723:

Resolution: Not A Problem
Status: Resolved  (was: Awaiting Feedback)

> Cassandra upgrade from 2.1.11 to 3.0.5 leads to unstable nodes (jemalloc to 
> blame)
> --
>
> Key: CASSANDRA-11723
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11723
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Stefano Ortolani
> Fix For: 3.0.x
>
>
> Upgrade seems fine, but any restart of the node might lead to a situation 
> where the node just dies after 30 seconds / 1 minute. 
> Nothing in the logs besides many "FailureDetector.java:456 - Ignoring 
> interval time of 3000892567 for /10.12.a.x" output every second (against all 
> other nodes) in debug.log plus some spurious GraphiteErrors/ReadRepair 
> notifications:
> {code:xml}
> DEBUG [GossipStage:1] 2016-05-05 22:29:03,921 FailureDetector.java:456 - 
> Ignoring interval time of 2373187360 for /10.12.a.x
> DEBUG [GossipStage:1] 2016-05-05 22:29:03,921 FailureDetector.java:456 - 
> Ignoring interval time of 2000276196 for /10.12.a.y
> DEBUG [ReadRepairStage:24] 2016-05-05 22:29:03,990 ReadCallback.java:234 - 
> Digest mismatch:
> org.apache.cassandra.service.DigestMismatchException: Mismatch for key 
> DecoratedKey(-152946356843306763, e859fdd2f264485f42030ce261e4e12e) 
> (d6e617ece3b7bec6138b52b8974b8cab vs 31becca666a62b3c4b2fc0bab9902718)
>   at 
> org.apache.cassandra.service.DigestResolver.resolve(DigestResolver.java:85) 
> ~[apache-cassandra-3.0.5.jar:3.0.5]
>   at 
> org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:225)
>  ~[apache-cassandra-3.0.5.jar:3.0.5]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_60]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_60]
>   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> DEBUG [GossipStage:1] 2016-05-05 22:29:04,841 FailureDetector.java:456 - 
> Ignoring interval time of 3000299340 for /10.12.33.5
> ERROR [metrics-graphite-reporter-1-thread-1] 2016-05-05 22:29:05,692 
> ScheduledReporter.java:119 - RuntimeException thrown from 
> GraphiteReporter#report. Exception was suppressed.
> java.lang.IllegalStateException: Unable to compute ceiling for max when 
> histogram overflowed
>   at 
> org.apache.cassandra.utils.EstimatedHistogram.rawMean(EstimatedHistogram.java:231)
>  ~[apache-cassandra-3.0.5.jar:3.0.5]
>   at 
> org.apache.cassandra.metrics.EstimatedHistogramReservoir$HistogramSnapshot.getMean(EstimatedHistogramReservoir.java:103)
>  ~[apache-cassandra-3.0.5.jar:3.0.5]
>   at 
> com.codahale.metrics.graphite.GraphiteReporter.reportHistogram(GraphiteReporter.java:252)
>  ~[metrics-graphite-3.1.0.jar:3.1.0]
>   at 
> com.codahale.metrics.graphite.GraphiteReporter.report(GraphiteReporter.java:166)
>  ~[metrics-graphite-3.1.0.jar:3.1.0]
>   at 
> com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) 
> ~[metrics-core-3.1.0.jar:3.1.0]
>   at 
> com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) 
> ~[metrics-core-3.1.0.jar:3.1.0]
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
> [na:1.8.0_60]
>   at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) 
> [na:1.8.0_60]
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>  [na:1.8.0_60]
>   at 
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>  [na:1.8.0_60]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  [na:1.8.0_60]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  [na:1.8.0_60]
>   at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
> {code}
> I know this is not much but nothing else gets to dmesg or to any other log. 
> Any suggestion how to debug this further?
> I upgraded two nodes so far, and it happened on both nodes.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-9639) size_estimates is inacurate in multi-dc clusters

2017-02-06 Thread Joshua McKenzie (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joshua McKenzie updated CASSANDRA-9639:
---
Reviewer: Paulo Motta

> size_estimates is inacurate in multi-dc clusters
> 
>
> Key: CASSANDRA-9639
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9639
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Sebastian Estevez
>Assignee: Chris Lohfink
>Priority: Minor
> Fix For: 3.0.x
>
>
> CASSANDRA-7688 introduced size_estimates to replace the thrift 
> describe_splits_ex command.
> Users have reported seeing estimates that are widely off in multi-dc clusters.
> system.size_estimates show the wrong range_start / range_end



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13041) Do not allow removal of a DC from system_auth replication settings if the DC has active Cassandra instances

2017-02-06 Thread Sam Tunnicliffe (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854216#comment-15854216
 ] 

Sam Tunnicliffe commented on CASSANDRA-13041:
-

Neither DropTest, nor the snippet pasted above actually emulate the behaviour 
of a real client as they call the statement's execute method directly. The 
restriction on dropping the keyspace, along with most other modifications to 
system tables, is enforced by 
{{ClientState::preventSystemKSSchemaModification}} which CQLTester tends to 
bypass by calling {{QueryProcessor::executeInternal}}. So a piece of internal 
code could perform modifications to the system tables (including system_auth), 
but a real client could not. CQLTester does have methods which exercise the 
client code path though, so we can fix the tests to use those.

Regarding the replication config, I don't see any harm in adding this 
constraint. Just to note though, it is possible to have authn/authz disabled 
for a particular DC, in which case having no replicas in that DC wouldn't cause 
a problem for connecting clients. I would imagine this is a pretty rare 
use-case though, and even in this case, it's generally such a tiny dataset that 
having it replicated even where it isn't strictly needed doesn't seem like much 
of a problem. 

If you're OK with this, I'd like to remove the changes to 
DropKeyspaceStatement, fix DropTest to use the same path as an actual client 
and add an entry in NEWS.txt. I've pushed a rebased branch with those changes 
and kicked off CI runs, if you wouldn't mind taking a look. 

||branch||testall||dtest||
|[13041-trunk|https://github.com/beobal/cassandra/tree/13041-trunk]|[testall|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-13041-trunk-testall]|[dtest|http://cassci.datastax.com/view/Dev/view/beobal/job/beobal-13041-trunk-dtest]|


> Do not allow removal of a DC from system_auth replication settings if the DC 
> has active Cassandra instances
> ---
>
> Key: CASSANDRA-13041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13041
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Nachiket Patil
>Assignee: Nachiket Patil
>Priority: Minor
> Fix For: 4.x
>
> Attachments: trunk.diff
>
>
> I don’t believe it is ever correct to remove a DC from the system_auth 
> replication settings while there are nodes up in that DC. Cassandra should 
> not allow this change if there are hosts which are currently members of the 
> cluster in that DC, as any request which is routed to these hosts will meet 
> an unavailable. Also dropping the keyspace system_auth should not be allowed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13041) Do not allow removal of a DC from system_auth replication settings if the DC has active Cassandra instances

2017-02-06 Thread Sam Tunnicliffe (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-13041:

Status: Awaiting Feedback  (was: In Progress)

> Do not allow removal of a DC from system_auth replication settings if the DC 
> has active Cassandra instances
> ---
>
> Key: CASSANDRA-13041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13041
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Nachiket Patil
>Assignee: Nachiket Patil
>Priority: Minor
> Fix For: 4.x
>
> Attachments: trunk.diff
>
>
> I don’t believe it is ever correct to remove a DC from the system_auth 
> replication settings while there are nodes up in that DC. Cassandra should 
> not allow this change if there are hosts which are currently members of the 
> cluster in that DC, as any request which is routed to these hosts will meet 
> an unavailable. Also dropping the keyspace system_auth should not be allowed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13041) Do not allow removal of a DC from system_auth replication settings if the DC has active Cassandra instances

2017-02-06 Thread Sam Tunnicliffe (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sam Tunnicliffe updated CASSANDRA-13041:

Status: In Progress  (was: Patch Available)

> Do not allow removal of a DC from system_auth replication settings if the DC 
> has active Cassandra instances
> ---
>
> Key: CASSANDRA-13041
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13041
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Distributed Metadata
>Reporter: Nachiket Patil
>Assignee: Nachiket Patil
>Priority: Minor
> Fix For: 4.x
>
> Attachments: trunk.diff
>
>
> I don’t believe it is ever correct to remove a DC from the system_auth 
> replication settings while there are nodes up in that DC. Cassandra should 
> not allow this change if there are hosts which are currently members of the 
> cluster in that DC, as any request which is routed to these hosts will meet 
> an unavailable. Also dropping the keyspace system_auth should not be allowed.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-11936) Add support for + and - operations on dates

2017-02-06 Thread Alex Petrov (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854201#comment-15854201
 ] 

Alex Petrov commented on CASSANDRA-11936:
-

I agree on all points. 

New patch looks great, along with the tests.

+1

> Add support for + and - operations on dates
> ---
>
> Key: CASSANDRA-11936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11936
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: CQL
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
> Fix For: 3.x
>
>
> For time series it can be interesting to allow queries with {{WHERE}} clause 
> like: {{... WHERE reading_time < now() - 2h}}
> In order to do that we need to add support for: {{+}} and {{-}} operation 
> with date.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-11936) Add support for + and - operations on dates

2017-02-06 Thread Alex Petrov (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-11936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Petrov updated CASSANDRA-11936:

Status: Ready to Commit  (was: Patch Available)

> Add support for + and - operations on dates
> ---
>
> Key: CASSANDRA-11936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11936
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: CQL
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
> Fix For: 3.x
>
>
> For time series it can be interesting to allow queries with {{WHERE}} clause 
> like: {{... WHERE reading_time < now() - 2h}}
> In order to do that we need to add support for: {{+}} and {{-}} operation 
> with date.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-10872) Debian Package does not prompt the user to review the config files; it just replaces them causing trouble (since the daemon starts by default)

2017-02-06 Thread Michael Shuler (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-10872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-10872:
---
Fix Version/s: (was: 3.0.x)
   (was: 2.2.x)
   (was: 2.1.x)
   (was: 3.x)

> Debian Package does not prompt the user to review the config files; it just 
> replaces them causing trouble (since the daemon starts by default)
> --
>
> Key: CASSANDRA-10872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10872
> Project: Cassandra
>  Issue Type: Bug
>  Components: Packaging
> Environment: Ubuntu 12.04, Cassandra 2.0.16 -> 2.0.17
>Reporter: Vasilis
>Assignee: Michael Shuler
>Priority: Critical
>  Labels: debian, packaging
>
> We run Cassandra 2.0.16 on Ubuntu 12.04 and were trying to upgrade to 2.0.17 
> due to CASSANDRA-9662. The problem is that during the upgrade the we were not 
> prompted how to handle cassandra-env.sh and cassandra-rackdc.properties. 
> The output from the upgrade:
> {panel}
> ...
> Setting up cassandra (2.0.17) ...
> Installing new version of config file /etc/cassandra/cassandra-env.sh ...
> Installing new version of config file 
> /etc/cassandra/cassandra-rackdc.properties ...
> vm.max_map_count = 1048575
> net.ipv4.tcp_keepalive_time = 300
> ...
> {panel}
> This meant that the nodes started automatically after the install with the 
> wrong DC name. 
> I don't think that these config files should have been replaced without the 
> admin being asked; this doesn't appear to comply with standard Debian 
> packages.
> Secondly if CASSANDRA-2356 was implemented the problem would not be as 
> severe; i.e. it would be possible to workaround this issue. Whereas 
> currently, there is no way to prevent the node when upgraded from starting in 
> the wrong DC.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-9320) test-burn target should be run occasionally

2017-02-06 Thread Michael Shuler (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-9320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-9320:
--
Fix Version/s: (was: 3.x)

> test-burn target should be run occasionally
> ---
>
> Key: CASSANDRA-9320
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9320
> Project: Cassandra
>  Issue Type: Test
>Reporter: Ariel Weisberg
>Assignee: Michael Shuler
>Priority: Minor
>
> The tests are all concurrency tests right now so they need to run on the 
> largest  # of cores we have available. The tests are not configured to run 
> very long right now, but the intent is that they run for longer periods (days 
> even).
> They aren't described as high value right now because the code under test 
> hasn't change since first introduced so we can defer setting this job up 
> until higher priority things are done.
> I think we should still run them at some low frequency so they don't rot or 
> some change doesn't sneak in that effects them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-8204) Configure multi-JDK8 build/test jenkins matrix job

2017-02-06 Thread Michael Shuler (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-8204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-8204:
--
Fix Version/s: (was: 3.x)

> Configure multi-JDK8 build/test jenkins matrix job
> --
>
> Key: CASSANDRA-8204
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8204
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: Testing
>Reporter: Michael Shuler
>Assignee: Michael Shuler
>
> Set up Oracle JDK-8 and OpenJDK-8 build/test matrix job in Jenkins.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12201) Burst Hour Compaction Strategy

2017-02-06 Thread Pedro Gordo (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-12201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854024#comment-15854024
 ] 

Pedro Gordo commented on CASSANDRA-12201:
-

I was unable to start this last year because as I was about to, I suffered a 
wrist injury which prevented me from working for more than six months. I'm now 
resuming work on this, although I'll still spend a few days getting up to speed 
with C*.

I studied on the data structure for Cassandra 2.0 but from what I know, there 
were significant changes to 3.0, so I'll need to consider now which version 
I'll be working on. Let me know your opinion on this, please.

> Burst Hour Compaction Strategy
> --
>
> Key: CASSANDRA-12201
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12201
> Project: Cassandra
>  Issue Type: New Feature
>  Components: Compaction
>Reporter: Pedro Gordo
>   Original Estimate: 1,008h
>  Remaining Estimate: 1,008h
>
> Although it may be subject to changes, for the moment I plan to create a 
> strategy that will revolve around taking advantage of periods of the day 
> where there's less I/O on the cluster. This time of the day will be called 
> “Burst Hour” (BH), and hence the strategy will be named “Burst Hour 
> Compaction Strategy” (BHCS). 
> The following process would be fired during BH:
> 1. Read all the SSTables and detect which partition keys are present in more 
> than a configurable value which I'll call referenced_sstable_limit. This 
> value will be three by default.
> 2. Group all the repeated keys with a reference to the SSTables containing 
> them.
> 3. Calculate the total size of the SSTables which will be merged for the 
> first partition key on the list created in step 2. If the size calculated is 
> bigger than property which I'll call max_sstable_size (also configurable), 
> more than one table will be created in step 4.
> 4. During the merge, the data will be streamed from SSTables up to a point 
> when we have a size close to max_sstable_size. After we reach this point, the 
> stream is paused, and the new SSTable will be closed, becoming immutable. 
> Repeat the streaming process until we've merged all tables for the partition 
> key that we're iterating.
> 5. Cycle through the rest of the collection created in step 2 and remove any 
> SSTables which don't exist anymore because they were merged in step 5. An 
> alternative course of action here would be to, instead of removing the 
> SSTable from the collection, to change its reference to the SSTable(s) which 
> was created in step 5. 
> 6. Repeat from step 3 to step 6 until we traversed the entirety of the 
> collection created in step 2.
> This strategy addresses three issues of the existing compaction strategies:
> - Due to max_sstable_size_limit, there's no need to reserve disc space for a 
> huge compaction, as it can happen on STCS.
> - The number of SSTables that we need to read from to reply to a read query 
> will be consistently maintained at a low level and controllable through the 
> referenced_sstable_limit property. This addresses the scenario of STCS when 
> we might have to read from a lot of SSTables.
> - It removes the dependency of a continuous high I/O of LCS.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13103) incorrect jvm metric names

2017-02-06 Thread Stefan Podkowinski (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefan Podkowinski updated CASSANDRA-13103:
---
Status: Ready to Commit  (was: Patch Available)

> incorrect jvm metric names
> --
>
> Key: CASSANDRA-13103
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13103
> Project: Cassandra
>  Issue Type: Bug
>  Components: Observability
>Reporter: Alain Rastoul
>Assignee: Alain Rastoul
>Priority: Minor
>  Labels: lhf
> Fix For: 3.9, 3.x
>
> Attachments: Fix-incorrect-jvm-metric-names-CASSANDRA-13103.patch
>
>
> Some jvm metrics have a double dot in name like:
> jvm.memory..total.max , jvm.memory..total.init (etc).
> it seems that an extra dot is added at the end of the name in 
> CassandraDaemon.java, around line 367 (in 3.0.10):
> ...
> // enable metrics provided by metrics-jvm.jar
> CassandraMetricsRegistry.Metrics.register("jvm.buffers.", new 
> BufferPoolMetricSet(ManagementFactory.getPlatformMBeanServer()));
> CassandraMetricsRegistry.Metrics.register("jvm.gc.", new 
> GarbageCollectorMetricSet());
> CassandraMetricsRegistry.Metrics.register("jvm.memory.", new 
> MemoryUsageGaugeSet());
> and also added in append method of MetricRegistry.
> Call stack is:
> MetricRegistry>>registerAll(String prefix, MetricSet metrics)
> MetricRegistry>>static String name(String name, String... names)
> MetricRegistry>>static void append(StringBuilder builder, String part)
> and in append the dot is also added:
> ...
> if(builder.length() > 0) {
> builder.append('.');
> }
> builder.append(part);
> ...
> The codahale MetricRegistry class seems to have no recent modification of 
> name or append methods, so it look like a small bug.
> May be the fix could be to simply not to add  the final dot in the metric 
> name, ie  "jvm.buffers"  instead of "jvm.buffers."



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-13103) incorrect jvm metric names

2017-02-06 Thread Stefan Podkowinski (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853952#comment-15853952
 ] 

Stefan Podkowinski edited comment on CASSANDRA-13103 at 2/6/17 1:05 PM:


Thanks for the patch, [~arastoul]! This is a little strange. I remember that I 
had to add that single dot character while working on CASSANDRA-12312, or else 
the dot would have been absent between the two names. But I can't really 
reproduce this and the code and netcat clearly indicates otherwise. So I'd 
suggest to go with the proposed patch (2.2 upwards), as the code is clearly not 
working as intended now.

||2.2||3.0||trunk||
|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13103-2.2]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13103-3.0]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13103-trunk]|
|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13103-2.2-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13103-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13103-trunk-dtest/]|
|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13103-2.2-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13103-3.0-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13103-trunk-testall/]|

(please note that trunk contains extra code for unit test)



was (Author: spo...@gmail.com):
Thanks for the patch, [~arastoul]! This is a little strange. I remember that I 
had to add that single dot character while working on CASSANDRA-12312, or else 
the dot would have been absent between the two names. But I can't really 
reproduce this and the code and netcat clearly indicates otherwise. So I'd 
suggest to go with the proposed patch (2.2 upwards), as the code is clearly not 
working as intended now.

||2.2||3.0||trunk||
|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13103-2.2]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13103-3.0]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13103-trunk]|
|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13103-2.2-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13103-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13103-trunk-dtest/]|
|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13103-2.2-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13103-3.0-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13103-trunk-testall/]|



> incorrect jvm metric names
> --
>
> Key: CASSANDRA-13103
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13103
> Project: Cassandra
>  Issue Type: Bug
>  Components: Observability
>Reporter: Alain Rastoul
>Assignee: Alain Rastoul
>Priority: Minor
>  Labels: lhf
> Fix For: 3.9, 3.x
>
> Attachments: Fix-incorrect-jvm-metric-names-CASSANDRA-13103.patch
>
>
> Some jvm metrics have a double dot in name like:
> jvm.memory..total.max , jvm.memory..total.init (etc).
> it seems that an extra dot is added at the end of the name in 
> CassandraDaemon.java, around line 367 (in 3.0.10):
> ...
> // enable metrics provided by metrics-jvm.jar
> CassandraMetricsRegistry.Metrics.register("jvm.buffers.", new 
> BufferPoolMetricSet(ManagementFactory.getPlatformMBeanServer()));
> CassandraMetricsRegistry.Metrics.register("jvm.gc.", new 
> GarbageCollectorMetricSet());
> CassandraMetricsRegistry.Metrics.register("jvm.memory.", new 
> MemoryUsageGaugeSet());
> and also added in append method of MetricRegistry.
> Call stack is:
> MetricRegistry>>registerAll(String prefix, MetricSet metrics)
> MetricRegistry>>static String name(String name, String... names)
> MetricRegistry>>static void append(StringBuilder builder, String part)
> and in append the dot is also added:
> ...
> if(builder.length() > 0) {
> builder.append('.');
> }
> builder.append(part);
> ...
> The codahale MetricRegistry class seems to have no recent modification of 
> name or append methods, so it look like a small bug.
> May be the fix could be to simply not to add  the final dot in the metric 
> name, ie  "jvm.buffers"  instead of "jvm.buffers."



--
This message was sent by Atlassian JIRA

[jira] [Commented] (CASSANDRA-13103) incorrect jvm metric names

2017-02-06 Thread Stefan Podkowinski (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853952#comment-15853952
 ] 

Stefan Podkowinski commented on CASSANDRA-13103:


Thanks for the patch, [~arastoul]! This is a little strange. I remember that I 
had to add that single dot character while working on CASSANDRA-12312, or else 
the dot would have been absent between the two names. But I can't really 
reproduce this and the code and netcat clearly indicates otherwise. So I'd 
suggest to go with the proposed patch (2.2 upwards), as the code is clearly not 
working as intended now.

||2.2||3.0||trunk||
|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13103-2.2]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13103-3.0]|[branch|https://github.com/spodkowinski/cassandra/tree/CASSANDRA-13103-trunk]|
|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13103-2.2-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13103-3.0-dtest/]|[dtest|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13103-trunk-dtest/]|
|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13103-2.2-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13103-3.0-testall/]|[testall|http://cassci.datastax.com/view/Dev/view/spodkowinski/job/spodkowinski-CASSANDRA-13103-trunk-testall/]|



> incorrect jvm metric names
> --
>
> Key: CASSANDRA-13103
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13103
> Project: Cassandra
>  Issue Type: Bug
>  Components: Observability
>Reporter: Alain Rastoul
>Assignee: Alain Rastoul
>Priority: Minor
>  Labels: lhf
> Fix For: 3.9, 3.x
>
> Attachments: Fix-incorrect-jvm-metric-names-CASSANDRA-13103.patch
>
>
> Some jvm metrics have a double dot in name like:
> jvm.memory..total.max , jvm.memory..total.init (etc).
> it seems that an extra dot is added at the end of the name in 
> CassandraDaemon.java, around line 367 (in 3.0.10):
> ...
> // enable metrics provided by metrics-jvm.jar
> CassandraMetricsRegistry.Metrics.register("jvm.buffers.", new 
> BufferPoolMetricSet(ManagementFactory.getPlatformMBeanServer()));
> CassandraMetricsRegistry.Metrics.register("jvm.gc.", new 
> GarbageCollectorMetricSet());
> CassandraMetricsRegistry.Metrics.register("jvm.memory.", new 
> MemoryUsageGaugeSet());
> and also added in append method of MetricRegistry.
> Call stack is:
> MetricRegistry>>registerAll(String prefix, MetricSet metrics)
> MetricRegistry>>static String name(String name, String... names)
> MetricRegistry>>static void append(StringBuilder builder, String part)
> and in append the dot is also added:
> ...
> if(builder.length() > 0) {
> builder.append('.');
> }
> builder.append(part);
> ...
> The codahale MetricRegistry class seems to have no recent modification of 
> name or append methods, so it look like a small bug.
> May be the fix could be to simply not to add  the final dot in the metric 
> name, ie  "jvm.buffers"  instead of "jvm.buffers."



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13161) testall failure in org.apache.cassandra.db.commitlog.CommitLogDescriptorTest.testVersions

2017-02-06 Thread Benjamin Lerer (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Lerer updated CASSANDRA-13161:
---
   Resolution: Fixed
Fix Version/s: 4.0
   Status: Resolved  (was: Ready to Commit)

Committed in trunk at 21730666929886be5300c782319a852e46b22763

> testall failure in 
> org.apache.cassandra.db.commitlog.CommitLogDescriptorTest.testVersions
> -
>
> Key: CASSANDRA-13161
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13161
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Sean McCarthy
>Assignee: Benjamin Lerer
>  Labels: test-failure, testall
> Fix For: 4.0
>
> Attachments: 
> TEST-org.apache.cassandra.db.commitlog.CommitLogDescriptorTest.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_testall/1374/testReport/org.apache.cassandra.db.commitlog/CommitLogDescriptorTest/testVersions
> {code}
> Error Message
> expected:<11> but was:<10>
> {code}{code}
> Stacktrace
> junit.framework.AssertionFailedError: expected:<11> but was:<10>
>   at 
> org.apache.cassandra.db.commitlog.CommitLogDescriptorTest.testVersions(CommitLogDescriptorTest.java:84)
> {code}
> Related Failures:
> http://cassci.datastax.com/job/trunk_testall/1374/testReport/org.apache.cassandra.db.commitlog/CommitLogDescriptorTest/testVersions_compression/



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

cassandra git commit: Increase commitlog version

2017-02-06 Thread blerer

Repository: cassandra
Updated Branches:
  refs/heads/trunk 45c92ba91 -> 217306669


Increase commitlog version

patch by Benjamin Lerer; reviewed by Joel Knighton for CASSANDRA-13161


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/21730666
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/21730666
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/21730666

Branch: refs/heads/trunk
Commit: 21730666929886be5300c782319a852e46b22763
Parents: 45c92ba
Author: Benjamin Lerer 
Authored: Mon Feb 6 13:20:54 2017 +0100
Committer: Benjamin Lerer 
Committed: Mon Feb 6 13:20:54 2017 +0100

--
 CHANGES.txt | 1 +
 .../org/apache/cassandra/db/commitlog/CommitLogDescriptor.java  | 5 -
 2 files changed, 5 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/21730666/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index ee9c14e..b339e6e 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 4.0
+ * Increase commitlog version (CASSANDRA-13161)
  * Make TableMetadata immutable, optimize Schema (CASSANDRA-9425)
  * Refactor ColumnCondition (CASSANDRA-12981)
  * Parallelize streaming of different keyspaces (CASSANDRA-4663)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/21730666/src/java/org/apache/cassandra/db/commitlog/CommitLogDescriptor.java
--
diff --git 
a/src/java/org/apache/cassandra/db/commitlog/CommitLogDescriptor.java 
b/src/java/org/apache/cassandra/db/commitlog/CommitLogDescriptor.java
index 0ab191d..dd9414a 100644
--- a/src/java/org/apache/cassandra/db/commitlog/CommitLogDescriptor.java
+++ b/src/java/org/apache/cassandra/db/commitlog/CommitLogDescriptor.java
@@ -59,13 +59,14 @@ public class CommitLogDescriptor
 
 // We don't support anything pre-3.0
 public static final int VERSION_30 = 6;
+public static final int VERSION_40 = 7;
 
 /**
  * Increment this number if there is a changes in the commit log disc 
layout or MessagingVersion changes.
  * Note: make sure to handle {@link #getMessagingVersion()}
  */
 @VisibleForTesting
-public static final int current_version = VERSION_30;
+public static final int current_version = VERSION_40;
 
 final int version;
 public final long id;
@@ -206,6 +207,8 @@ public class CommitLogDescriptor
 {
 case VERSION_30:
 return MessagingService.VERSION_30;
+case VERSION_40:
+return MessagingService.VERSION_40;
 default:
 throw new IllegalStateException("Unknown commitlog version " + 
version);
 }

[jira] [Commented] (CASSANDRA-12966) Gossip thread slows down when using batch commit log

2017-02-06 Thread Stefan Podkowinski (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-12966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853883#comment-15853883
 ] 

Stefan Podkowinski commented on CASSANDRA-12966:


I'm +1 assuming tests look fine after rebasing. Maybe check if 
{{DatabaseDescriptor.createAllDirectories()}} is really required, as this is 
already called by the {{Keyspace}} static initializer.

> Gossip thread slows down when using batch commit log
> 
>
> Key: CASSANDRA-12966
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12966
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
>
> When using batch commit log mode, the Gossip thread slows down when peers 
> after a node bounces. This is because we perform a bunch of updates to the 
> peers table via {{SystemKeyspace.updatePeerInfo}}, which is a synchronized 
> method. How quickly each one of those individual updates takes depends on how 
> busy the system is at the time wrt write traffic. If the system is largely 
> quiescent, each update will be relatively quick (just waiting for the fsync). 
> If the system is getting a lot of writes, and depending on the 
> commitlog_sync_batch_window_in_ms, each of the Gossip thread's updates can 
> get stuck in the backlog, which causes the Gossip thread to stop processing. 
> We have observed in large clusters that a rolling restart causes triggers and 
> exacerbates this behavior. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-11936) Add support for + and - operations on dates

2017-02-06 Thread Benjamin Lerer (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853843#comment-15853843
 ] 

Benjamin Lerer commented on CASSANDRA-11936:


I created a new branch for trunk and made the fixes to it:

||trunk|[branch|https://github.com/apache/cassandra/compare/trunk...blerer:11936-trunk]|[utests|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-11936-trunk-testall/]|[dtests|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-11936-trunk-dtest/]|

bq. Do you think it's better skip supporting for simple date + sub-day sums? 
It'll currently throw an exception, although it could've been just an identity 
function on the first arg..

We discussed that subjet off-line and come to the conclusion that it was less 
misleading to reject those type of operations.
If somebody really need to add sub-day duration to a date, he can do: 
{{toTimestamp(myDate) + 2h30m}}

bq. looks like {{to/fromTimeInMillis}} is a bit too specific to be pulled up to 
the temporal, since it's supported by just a single temporal type here we can 
just make it static. Or maybe the semantics/ meaning of their particular method 
is very different from instance to instance.

The  {{to/fromTimeInMillis}} exists in {{SimpleDateType}}, {{TimestampType}} 
and {{TimeuuidType}}. They were simply already existing before.

bq. Just to consider that as an idea, for example, we could also allow 
subtracting date from date and yield duration type as a result. Do you think it 
can be useful?

As the duration type as a lot of limitations, I am not sure of how much useful 
it could be. I would be in favor of waiting to see if somebody really needs it 
before adding it.

> Add support for + and - operations on dates
> ---
>
> Key: CASSANDRA-11936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11936
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: CQL
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
> Fix For: 3.x
>
>
> For time series it can be interesting to allow queries with {{WHERE}} clause 
> like: {{... WHERE reading_time < now() - 2h}}
> In order to do that we need to add support for: {{+}} and {{-}} operation 
> with date.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-8460) Make it possible to move non-compacting sstables to slow/big storage in DTCS

2017-02-06 Thread Pavel Trukhanov (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-8460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853810#comment-15853810
 ] 

Pavel Trukhanov commented on CASSANDRA-8460:


Any plans on that one? 

And any thoughts with regards to TWCS?

[~bdeggleston] ? 

> Make it possible to move non-compacting sstables to slow/big storage in DTCS
> 
>
> Key: CASSANDRA-8460
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8460
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: Marcus Eriksson
>  Labels: doc-impacting, dtcs
> Fix For: 3.x
>
>
> It would be nice if we could configure DTCS to have a set of extra data 
> directories where we move the sstables once they are older than 
> max_sstable_age_days. 
> This would enable users to have a quick, small SSD for hot, new data, and big 
> spinning disks for data that is rarely read and never compacted.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13171) Setting -Dcassandra.join_ring=false is ambiguous

2017-02-06 Thread Stefan Podkowinski (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853690#comment-15853690
 ] 

Stefan Podkowinski commented on CASSANDRA-13171:


Did you use {{-Dcassandra.start_gossip=false}}? I'm missing a "Not starting 
gossip as requested." message from your logs.

> Setting -Dcassandra.join_ring=false is ambiguous
> 
>
> Key: CASSANDRA-13171
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13171
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Joe Olson
> Attachments: log2.txt, log.txt
>
>
> Setting -Dcassandra.join_ring=false in /etc/cassandra/jvm.options has 
> questionable results. From the log snippet below, the value is set to false, 
> and read. However, everything seems to happen as if it weren'tgossip 
> occurs, and streaming data via a bulk load comes in.
> (see attached log)
> I really need this node not to join the cluster, because it is behind on its 
> compaction, and will immediately crash (too many files open - OS ulimit is 
> already set to 30 temporarily). I know of no other way to do an "offline 
> compaction"



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13008) Add vm.max_map_count StartupCheck

2017-02-06 Thread Stefania (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15853687#comment-15853687
 ] 

Stefania commented on CASSANDRA-13008:
--

Sure, I'll try to review it in the next few days.

> Add vm.max_map_count StartupCheck
> -
>
> Key: CASSANDRA-13008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13008
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 3.0.x
>
> Attachments: 13008-3.0.txt
>
>
> It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563)
> When the max_map_count is low, it throws OOM exception, which is hard to link 
> to the real issue of vm.max_map_count.
> The problem happened when we tried to remove one node, all the other nodes in 
> cluster crashed. As each node was trying to load more local SSTable files for 
> streaming.
> I would suggest to add a StartupCheck for max_map_count, at least it could 
> give a warning message to help the debug.
> {code}
> ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to 
> be unstable.  Exiting forcefully due to:
> java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) 
> ~[na:1.8.0_112]
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (CASSANDRA-13185) cqlsh COPY doesn't support dates before 1900 or after 9999

2017-02-06 Thread Stefania (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania reassigned CASSANDRA-13185:


Assignee: Stefania

> cqlsh COPY doesn't support dates before 1900 or after 
> --
>
> Key: CASSANDRA-13185
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13185
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Tyler Hobbs
>Assignee: Stefania
> Fix For: 3.0.x, 3.x
>
>
> Although we fixed this problem for standard queries in CASSANDRA-10625, it 
> still exists for COPY.  In CASSANDRA-10625, we replaced datetimes outside of 
> the supported time range with a simple milliseconds-since-epoch long.  We may 
> not want to use the same solution for COPY, because we wouldn't be able to 
> load the same data back in through COPY.  Having consistency in the format of 
> values and support for loading exported data seems more important for COPY.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-11936) Add support for + and - operations on dates

2017-02-06 Thread Alex Petrov (JIRA)


[ 
https://issues.apache.org/jira/browse/CASSANDRA-11936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852007#comment-15852007
 ] 

Alex Petrov edited comment on CASSANDRA-11936 at 2/6/17 8:34 AM:
-

Thank you for the patch, a couple of very small things:

  * Do you think it's better skip supporting for simple date + sub-day sums? 
It'll currently throw an exception, although it could've been just an identity 
function on the first arg..
  * {{hasMillisecondPrecision}} has a [copy-pasted 
comment|https://github.com/apache/cassandra/compare/trunk...blerer:11936-3.X#diff-5cadfa6d134cd8bbd2c32874ac577c55R369]
  * the table 
[here|https://github.com/apache/cassandra/compare/trunk...blerer:11936-3.X#diff-1821fccd59a15282b84fe87315856c88R81]
 looks a little bit redundant. Wouldn't it be better to just specify the 
available conversions?
  * the {{addTo}} and {{substractFrom}} methods 
[here|https://github.com/apache/cassandra/compare/trunk...blerer:11936-3.X#diff-5cadfa6d134cd8bbd2c32874ac577c55R272]
 can be generalised and the code can be reused. We can use just one method and 
pass the sign.
  * looks like {{to/fromTimeInMillis}} is a bit too specific to be pulled up to 
the temporal, since it's supported by just a single temporal type 
[here|https://github.com/apache/cassandra/compare/trunk...blerer:11936-3.X#diff-218693a72559ba82aaf3fb033ab4c68fR47]
 we can just make it static. Or maybe the semantics/ meaning of their 
particular method is very different from instance to instance.

Just to consider that as an idea, for example, we could also allow subtracting 
date from date and yield duration type as a result. Do you think it can be 
useful?

Also it seems that we should start thinking about how to implement overloads in 
CQL. Currently the operation seems to be "type-aware" ({{executeOnTemporals}} 
method suggests it). We could just register all permutations with their 
implementations as independent functions (a bit like sum/avg aggregate 
functions are implemented). 


was (Author: ifesdjeen):
Thank you for the patch, a couple of very small things:

  * Why does {{TimeType}} not support to/from nanoseconds conversion 
(subsequently, also addition and subtraction operations..
  * Do you think it's better skip supporting for simple date + sub-day sums? 
It'll currently throw an exception, although it could've been just an identity 
function on the first arg..
  * {{hasMillisecondPrecision}} has a [copy-pasted 
comment|https://github.com/apache/cassandra/compare/cassandra-3.X...blerer:11936-3.X#diff-5cadfa6d134cd8bbd2c32874ac577c55R366]

> Add support for + and - operations on dates
> ---
>
> Key: CASSANDRA-11936
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11936
> Project: Cassandra
>  Issue Type: Sub-task
>  Components: CQL
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
> Fix For: 3.x
>
>
> For time series it can be interesting to allow queries with {{WHERE}} clause 
> like: {{... WHERE reading_time < now() - 2h}}
> In order to do that we need to add support for: {{+}} and {{-}} operation 
> with date.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13008) Add vm.max_map_count StartupCheck

2017-02-06 Thread Stefania (JIRA)


 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stefania updated CASSANDRA-13008:
-
Reviewer: Stefania

> Add vm.max_map_count StartupCheck
> -
>
> Key: CASSANDRA-13008
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13008
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Configuration
>Reporter: Jay Zhuang
>Assignee: Jay Zhuang
>Priority: Minor
> Fix For: 3.0.x
>
> Attachments: 13008-3.0.txt
>
>
> It's recommended to set {{vm.max_map_count}} to 1048575 (CASSANDRA-3563)
> When the max_map_count is low, it throws OOM exception, which is hard to link 
> to the real issue of vm.max_map_count.
> The problem happened when we tried to remove one node, all the other nodes in 
> cluster crashed. As each node was trying to load more local SSTable files for 
> streaming.
> I would suggest to add a StartupCheck for max_map_count, at least it could 
> give a warning message to help the debug.
> {code}
> ERROR [STREAM-IN-] JVMStabilityInspector.java:140 - JVM state determined to 
> be unstable.  Exiting forcefully due to:
> java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_112]
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) 
> ~[na:1.8.0_112]
> at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:152) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:280)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:216)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:173)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:70) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:58) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:96) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.CompressedSegmentedFile.(CompressedSegmentedFile.java:47)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTableWriter.java:276)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWriter.java:50)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:313)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:173)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:184)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.format.SSTableWriter.finish(SSTableWriter.java:213)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.finish(SimpleSSTableMultiWriter.java:56)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamReceiveTask.received(StreamReceiveTask.java:109)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:599) 
> ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:482)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at 
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:296)
>  ~[apache-cassandra-3.0.10.jar:3.0.10]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

76 matches

Mail list logo