[jira] [Commented] (CASSANDRA-13241) Lower default chunk_length_in_kb from 64kb to 4kb

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887451#comment-15887451
 ] 

Benjamin Roth commented on CASSANDRA-13241:
---

[~aweisberg] I didn't really get the point of your comment. Would you like to 
explain?

[~jjirsa]
I understand you consideration. A default value should avoid worst cases for 
most or all people and not optimize one case. So maybe yes, we could choose sth 
in between.
Do you see a way to offer a recommendation to users similar to the comments of 
cassandra.yaml. IMHO this table option is somewhat hidden for the average user 
but may have a huge impact on your overall server load and your latency.

> Lower default chunk_length_in_kb from 64kb to 4kb
> -
>
> Key: CASSANDRA-13241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13241
> Project: Cassandra
>  Issue Type: Wish
>  Components: Core
>Reporter: Benjamin Roth
>
> Having a too low chunk size may result in some wasted disk space. A too high 
> chunk size may lead to massive overreads and may have a critical impact on 
> overall system performance.
> In my case, the default chunk size lead to peak read IOs of up to 1GB/s and 
> avg reads of 200MB/s. After lowering chunksize (of course aligned with read 
> ahead), the avg read IO went below 20 MB/s, rather 10-15MB/s.
> The risk of (physical) overreads is increasing with lower (page cache size) / 
> (total data size) ratio.
> High chunk sizes are mostly appropriate for bigger payloads pre request but 
> if the model consists rather of small rows or small resultsets, the read 
> overhead with 64kb chunk size is insanely high. This applies for example for 
> (small) skinny rows.
> Please also see here:
> https://groups.google.com/forum/#!topic/scylladb-dev/j_qXSP-6-gY
> To give you some insights what a difference it can make (460GB data, 128GB 
> RAM):
> - Latency of a quite large CF: https://cl.ly/1r3e0W0S393L
> - Disk throughput: https://cl.ly/2a0Z250S1M3c
> - This shows, that the request distribution remained the same, so no "dynamic 
> snitch magic": https://cl.ly/3E0t1T1z2c0J



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (CASSANDRA-13010) nodetool compactionstats should say which disk a compaction is writing to

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa reassigned CASSANDRA-13010:
--

Assignee: Alex Lourie

> nodetool compactionstats should say which disk a compaction is writing to
> -
>
> Key: CASSANDRA-13010
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13010
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jon Haddad
>Assignee: Alex Lourie
>  Labels: lhf
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13010) nodetool compactionstats should say which disk a compaction is writing to

[Alex Lourie (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887049#comment-15887049
 ] 

Alex Lourie commented on CASSANDRA-13010:
-

[~jjirsa] Thanks!

[~rustyrazorblade] please be gentle :-)

> nodetool compactionstats should say which disk a compaction is writing to
> -
>
> Key: CASSANDRA-13010
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13010
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jon Haddad
>  Labels: lhf
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13010) nodetool compactionstats should say which disk a compaction is writing to

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13010:
---
Reviewer: Jon Haddad
  Status: Patch Available  (was: Open)

> nodetool compactionstats should say which disk a compaction is writing to
> -
>
> Key: CASSANDRA-13010
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13010
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jon Haddad
>  Labels: lhf
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13010) nodetool compactionstats should say which disk a compaction is writing to

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15887028#comment-15887028
 ] 

Jeff Jirsa commented on CASSANDRA-13010:


[~rustyrazorblade] says he's happy to review for you.



> nodetool compactionstats should say which disk a compaction is writing to
> -
>
> Key: CASSANDRA-13010
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13010
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jon Haddad
>  Labels: lhf
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13038) 33% of compaction time spent in StreamingHistogram.update()

[Jeff Jirsa (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeff Jirsa updated CASSANDRA-13038:
---
   Resolution: Fixed
 Reviewer: Nate McCall
Fix Version/s: 3.11.0
   3.0.12
   Status: Resolved  (was: Patch Available)

Thanks [~iksaif] and [~zznate] - committed as 
{{a5ce963117acf5e4cf0a31057551f2f42385c398}} to 3.0.12 and merged to 3.11.0 and 
trunk. 

> 33% of compaction time spent in StreamingHistogram.update()
> ---
>
> Key: CASSANDRA-13038
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13038
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Corentin Chary
>Assignee: Jeff Jirsa
> Fix For: 3.0.12, 3.11.0
>
> Attachments: compaction-speedup.patch, 
> compaction-streaminghistrogram.png, profiler-snapshot.nps
>
>
> With the following table, that contains a *lot* of cells: 
> {code}
> CREATE TABLE biggraphite.datapoints_11520p_60s (
> metric uuid,
> time_start_ms bigint,
> offset smallint,
> count int,
> value double,
> PRIMARY KEY ((metric, time_start_ms), offset)
> ) WITH CLUSTERING ORDER BY (offset DESC);
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 
> 'compaction_window_size': '6', 'compaction_window_unit': 'HOURS', 
> 'max_threshold': '32', 'min_threshold': '6'}
> Keyspace : biggraphite
> Read Count: 1822
> Read Latency: 1.8870054884742042 ms.
> Write Count: 2212271647
> Write Latency: 0.027705127678653473 ms.
> Pending Flushes: 0
> Table: datapoints_11520p_60s
> SSTable count: 47
> Space used (live): 300417555945
> Space used (total): 303147395017
> Space used by snapshots (total): 0
> Off heap memory used (total): 207453042
> SSTable Compression Ratio: 0.4955200053039823
> Number of keys (estimate): 16343723
> Memtable cell count: 220576
> Memtable data size: 17115128
> Memtable off heap memory used: 0
> Memtable switch count: 2872
> Local read count: 0
> Local read latency: NaN ms
> Local write count: 1103167888
> Local write latency: 0.025 ms
> Pending flushes: 0
> Percent repaired: 0.0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 105118296
> Bloom filter off heap memory used: 106547192
> Index summary off heap memory used: 27730962
> Compression metadata off heap memory used: 73174888
> Compacted partition minimum bytes: 61
> Compacted partition maximum bytes: 51012
> Compacted partition mean bytes: 7899
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> It looks like a good chunk of the compaction time is lost in 
> StreamingHistogram.update() (which is used to store the estimated tombstone 
> drop times).
> This could be caused by a huge number of different deletion times which would 
> makes the bin huge but it this histogram should be capped to 100 keys. It's 
> more likely caused by the huge number of cells.
> A simple solutions could be to only take into accounts part of the cells, the 
> fact the this table has a TWCS also gives us an additional hint that sampling 
> deletion times would be fine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[4/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[jjirsa]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/942b83ca
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/942b83ca
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/942b83ca

Branch: refs/heads/trunk
Commit: 942b83ca93d6d2380a830cb775e8506cc1cf0324
Parents: 1dc1aa1 a5ce963
Author: Jeff Jirsa 
Authored: Mon Feb 27 17:22:31 2017 -0800
Committer: Jeff Jirsa 
Committed: Mon Feb 27 17:23:17 2017 -0800

--
 CHANGES.txt |   1 +
 .../apache/cassandra/io/sstable/SSTable.java|   2 +
 .../io/sstable/metadata/MetadataCollector.java  |   2 +-
 .../cassandra/utils/StreamingHistogram.java | 133 --
 .../microbench/StreamingHistogramBench.java | 405 +++
 .../db/compaction/CompactionsTest.java  |   3 +
 .../DateTieredCompactionStrategyTest.java   |   4 +
 .../LeveledCompactionStrategyTest.java  |   4 +
 .../SizeTieredCompactionStrategyTest.java   |   4 +
 .../cassandra/db/compaction/TTLExpiryTest.java  |   4 +
 .../TimeWindowCompactionStrategyTest.java   |   4 +
 .../cassandra/utils/StreamingHistogramTest.java |  40 +-
 12 files changed, 569 insertions(+), 37 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/942b83ca/CHANGES.txt
--
diff --cc CHANGES.txt
index f5b9d28,1100bfd..1810cae
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,13 -1,5 +1,14 @@@
 -3.0.12
 +3.11.0
 + * Fix equality comparisons of columns using the duration type 
(CASSANDRA-13174)
 + * Obfuscate password in stress-graphs (CASSANDRA-12233)
 + * Move to FastThreadLocalThread and FastThreadLocal (CASSANDRA-13034)
 + * nodetool stopdaemon errors out (CASSANDRA-13030)
 + * Tables in system_distributed should not use gcgs of 0 (CASSANDRA-12954)
 + * Fix primary index calculation for SASI (CASSANDRA-12910)
 + * More fixes to the TokenAllocator (CASSANDRA-12990)
 + * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
 +Merged from 3.0:
+  * Faster StreamingHistogram (CASSANDRA-13038)
   * Legacy deserializer can create unexpected boundary range tombstones 
(CASSANDRA-13237)
   * Remove unnecessary assertion from AntiCompactionTest (CASSANDRA-13070)
   * Fix cqlsh COPY for dates before 1900 (CASSANDRA-13185)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/942b83ca/src/java/org/apache/cassandra/io/sstable/SSTable.java
--
diff --cc src/java/org/apache/cassandra/io/sstable/SSTable.java
index 601f5a0,1e4488c..fdcd17f
--- a/src/java/org/apache/cassandra/io/sstable/SSTable.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTable.java
@@@ -59,7 -58,10 +59,9 @@@ public abstract class SSTabl
  {
  static final Logger logger = LoggerFactory.getLogger(SSTable.class);
  
 -
  public static final int TOMBSTONE_HISTOGRAM_BIN_SIZE = 100;
+ public static final int TOMBSTONE_HISTOGRAM_SPOOL_SIZE = 10;
+ public static final int TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS = 
Integer.valueOf(System.getProperty("cassandra.streaminghistogram.roundseconds", 
"60"));
  
  public final Descriptor descriptor;
  protected final Set components;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/942b83ca/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/942b83ca/src/java/org/apache/cassandra/utils/StreamingHistogram.java
--
diff --cc src/java/org/apache/cassandra/utils/StreamingHistogram.java
index a500450,fffa73e..6fde931
--- a/src/java/org/apache/cassandra/utils/StreamingHistogram.java
+++ b/src/java/org/apache/cassandra/utils/StreamingHistogram.java
@@@ -39,11 -39,11 +39,14 @@@ public class StreamingHistogra
  public static final StreamingHistogramSerializer serializer = new 
StreamingHistogramSerializer();
  
  // TreeMap to hold bins of histogram.
 -private final TreeMap bin;
 +// The key is a numeric type so we can avoid boxing/unboxing streams of 
different key types
 +// The value is a unboxed long array always of length == 1
 +// Serialized Histograms always writes with double keys for backwards 
compatibility
 +private final TreeMap bin;
  
+ // Keep a second, larger buffer to spool data in, before finalizing it 
into `bin`
 -private final TreeMap spool;
++private final TreeMap spool;
+ 
  // maximum bin size for this histogram

[2/6] cassandra git commit: Faster streaming histograms

[jjirsa]

Faster streaming histograms

In ttl-heavy use cases (especially tables with default time to live set), the
streaming histograms calculated during compaction and streaming are very 
inefficient.

This patch addresses that in two ways:
1) It creates a system property -Dcassandra.streaminghistogram.roundseconds=60,
and rounds all histograms to the next highest multiple of that value, and
2) Rather than maintaining a histogram of 100 bins that have to be merged
on every new value, we keep a temporary spool of 10 bins, and merge
down to the 100 bin final histogram only when the temporary spool overflows.

Patch by Jeff Jirsa; Reviewed by Nate McCall for CASSANDRA-13038


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a5ce9631
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a5ce9631
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a5ce9631

Branch: refs/heads/cassandra-3.11
Commit: a5ce963117acf5e4cf0a31057551f2f42385c398
Parents: ab71748
Author: Jeff Jirsa 
Authored: Mon Feb 13 19:51:58 2017 -0800
Committer: Jeff Jirsa 
Committed: Mon Feb 27 17:21:48 2017 -0800

--
 CHANGES.txt |   1 +
 .../apache/cassandra/io/sstable/SSTable.java|   3 +
 .../io/sstable/metadata/MetadataCollector.java  |   2 +-
 .../cassandra/utils/StreamingHistogram.java | 109 +++--
 .../microbench/StreamingHistogramBench.java | 405 +++
 .../db/compaction/CompactionsTest.java  |   3 +
 .../DateTieredCompactionStrategyTest.java   |   4 +
 .../LeveledCompactionStrategyTest.java  |   4 +
 .../SizeTieredCompactionStrategyTest.java   |   4 +
 .../cassandra/db/compaction/TTLExpiryTest.java  |   4 +
 .../TimeWindowCompactionStrategyTest.java   |   4 +
 .../cassandra/utils/StreamingHistogramTest.java |  39 +-
 12 files changed, 551 insertions(+), 31 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a5ce9631/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 386029e..1100bfd 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.12
+ * Faster StreamingHistogram (CASSANDRA-13038)
  * Legacy deserializer can create unexpected boundary range tombstones 
(CASSANDRA-13237)
  * Remove unnecessary assertion from AntiCompactionTest (CASSANDRA-13070)
  * Fix cqlsh COPY for dates before 1900 (CASSANDRA-13185)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a5ce9631/src/java/org/apache/cassandra/io/sstable/SSTable.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTable.java 
b/src/java/org/apache/cassandra/io/sstable/SSTable.java
index 923ef82..1e4488c 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTable.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTable.java
@@ -58,7 +58,10 @@ public abstract class SSTable
 {
 static final Logger logger = LoggerFactory.getLogger(SSTable.class);
 
+
 public static final int TOMBSTONE_HISTOGRAM_BIN_SIZE = 100;
+public static final int TOMBSTONE_HISTOGRAM_SPOOL_SIZE = 10;
+public static final int TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS = 
Integer.valueOf(System.getProperty("cassandra.streaminghistogram.roundseconds", 
"60"));
 
 public final Descriptor descriptor;
 protected final Set components;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a5ce9631/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
--
diff --git 
a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java 
b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
index 4a67623..3b13cf4 100644
--- a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
+++ b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
@@ -61,7 +61,7 @@ public class MetadataCollector implements 
PartitionStatisticsCollector
 
 static StreamingHistogram defaultTombstoneDropTimeHistogram()
 {
-return new StreamingHistogram(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE);
+return new StreamingHistogram(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE, 
SSTable.TOMBSTONE_HISTOGRAM_SPOOL_SIZE, 
SSTable.TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS);
 }
 
 public static StatsMetadata defaultStatsMetadata()

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a5ce9631/src/java/org/apache/cassandra/utils/StreamingHistogram.java
--
diff --git a/src/java/org/apache/cassandra/utils/StreamingHistogram.java 

[6/6] cassandra git commit: Merge branch 'cassandra-3.11' into trunk

[jjirsa]

Merge branch 'cassandra-3.11' into trunk


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/d24f4c68
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/d24f4c68
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/d24f4c68

Branch: refs/heads/trunk
Commit: d24f4c68d0889e0934f30c4590657bda5dc063c4
Parents: 57f5e36 942b83c
Author: Jeff Jirsa 
Authored: Mon Feb 27 17:23:28 2017 -0800
Committer: Jeff Jirsa 
Committed: Mon Feb 27 17:24:15 2017 -0800

--
 CHANGES.txt |   1 +
 .../apache/cassandra/io/sstable/SSTable.java|   2 +
 .../io/sstable/metadata/MetadataCollector.java  |   2 +-
 .../cassandra/utils/StreamingHistogram.java | 133 --
 .../microbench/StreamingHistogramBench.java | 405 +++
 .../db/compaction/CompactionsTest.java  |   3 +
 .../DateTieredCompactionStrategyTest.java   |   4 +
 .../LeveledCompactionStrategyTest.java  |   4 +
 .../SizeTieredCompactionStrategyTest.java   |   4 +
 .../cassandra/db/compaction/TTLExpiryTest.java  |   4 +
 .../TimeWindowCompactionStrategyTest.java   |   4 +
 .../cassandra/utils/StreamingHistogramTest.java |  40 +-
 12 files changed, 569 insertions(+), 37 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/d24f4c68/CHANGES.txt
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d24f4c68/src/java/org/apache/cassandra/io/sstable/SSTable.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d24f4c68/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d24f4c68/test/unit/org/apache/cassandra/db/compaction/CompactionsTest.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d24f4c68/test/unit/org/apache/cassandra/db/compaction/DateTieredCompactionStrategyTest.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d24f4c68/test/unit/org/apache/cassandra/db/compaction/LeveledCompactionStrategyTest.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d24f4c68/test/unit/org/apache/cassandra/db/compaction/SizeTieredCompactionStrategyTest.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/d24f4c68/test/unit/org/apache/cassandra/db/compaction/TTLExpiryTest.java
--
diff --cc test/unit/org/apache/cassandra/db/compaction/TTLExpiryTest.java
index 6e4ba0b,78ba707..615b206
--- a/test/unit/org/apache/cassandra/db/compaction/TTLExpiryTest.java
+++ b/test/unit/org/apache/cassandra/db/compaction/TTLExpiryTest.java
@@@ -58,19 -57,23 +58,23 @@@ public class TTLExpiryTes
  public static void defineSchema() throws ConfigurationException
  {
  SchemaLoader.prepareServer();
+ 
+ // Disable tombstone histogram rounding for tests
+ System.setProperty("cassandra.streaminghistogram.roundseconds", "1");
+ 
  SchemaLoader.createKeyspace(KEYSPACE1,
  KeyspaceParams.simple(1),
 -CFMetaData.Builder.create(KEYSPACE1, 
CF_STANDARD1)
 -  
.addPartitionKey("pKey", AsciiType.instance)
 -  
.addRegularColumn("col1", AsciiType.instance)
 -  
.addRegularColumn("col", AsciiType.instance)
 -  
.addRegularColumn("col311", AsciiType.instance)
 -  
.addRegularColumn("col2", AsciiType.instance)
 -  
.addRegularColumn("col3", AsciiType.instance)
 -  
.addRegularColumn("col7", AsciiType.instance)
 -  
.addRegularColumn("col8", MapType.getInstance(AsciiType.instance, 
AsciiType.instance, true))
 -  
.addRegularColumn("shadow", AsciiType.instance)
 -  
.build().gcGraceSeconds(0));
 +TableMetadata.builder(KEYSPACE1, 

[3/6] cassandra git commit: Faster streaming histograms

[jjirsa]

Faster streaming histograms

In ttl-heavy use cases (especially tables with default time to live set), the
streaming histograms calculated during compaction and streaming are very 
inefficient.

This patch addresses that in two ways:
1) It creates a system property -Dcassandra.streaminghistogram.roundseconds=60,
and rounds all histograms to the next highest multiple of that value, and
2) Rather than maintaining a histogram of 100 bins that have to be merged
on every new value, we keep a temporary spool of 10 bins, and merge
down to the 100 bin final histogram only when the temporary spool overflows.

Patch by Jeff Jirsa; Reviewed by Nate McCall for CASSANDRA-13038


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a5ce9631
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a5ce9631
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a5ce9631

Branch: refs/heads/trunk
Commit: a5ce963117acf5e4cf0a31057551f2f42385c398
Parents: ab71748
Author: Jeff Jirsa 
Authored: Mon Feb 13 19:51:58 2017 -0800
Committer: Jeff Jirsa 
Committed: Mon Feb 27 17:21:48 2017 -0800

--
 CHANGES.txt |   1 +
 .../apache/cassandra/io/sstable/SSTable.java|   3 +
 .../io/sstable/metadata/MetadataCollector.java  |   2 +-
 .../cassandra/utils/StreamingHistogram.java | 109 +++--
 .../microbench/StreamingHistogramBench.java | 405 +++
 .../db/compaction/CompactionsTest.java  |   3 +
 .../DateTieredCompactionStrategyTest.java   |   4 +
 .../LeveledCompactionStrategyTest.java  |   4 +
 .../SizeTieredCompactionStrategyTest.java   |   4 +
 .../cassandra/db/compaction/TTLExpiryTest.java  |   4 +
 .../TimeWindowCompactionStrategyTest.java   |   4 +
 .../cassandra/utils/StreamingHistogramTest.java |  39 +-
 12 files changed, 551 insertions(+), 31 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a5ce9631/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 386029e..1100bfd 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.12
+ * Faster StreamingHistogram (CASSANDRA-13038)
  * Legacy deserializer can create unexpected boundary range tombstones 
(CASSANDRA-13237)
  * Remove unnecessary assertion from AntiCompactionTest (CASSANDRA-13070)
  * Fix cqlsh COPY for dates before 1900 (CASSANDRA-13185)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a5ce9631/src/java/org/apache/cassandra/io/sstable/SSTable.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTable.java 
b/src/java/org/apache/cassandra/io/sstable/SSTable.java
index 923ef82..1e4488c 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTable.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTable.java
@@ -58,7 +58,10 @@ public abstract class SSTable
 {
 static final Logger logger = LoggerFactory.getLogger(SSTable.class);
 
+
 public static final int TOMBSTONE_HISTOGRAM_BIN_SIZE = 100;
+public static final int TOMBSTONE_HISTOGRAM_SPOOL_SIZE = 10;
+public static final int TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS = 
Integer.valueOf(System.getProperty("cassandra.streaminghistogram.roundseconds", 
"60"));
 
 public final Descriptor descriptor;
 protected final Set components;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a5ce9631/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
--
diff --git 
a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java 
b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
index 4a67623..3b13cf4 100644
--- a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
+++ b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
@@ -61,7 +61,7 @@ public class MetadataCollector implements 
PartitionStatisticsCollector
 
 static StreamingHistogram defaultTombstoneDropTimeHistogram()
 {
-return new StreamingHistogram(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE);
+return new StreamingHistogram(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE, 
SSTable.TOMBSTONE_HISTOGRAM_SPOOL_SIZE, 
SSTable.TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS);
 }
 
 public static StatsMetadata defaultStatsMetadata()

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a5ce9631/src/java/org/apache/cassandra/utils/StreamingHistogram.java
--
diff --git a/src/java/org/apache/cassandra/utils/StreamingHistogram.java 
b/src/java/org/apache/cassandra/utils/StreamingHistogram.java

[1/6] cassandra git commit: Faster streaming histograms

[jjirsa]

Repository: cassandra
Updated Branches:
  refs/heads/cassandra-3.0 ab7174849 -> a5ce96311
  refs/heads/cassandra-3.11 1dc1aa198 -> 942b83ca9
  refs/heads/trunk 57f5e3671 -> d24f4c68d


Faster streaming histograms

In ttl-heavy use cases (especially tables with default time to live set), the
streaming histograms calculated during compaction and streaming are very 
inefficient.

This patch addresses that in two ways:
1) It creates a system property -Dcassandra.streaminghistogram.roundseconds=60,
and rounds all histograms to the next highest multiple of that value, and
2) Rather than maintaining a histogram of 100 bins that have to be merged
on every new value, we keep a temporary spool of 10 bins, and merge
down to the 100 bin final histogram only when the temporary spool overflows.

Patch by Jeff Jirsa; Reviewed by Nate McCall for CASSANDRA-13038


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/a5ce9631
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/a5ce9631
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/a5ce9631

Branch: refs/heads/cassandra-3.0
Commit: a5ce963117acf5e4cf0a31057551f2f42385c398
Parents: ab71748
Author: Jeff Jirsa 
Authored: Mon Feb 13 19:51:58 2017 -0800
Committer: Jeff Jirsa 
Committed: Mon Feb 27 17:21:48 2017 -0800

--
 CHANGES.txt |   1 +
 .../apache/cassandra/io/sstable/SSTable.java|   3 +
 .../io/sstable/metadata/MetadataCollector.java  |   2 +-
 .../cassandra/utils/StreamingHistogram.java | 109 +++--
 .../microbench/StreamingHistogramBench.java | 405 +++
 .../db/compaction/CompactionsTest.java  |   3 +
 .../DateTieredCompactionStrategyTest.java   |   4 +
 .../LeveledCompactionStrategyTest.java  |   4 +
 .../SizeTieredCompactionStrategyTest.java   |   4 +
 .../cassandra/db/compaction/TTLExpiryTest.java  |   4 +
 .../TimeWindowCompactionStrategyTest.java   |   4 +
 .../cassandra/utils/StreamingHistogramTest.java |  39 +-
 12 files changed, 551 insertions(+), 31 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/a5ce9631/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 386029e..1100bfd 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
 3.0.12
+ * Faster StreamingHistogram (CASSANDRA-13038)
  * Legacy deserializer can create unexpected boundary range tombstones 
(CASSANDRA-13237)
  * Remove unnecessary assertion from AntiCompactionTest (CASSANDRA-13070)
  * Fix cqlsh COPY for dates before 1900 (CASSANDRA-13185)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a5ce9631/src/java/org/apache/cassandra/io/sstable/SSTable.java
--
diff --git a/src/java/org/apache/cassandra/io/sstable/SSTable.java 
b/src/java/org/apache/cassandra/io/sstable/SSTable.java
index 923ef82..1e4488c 100644
--- a/src/java/org/apache/cassandra/io/sstable/SSTable.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTable.java
@@ -58,7 +58,10 @@ public abstract class SSTable
 {
 static final Logger logger = LoggerFactory.getLogger(SSTable.class);
 
+
 public static final int TOMBSTONE_HISTOGRAM_BIN_SIZE = 100;
+public static final int TOMBSTONE_HISTOGRAM_SPOOL_SIZE = 10;
+public static final int TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS = 
Integer.valueOf(System.getProperty("cassandra.streaminghistogram.roundseconds", 
"60"));
 
 public final Descriptor descriptor;
 protected final Set components;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a5ce9631/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
--
diff --git 
a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java 
b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
index 4a67623..3b13cf4 100644
--- a/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
+++ b/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
@@ -61,7 +61,7 @@ public class MetadataCollector implements 
PartitionStatisticsCollector
 
 static StreamingHistogram defaultTombstoneDropTimeHistogram()
 {
-return new StreamingHistogram(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE);
+return new StreamingHistogram(SSTable.TOMBSTONE_HISTOGRAM_BIN_SIZE, 
SSTable.TOMBSTONE_HISTOGRAM_SPOOL_SIZE, 
SSTable.TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS);
 }
 
 public static StatsMetadata defaultStatsMetadata()

http://git-wip-us.apache.org/repos/asf/cassandra/blob/a5ce9631/src/java/org/apache/cassandra/utils/StreamingHistogram.java

[5/6] cassandra git commit: Merge branch 'cassandra-3.0' into cassandra-3.11

[jjirsa]

Merge branch 'cassandra-3.0' into cassandra-3.11


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/942b83ca
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/942b83ca
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/942b83ca

Branch: refs/heads/cassandra-3.11
Commit: 942b83ca93d6d2380a830cb775e8506cc1cf0324
Parents: 1dc1aa1 a5ce963
Author: Jeff Jirsa 
Authored: Mon Feb 27 17:22:31 2017 -0800
Committer: Jeff Jirsa 
Committed: Mon Feb 27 17:23:17 2017 -0800

--
 CHANGES.txt |   1 +
 .../apache/cassandra/io/sstable/SSTable.java|   2 +
 .../io/sstable/metadata/MetadataCollector.java  |   2 +-
 .../cassandra/utils/StreamingHistogram.java | 133 --
 .../microbench/StreamingHistogramBench.java | 405 +++
 .../db/compaction/CompactionsTest.java  |   3 +
 .../DateTieredCompactionStrategyTest.java   |   4 +
 .../LeveledCompactionStrategyTest.java  |   4 +
 .../SizeTieredCompactionStrategyTest.java   |   4 +
 .../cassandra/db/compaction/TTLExpiryTest.java  |   4 +
 .../TimeWindowCompactionStrategyTest.java   |   4 +
 .../cassandra/utils/StreamingHistogramTest.java |  40 +-
 12 files changed, 569 insertions(+), 37 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/942b83ca/CHANGES.txt
--
diff --cc CHANGES.txt
index f5b9d28,1100bfd..1810cae
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,13 -1,5 +1,14 @@@
 -3.0.12
 +3.11.0
 + * Fix equality comparisons of columns using the duration type 
(CASSANDRA-13174)
 + * Obfuscate password in stress-graphs (CASSANDRA-12233)
 + * Move to FastThreadLocalThread and FastThreadLocal (CASSANDRA-13034)
 + * nodetool stopdaemon errors out (CASSANDRA-13030)
 + * Tables in system_distributed should not use gcgs of 0 (CASSANDRA-12954)
 + * Fix primary index calculation for SASI (CASSANDRA-12910)
 + * More fixes to the TokenAllocator (CASSANDRA-12990)
 + * NoReplicationTokenAllocator should work with zero replication factor 
(CASSANDRA-12983)
 +Merged from 3.0:
+  * Faster StreamingHistogram (CASSANDRA-13038)
   * Legacy deserializer can create unexpected boundary range tombstones 
(CASSANDRA-13237)
   * Remove unnecessary assertion from AntiCompactionTest (CASSANDRA-13070)
   * Fix cqlsh COPY for dates before 1900 (CASSANDRA-13185)

http://git-wip-us.apache.org/repos/asf/cassandra/blob/942b83ca/src/java/org/apache/cassandra/io/sstable/SSTable.java
--
diff --cc src/java/org/apache/cassandra/io/sstable/SSTable.java
index 601f5a0,1e4488c..fdcd17f
--- a/src/java/org/apache/cassandra/io/sstable/SSTable.java
+++ b/src/java/org/apache/cassandra/io/sstable/SSTable.java
@@@ -59,7 -58,10 +59,9 @@@ public abstract class SSTabl
  {
  static final Logger logger = LoggerFactory.getLogger(SSTable.class);
  
 -
  public static final int TOMBSTONE_HISTOGRAM_BIN_SIZE = 100;
+ public static final int TOMBSTONE_HISTOGRAM_SPOOL_SIZE = 10;
+ public static final int TOMBSTONE_HISTOGRAM_TTL_ROUND_SECONDS = 
Integer.valueOf(System.getProperty("cassandra.streaminghistogram.roundseconds", 
"60"));
  
  public final Descriptor descriptor;
  protected final Set components;

http://git-wip-us.apache.org/repos/asf/cassandra/blob/942b83ca/src/java/org/apache/cassandra/io/sstable/metadata/MetadataCollector.java
--

http://git-wip-us.apache.org/repos/asf/cassandra/blob/942b83ca/src/java/org/apache/cassandra/utils/StreamingHistogram.java
--
diff --cc src/java/org/apache/cassandra/utils/StreamingHistogram.java
index a500450,fffa73e..6fde931
--- a/src/java/org/apache/cassandra/utils/StreamingHistogram.java
+++ b/src/java/org/apache/cassandra/utils/StreamingHistogram.java
@@@ -39,11 -39,11 +39,14 @@@ public class StreamingHistogra
  public static final StreamingHistogramSerializer serializer = new 
StreamingHistogramSerializer();
  
  // TreeMap to hold bins of histogram.
 -private final TreeMap bin;
 +// The key is a numeric type so we can avoid boxing/unboxing streams of 
different key types
 +// The value is a unboxed long array always of length == 1
 +// Serialized Histograms always writes with double keys for backwards 
compatibility
 +private final TreeMap bin;
  
+ // Keep a second, larger buffer to spool data in, before finalizing it 
into `bin`
 -private final TreeMap spool;
++private final TreeMap spool;
+ 
  // maximum bin size for this 

[jira] [Commented] (CASSANDRA-13010) nodetool compactionstats should say which disk a compaction is writing to

[Alex Lourie (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886937#comment-15886937
 ] 

Alex Lourie commented on CASSANDRA-13010:
-

[~jjirsa] I've created a branch with my work at 
https://github.com/apache/cassandra/compare/trunk...alourie:CASSANDRA-13010, 
please provide feedback and whether I'm on the right path.

Thanks a lot!

> nodetool compactionstats should say which disk a compaction is writing to
> -
>
> Key: CASSANDRA-13010
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13010
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jon Haddad
>  Labels: lhf
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-12906) Update doco with new getting started with contribution section

[Nate McCall (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nate McCall updated CASSANDRA-12906:

   Resolution: Fixed
Fix Version/s: 4.0
   Status: Resolved  (was: Patch Available)

Oops! Sorry - just committed. Thanks [~slater_ben]!

> Update doco with new getting started with contribution section
> --
>
> Key: CASSANDRA-12906
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12906
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Documentation and Website
>Reporter: Ben Slater
>Assignee: Ben Slater
>Priority: Minor
> Fix For: 4.0
>
> Attachments: CASSANDRA-12906v2-trunk.patch
>
>
> Following discussion on the mailing list about how to get more community 
> input it seemed to be agreed that adding some doco emphasising contributions 
> other than creating new features would be a good idea.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

cassandra git commit: Documentation updates per CASSANDRA-12906

[zznate]

Repository: cassandra
Updated Branches:
  refs/heads/trunk 8bfa82769 -> 57f5e3671


Documentation updates per CASSANDRA-12906

Patch by Ben Slater; reviewed by Nate McCall


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/57f5e367
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/57f5e367
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/57f5e367

Branch: refs/heads/trunk
Commit: 57f5e36713e8d6b5f67022f53439e0d458790469
Parents: 8bfa827
Author: Ben Slater 
Authored: Thu Nov 17 14:59:35 2016 +1100
Committer: Nate McCall 
Committed: Tue Feb 28 12:18:48 2017 +1300

--
 doc/source/development/gettingstarted.rst | 60 ++
 doc/source/development/how_to_review.rst  |  2 +
 doc/source/development/index.rst  |  5 ++-
 doc/source/development/patches.rst|  1 +
 doc/source/development/testing.rst|  1 +
 5 files changed, 67 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/57f5e367/doc/source/development/gettingstarted.rst
--
diff --git a/doc/source/development/gettingstarted.rst 
b/doc/source/development/gettingstarted.rst
new file mode 100644
index 000..b5005ab
--- /dev/null
+++ b/doc/source/development/gettingstarted.rst
@@ -0,0 +1,60 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements.  See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership.  The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License.  You may obtain a copy of the License at
+..
+.. http://www.apache.org/licenses/LICENSE-2.0
+..
+.. Unless required by applicable law or agreed to in writing, software
+.. distributed under the License is distributed on an "AS IS" BASIS,
+.. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+.. See the License for the specific language governing permissions and
+.. limitations under the License.
+
+.. highlight:: none
+..  _gettingstarted:
+
+Getting Started
+*
+
+Initial Contributions
+
+
+Writing a new feature is just one way to contribute to the Cassandra project.  
In fact, making sure that supporting tasks, such as QA, documentation and 
helping users, keep up with the development of new features is an ongoing 
challenge for the project (and most open source projects). So, before firing up 
your IDE to create that new feature, we'd suggest you consider some of the 
following activities as a way of introducing yourself to the project and 
getting to know how things work.
+ * Add to or update the documentation
+ * Answer questions on the user list
+ * Review and test a submitted patch
+ * Investigate and fix a reported bug
+ * Create unit tests and d-tests
+
+Updating documentation
+
+
+The Cassandra documentation is maintained in the Cassandra source repository 
along with the Cassandra code base. To submit changes to the documentation, 
follow the standard process for submitting a patch (:ref:`patches`).
+
+Answering questions on the user list
+
+
+Subscribe to the user list, look out for some questions you know the answer to 
and reply with an answer. Simple as that!
+See the `community `_ page for details 
on how to subscribe to the mailing list.
+
+Reviewing and testing a submitted patch
+
+
+Reviewing patches is not the sole domain of committers, if others have 
reviewed a patch it can reduce the load on the committers allowing them to 
write more great features or review more patches. Follow the instructions in 
:ref:`_development_how_to_review` or create a build with the patch and test it 
with your own workload. Add a comment to the JIRA ticket to let others know 
what you have done and the results of your work. (For example, "I tested this 
performance enhacement on our application's standard production load test and 
found a 3% improvement.")
+
+Investigate and/or fix a reported bug
+
+
+Often, the hardest work in fixing a bug is reproducing it. Even if you don't 
have the knowledge to produce a fix, figuring out a way to reliable reproduce 
an issues can be a massive contribution to getting a bug fixed. Document your 
method of reproduction in a JIRA comment or, better yet, produce an automated 
test that reproduces the issue and attach it to the ticket. If you go as far as 
producing a fix, follow the process for submitting a patch 

[jira] [Commented] (CASSANDRA-12906) Update doco with new getting started with contribution section

[Ben Slater (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886718#comment-15886718
 ] 

Ben Slater commented on CASSANDRA-12906:


[~zznate] - just remembered this one and thought I'd give you a ping in case 
you have time to look

> Update doco with new getting started with contribution section
> --
>
> Key: CASSANDRA-12906
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12906
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Documentation and Website
>Reporter: Ben Slater
>Assignee: Ben Slater
>Priority: Minor
> Attachments: CASSANDRA-12906v2-trunk.patch
>
>
> Following discussion on the mailing list about how to get more community 
> input it seemed to be agreed that adding some doco emphasising contributions 
> other than creating new features would be a good idea.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13257) Add repair streaming preview

[Blake Eggleston (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886692#comment-15886692
 ] 

Blake Eggleston commented on CASSANDRA-13257:
-

bq. Is it possible to add "dry-run" feature to stremaing instead of 
TreeDifference?

That’s a really good idea. It will also be more accurate than TreeDifference, 
since it will measure actual sstable sizes, not the size of the partitions that 
come out of the validation compaction.

> Add repair streaming preview
> 
>
> Key: CASSANDRA-13257
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13257
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>
> It would be useful to be able to estimate the amount of repair streaming that 
> needs to be done, without actually doing any streaming. Our main motivation 
> for this having something this is validating CASSANDRA-9143 in production, 
> but I’d imagine it could also be a useful tool in troubleshooting.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13241) Lower default chunk_length_in_kb from 64kb to 4kb

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886689#comment-15886689
 ] 

Jeff Jirsa commented on CASSANDRA-13241:


I suspect 1-2TB/node / 128GB RAM would really nice suggestions that many people 
dont follow in practice (at least in 2017). Past employer we ran "lots" of 
nodes (multiple petabytes worth) with 3-4T of data on 32G of RAM, and I've 
heard of people trying to go as dense as 8-10TB/node (DTCS/TWCS, in particular, 
are designed to let you run right up to the edge of your disk capacity, and 
don't have the tons-of-open-files problem that a very dense LCS node might 
have). 

It IS true that a few extra GB of RAM to get faster IO is a tradeoff that some 
people would glady take, and with dense nodes that may be really important 
(since you're less likely to fit everything in page cache). It's also true that 
if you have 32G of RAM (say you're using an AWS c4.4xl or m4.2xl or r4.xl ), 
that extra 2GB of RAM may be a big deal). 

I'm not saying I object to changing the default, I'm just saying I don't think 
we should just jump to 4k because it's faster. 

> Lower default chunk_length_in_kb from 64kb to 4kb
> -
>
> Key: CASSANDRA-13241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13241
> Project: Cassandra
>  Issue Type: Wish
>  Components: Core
>Reporter: Benjamin Roth
>
> Having a too low chunk size may result in some wasted disk space. A too high 
> chunk size may lead to massive overreads and may have a critical impact on 
> overall system performance.
> In my case, the default chunk size lead to peak read IOs of up to 1GB/s and 
> avg reads of 200MB/s. After lowering chunksize (of course aligned with read 
> ahead), the avg read IO went below 20 MB/s, rather 10-15MB/s.
> The risk of (physical) overreads is increasing with lower (page cache size) / 
> (total data size) ratio.
> High chunk sizes are mostly appropriate for bigger payloads pre request but 
> if the model consists rather of small rows or small resultsets, the read 
> overhead with 64kb chunk size is insanely high. This applies for example for 
> (small) skinny rows.
> Please also see here:
> https://groups.google.com/forum/#!topic/scylladb-dev/j_qXSP-6-gY
> To give you some insights what a difference it can make (460GB data, 128GB 
> RAM):
> - Latency of a quite large CF: https://cl.ly/1r3e0W0S393L
> - Disk throughput: https://cl.ly/2a0Z250S1M3c
> - This shows, that the request distribution remained the same, so no "dynamic 
> snitch magic": https://cl.ly/3E0t1T1z2c0J



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13241) Lower default chunk_length_in_kb from 64kb to 4kb

[Ariel Weisberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886681#comment-15886681
 ] 

Ariel Weisberg commented on CASSANDRA-13241:


It's 16x more chunks not 4x right?

We can get to 20 per chunk without thinking too hard, but the fact that 
compression can result in some pages being larger than 4k is a problem. It's 
really 24 bits per chunk if that has to be supported.

> Lower default chunk_length_in_kb from 64kb to 4kb
> -
>
> Key: CASSANDRA-13241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13241
> Project: Cassandra
>  Issue Type: Wish
>  Components: Core
>Reporter: Benjamin Roth
>
> Having a too low chunk size may result in some wasted disk space. A too high 
> chunk size may lead to massive overreads and may have a critical impact on 
> overall system performance.
> In my case, the default chunk size lead to peak read IOs of up to 1GB/s and 
> avg reads of 200MB/s. After lowering chunksize (of course aligned with read 
> ahead), the avg read IO went below 20 MB/s, rather 10-15MB/s.
> The risk of (physical) overreads is increasing with lower (page cache size) / 
> (total data size) ratio.
> High chunk sizes are mostly appropriate for bigger payloads pre request but 
> if the model consists rather of small rows or small resultsets, the read 
> overhead with 64kb chunk size is insanely high. This applies for example for 
> (small) skinny rows.
> Please also see here:
> https://groups.google.com/forum/#!topic/scylladb-dev/j_qXSP-6-gY
> To give you some insights what a difference it can make (460GB data, 128GB 
> RAM):
> - Latency of a quite large CF: https://cl.ly/1r3e0W0S393L
> - Disk throughput: https://cl.ly/2a0Z250S1M3c
> - This shows, that the request distribution remained the same, so no "dynamic 
> snitch magic": https://cl.ly/3E0t1T1z2c0J



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13241) Lower default chunk_length_in_kb from 64kb to 4kb

[Ryan Svihla (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886677#comment-15886677
 ] 

Ryan Svihla commented on CASSANDRA-13241:
-

Density wise it depends on the use case and the needs, worked on a ton of 
clusters over 2TB per node though.


> Lower default chunk_length_in_kb from 64kb to 4kb
> -
>
> Key: CASSANDRA-13241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13241
> Project: Cassandra
>  Issue Type: Wish
>  Components: Core
>Reporter: Benjamin Roth
>
> Having a too low chunk size may result in some wasted disk space. A too high 
> chunk size may lead to massive overreads and may have a critical impact on 
> overall system performance.
> In my case, the default chunk size lead to peak read IOs of up to 1GB/s and 
> avg reads of 200MB/s. After lowering chunksize (of course aligned with read 
> ahead), the avg read IO went below 20 MB/s, rather 10-15MB/s.
> The risk of (physical) overreads is increasing with lower (page cache size) / 
> (total data size) ratio.
> High chunk sizes are mostly appropriate for bigger payloads pre request but 
> if the model consists rather of small rows or small resultsets, the read 
> overhead with 64kb chunk size is insanely high. This applies for example for 
> (small) skinny rows.
> Please also see here:
> https://groups.google.com/forum/#!topic/scylladb-dev/j_qXSP-6-gY
> To give you some insights what a difference it can make (460GB data, 128GB 
> RAM):
> - Latency of a quite large CF: https://cl.ly/1r3e0W0S393L
> - Disk throughput: https://cl.ly/2a0Z250S1M3c
> - This shows, that the request distribution remained the same, so no "dynamic 
> snitch magic": https://cl.ly/3E0t1T1z2c0J



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13241) Lower default chunk_length_in_kb from 64kb to 4kb

[Benjamin Roth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Benjamin Roth updated CASSANDRA-13241:
--

Hm. I read recommendations that a single node should not have a load of
more than 1-2 TB per node. And I read recommendations of having at least
128gb RAM. If I pay 2gb for a recommended max load to have a MUCH better
performance on uncached io that makes more than 80% of recommended sizings
(on equally hot data) it seems a quite fair price to me.
If there is much less hot data it probably still works as you only deal
page cache for faster io. The fewer hot data the fewer page cache is
required.

Did I miss a point?

Btw 4kb worked perfectly for me with 460gb load/128gb RAM. 64kb did not
work well. Really.




> Lower default chunk_length_in_kb from 64kb to 4kb
> -
>
> Key: CASSANDRA-13241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13241
> Project: Cassandra
>  Issue Type: Wish
>  Components: Core
>Reporter: Benjamin Roth
>
> Having a too low chunk size may result in some wasted disk space. A too high 
> chunk size may lead to massive overreads and may have a critical impact on 
> overall system performance.
> In my case, the default chunk size lead to peak read IOs of up to 1GB/s and 
> avg reads of 200MB/s. After lowering chunksize (of course aligned with read 
> ahead), the avg read IO went below 20 MB/s, rather 10-15MB/s.
> The risk of (physical) overreads is increasing with lower (page cache size) / 
> (total data size) ratio.
> High chunk sizes are mostly appropriate for bigger payloads pre request but 
> if the model consists rather of small rows or small resultsets, the read 
> overhead with 64kb chunk size is insanely high. This applies for example for 
> (small) skinny rows.
> Please also see here:
> https://groups.google.com/forum/#!topic/scylladb-dev/j_qXSP-6-gY
> To give you some insights what a difference it can make (460GB data, 128GB 
> RAM):
> - Latency of a quite large CF: https://cl.ly/1r3e0W0S393L
> - Disk throughput: https://cl.ly/2a0Z250S1M3c
> - This shows, that the request distribution remained the same, so no "dynamic 
> snitch magic": https://cl.ly/3E0t1T1z2c0J



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-13241) Lower default chunk_length_in_kb from 64kb to 4kb

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886621#comment-15886621
 ] 

Jeff Jirsa edited comment on CASSANDRA-13241 at 2/27/17 10:20 PM:
--

{quote}
Can the increased offheap requirements be expressed in a formula?
{quote}

[8 bytes per compression 
chunk|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java#L172-L205].
 4x as many chunks = 4x as much offheap required to store them. AFAIK, it's a 
naive encoding. [~aweisberg] suggested in IRC that a more efficient encoding 
may make such a tradeoff much more tolerable. 

If you have 1TB of uncompressed data, with 64k chunks that's 16M chunks at 8 
bytes each (128MB).
With 16k chunks, that's 512MB.
With 4k chunks, it's 2G.

Per terabyte of data (pre-compression). 



was (Author: jjirsa):
{quote}
Can the increased offheap requirements be expressed in a formula?
{quote}

[8 bytes per compression 
chunk|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java#L172-L205].
 4x as many chunks = 4x as much offheap required to store them. AFAIK, it's a 
naive encoding. [~aweisberg] suggested in IRC that a more efficient encoding 
may make such a tradeoff much more tolerable. 



> Lower default chunk_length_in_kb from 64kb to 4kb
> -
>
> Key: CASSANDRA-13241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13241
> Project: Cassandra
>  Issue Type: Wish
>  Components: Core
>Reporter: Benjamin Roth
>
> Having a too low chunk size may result in some wasted disk space. A too high 
> chunk size may lead to massive overreads and may have a critical impact on 
> overall system performance.
> In my case, the default chunk size lead to peak read IOs of up to 1GB/s and 
> avg reads of 200MB/s. After lowering chunksize (of course aligned with read 
> ahead), the avg read IO went below 20 MB/s, rather 10-15MB/s.
> The risk of (physical) overreads is increasing with lower (page cache size) / 
> (total data size) ratio.
> High chunk sizes are mostly appropriate for bigger payloads pre request but 
> if the model consists rather of small rows or small resultsets, the read 
> overhead with 64kb chunk size is insanely high. This applies for example for 
> (small) skinny rows.
> Please also see here:
> https://groups.google.com/forum/#!topic/scylladb-dev/j_qXSP-6-gY
> To give you some insights what a difference it can make (460GB data, 128GB 
> RAM):
> - Latency of a quite large CF: https://cl.ly/1r3e0W0S393L
> - Disk throughput: https://cl.ly/2a0Z250S1M3c
> - This shows, that the request distribution remained the same, so no "dynamic 
> snitch magic": https://cl.ly/3E0t1T1z2c0J



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13241) Lower default chunk_length_in_kb from 64kb to 4kb

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886621#comment-15886621
 ] 

Jeff Jirsa commented on CASSANDRA-13241:


{quote}
Can the increased offheap requirements be expressed in a formula?
{quote}

[8 bytes per compression 
chunk|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/io/compress/CompressionMetadata.java#L172-L205].
 4x as many chunks = 4x as much offheap required to store them. AFAIK, it's a 
naive encoding. [~aweisberg] suggested in IRC that a more efficient encoding 
may make such a tradeoff much more tolerable. 



> Lower default chunk_length_in_kb from 64kb to 4kb
> -
>
> Key: CASSANDRA-13241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13241
> Project: Cassandra
>  Issue Type: Wish
>  Components: Core
>Reporter: Benjamin Roth
>
> Having a too low chunk size may result in some wasted disk space. A too high 
> chunk size may lead to massive overreads and may have a critical impact on 
> overall system performance.
> In my case, the default chunk size lead to peak read IOs of up to 1GB/s and 
> avg reads of 200MB/s. After lowering chunksize (of course aligned with read 
> ahead), the avg read IO went below 20 MB/s, rather 10-15MB/s.
> The risk of (physical) overreads is increasing with lower (page cache size) / 
> (total data size) ratio.
> High chunk sizes are mostly appropriate for bigger payloads pre request but 
> if the model consists rather of small rows or small resultsets, the read 
> overhead with 64kb chunk size is insanely high. This applies for example for 
> (small) skinny rows.
> Please also see here:
> https://groups.google.com/forum/#!topic/scylladb-dev/j_qXSP-6-gY
> To give you some insights what a difference it can make (460GB data, 128GB 
> RAM):
> - Latency of a quite large CF: https://cl.ly/1r3e0W0S393L
> - Disk throughput: https://cl.ly/2a0Z250S1M3c
> - This shows, that the request distribution remained the same, so no "dynamic 
> snitch magic": https://cl.ly/3E0t1T1z2c0J



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13038) 33% of compaction time spent in StreamingHistogram.update()

[Nate McCall (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886572#comment-15886572
 ] 

Nate McCall commented on CASSANDRA-13038:
-

Thanks for the fixes [~jjirsa] +1 on the current patch. 

> 33% of compaction time spent in StreamingHistogram.update()
> ---
>
> Key: CASSANDRA-13038
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13038
> Project: Cassandra
>  Issue Type: Bug
>  Components: Compaction
>Reporter: Corentin Chary
>Assignee: Jeff Jirsa
> Attachments: compaction-speedup.patch, 
> compaction-streaminghistrogram.png, profiler-snapshot.nps
>
>
> With the following table, that contains a *lot* of cells: 
> {code}
> CREATE TABLE biggraphite.datapoints_11520p_60s (
> metric uuid,
> time_start_ms bigint,
> offset smallint,
> count int,
> value double,
> PRIMARY KEY ((metric, time_start_ms), offset)
> ) WITH CLUSTERING ORDER BY (offset DESC);
> AND compaction = {'class': 
> 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 
> 'compaction_window_size': '6', 'compaction_window_unit': 'HOURS', 
> 'max_threshold': '32', 'min_threshold': '6'}
> Keyspace : biggraphite
> Read Count: 1822
> Read Latency: 1.8870054884742042 ms.
> Write Count: 2212271647
> Write Latency: 0.027705127678653473 ms.
> Pending Flushes: 0
> Table: datapoints_11520p_60s
> SSTable count: 47
> Space used (live): 300417555945
> Space used (total): 303147395017
> Space used by snapshots (total): 0
> Off heap memory used (total): 207453042
> SSTable Compression Ratio: 0.4955200053039823
> Number of keys (estimate): 16343723
> Memtable cell count: 220576
> Memtable data size: 17115128
> Memtable off heap memory used: 0
> Memtable switch count: 2872
> Local read count: 0
> Local read latency: NaN ms
> Local write count: 1103167888
> Local write latency: 0.025 ms
> Pending flushes: 0
> Percent repaired: 0.0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 105118296
> Bloom filter off heap memory used: 106547192
> Index summary off heap memory used: 27730962
> Compression metadata off heap memory used: 73174888
> Compacted partition minimum bytes: 61
> Compacted partition maximum bytes: 51012
> Compacted partition mean bytes: 7899
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
> Dropped Mutations: 0
> {code}
> It looks like a good chunk of the compaction time is lost in 
> StreamingHistogram.update() (which is used to store the estimated tombstone 
> drop times).
> This could be caused by a huge number of different deletion times which would 
> makes the bin huge but it this histogram should be capped to 100 keys. It's 
> more likely caused by the huge number of cells.
> A simple solutions could be to only take into accounts part of the cells, the 
> fact the this table has a TWCS also gives us an additional hint that sampling 
> deletion times would be fine.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13241) Lower default chunk_length_in_kb from 64kb to 4kb

[Jeremy Hanna (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886561#comment-15886561
 ] 

Jeremy Hanna commented on CASSANDRA-13241:
--

I would think that if CASSANDRA-10995 got in (which I would think would be 
reasonable), then it would make for a stronger case for 4K by default.

> Lower default chunk_length_in_kb from 64kb to 4kb
> -
>
> Key: CASSANDRA-13241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13241
> Project: Cassandra
>  Issue Type: Wish
>  Components: Core
>Reporter: Benjamin Roth
>
> Having a too low chunk size may result in some wasted disk space. A too high 
> chunk size may lead to massive overreads and may have a critical impact on 
> overall system performance.
> In my case, the default chunk size lead to peak read IOs of up to 1GB/s and 
> avg reads of 200MB/s. After lowering chunksize (of course aligned with read 
> ahead), the avg read IO went below 20 MB/s, rather 10-15MB/s.
> The risk of (physical) overreads is increasing with lower (page cache size) / 
> (total data size) ratio.
> High chunk sizes are mostly appropriate for bigger payloads pre request but 
> if the model consists rather of small rows or small resultsets, the read 
> overhead with 64kb chunk size is insanely high. This applies for example for 
> (small) skinny rows.
> Please also see here:
> https://groups.google.com/forum/#!topic/scylladb-dev/j_qXSP-6-gY
> To give you some insights what a difference it can make (460GB data, 128GB 
> RAM):
> - Latency of a quite large CF: https://cl.ly/1r3e0W0S393L
> - Disk throughput: https://cl.ly/2a0Z250S1M3c
> - This shows, that the request distribution remained the same, so no "dynamic 
> snitch magic": https://cl.ly/3E0t1T1z2c0J



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13241) Lower default chunk_length_in_kb from 64kb to 4kb

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886536#comment-15886536
 ] 

Benjamin Roth commented on CASSANDRA-13241:
---

According to percona 
(https://www.percona.com/blog/2016/03/09/evaluating-database-compression-methods/)
 and my own experience, the impact on compression ratio is not that big with 
lz4.

Can the increased offheap requirements be expressed in a formula?

> Lower default chunk_length_in_kb from 64kb to 4kb
> -
>
> Key: CASSANDRA-13241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13241
> Project: Cassandra
>  Issue Type: Wish
>  Components: Core
>Reporter: Benjamin Roth
>
> Having a too low chunk size may result in some wasted disk space. A too high 
> chunk size may lead to massive overreads and may have a critical impact on 
> overall system performance.
> In my case, the default chunk size lead to peak read IOs of up to 1GB/s and 
> avg reads of 200MB/s. After lowering chunksize (of course aligned with read 
> ahead), the avg read IO went below 20 MB/s, rather 10-15MB/s.
> The risk of (physical) overreads is increasing with lower (page cache size) / 
> (total data size) ratio.
> High chunk sizes are mostly appropriate for bigger payloads pre request but 
> if the model consists rather of small rows or small resultsets, the read 
> overhead with 64kb chunk size is insanely high. This applies for example for 
> (small) skinny rows.
> Please also see here:
> https://groups.google.com/forum/#!topic/scylladb-dev/j_qXSP-6-gY
> To give you some insights what a difference it can make (460GB data, 128GB 
> RAM):
> - Latency of a quite large CF: https://cl.ly/1r3e0W0S393L
> - Disk throughput: https://cl.ly/2a0Z250S1M3c
> - This shows, that the request distribution remained the same, so no "dynamic 
> snitch magic": https://cl.ly/3E0t1T1z2c0J



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13278) Update build.xml and build.properties.default maven repos

[Michael Shuler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Shuler updated CASSANDRA-13278:
---
Labels: lhf  (was: )

> Update build.xml and build.properties.default maven repos
> -
>
> Key: CASSANDRA-13278
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13278
> Project: Cassandra
>  Issue Type: Bug
>  Components: Build
>Reporter: Michael Shuler
>Priority: Minor
>  Labels: lhf
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.11.x, 4.x
>
>
> Only 2 of the 5 urls in build.properties.default are currently valid.
> java.net2, jclouds, and oauth urls all 404.
> {noformat}
> $ git grep remoteRepository
> build.properties.default:artifact.remoteRepository.central: 
> http://repo1.maven.org/maven2
> build.properties.default:artifact.remoteRepository.java.net2:   
> http://download.java.net/maven/2
> build.properties.default:artifact.remoteRepository.apache:  
> https://repository.apache.org/content/repositories/releases
> build.properties.default:artifact.remoteRepository.jclouds: 
> http://jclouds.googlecode.com/svn/repo
> build.properties.default:artifact.remoteRepository.oauth:   
> http://oauth.googlecode.com/svn/code/maven
> build.xml:   url="${artifact.remoteRepository.central}"/>
> build.xml:   url="${artifact.remoteRepository.java.net2}"/>
> build.xml:   url="${artifact.remoteRepository.apache}"/>
> build.xml:  
> build.xml:  
> build.xml:  
> build.xml:  
> build.xml:  
> build.xml:  
> build.xml:  
> build.xml:
> build.xml:
> build.xml:
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (CASSANDRA-13278) Update build.xml and build.properties.default maven repos

[Michael Shuler (JIRA)]

Michael Shuler created CASSANDRA-13278:
--

 Summary: Update build.xml and build.properties.default maven repos
 Key: CASSANDRA-13278
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13278
 Project: Cassandra
  Issue Type: Bug
  Components: Build
Reporter: Michael Shuler
Priority: Minor
 Fix For: 2.1.x, 2.2.x, 3.0.x, 3.11.x, 4.x


Only 2 of the 5 urls in build.properties.default are currently valid.

java.net2, jclouds, and oauth urls all 404.

{noformat}
$ git grep remoteRepository
build.properties.default:artifact.remoteRepository.central: 
http://repo1.maven.org/maven2
build.properties.default:artifact.remoteRepository.java.net2:   
http://download.java.net/maven/2
build.properties.default:artifact.remoteRepository.apache:  
https://repository.apache.org/content/repositories/releases
build.properties.default:artifact.remoteRepository.jclouds: 
http://jclouds.googlecode.com/svn/repo
build.properties.default:artifact.remoteRepository.oauth:   
http://oauth.googlecode.com/svn/code/maven
build.xml:  
build.xml:  
build.xml:  
build.xml:  
build.xml:  
build.xml:  
build.xml:  
build.xml:  
build.xml:  
build.xml:  
build.xml:
build.xml:
build.xml:
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13241) Lower default chunk_length_in_kb from 64kb to 4kb

[Jeff Jirsa (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886529#comment-15886529
 ] 

Jeff Jirsa commented on CASSANDRA-13241:


4k chunks will will give much better IO for sstables not in page cache, but 
come at the cost of significant offheap memory requirements, and compression 
ratios will suffer. There might be a better default, but I'm not sure going all 
the way to 4k is the right answer. 

{quote}
Thanks for your vote, but ... maybe this is a stupid question: Who will finally 
decide if that change is accepted?
{quote}

Generally, a committer can push it as long as they have a +1 vote. However, for 
something like this, most committers will (should) look for consensus among the 
other committers. Ultimately, the final say will come from consensus among the 
PMC when it goes to voting for a release - if a member of the PMC ultimately 
decides it doesn't like the change, that member can/will/should vote -1 on the 
release until the commit is removed. 


> Lower default chunk_length_in_kb from 64kb to 4kb
> -
>
> Key: CASSANDRA-13241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13241
> Project: Cassandra
>  Issue Type: Wish
>  Components: Core
>Reporter: Benjamin Roth
>
> Having a too low chunk size may result in some wasted disk space. A too high 
> chunk size may lead to massive overreads and may have a critical impact on 
> overall system performance.
> In my case, the default chunk size lead to peak read IOs of up to 1GB/s and 
> avg reads of 200MB/s. After lowering chunksize (of course aligned with read 
> ahead), the avg read IO went below 20 MB/s, rather 10-15MB/s.
> The risk of (physical) overreads is increasing with lower (page cache size) / 
> (total data size) ratio.
> High chunk sizes are mostly appropriate for bigger payloads pre request but 
> if the model consists rather of small rows or small resultsets, the read 
> overhead with 64kb chunk size is insanely high. This applies for example for 
> (small) skinny rows.
> Please also see here:
> https://groups.google.com/forum/#!topic/scylladb-dev/j_qXSP-6-gY
> To give you some insights what a difference it can make (460GB data, 128GB 
> RAM):
> - Latency of a quite large CF: https://cl.ly/1r3e0W0S393L
> - Disk throughput: https://cl.ly/2a0Z250S1M3c
> - This shows, that the request distribution remained the same, so no "dynamic 
> snitch magic": https://cl.ly/3E0t1T1z2c0J



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13241) Lower default chunk_length_in_kb from 64kb to 4kb

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886517#comment-15886517
 ] 

Benjamin Roth commented on CASSANDRA-13241:
---

No worries. Your patch answered my questions implicitly. Thanks!

> Lower default chunk_length_in_kb from 64kb to 4kb
> -
>
> Key: CASSANDRA-13241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13241
> Project: Cassandra
>  Issue Type: Wish
>  Components: Core
>Reporter: Benjamin Roth
>
> Having a too low chunk size may result in some wasted disk space. A too high 
> chunk size may lead to massive overreads and may have a critical impact on 
> overall system performance.
> In my case, the default chunk size lead to peak read IOs of up to 1GB/s and 
> avg reads of 200MB/s. After lowering chunksize (of course aligned with read 
> ahead), the avg read IO went below 20 MB/s, rather 10-15MB/s.
> The risk of (physical) overreads is increasing with lower (page cache size) / 
> (total data size) ratio.
> High chunk sizes are mostly appropriate for bigger payloads pre request but 
> if the model consists rather of small rows or small resultsets, the read 
> overhead with 64kb chunk size is insanely high. This applies for example for 
> (small) skinny rows.
> Please also see here:
> https://groups.google.com/forum/#!topic/scylladb-dev/j_qXSP-6-gY
> To give you some insights what a difference it can make (460GB data, 128GB 
> RAM):
> - Latency of a quite large CF: https://cl.ly/1r3e0W0S393L
> - Disk throughput: https://cl.ly/2a0Z250S1M3c
> - This shows, that the request distribution remained the same, so no "dynamic 
> snitch magic": https://cl.ly/3E0t1T1z2c0J



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13271) Reduce lock contention on instance factories of ListType and SetType

[Robert Stupp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp updated CASSANDRA-13271:
-
Status: Open  (was: Patch Available)

> Reduce lock contention on instance factories of ListType and SetType
> 
>
> Key: CASSANDRA-13271
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13271
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: vincent royer
>Priority: Minor
>  Labels: performance
> Fix For: 4.x
>
> Attachments: 
> 0001-CASSANDRA-13271-singleton-factory-concurrency-opimiz.patch
>
>
> By doing some performance tests, i noticed that getInstance() in 
> org.apache.cassandra.db.marshal.ListType and SetType could suffer from lock 
> contention on the singleton factory getInstance(). Here is a proposal to 
> reduce lock contention by using a ConcurrentMap and the putIfAbsent method 
> rather than a synchronized method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-13241) Lower default chunk_length_in_kb from 64kb to 4kb

[Ben Bromhead (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886494#comment-15886494
 ] 

Ben Bromhead edited comment on CASSANDRA-13241 at 2/27/17 8:43 PM:
---

I had a quick look at the original SSTable compression ticket 
https://issues.apache.org/jira/browse/CASSANDRA-47 and I can't see any specific 
reasons for the choice of 64kb. Maybe the folks originally working on that 
ticket could comment if there is some reason I'm missing. 
 
Irrespective I've included trivial patch

||Branch||
|[4.0|https://github.com/apache/cassandra/compare/trunk...benbromhead:13241]

Edit: Sorry I didn't see your most recent comment, I've included the trivial 
patch for what it's worth :)


was (Author: benbromhead):
I had a quick look at the original SSTable compression ticket 
https://issues.apache.org/jira/browse/CASSANDRA-47 and I can't see any specific 
reasons for the choice of 64kb. Maybe the folks originally working on that 
ticket could comment if there is some reason I'm missing. 
 
Irrespective I've included trivial patch

||Branch||
|[4.0|https://github.com/apache/cassandra/compare/trunk...benbromhead:13241]

> Lower default chunk_length_in_kb from 64kb to 4kb
> -
>
> Key: CASSANDRA-13241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13241
> Project: Cassandra
>  Issue Type: Wish
>  Components: Core
>Reporter: Benjamin Roth
>
> Having a too low chunk size may result in some wasted disk space. A too high 
> chunk size may lead to massive overreads and may have a critical impact on 
> overall system performance.
> In my case, the default chunk size lead to peak read IOs of up to 1GB/s and 
> avg reads of 200MB/s. After lowering chunksize (of course aligned with read 
> ahead), the avg read IO went below 20 MB/s, rather 10-15MB/s.
> The risk of (physical) overreads is increasing with lower (page cache size) / 
> (total data size) ratio.
> High chunk sizes are mostly appropriate for bigger payloads pre request but 
> if the model consists rather of small rows or small resultsets, the read 
> overhead with 64kb chunk size is insanely high. This applies for example for 
> (small) skinny rows.
> Please also see here:
> https://groups.google.com/forum/#!topic/scylladb-dev/j_qXSP-6-gY
> To give you some insights what a difference it can make (460GB data, 128GB 
> RAM):
> - Latency of a quite large CF: https://cl.ly/1r3e0W0S393L
> - Disk throughput: https://cl.ly/2a0Z250S1M3c
> - This shows, that the request distribution remained the same, so no "dynamic 
> snitch magic": https://cl.ly/3E0t1T1z2c0J



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13241) Lower default chunk_length_in_kb from 64kb to 4kb

[Ben Bromhead (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886494#comment-15886494
 ] 

Ben Bromhead commented on CASSANDRA-13241:
--

I had a quick look at the original SSTable compression ticket 
https://issues.apache.org/jira/browse/CASSANDRA-47 and I can't see any specific 
reasons for the choice of 64kb. Maybe the folks originally working on that 
ticket could comment if there is some reason I'm missing. 
 
Irrespective I've included trivial patch

||Branch||
|[4.0|https://github.com/apache/cassandra/compare/trunk...benbromhead:13241]

> Lower default chunk_length_in_kb from 64kb to 4kb
> -
>
> Key: CASSANDRA-13241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13241
> Project: Cassandra
>  Issue Type: Wish
>  Components: Core
>Reporter: Benjamin Roth
>
> Having a too low chunk size may result in some wasted disk space. A too high 
> chunk size may lead to massive overreads and may have a critical impact on 
> overall system performance.
> In my case, the default chunk size lead to peak read IOs of up to 1GB/s and 
> avg reads of 200MB/s. After lowering chunksize (of course aligned with read 
> ahead), the avg read IO went below 20 MB/s, rather 10-15MB/s.
> The risk of (physical) overreads is increasing with lower (page cache size) / 
> (total data size) ratio.
> High chunk sizes are mostly appropriate for bigger payloads pre request but 
> if the model consists rather of small rows or small resultsets, the read 
> overhead with 64kb chunk size is insanely high. This applies for example for 
> (small) skinny rows.
> Please also see here:
> https://groups.google.com/forum/#!topic/scylladb-dev/j_qXSP-6-gY
> To give you some insights what a difference it can make (460GB data, 128GB 
> RAM):
> - Latency of a quite large CF: https://cl.ly/1r3e0W0S393L
> - Disk throughput: https://cl.ly/2a0Z250S1M3c
> - This shows, that the request distribution remained the same, so no "dynamic 
> snitch magic": https://cl.ly/3E0t1T1z2c0J



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13241) Lower default chunk_length_in_kb from 64kb to 4kb

[Benjamin Roth (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886402#comment-15886402
 ] 

Benjamin Roth commented on CASSANDRA-13241:
---

Thanks for your vote, but ... maybe this is a stupid question: Who will finally 
decide if that change is accepted?
I think I could make a patch pretty easily but how does change management work?

> Lower default chunk_length_in_kb from 64kb to 4kb
> -
>
> Key: CASSANDRA-13241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13241
> Project: Cassandra
>  Issue Type: Wish
>  Components: Core
>Reporter: Benjamin Roth
>
> Having a too low chunk size may result in some wasted disk space. A too high 
> chunk size may lead to massive overreads and may have a critical impact on 
> overall system performance.
> In my case, the default chunk size lead to peak read IOs of up to 1GB/s and 
> avg reads of 200MB/s. After lowering chunksize (of course aligned with read 
> ahead), the avg read IO went below 20 MB/s, rather 10-15MB/s.
> The risk of (physical) overreads is increasing with lower (page cache size) / 
> (total data size) ratio.
> High chunk sizes are mostly appropriate for bigger payloads pre request but 
> if the model consists rather of small rows or small resultsets, the read 
> overhead with 64kb chunk size is insanely high. This applies for example for 
> (small) skinny rows.
> Please also see here:
> https://groups.google.com/forum/#!topic/scylladb-dev/j_qXSP-6-gY
> To give you some insights what a difference it can make (460GB data, 128GB 
> RAM):
> - Latency of a quite large CF: https://cl.ly/1r3e0W0S393L
> - Disk throughput: https://cl.ly/2a0Z250S1M3c
> - This shows, that the request distribution remained the same, so no "dynamic 
> snitch magic": https://cl.ly/3E0t1T1z2c0J



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-13271) Reduce lock contention on instance factories of ListType and SetType

[Robert Stupp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886318#comment-15886318
 ] 

Robert Stupp edited comment on CASSANDRA-13271 at 2/27/17 7:33 PM:
---

The changes for {{ListType}} and {{SetType}} look good so far. However, you 
miss some similar one in {{MapType}} and {{ReversedType}} plus the 
corresponding {{List/Set/MapSerializer}} classes, that use the same pattern.
Can you provide a branch on GitHub instead of a patch file?

EDIT: Also {{DynamicCompositeType}}. You might also switch the pattern to 
something like this:
{code}
X t t = internMap.get(key);
if (t == null)
t = internMap.computeIfAbsent(key, (key) -> ...);
{code}
In the rare case when an instance is missing, we can afford the extra overhead 
but favor the cleaner code.


was (Author: snazy):
The changes for {{ListType}} and {{SetType}} look good so far. However, you 
miss some similar one in {{MapType}} and {{ReversedType}} plus the 
corresponding {{List/Set/MapSerializer}} classes, that use the same pattern.
Can you provide a branch on GitHub instead of a patch file?

EDIT: Also {{DynamicCompositeType}}. You might also switch the pattern to 
something like this:
{code}
X t t = internMap.get(key);
if (t == null)
internMap.computeIfAbsent(key, (key) -> ...);
{code}
In the rare case when an instance is missing, we can afford the extra overhead 
but favor the cleaner code.

> Reduce lock contention on instance factories of ListType and SetType
> 
>
> Key: CASSANDRA-13271
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13271
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: vincent royer
>Priority: Minor
>  Labels: performance
> Fix For: 4.x
>
> Attachments: 
> 0001-CASSANDRA-13271-singleton-factory-concurrency-opimiz.patch
>
>
> By doing some performance tests, i noticed that getInstance() in 
> org.apache.cassandra.db.marshal.ListType and SetType could suffer from lock 
> contention on the singleton factory getInstance(). Here is a proposal to 
> reduce lock contention by using a ConcurrentMap and the putIfAbsent method 
> rather than a synchronized method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-13271) Reduce lock contention on instance factories of ListType and SetType

[Robert Stupp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886318#comment-15886318
 ] 

Robert Stupp edited comment on CASSANDRA-13271 at 2/27/17 7:33 PM:
---

The changes for {{ListType}} and {{SetType}} look good so far. However, you 
miss some similar one in {{MapType}} and {{ReversedType}} plus the 
corresponding {{List/Set/MapSerializer}} classes, that use the same pattern.
Can you provide a branch on GitHub instead of a patch file?

EDIT: Also {{DynamicCompositeType}}. You might also switch the pattern to 
something like this:
{code}
X t t = internMap.get(key);
if (t == null)
internMap.computeIfAbsent(key, (key) -> ...);
{code}
In the rare case when an instance is missing, we can afford the extra overhead 
but favor the cleaner code.


was (Author: snazy):
The changes for {{ListType}} and {{SetType}} look good so far. However, you 
miss some similar one in {{MapType}} and {{ReversedType}} plus the 
corresponding {{List/Set/MapSerializer}} classes, that use the same pattern.
Can you provide a branch on GitHub instead of a patch file?

> Reduce lock contention on instance factories of ListType and SetType
> 
>
> Key: CASSANDRA-13271
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13271
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: vincent royer
>Priority: Minor
>  Labels: performance
> Fix For: 4.x
>
> Attachments: 
> 0001-CASSANDRA-13271-singleton-factory-concurrency-opimiz.patch
>
>
> By doing some performance tests, i noticed that getInstance() in 
> org.apache.cassandra.db.marshal.ListType and SetType could suffer from lock 
> contention on the singleton factory getInstance(). Here is a proposal to 
> reduce lock contention by using a ConcurrentMap and the putIfAbsent method 
> rather than a synchronized method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-12988) make the consistency level for user-level auth reads and writes configurable

[Robert Stupp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp updated CASSANDRA-12988:
-
Status: Open  (was: Patch Available)

> make the consistency level for user-level auth reads and writes configurable
> 
>
> Key: CASSANDRA-12988
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12988
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
>
> Most reads for the auth-related tables execute at {{LOCAL_ONE}}. We'd like to 
> make it configurable, with the default still being {{LOCAL_ONE}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12988) make the consistency level for user-level auth reads and writes configurable

[Robert Stupp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886362#comment-15886362
 ] 

Robert Stupp commented on CASSANDRA-12988:
--

Generally, I think this idea is fine. Since it's a new feature, I fear we have 
to target 4.0/4.x for this.
The patch breaks {{DatabaseDescriptorRefTest}}, which ensures that daemon 
components are not started just by accessing lovely {{DatabaseDescriptor}} - 
for offline tools. {{ConsistencyLevel}} is one of the classes that would 
indirectly start unwanted threads or load unwanted classes. TL;DR I think it's 
necessary to refactor this a bit.

> make the consistency level for user-level auth reads and writes configurable
> 
>
> Key: CASSANDRA-12988
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12988
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
>
> Most reads for the auth-related tables execute at {{LOCAL_ONE}}. We'd like to 
> make it configurable, with the default still being {{LOCAL_ONE}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13241) Lower default chunk_length_in_kb from 64kb to 4kb

[Ben Bromhead (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886361#comment-15886361
 ] 

Ben Bromhead commented on CASSANDRA-13241:
--

We generally end up recommending to our customers they reduce their default 
chunk_length_in_kb for most applications generally to be around the average 
size of their reads (dependent on end latency goals) with a floor of the 
underlying disks smallest read unit (generally for SSDs this is the page size, 
rather than block size iirc). This ends up being anywhere from 2kb - 16kb 
depending on hardware. I would say driving higher IOPs/lower latencies through 
the disk rather than throughput is much more aligned with the standard use 
cases for Cassandra.

4kb is pretty common and I would be very happy with it as the default chunk 
length, especially given that SSDs are a pretty much standard recommendation 
for C*. Increasing the chunk length for better compression whilst sacrificing 
read perf should be opt-in rather than default.

+1


> Lower default chunk_length_in_kb from 64kb to 4kb
> -
>
> Key: CASSANDRA-13241
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13241
> Project: Cassandra
>  Issue Type: Wish
>  Components: Core
>Reporter: Benjamin Roth
>
> Having a too low chunk size may result in some wasted disk space. A too high 
> chunk size may lead to massive overreads and may have a critical impact on 
> overall system performance.
> In my case, the default chunk size lead to peak read IOs of up to 1GB/s and 
> avg reads of 200MB/s. After lowering chunksize (of course aligned with read 
> ahead), the avg read IO went below 20 MB/s, rather 10-15MB/s.
> The risk of (physical) overreads is increasing with lower (page cache size) / 
> (total data size) ratio.
> High chunk sizes are mostly appropriate for bigger payloads pre request but 
> if the model consists rather of small rows or small resultsets, the read 
> overhead with 64kb chunk size is insanely high. This applies for example for 
> (small) skinny rows.
> Please also see here:
> https://groups.google.com/forum/#!topic/scylladb-dev/j_qXSP-6-gY
> To give you some insights what a difference it can make (460GB data, 128GB 
> RAM):
> - Latency of a quite large CF: https://cl.ly/1r3e0W0S393L
> - Disk throughput: https://cl.ly/2a0Z250S1M3c
> - This shows, that the request distribution remained the same, so no "dynamic 
> snitch magic": https://cl.ly/3E0t1T1z2c0J



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13271) Reduce lock contention on instance factories of ListType and SetType

[Aleksey Yeschenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-13271:
--
Fix Version/s: (was: 3.0.x)
   4.x

> Reduce lock contention on instance factories of ListType and SetType
> 
>
> Key: CASSANDRA-13271
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13271
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: vincent royer
>Priority: Minor
>  Labels: performance
> Fix For: 4.x
>
> Attachments: 
> 0001-CASSANDRA-13271-singleton-factory-concurrency-opimiz.patch
>
>
> By doing some performance tests, i noticed that getInstance() in 
> org.apache.cassandra.db.marshal.ListType and SetType could suffer from lock 
> contention on the singleton factory getInstance(). Here is a proposal to 
> reduce lock contention by using a ConcurrentMap and the putIfAbsent method 
> rather than a synchronized method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-12988) make the consistency level for user-level auth reads and writes configurable

[Robert Stupp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp updated CASSANDRA-12988:
-
Reviewer: Robert Stupp

> make the consistency level for user-level auth reads and writes configurable
> 
>
> Key: CASSANDRA-12988
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12988
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Jason Brown
>Assignee: Jason Brown
>Priority: Minor
>
> Most reads for the auth-related tables execute at {{LOCAL_ONE}}. We'd like to 
> make it configurable, with the default still being {{LOCAL_ONE}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-12540) Password Management: Hardcoded Password

[Robert Stupp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp updated CASSANDRA-12540:
-
Status: Open  (was: Patch Available)

> Password Management: Hardcoded Password
> ---
>
> Key: CASSANDRA-12540
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12540
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Eduardo Aguinaga
>Assignee: Stefan Podkowinski
> Attachments: 12540-trunk.patch
>
>
> Overview:
> In May through June of 2016 a static analysis was performed on version 3.0.5 
> of the Cassandra source code. The analysis included an automated analysis 
> using HP Fortify v4.21 SCA and a manual analysis utilizing SciTools 
> Understand v4. The results of that analysis includes the issue below.
> Issue:
> In the file EncryptionOptions.java there are hard coded passwords on lines 23 
> and 25.
> {code:java}
> EncryptionOptions.java, lines 20-30:
> 20 public abstract class EncryptionOptions
> 21 {
> 22 public String keystore = "conf/.keystore";
> 23 public String keystore_password = "cassandra";
> 24 public String truststore = "conf/.truststore";
> 25 public String truststore_password = "cassandra";
> 26 public String[] cipher_suites = {
> 27 "TLS_RSA_WITH_AES_128_CBC_SHA", "TLS_RSA_WITH_AES_256_CBC_SHA",
> 28 "TLS_DHE_RSA_WITH_AES_128_CBC_SHA", 
> "TLS_DHE_RSA_WITH_AES_256_CBC_SHA",
> 29 "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA", 
> "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA" 
> 30 };
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12540) Password Management: Hardcoded Password

[Robert Stupp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886344#comment-15886344
 ] 

Robert Stupp commented on CASSANDRA-12540:
--

We should target 4.0 for this one as it changes existing behaviour (although 
it's not a "good" behaviour).
Haven't looked thoroughly at the patch yet, but it seems that some null-checks 
are needed, in case the password(s) are not configured - looks, like it will 
NPE with this patch.
Can you add a dtest that verifies both a good and the bad configurations (i.e. 
with and without passwords)?
Similar to CASSANDRA-12773, the CI results have been removed meanwhile. Can you 
rebase and re-run CI?

> Password Management: Hardcoded Password
> ---
>
> Key: CASSANDRA-12540
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12540
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Eduardo Aguinaga
>Assignee: Stefan Podkowinski
> Attachments: 12540-trunk.patch
>
>
> Overview:
> In May through June of 2016 a static analysis was performed on version 3.0.5 
> of the Cassandra source code. The analysis included an automated analysis 
> using HP Fortify v4.21 SCA and a manual analysis utilizing SciTools 
> Understand v4. The results of that analysis includes the issue below.
> Issue:
> In the file EncryptionOptions.java there are hard coded passwords on lines 23 
> and 25.
> {code:java}
> EncryptionOptions.java, lines 20-30:
> 20 public abstract class EncryptionOptions
> 21 {
> 22 public String keystore = "conf/.keystore";
> 23 public String keystore_password = "cassandra";
> 24 public String truststore = "conf/.truststore";
> 25 public String truststore_password = "cassandra";
> 26 public String[] cipher_suites = {
> 27 "TLS_RSA_WITH_AES_128_CBC_SHA", "TLS_RSA_WITH_AES_256_CBC_SHA",
> 28 "TLS_DHE_RSA_WITH_AES_128_CBC_SHA", 
> "TLS_DHE_RSA_WITH_AES_256_CBC_SHA",
> 29 "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA", 
> "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA" 
> 30 };
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-12773) cassandra-stress error for one way SSL

[Robert Stupp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp updated CASSANDRA-12773:
-
Status: Open  (was: Patch Available)

> cassandra-stress error for one way SSL 
> ---
>
> Key: CASSANDRA-12773
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12773
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Jane Deng
>Assignee: Stefan Podkowinski
> Fix For: 2.2.x
>
> Attachments: 12773-2.2.patch
>
>
> CASSANDRA-9325 added keystore/truststore configuration into cassandra-stress. 
> However, for one way ssl (require_client_auth=false), there is no need to 
> pass keystore info into ssloptions. Cassadra-stress errored out:
> {noformat}
> java.lang.RuntimeException: java.io.IOException: Error creating the 
> initializing the SSL Context 
> at 
> org.apache.cassandra.stress.settings.StressSettings.getJavaDriverClient(StressSettings.java:200)
>  
> at 
> org.apache.cassandra.stress.settings.SettingsSchema.createKeySpacesNative(SettingsSchema.java:79)
>  
> at 
> org.apache.cassandra.stress.settings.SettingsSchema.createKeySpaces(SettingsSchema.java:69)
>  
> at 
> org.apache.cassandra.stress.settings.StressSettings.maybeCreateKeyspaces(StressSettings.java:207)
>  
> at org.apache.cassandra.stress.StressAction.run(StressAction.java:55) 
> at org.apache.cassandra.stress.Stress.main(Stress.java:117) 
> Caused by: java.io.IOException: Error creating the initializing the SSL 
> Context 
> at 
> org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:151)
>  
> at 
> org.apache.cassandra.stress.util.JavaDriverClient.connect(JavaDriverClient.java:128)
>  
> at 
> org.apache.cassandra.stress.settings.StressSettings.getJavaDriverClient(StressSettings.java:191)
>  
> ... 5 more 
> Caused by: java.io.IOException: Keystore was tampered with, or password was 
> incorrect 
> at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772) 
> at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55) 
> at java.security.KeyStore.load(KeyStore.java:1445) 
> at 
> org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:129)
>  
> ... 7 more 
> Caused by: java.security.UnrecoverableKeyException: Password verification 
> failed 
> at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770) 
> ... 10 more
> {noformat}
> It's a bug from CASSANDRA-9325. When the keystore is absent, the keystore is 
> assigned to the path of the truststore, but the password isn't taken care.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-12540) Password Management: Hardcoded Password

[Robert Stupp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp updated CASSANDRA-12540:
-
Reviewer: Robert Stupp

> Password Management: Hardcoded Password
> ---
>
> Key: CASSANDRA-12540
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12540
> Project: Cassandra
>  Issue Type: Sub-task
>Reporter: Eduardo Aguinaga
>Assignee: Stefan Podkowinski
> Attachments: 12540-trunk.patch
>
>
> Overview:
> In May through June of 2016 a static analysis was performed on version 3.0.5 
> of the Cassandra source code. The analysis included an automated analysis 
> using HP Fortify v4.21 SCA and a manual analysis utilizing SciTools 
> Understand v4. The results of that analysis includes the issue below.
> Issue:
> In the file EncryptionOptions.java there are hard coded passwords on lines 23 
> and 25.
> {code:java}
> EncryptionOptions.java, lines 20-30:
> 20 public abstract class EncryptionOptions
> 21 {
> 22 public String keystore = "conf/.keystore";
> 23 public String keystore_password = "cassandra";
> 24 public String truststore = "conf/.truststore";
> 25 public String truststore_password = "cassandra";
> 26 public String[] cipher_suites = {
> 27 "TLS_RSA_WITH_AES_128_CBC_SHA", "TLS_RSA_WITH_AES_256_CBC_SHA",
> 28 "TLS_DHE_RSA_WITH_AES_128_CBC_SHA", 
> "TLS_DHE_RSA_WITH_AES_256_CBC_SHA",
> 29 "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA", 
> "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA" 
> 30 };
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12773) cassandra-stress error for one way SSL

[Robert Stupp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886335#comment-15886335
 ] 

Robert Stupp commented on CASSANDRA-12773:
--

Patch looks good so far. However, the CI results are no longer accessible. Can 
you rebase and re-run CI?

> cassandra-stress error for one way SSL 
> ---
>
> Key: CASSANDRA-12773
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12773
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Jane Deng
>Assignee: Stefan Podkowinski
> Fix For: 2.2.x
>
> Attachments: 12773-2.2.patch
>
>
> CASSANDRA-9325 added keystore/truststore configuration into cassandra-stress. 
> However, for one way ssl (require_client_auth=false), there is no need to 
> pass keystore info into ssloptions. Cassadra-stress errored out:
> {noformat}
> java.lang.RuntimeException: java.io.IOException: Error creating the 
> initializing the SSL Context 
> at 
> org.apache.cassandra.stress.settings.StressSettings.getJavaDriverClient(StressSettings.java:200)
>  
> at 
> org.apache.cassandra.stress.settings.SettingsSchema.createKeySpacesNative(SettingsSchema.java:79)
>  
> at 
> org.apache.cassandra.stress.settings.SettingsSchema.createKeySpaces(SettingsSchema.java:69)
>  
> at 
> org.apache.cassandra.stress.settings.StressSettings.maybeCreateKeyspaces(StressSettings.java:207)
>  
> at org.apache.cassandra.stress.StressAction.run(StressAction.java:55) 
> at org.apache.cassandra.stress.Stress.main(Stress.java:117) 
> Caused by: java.io.IOException: Error creating the initializing the SSL 
> Context 
> at 
> org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:151)
>  
> at 
> org.apache.cassandra.stress.util.JavaDriverClient.connect(JavaDriverClient.java:128)
>  
> at 
> org.apache.cassandra.stress.settings.StressSettings.getJavaDriverClient(StressSettings.java:191)
>  
> ... 5 more 
> Caused by: java.io.IOException: Keystore was tampered with, or password was 
> incorrect 
> at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772) 
> at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55) 
> at java.security.KeyStore.load(KeyStore.java:1445) 
> at 
> org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:129)
>  
> ... 7 more 
> Caused by: java.security.UnrecoverableKeyException: Password verification 
> failed 
> at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770) 
> ... 10 more
> {noformat}
> It's a bug from CASSANDRA-9325. When the keystore is absent, the keystore is 
> assigned to the path of the truststore, but the password isn't taken care.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (CASSANDRA-12773) cassandra-stress error for one way SSL

[Robert Stupp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-12773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp reassigned CASSANDRA-12773:


 Assignee: Stefan Podkowinski
 Reviewer: Robert Stupp
Fix Version/s: 2.2.x

> cassandra-stress error for one way SSL 
> ---
>
> Key: CASSANDRA-12773
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12773
> Project: Cassandra
>  Issue Type: Bug
>  Components: Tools
>Reporter: Jane Deng
>Assignee: Stefan Podkowinski
> Fix For: 2.2.x
>
> Attachments: 12773-2.2.patch
>
>
> CASSANDRA-9325 added keystore/truststore configuration into cassandra-stress. 
> However, for one way ssl (require_client_auth=false), there is no need to 
> pass keystore info into ssloptions. Cassadra-stress errored out:
> {noformat}
> java.lang.RuntimeException: java.io.IOException: Error creating the 
> initializing the SSL Context 
> at 
> org.apache.cassandra.stress.settings.StressSettings.getJavaDriverClient(StressSettings.java:200)
>  
> at 
> org.apache.cassandra.stress.settings.SettingsSchema.createKeySpacesNative(SettingsSchema.java:79)
>  
> at 
> org.apache.cassandra.stress.settings.SettingsSchema.createKeySpaces(SettingsSchema.java:69)
>  
> at 
> org.apache.cassandra.stress.settings.StressSettings.maybeCreateKeyspaces(StressSettings.java:207)
>  
> at org.apache.cassandra.stress.StressAction.run(StressAction.java:55) 
> at org.apache.cassandra.stress.Stress.main(Stress.java:117) 
> Caused by: java.io.IOException: Error creating the initializing the SSL 
> Context 
> at 
> org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:151)
>  
> at 
> org.apache.cassandra.stress.util.JavaDriverClient.connect(JavaDriverClient.java:128)
>  
> at 
> org.apache.cassandra.stress.settings.StressSettings.getJavaDriverClient(StressSettings.java:191)
>  
> ... 5 more 
> Caused by: java.io.IOException: Keystore was tampered with, or password was 
> incorrect 
> at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772) 
> at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55) 
> at java.security.KeyStore.load(KeyStore.java:1445) 
> at 
> org.apache.cassandra.security.SSLFactory.createSSLContext(SSLFactory.java:129)
>  
> ... 7 more 
> Caused by: java.security.UnrecoverableKeyException: Password verification 
> failed 
> at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770) 
> ... 10 more
> {noformat}
> It's a bug from CASSANDRA-9325. When the keystore is absent, the keystore is 
> assigned to the path of the truststore, but the password isn't taken care.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13271) Reduce lock contention on instance factories of ListType and SetType

[Robert Stupp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886318#comment-15886318
 ] 

Robert Stupp commented on CASSANDRA-13271:
--

The changes for {{ListType}} and {{SetType}} look good so far. However, you 
miss some similar one in {{MapType}} and {{ReversedType}} plus the 
corresponding {{List/Set/MapSerializer}} classes, that use the same pattern.
Can you provide a branch on GitHub instead of a patch file?

> Reduce lock contention on instance factories of ListType and SetType
> 
>
> Key: CASSANDRA-13271
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13271
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: vincent royer
>Priority: Minor
>  Labels: performance
> Fix For: 3.0.x
>
> Attachments: 
> 0001-CASSANDRA-13271-singleton-factory-concurrency-opimiz.patch
>
>
> By doing some performance tests, i noticed that getInstance() in 
> org.apache.cassandra.db.marshal.ListType and SetType could suffer from lock 
> contention on the singleton factory getInstance(). Here is a proposal to 
> reduce lock contention by using a ConcurrentMap and the putIfAbsent method 
> rather than a synchronized method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13271) Reduce lock contention on instance factories of ListType and SetType

[Robert Stupp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp updated CASSANDRA-13271:
-
Fix Version/s: (was: 3.11.x)

> Reduce lock contention on instance factories of ListType and SetType
> 
>
> Key: CASSANDRA-13271
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13271
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: vincent royer
>Priority: Minor
>  Labels: performance
> Fix For: 3.0.x
>
> Attachments: 
> 0001-CASSANDRA-13271-singleton-factory-concurrency-opimiz.patch
>
>
> By doing some performance tests, i noticed that getInstance() in 
> org.apache.cassandra.db.marshal.ListType and SetType could suffer from lock 
> contention on the singleton factory getInstance(). Here is a proposal to 
> reduce lock contention by using a ConcurrentMap and the putIfAbsent method 
> rather than a synchronized method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13271) Reduce lock contention on instance factories of ListType and SetType

[Robert Stupp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp updated CASSANDRA-13271:
-
Reviewer: Robert Stupp

> Reduce lock contention on instance factories of ListType and SetType
> 
>
> Key: CASSANDRA-13271
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13271
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: vincent royer
>Priority: Minor
>  Labels: performance
> Fix For: 3.0.x
>
> Attachments: 
> 0001-CASSANDRA-13271-singleton-factory-concurrency-opimiz.patch
>
>
> By doing some performance tests, i noticed that getInstance() in 
> org.apache.cassandra.db.marshal.ListType and SetType could suffer from lock 
> contention on the singleton factory getInstance(). Here is a proposal to 
> reduce lock contention by using a ConcurrentMap and the putIfAbsent method 
> rather than a synchronized method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (CASSANDRA-13277) Duplicate results with secondary index on static column

[Romain Hardouin (JIRA)]

Romain Hardouin created CASSANDRA-13277:
---

 Summary: Duplicate results with secondary index on static column
 Key: CASSANDRA-13277
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13277
 Project: Cassandra
  Issue Type: Bug
Reporter: Romain Hardouin


As a follow up of 
http://www.mail-archive.com/user@cassandra.apache.org/msg50816.html 

Duplicate results appear with secondary index on static column with RF > 1.
Number of results vary depending on consistency level.

Here is a CCM session to reproduce the issue:
{code}
romain@debian:~$ ccm create 39 -n 3 -v 3.9 -s
Current cluster is now: 39
romain@debian:~$ ccm node1 cqlsh
Connected to 39 at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': 2};
cqlsh> CREATE TABLE test.idx_static (id text, id2 bigint static, added 
timestamp, source text static, dest text, primary key (id, added));
cqlsh> CREATE index ON test.idx_static (id2);
cqlsh> INSERT INTO test.idx_static (id, id2, added, source, dest) values 
('id1', 22,'2017-01-28', 'src1', 'dst1');
cqlsh> SELECT * FROM test.idx_static where id2=22;

 id  | added   | id2 | source | dest
-+-+-++--
 id1 | 2017-01-27 23:00:00.00+ |  22 |   src1 | dst1
 id1 | 2017-01-27 23:00:00.00+ |  22 |   src1 | dst1

(2 rows)
cqlsh> CONSISTENCY ALL 
Consistency level set to ALL.
cqlsh> SELECT * FROM test.idx_static where id2=22;

 id  | added   | id2 | source | dest
-+-+-++--
 id1 | 2017-01-27 23:00:00.00+ |  22 |   src1 | dst1
 id1 | 2017-01-27 23:00:00.00+ |  22 |   src1 | dst1
 id1 | 2017-01-27 23:00:00.00+ |  22 |   src1 | dst1

(3 rows)
{code}

When RF matches the number of nodes, it works as expected.

Example with RF=3 and 3 nodes:
{code}
romain@debian:~$ ccm create 39 -n 3 -v 3.9 -s
Current cluster is now: 39

romain@debian:~$ ccm node1 cqlsh
Connected to 39 at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.

cqlsh> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': 3};
cqlsh> CREATE TABLE test.idx_static (id text, id2 bigint static, added 
timestamp, source text static, dest text, primary key (id, added));
cqlsh> CREATE index ON test.idx_static (id2);
cqlsh> INSERT INTO test.idx_static (id, id2, added, source, dest) values 
('id1', 22,'2017-01-28', 'src1', 'dst1');
cqlsh> SELECT * FROM test.idx_static where id2=22;

 id  | added   | id2 | source | dest
-+-+-++--
 id1 | 2017-01-27 23:00:00.00+ |  22 |   src1 | dst1

(1 rows)
cqlsh> CONSISTENCY all
Consistency level set to ALL.
cqlsh> SELECT * FROM test.idx_static where id2=22;

 id  | added   | id2 | source | dest
-+-+-++--
 id1 | 2017-01-27 23:00:00.00+ |  22 |   src1 | dst1

(1 rows)
{code}

Example with RF = 2 and 2 nodes:

{code}
romain@debian:~$ ccm create 39 -n 2 -v 3.9 -s
Current cluster is now: 39
romain@debian:~$ ccm node1 cqlsh
Connected to 39 at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]
Use HELP for help.
cqlsh> CREATE KEYSPACE test WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': 2};
cqlsh> CREATE TABLE test.idx_static (id text, id2 bigint static, added 
timestamp, source text static, dest text, primary key (id, added));
cqlsh> INSERT INTO test.idx_static (id, id2, added, source, dest) values 
('id1', 22,'2017-01-28', 'src1', 'dst1');
cqlsh> CREATE index ON test.idx_static (id2);
cqlsh> INSERT INTO test.idx_static (id, id2, added, source, dest) values 
('id1', 22,'2017-01-28', 'src1', 'dst1');
cqlsh> SELECT * FROM test.idx_static where id2=22;

 id  | added   | id2 | source | dest
-+-+-++--
 id1 | 2017-01-27 23:00:00.00+ |  22 |   src1 | dst1

(1 rows)
cqlsh> CONSISTENCY ALL 
Consistency level set to ALL.
cqlsh> SELECT * FROM test.idx_static where id2=22;

 id  | added   | id2 | source | dest
-+-+-++--
 id1 | 2017-01-27 23:00:00.00+ |  22 |   src1 | dst1

(1 rows)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (CASSANDRA-13276) Regression on CASSANDRA-11416: can't load snapshots of tables with dropped columns

[Matt Kopit (JIRA)]

Matt Kopit created CASSANDRA-13276:
--

 Summary: Regression on CASSANDRA-11416: can't load snapshots of 
tables with dropped columns
 Key: CASSANDRA-13276
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13276
 Project: Cassandra
  Issue Type: Bug
Reporter: Matt Kopit


I'm running Cassandra 3.10 and running into the exact same issue described in 
CASSANDRA-11416: 

1. A table is created with columns 'a' and 'b'
2. Data is written to the table
3. Drop column 'b'
4. Take a snapshot
5. Drop the table
6. Run the snapshot schema.cql to recreate the table and the run the alter
7. Try to restore the snapshot data using sstableloader

sstableloader yields the error:
java.lang.RuntimeException: Unknown column b during deserialization



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-13042) The two cassandra nodes suddenly encounter hints each other and failed replaying.

[Greg Doermann (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886128#comment-15886128
 ] 

Greg Doermann edited comment on CASSANDRA-13042 at 2/27/17 5:06 PM:


We are getting a constant loop of restarts on the affected servers.  It tries 
to send the server the hints and ends up restarting when all of them fail:

{code}
INFO  [HintedHandoff:5] 2017-02-27 16:52:13,959 HintedHandOffManager.java:367 - 
Started hinted handoff for host: ---000- with IP: 
/1.1.1.1
ERROR [HintedHandoff:10] 2017-02-27 16:52:13,961 CassandraDaemon.java:185 - 
Exception in thread Thread[HintedHandoff:10,1,main]
java.lang.AssertionError: null
at 
org.apache.cassandra.net.WriteCallbackInfo.(WriteCallbackInfo.java:49) 
~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:639)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:703) 
~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:474)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:354)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:93)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:566)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_74]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
~[na:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_74]
INFO  [HintedHandoff:5] 2017-02-27 16:52:19,465 HintedHandOffManager.java:486 - 
Failed replaying hints to /1.1.1.1; aborting (523 delivered), error : Operation 
timed out - received only 0 responses.
{code}



was (Author: gdoermann):
We are getting a constant loop of restarts on the affected servers.  It tries 
to send the server the hints and ends up restarting when all of them fail:

{quote}
INFO  [HintedHandoff:5] 2017-02-27 16:52:13,959 HintedHandOffManager.java:367 - 
Started hinted handoff for host: ---000- with IP: 
/1.1.1.1
ERROR [HintedHandoff:10] 2017-02-27 16:52:13,961 CassandraDaemon.java:185 - 
Exception in thread Thread[HintedHandoff:10,1,main]
java.lang.AssertionError: null
at 
org.apache.cassandra.net.WriteCallbackInfo.(WriteCallbackInfo.java:49) 
~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:639)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:703) 
~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:474)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:354)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:93)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:566)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_74]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
~[na:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_74]
INFO  [HintedHandoff:5] 2017-02-27 16:52:19,465 HintedHandOffManager.java:486 - 
Failed replaying hints to /1.1.1.1; aborting (523 delivered), error : Operation 
timed out - received only 0 responses.
{quote}


> The two cassandra nodes suddenly encounter hints each other and failed 
> replaying.
> -
>
> Key: CASSANDRA-13042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13042
> Project: Cassandra
>  Issue Type: Bug
>Reporter: YheonHo.Choi
>Priority: Critical
> Attachments: out_2.2.2.1.txt, out_2.2.2.2.txt
>
>
> Although there are no changes to cassandra, two node suddenly encounter hints 
> and failed replaying.
> Any commands like disablethrift, disablegossip can not solve the above 
> problem and the only way was restart.
> When we check the status of cluster, all nodes are looks UN but 
> describecluster show unreachable each other.
> Here's the state 

[jira] [Comment Edited] (CASSANDRA-13042) The two cassandra nodes suddenly encounter hints each other and failed replaying.

[Greg Doermann (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886128#comment-15886128
 ] 

Greg Doermann edited comment on CASSANDRA-13042 at 2/27/17 5:05 PM:


We are getting a constant loop of restarts on the affected servers.  It tries 
to send the server the hints and ends up restarting when all of them fail:

{quote}
INFO  [HintedHandoff:5] 2017-02-27 16:52:13,959 HintedHandOffManager.java:367 - 
Started hinted handoff for host: ---000- with IP: 
/1.1.1.1
ERROR [HintedHandoff:10] 2017-02-27 16:52:13,961 CassandraDaemon.java:185 - 
Exception in thread Thread[HintedHandoff:10,1,main]
java.lang.AssertionError: null
at 
org.apache.cassandra.net.WriteCallbackInfo.(WriteCallbackInfo.java:49) 
~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:639)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:703) 
~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:474)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:354)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:93)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:566)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_74]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
~[na:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_74]
INFO  [HintedHandoff:5] 2017-02-27 16:52:19,465 HintedHandOffManager.java:486 - 
Failed replaying hints to /1.1.1.1; aborting (523 delivered), error : Operation 
timed out - received only 0 responses.
{quote}



was (Author: gdoermann):
We are getting a constant loop of restarts on the affected servers.  It tries 
to send the server the hints and ends up restarting when all of them fail:

```INFO  [HintedHandoff:5] 2017-02-27 16:52:13,959 
HintedHandOffManager.java:367 - Started hinted handoff for host: 
---000- with IP: /1.1.1.1
ERROR [HintedHandoff:10] 2017-02-27 16:52:13,961 CassandraDaemon.java:185 - 
Exception in thread Thread[HintedHandoff:10,1,main]
java.lang.AssertionError: null
at 
org.apache.cassandra.net.WriteCallbackInfo.(WriteCallbackInfo.java:49) 
~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:639)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:703) 
~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:474)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:354)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:93)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:566)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_74]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
~[na:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_74]
INFO  [HintedHandoff:5] 2017-02-27 16:52:19,465 HintedHandOffManager.java:486 - 
Failed replaying hints to /1.1.1.1; aborting (523 delivered), error : Operation 
timed out - received only 0 responses.
```


> The two cassandra nodes suddenly encounter hints each other and failed 
> replaying.
> -
>
> Key: CASSANDRA-13042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13042
> Project: Cassandra
>  Issue Type: Bug
>Reporter: YheonHo.Choi
>Priority: Critical
> Attachments: out_2.2.2.1.txt, out_2.2.2.2.txt
>
>
> Although there are no changes to cassandra, two node suddenly encounter hints 
> and failed replaying.
> Any commands like disablethrift, disablegossip can not solve the above 
> problem and the only way was restart.
> When we check the status of cluster, all nodes are looks UN but 
> describecluster show unreachable each other.
> Here's the state of the 

[jira] [Commented] (CASSANDRA-13042) The two cassandra nodes suddenly encounter hints each other and failed replaying.

[Greg Doermann (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886128#comment-15886128
 ] 

Greg Doermann commented on CASSANDRA-13042:
---

We are getting a constant loop of restarts on the affected servers.  It tries 
to send the server the hints and ends up restarting when all of them fail:

```INFO  [HintedHandoff:5] 2017-02-27 16:52:13,959 
HintedHandOffManager.java:367 - Started hinted handoff for host: 
---000- with IP: /1.1.1.1
ERROR [HintedHandoff:10] 2017-02-27 16:52:13,961 CassandraDaemon.java:185 - 
Exception in thread Thread[HintedHandoff:10,1,main]
java.lang.AssertionError: null
at 
org.apache.cassandra.net.WriteCallbackInfo.(WriteCallbackInfo.java:49) 
~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:639)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:703) 
~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:474)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:354)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:93)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:566)
 ~[apache-cassandra-2.2.5.jar:2.2.5]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_74]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
~[na:1.8.0_74]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_74]
INFO  [HintedHandoff:5] 2017-02-27 16:52:19,465 HintedHandOffManager.java:486 - 
Failed replaying hints to /1.1.1.1; aborting (523 delivered), error : Operation 
timed out - received only 0 responses.
```


> The two cassandra nodes suddenly encounter hints each other and failed 
> replaying.
> -
>
> Key: CASSANDRA-13042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13042
> Project: Cassandra
>  Issue Type: Bug
>Reporter: YheonHo.Choi
>Priority: Critical
> Attachments: out_2.2.2.1.txt, out_2.2.2.2.txt
>
>
> Although there are no changes to cassandra, two node suddenly encounter hints 
> and failed replaying.
> Any commands like disablethrift, disablegossip can not solve the above 
> problem and the only way was restart.
> When we check the status of cluster, all nodes are looks UN but 
> describecluster show unreachable each other.
> Here's the state of the cassandra during the above problem occurred.
> IP addresses in report anonymized:
> cassandra version: 2.2.5
> node 1 = 1.1.1.1
> node 2 = 1.1.1.2
> others = x.x.x.x
> system.log
> {code}
> ## result of nodetool status on 1.1.1.1
> INFO  [HintedHandoff:1] 2016-11-24 06:15:07,969 HintedHandOffManager.java:367 
> - Started hinted handoff for host: 8caa54f3-7d67-40d6-b224-8c64a1d289be with 
> IP: /1.1.1.2
> INFO  [HintedHandoff:1] 2016-11-24 06:15:09,969 HintedHandOffManager.java:486 
> - Failed replaying hints to /1.1.1.2; aborting (0 delivered), error : 
> Operation timed out - received only 0 responses.
> INFO  [HintedHandoff:2] 2016-11-24 06:25:09,736 HintedHandOffManager.java:367 
> - Started hinted handoff for host: 8caa54f3-7d67-40d6-b224-8c64a1d289be with 
> IP: /1.1.1.2
> INFO  [HintedHandoff:2] 2016-11-24 06:25:11,738 HintedHandOffManager.java:486 
> - Failed replaying hints to /1.1.1.2; aborting (0 delivered), error : 
> Operation timed out - received only 0 responses.
> WARN  [MemtableFlushWriter:55270] 2016-11-24 06:25:12,625 
> BigTableWriter.java:184 - Writing large partition 
> system/hints:d640677d-f354-aa8c-be89-d2a1648c24b2 (109029803 bytes)
> WARN  [CompactionExecutor:37908] 2016-11-24 06:35:23,682 
> BigTableWriter.java:184 - Writing large partition 
> system/hints:8caa54f3-7d67-40d6-b224-8c64a1d289be (250651758 bytes)
> INFO  [HintedHandoff:1] 2016-11-24 06:35:23,727 HintedHandOffManager.java:367 
> - Started hinted handoff for host: 8caa54f3-7d67-40d6-b224-8c64a1d289be with 
> IP: /1.1.1.2
> INFO  [HintedHandoff:1] 2016-11-24 06:35:25,728 HintedHandOffManager.java:486 
> - Failed replaying hints to /1.1.1.2; aborting (0 delivered), error : 
> Operation timed out - received only 0 responses.
> WARN  [CompactionExecutor:37909] 2016-11-24 06:45:53,615 
> BigTableWriter.java:184 - Writing large partition 
> system/hints:8caa54f3-7d67-40d6-b224-8c64a1d289be (340801514 bytes)
> INFO  [HintedHandoff:2] 2016-11-24 06:45:53,718 

[jira] [Updated] (CASSANDRA-13090) Coalescing strategy sleeps too much

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg updated CASSANDRA-13090:
---
Status: Patch Available  (was: Reopened)

> Coalescing strategy sleeps too much
> ---
>
> Key: CASSANDRA-13090
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13090
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Corentin Chary
>Assignee: Corentin Chary
> Fix For: 3.11.0, 4.0, 3.0.11, 2.2.9
>
> Attachments: 0001-Fix-wait-time-coalescing-CASSANDRA-13090-2.patch, 
> 0001-Fix-wait-time-coalescing-CASSANDRA-13090.patch
>
>
> With the current code maybeSleep is called even if we managed to take 
> maxItems out of the backlog. In this case we should really avoid sleeping 
> because it means that backlog is building up.
> I'll send a patch shortly.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Reopened] (CASSANDRA-13090) Coalescing strategy sleeps too much

[Ariel Weisberg (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ariel Weisberg reopened CASSANDRA-13090:


> Coalescing strategy sleeps too much
> ---
>
> Key: CASSANDRA-13090
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13090
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Corentin Chary
>Assignee: Corentin Chary
> Fix For: 2.2.9, 3.0.11, 3.11.0, 4.0
>
> Attachments: 0001-Fix-wait-time-coalescing-CASSANDRA-13090-2.patch, 
> 0001-Fix-wait-time-coalescing-CASSANDRA-13090.patch
>
>
> With the current code maybeSleep is called even if we managed to take 
> maxItems out of the backlog. In this case we should really avoid sleeping 
> because it means that backlog is building up.
> I'll send a patch shortly.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13090) Coalescing strategy sleeps too much

[Michael Shuler (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886058#comment-15886058
 ] 

Michael Shuler commented on CASSANDRA-13090:


The CHANGE.txt entry for this ticket is under the incorrect releases in the 
cassandra-2.2, cassandra-3.0, cassandra-3.11, and trunk branches. This was 
committed during release votes were -tentative tagged.  The JIRA fix versions 
for 2.2.x and 3.0.x should be corrected, as well, since this commit should be 
listed in CHANGES.txt under:
- 2.2.10 (currently under 2.2.9)
- 3.0.12 (currently under 3.0.11)
- 3.11.0 (currently under 3.10 in both cassandra-3.11 and trunk branches 
CHANGES.txt)

> Coalescing strategy sleeps too much
> ---
>
> Key: CASSANDRA-13090
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13090
> Project: Cassandra
>  Issue Type: Bug
>  Components: Streaming and Messaging
>Reporter: Corentin Chary
>Assignee: Corentin Chary
> Fix For: 2.2.9, 3.0.11, 3.11.0, 4.0
>
> Attachments: 0001-Fix-wait-time-coalescing-CASSANDRA-13090-2.patch, 
> 0001-Fix-wait-time-coalescing-CASSANDRA-13090.patch
>
>
> With the current code maybeSleep is called even if we managed to take 
> maxItems out of the backlog. In this case we should really avoid sleeping 
> because it means that backlog is building up.
> I'll send a patch shortly.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13042) The two cassandra nodes suddenly encounter hints each other and failed replaying.

[Greg Doermann (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886023#comment-15886023
 ] 

Greg Doermann commented on CASSANDRA-13042:
---

+1 on this one. We are also having this problem on 2.2.5 every single day. 

> The two cassandra nodes suddenly encounter hints each other and failed 
> replaying.
> -
>
> Key: CASSANDRA-13042
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13042
> Project: Cassandra
>  Issue Type: Bug
>Reporter: YheonHo.Choi
>Priority: Critical
> Attachments: out_2.2.2.1.txt, out_2.2.2.2.txt
>
>
> Although there are no changes to cassandra, two node suddenly encounter hints 
> and failed replaying.
> Any commands like disablethrift, disablegossip can not solve the above 
> problem and the only way was restart.
> When we check the status of cluster, all nodes are looks UN but 
> describecluster show unreachable each other.
> Here's the state of the cassandra during the above problem occurred.
> IP addresses in report anonymized:
> cassandra version: 2.2.5
> node 1 = 1.1.1.1
> node 2 = 1.1.1.2
> others = x.x.x.x
> system.log
> {code}
> ## result of nodetool status on 1.1.1.1
> INFO  [HintedHandoff:1] 2016-11-24 06:15:07,969 HintedHandOffManager.java:367 
> - Started hinted handoff for host: 8caa54f3-7d67-40d6-b224-8c64a1d289be with 
> IP: /1.1.1.2
> INFO  [HintedHandoff:1] 2016-11-24 06:15:09,969 HintedHandOffManager.java:486 
> - Failed replaying hints to /1.1.1.2; aborting (0 delivered), error : 
> Operation timed out - received only 0 responses.
> INFO  [HintedHandoff:2] 2016-11-24 06:25:09,736 HintedHandOffManager.java:367 
> - Started hinted handoff for host: 8caa54f3-7d67-40d6-b224-8c64a1d289be with 
> IP: /1.1.1.2
> INFO  [HintedHandoff:2] 2016-11-24 06:25:11,738 HintedHandOffManager.java:486 
> - Failed replaying hints to /1.1.1.2; aborting (0 delivered), error : 
> Operation timed out - received only 0 responses.
> WARN  [MemtableFlushWriter:55270] 2016-11-24 06:25:12,625 
> BigTableWriter.java:184 - Writing large partition 
> system/hints:d640677d-f354-aa8c-be89-d2a1648c24b2 (109029803 bytes)
> WARN  [CompactionExecutor:37908] 2016-11-24 06:35:23,682 
> BigTableWriter.java:184 - Writing large partition 
> system/hints:8caa54f3-7d67-40d6-b224-8c64a1d289be (250651758 bytes)
> INFO  [HintedHandoff:1] 2016-11-24 06:35:23,727 HintedHandOffManager.java:367 
> - Started hinted handoff for host: 8caa54f3-7d67-40d6-b224-8c64a1d289be with 
> IP: /1.1.1.2
> INFO  [HintedHandoff:1] 2016-11-24 06:35:25,728 HintedHandOffManager.java:486 
> - Failed replaying hints to /1.1.1.2; aborting (0 delivered), error : 
> Operation timed out - received only 0 responses.
> WARN  [CompactionExecutor:37909] 2016-11-24 06:45:53,615 
> BigTableWriter.java:184 - Writing large partition 
> system/hints:8caa54f3-7d67-40d6-b224-8c64a1d289be (340801514 bytes)
> INFO  [HintedHandoff:2] 2016-11-24 06:45:53,718 HintedHandOffManager.java:367 
> - Started hinted handoff for host: 8caa54f3-7d67-40d6-b224-8c64a1d289be with 
> IP: /1.1.1.2
> INFO  [HintedHandoff:2] 2016-11-24 06:45:55,719 HintedHandOffManager.java:486 
> - Failed replaying hints to /1.1.1.2; aborting (0 delivered), error : 
> Operation timed out - received only 0 responses.
> WARN  [CompactionExecutor:37912] 2016-11-24 06:56:20,884 
> BigTableWriter.java:184 - Writing large partition 
> system/hints:8caa54f3-7d67-40d6-b224-8c64a1d289be (472465093 bytes)
> INFO  [HintedHandoff:1] 2016-11-24 06:56:20,966 HintedHandOffManager.java:367 
> - Started hinted handoff for host: 8caa54f3-7d67-40d6-b224-8c64a1d289be with 
> IP: /1.1.1.2
> INFO  [HintedHandoff:1] 2016-11-24 06:56:22,967 HintedHandOffManager.java:486 
> - Failed replaying hints to /1.1.1.2; aborting (0 delivered), error : 
> Operation timed out - received only 0 responses.
> WARN  [CompactionExecutor:37911] 2016-11-24 07:07:12,568 
> BigTableWriter.java:184 - Writing large partition 
> system/hints:8caa54f3-7d67-40d6-b224-8c64a1d289be (577392172 bytes)
> INFO  [HintedHandoff:2] 2016-11-24 07:07:12,643 HintedHandOffManager.java:367 
> - Started hinted handoff for host: 8caa54f3-7d67-40d6-b224-8c64a1d289be with 
> IP: /1.1.1.2
> INFO  [HintedHandoff:2] 2016-11-24 07:07:14,643 HintedHandOffManager.java:486 
> - Failed replaying hints to /1.1.1.2; aborting (0 delivered), error : 
> Operation timed out - received only 0 responses.
> INFO  [IndexSummaryManager:1] 2016-11-24 07:09:15,929 
> IndexSummaryRedistribution.java:74 - Redistributing index summaries
> ## result of nodetool status on 1.1.1.2
> INFO  [HintedHandoff:1] 2016-11-24 06:11:37,300 HintedHandOffManager.java:367 
> - Started hinted handoff for host: b79124e9-394c-4400-a8e7-a0c94aec6878 with 
> IP: /1.1.1.1
> INFO  [HintedHandoff:1] 2016-11-24 06:11:39,301 

[jira] [Comment Edited] (CASSANDRA-13274) Fix rolling upgrade to 4.0+

[Robert Stupp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885961#comment-15885961
 ] 

Robert Stupp edited comment on CASSANDRA-13274 at 2/27/17 4:01 PM:
---

||cassandra-3.0|[branch|https://github.com/apache/cassandra/compare/cassandra-3.0...snazy:13274-rolling-upgrade-3.0]|[testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.0-testall/lastSuccessfulBuild/]|[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.0-dtest/lastSuccessfulBuild/]
||cassandra-3.11|[branch|https://github.com/apache/cassandra/compare/cassandra-3.11...snazy:13274-rolling-upgrade-3.11]|[testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.11-testall/lastSuccessfulBuild/]|[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.11-dtest/lastSuccessfulBuild/]
||trunk|[branch|https://github.com/apache/cassandra/compare/trunk...snazy:13274-rolling-upgrade-trunk]|[testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-trunk-testall/lastSuccessfulBuild/]|[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-trunk-dtest/lastSuccessfulBuild/]

CI triggered.

Also tested a few CQL upgrade tests from "legacy" 3.0 to current trunk (to 
verify the upgrade issue locally) and this patch to trunk (to verify the fix).

Also added an entry to {{NEWS.txt}} in the trunk branch regarding rolling 
upgrades.

Reviewer hint: for {{rolling_upgrade_test}}, it's necessary to disable the 
{{nodetool upgradesstables }} (comment out {{node.nodetool('upgradesstables 
-a')}} in {{upgrade_through_versions_test.py}} line#446) invocation until 
CASSANDRA-13059 is fixed.


was (Author: snazy):
||cassandra-3.0|[branch|https://github.com/apache/cassandra/compare/cassandra-3.0...snazy:13274-rolling-upgrade-3.0]|[testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.0-testall/lastSuccessfulBuild/]|[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.0-dtest/lastSuccessfulBuild/]
||cassandra-3.11|[branch|https://github.com/apache/cassandra/compare/cassandra-3.11...snazy:13274-rolling-upgrade-3.11]|[testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.11-testall/lastSuccessfulBuild/]|[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.11-dtest/lastSuccessfulBuild/]
||trunk|[branch|https://github.com/apache/cassandra/compare/trunk...snazy:13274-rolling-upgrade-trunk]|[testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-trunk-testall/lastSuccessfulBuild/]|[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-trunk-dtest/lastSuccessfulBuild/]

CI triggered.

Also tested a few CQL upgrade tests from "legacy" 3.0 to current trunk (to 
verify the upgrade issue locally) and this patch to trunk (to verify the fix).

Also added an entry to {{NEWS.txt}} in the trunk branch regarding rolling 
upgrades.

> Fix rolling upgrade to 4.0+
> ---
>
> Key: CASSANDRA-13274
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13274
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> A rolling upgrade from 3.* to 4.0 (messaging version {{11}}) unveils a 
> regression caused by CASSANDRA-11128.
> Generally, we store all possible options/attributes including the default 
> values in the schema. This causes (expected) schema-version-mismatches during 
> rolling upgrades and therefore we prevent schema pulls/pushes in this 
> situation, which has been broken by CASSANDRA-11128.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-10520) Compressed writer and reader should support non-compressed data.

[Robert Stupp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886000#comment-15886000
 ] 

Robert Stupp commented on CASSANDRA-10520:
--

(Opened CASSANDRA-13274 for the follow-up to fix the rolling-upgrade issue.)

I've got a slight preference for Sylvain's proposal to not touch existing 
tables but enable it on new tables.
This is a nice improvement for 4.0+ sstables and all values for 
{{min_compress_ratio}} go through the changed code anyway - want to say: I'm 
open to both proposals.

> Compressed writer and reader should support non-compressed data.
> 
>
> Key: CASSANDRA-10520
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10520
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Local Write-Read Paths
>Reporter: Branimir Lambov
>Assignee: Branimir Lambov
>  Labels: messaging-service-bump-required
> Fix For: 4.x
>
> Attachments: ReadWriteTestCompression.java
>
>
> Compressing uncompressible data, as done, for instance, to write SSTables 
> during stress-tests, results in chunks larger than 64k which are a problem 
> for the buffer pooling mechanisms employed by the 
> {{CompressedRandomAccessReader}}. This results in non-negligible performance 
> issues due to excessive memory allocation.
> To solve this problem and avoid decompression delays in the cases where it 
> does not provide benefits, I think we should allow compressed files to store 
> uncompressed chunks as alternative to compressed data. Such a chunk could be 
> written after compression returns a buffer larger than, for example, 90% of 
> the input, and would not result in additional delays in writing. On reads it 
> could be recognized by size (using a single global threshold constant in the 
> compression metadata) and data could be directly transferred into the 
> decompressed buffer, skipping the decompression step and ensuring a 64k 
> buffer for compressed data always suffices.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-13274) Fix rolling upgrade to 4.0+

[Robert Stupp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885961#comment-15885961
 ] 

Robert Stupp edited comment on CASSANDRA-13274 at 2/27/17 3:36 PM:
---

||cassandra-3.0|[branch|https://github.com/apache/cassandra/compare/cassandra-3.0...snazy:13274-rolling-upgrade-3.0]|[testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.0-testall/lastSuccessfulBuild/]|[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.0-dtest/lastSuccessfulBuild/]
||cassandra-3.11|[branch|https://github.com/apache/cassandra/compare/cassandra-3.11...snazy:13274-rolling-upgrade-3.11]|[testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.11-testall/lastSuccessfulBuild/]|[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.11-dtest/lastSuccessfulBuild/]
||trunk|[branch|https://github.com/apache/cassandra/compare/trunk...snazy:13274-rolling-upgrade-trunk]|[testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-trunk-testall/lastSuccessfulBuild/]|[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-trunk-dtest/lastSuccessfulBuild/]

CI triggered.

Also tested a few CQL upgrade tests from "legacy" 3.0 to current trunk (to 
verify the upgrade issue locally) and this patch to trunk (to verify the fix).

Also added an entry to {{NEWS.txt}} in the trunk branch regarding rolling 
upgrades.


was (Author: snazy):
||cassandra-3.0|[branch|https://github.com/apache/cassandra/compare/cassandra-3.0...snazy:13274-rolling-upgrade-3.0]|[testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.0-testall/lastSuccessfulBuild/]|[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.0-dtest/lastSuccessfulBuild/]
||cassandra-3.11|[branch|https://github.com/apache/cassandra/compare/cassandra-3.11...snazy:13274-rolling-upgrade-3.11]|[testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.11-testall/lastSuccessfulBuild/]|[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.11-dtest/lastSuccessfulBuild/]
||trunk|[branch|https://github.com/apache/cassandra/compare/trunk...snazy:13274-rolling-upgrade-trunk]|[testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-trunk-testall/lastSuccessfulBuild/]|[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-trunk-dtest/lastSuccessfulBuild/]

CI triggered.

Also tested a few CQL upgrade tests from "legacy" 3.0 to current trunk (to 
verify the upgrade issue locally) and this patch to trunk (to verify the fix).

> Fix rolling upgrade to 4.0+
> ---
>
> Key: CASSANDRA-13274
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13274
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> A rolling upgrade from 3.* to 4.0 (messaging version {{11}}) unveils a 
> regression caused by CASSANDRA-11128.
> Generally, we store all possible options/attributes including the default 
> values in the schema. This causes (expected) schema-version-mismatches during 
> rolling upgrades and therefore we prevent schema pulls/pushes in this 
> situation, which has been broken by CASSANDRA-11128.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (CASSANDRA-13274) Fix rolling upgrade to 4.0+

[Robert Stupp (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Stupp updated CASSANDRA-13274:
-
Status: Patch Available  (was: Open)

||cassandra-3.0|[branch|https://github.com/apache/cassandra/compare/cassandra-3.0...snazy:13274-rolling-upgrade-3.0]|[testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.0-testall/lastSuccessfulBuild/]|[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.0-dtest/lastSuccessfulBuild/]
||cassandra-3.11|[branch|https://github.com/apache/cassandra/compare/cassandra-3.11...snazy:13274-rolling-upgrade-3.11]|[testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.11-testall/lastSuccessfulBuild/]|[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-3.11-dtest/lastSuccessfulBuild/]
||trunk|[branch|https://github.com/apache/cassandra/compare/trunk...snazy:13274-rolling-upgrade-trunk]|[testall|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-trunk-testall/lastSuccessfulBuild/]|[dtest|http://cassci.datastax.com/view/Dev/view/snazy/job/snazy-13274-rolling-upgrade-trunk-dtest/lastSuccessfulBuild/]

CI triggered.

Also tested a few CQL upgrade tests from "legacy" 3.0 to current trunk (to 
verify the upgrade issue locally) and this patch to trunk (to verify the fix).

> Fix rolling upgrade to 4.0+
> ---
>
> Key: CASSANDRA-13274
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13274
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> A rolling upgrade from 3.* to 4.0 (messaging version {{11}}) unveils a 
> regression caused by CASSANDRA-11128.
> Generally, we store all possible options/attributes including the default 
> values in the schema. This causes (expected) schema-version-mismatches during 
> rolling upgrades and therefore we prevent schema pulls/pushes in this 
> situation, which has been broken by CASSANDRA-11128.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13274) Fix rolling upgrade to 4.0+

[Robert Stupp (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885919#comment-15885919
 ] 

Robert Stupp commented on CASSANDRA-13274:
--

CASSANDRA-11138 sets the messaging-version for any node to {{Math.min(version, 
current_version}}. I.e. a newer node (aka C*4.0 using messaging-version 11) 
would appear as messaging-version 10 in 3.* nodes. That basically allows 
migration-tasks (schema pull/push) between different messaging versions.

> Fix rolling upgrade to 4.0+
> ---
>
> Key: CASSANDRA-13274
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13274
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> A rolling upgrade from 3.* to 4.0 (messaging version {{11}}) unveils a 
> regression caused by CASSANDRA-11128.
> Generally, we store all possible options/attributes including the default 
> values in the schema. This causes (expected) schema-version-mismatches during 
> rolling upgrades and therefore we prevent schema pulls/pushes in this 
> situation, which has been broken by CASSANDRA-11128.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-12965) StreamReceiveTask causing high CPU utilization during repair

[liangsibin (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885877#comment-15885877
 ] 

liangsibin edited comment on CASSANDRA-12965 at 2/27/17 2:48 PM:
-

when I use four nodes bulkload data into cassandra, sometimes one of the nodes 
cpu usage is almost 100%, which make bulkload very slow.Is there any method can 
solve this problem? 
cassandra 2.1.14
help


was (Author: bruceliang):
when I use four nodes bulkload data into cassandra, sometimes one of the nodes 
cpu usage is almost 100%, which make bulkload very slow.Is there any method can 
solve this problem? 
help

> StreamReceiveTask causing high CPU utilization during repair
> 
>
> Key: CASSANDRA-12965
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12965
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Randy Fradin
>
> During a full repair run, I observed one node in my cluster using 100% cpu 
> (100% of all cores on a 48-core machine). When I took a stack trace I found 
> exactly 48 running StreamReceiveTask threads. Each was in the same block of 
> code in StreamReceiveTask.OnCompletionRunnable:
> {noformat}
> "StreamReceiveTask:8077" #1511134 daemon prio=5 os_prio=0 
> tid=0x7f01520a8800 nid=0x6e77 runnable [0x7f020dfae000]
>java.lang.Thread.State: RUNNABLE
> at java.util.ComparableTimSort.binarySort(ComparableTimSort.java:258)
> at java.util.ComparableTimSort.sort(ComparableTimSort.java:203)
> at java.util.Arrays.sort(Arrays.java:1312)
> at java.util.Arrays.sort(Arrays.java:1506)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:141)
> at 
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:257)
> at 
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:280)
> at 
> org.apache.cassandra.utils.IntervalTree.(IntervalTree.java:72)
> at 
> org.apache.cassandra.db.DataTracker$SSTableIntervalTree.(DataTracker.java:590)
> at 
> org.apache.cassandra.db.DataTracker$SSTableIntervalTree.(DataTracker.java:584)
> at 
> org.apache.cassandra.db.DataTracker.buildIntervalTree(DataTracker.java:565)
> at 
> org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:761)
> at 
> org.apache.cassandra.db.DataTracker.addSSTablesToTracker(DataTracker.java:428)
> at 
> org.apache.cassandra.db.DataTracker.addSSTables(DataTracker.java:283)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.addSSTables(ColumnFamilyStore.java:1422)
> at 
> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:148)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> All 48 threads were in ColumnFamilyStore.addSSTables(), and specifically in 
> the IntervalNode constructor called from the IntervalTree constructor.
> It stayed this way for maybe an hour before we restarted the node. The repair 
> was also generating thousands (20,000+) of tiny SSTables in a table that 
> previously had just 20.
> I don't know enough about SSTables and ColumnFamilyStore to know if all this 
> CPU work is necessary or a bug, but I did notice that these tasks are run on 
> a thread pool constructed in StreamReceiveTask.java, so perhaps this pool 
> should have a thread count max less than the number of processors on the 
> machine, at least for machines with a lot of processors. Any reason not to do 
> that? Any ideas for a reasonable # or formula to cap the thread count?
> Some additional info: We have never run incremental repair on this cluster, 
> so that is not a factor. All our tables use LCS. Unfortunately I don't have 
> the log files from the period saved.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-12965) StreamReceiveTask causing high CPU utilization during repair

[liangsibin (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-12965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885877#comment-15885877
 ] 

liangsibin commented on CASSANDRA-12965:


when I use four nodes bulkload data into cassandra, sometimes one of the nodes 
cpu usage is almost 100%, which make bulkload very slow.Is there any method can 
solve this problem? 
help

> StreamReceiveTask causing high CPU utilization during repair
> 
>
> Key: CASSANDRA-12965
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12965
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Randy Fradin
>
> During a full repair run, I observed one node in my cluster using 100% cpu 
> (100% of all cores on a 48-core machine). When I took a stack trace I found 
> exactly 48 running StreamReceiveTask threads. Each was in the same block of 
> code in StreamReceiveTask.OnCompletionRunnable:
> {noformat}
> "StreamReceiveTask:8077" #1511134 daemon prio=5 os_prio=0 
> tid=0x7f01520a8800 nid=0x6e77 runnable [0x7f020dfae000]
>java.lang.Thread.State: RUNNABLE
> at java.util.ComparableTimSort.binarySort(ComparableTimSort.java:258)
> at java.util.ComparableTimSort.sort(ComparableTimSort.java:203)
> at java.util.Arrays.sort(Arrays.java:1312)
> at java.util.Arrays.sort(Arrays.java:1506)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:141)
> at 
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:257)
> at 
> org.apache.cassandra.utils.IntervalTree$IntervalNode.(IntervalTree.java:280)
> at 
> org.apache.cassandra.utils.IntervalTree.(IntervalTree.java:72)
> at 
> org.apache.cassandra.db.DataTracker$SSTableIntervalTree.(DataTracker.java:590)
> at 
> org.apache.cassandra.db.DataTracker$SSTableIntervalTree.(DataTracker.java:584)
> at 
> org.apache.cassandra.db.DataTracker.buildIntervalTree(DataTracker.java:565)
> at 
> org.apache.cassandra.db.DataTracker$View.replace(DataTracker.java:761)
> at 
> org.apache.cassandra.db.DataTracker.addSSTablesToTracker(DataTracker.java:428)
> at 
> org.apache.cassandra.db.DataTracker.addSSTables(DataTracker.java:283)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.addSSTables(ColumnFamilyStore.java:1422)
> at 
> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:148)
> at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> All 48 threads were in ColumnFamilyStore.addSSTables(), and specifically in 
> the IntervalNode constructor called from the IntervalTree constructor.
> It stayed this way for maybe an hour before we restarted the node. The repair 
> was also generating thousands (20,000+) of tiny SSTables in a table that 
> previously had just 20.
> I don't know enough about SSTables and ColumnFamilyStore to know if all this 
> CPU work is necessary or a bug, but I did notice that these tasks are run on 
> a thread pool constructed in StreamReceiveTask.java, so perhaps this pool 
> should have a thread count max less than the number of processors on the 
> machine, at least for machines with a lot of processors. Any reason not to do 
> that? Any ideas for a reasonable # or formula to cap the thread count?
> Some additional info: We have never run incremental repair on this cluster, 
> so that is not a factor. All our tables use LCS. Unfortunately I don't have 
> the log files from the period saved.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (CASSANDRA-13275) Cassandra throws an exception during CQL select query filtering on map key

[Abderrahmane CHRAIBI (JIRA)]

Abderrahmane CHRAIBI created CASSANDRA-13275:


 Summary: Cassandra throws an exception during CQL select query 
filtering on map key 
 Key: CASSANDRA-13275
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13275
 Project: Cassandra
  Issue Type: Bug
  Components: CQL
Reporter: Abderrahmane CHRAIBI


Env : [cqlsh 5.0.1 | Cassandra 3.9 | CQL spec 3.4.2 | Native protocol v4]

Using this table structure : 
CREATE TABLE mytable (
mymap frozen>> PRIMARY KEY
)

Executing : 
select * from mytable where mymap contains key UUID;

Within cqlsh shows this message :
ServerError: java.lang.UnsupportedOperationException

system.log:
java.lang.UnsupportedOperationException: null
at 
org.apache.cassandra.cql3.restrictions.SingleColumnRestriction$ContainsRestriction.appendTo(SingleColumnRestriction.java:456)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.restrictions.PartitionKeySingleRestrictionSet.values(PartitionKeySingleRestrictionSet.java:86)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.restrictions.StatementRestrictions.getPartitionKeys(StatementRestrictions.java:585)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.statements.SelectStatement.getSliceCommands(SelectStatement.java:474)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.statements.SelectStatement.getQuery(SelectStatement.java:262)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:227)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:76)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:188)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:219) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:204) 
~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115)
 ~[apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:513)
 [apache-cassandra-3.9.jar:3.9]
at 
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:407)
 [apache-cassandra-3.9.jar:3.9]
at 
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
 [netty-all-4.0.39.Final.jar:4.0.39.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:366)
 [netty-all-4.0.39.Final.jar:4.0.39.Final]
at 
io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
 [netty-all-4.0.39.Final.jar:4.0.39.Final]
at 
io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:357)
 [netty-all-4.0.39.Final.jar:4.0.39.Final]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[na:1.8.0_121]
at 
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
 [apache-cassandra-3.9.jar:3.9]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) 
[apache-cassandra-3.9.jar:3.9]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_121]




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-13010) nodetool compactionstats should say which disk a compaction is writing to

[Alex Lourie (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884908#comment-15884908
 ] 

Alex Lourie edited comment on CASSANDRA-13010 at 2/27/17 2:21 PM:
--

Thanks [~jjirsa]!

I've started by adding a targetDirectory to the *CompactionInfo* class. Then 
I'm working backwards handling new instantiations with appropriate path 
parameters.

The problem I got at the moment is finding the correct *path* parameter for 
every such new call.  For instance, in 
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/index/internal/CollatedViewIndexBuilder.java#L54;
 I can't find a proper argument to add as a target directory.

Am I going the right direction? I've tried running the test with building ccm 
cluster and causing it to run compactions, but the target dir shown for a 
running compaction is empty, so I probably do something wrong somewhere.

Thanks.


was (Author: alourie):
Thanks [~jjirsa]!

I've started by adding a targetDirectory to the *CompactionInfo* object. Then 
I'm working backwards handling new instantiations with appropriate path 
parameters.

The problem I got at the moment is finding the correct *path* parameter for 
every such new call.  For instance, in 
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/index/internal/CollatedViewIndexBuilder.java#L54;
 I can't find a proper argument to add as a target directory.

Am I going the right direction? I've tried running the test with building ccm 
cluster and causing it to run compactions, but the target dir shown for a 
running compaction is empty, so I probably do something wrong somewhere.

Thanks.

> nodetool compactionstats should say which disk a compaction is writing to
> -
>
> Key: CASSANDRA-13010
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13010
> Project: Cassandra
>  Issue Type: New Feature
>Reporter: Jon Haddad
>  Labels: lhf
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13274) Fix rolling upgrade to 4.0+

[Aleksey Yeschenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885831#comment-15885831
 ] 

Aleksey Yeschenko commented on CASSANDRA-13274:
---

Not sure I understand the issue.

Could you please be more clear regarding what is broken, what the expected 
behaviour is, and what is the behaviour you are observing?

> Fix rolling upgrade to 4.0+
> ---
>
> Key: CASSANDRA-13274
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13274
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Critical
> Fix For: 3.0.x, 3.11.x
>
>
> A rolling upgrade from 3.* to 4.0 (messaging version {{11}}) unveils a 
> regression caused by CASSANDRA-11128.
> Generally, we store all possible options/attributes including the default 
> values in the schema. This causes (expected) schema-version-mismatches during 
> rolling upgrades and therefore we prevent schema pulls/pushes in this 
> situation, which has been broken by CASSANDRA-11128.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (CASSANDRA-13266) Bulk loading sometimes is very slow?

[liangsibin (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885763#comment-15885763
 ] 

liangsibin edited comment on CASSANDRA-13266 at 2/27/17 1:18 PM:
-

I can see a cassandra node hold cpu 100%，when the load speed becomes very slow!


was (Author: bruceliang):
I can see a cassandra node hold cpu 100%

> Bulk loading sometimes is very slow?
> 
>
> Key: CASSANDRA-13266
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13266
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: liangsibin
>
> When I bulkload sstable created with CQLSSTableWriter, it's sometimes very 
> slow.  
> CQLSSTableWriter withBufferSizeInMB 32MB
> use 2 nodes write SSTable and bulkload
> 1、Use CQLSSTableWriter create SSTable （60 threads）
> 2、When the directory over 10 rows，bulkload the directory （20 threads）
> the normal bulkload speed is about 70M/s per node，and bulkload 141G SStables 
> per node cost 90 minutes but sometimes is very slow，the same data cost 4 
> hours why?
> here is the code bulkload sstable
> {code:java}
> public class JmxBulkLoader {
>   
> static final Logger LOGGER = LoggerFactory.getLogger(JmxBulkLoader.class);
>   private JMXConnector connector;
>   private StorageServiceMBean storageBean;
>   private Timer timer = new Timer();
>   public JmxBulkLoader(String host, int port) throws Exception {
>   connect(host, port);
>   }
>   private void connect(String host, int port) throws IOException, 
> MalformedObjectNameException {
>   JMXServiceURL jmxUrl = new JMXServiceURL(
>   
> String.format("service:jmx:rmi:///jndi/rmi://%s:%d/jmxrmi", host, port));
>   Map env = new HashMap();
>   connector = JMXConnectorFactory.connect(jmxUrl, env);
>   MBeanServerConnection mbeanServerConn = 
> connector.getMBeanServerConnection();
>   ObjectName name = new 
> ObjectName("org.apache.cassandra.db:type=StorageService");
>   storageBean = JMX.newMBeanProxy(mbeanServerConn, name, 
> StorageServiceMBean.class);
>   }
>   public void close() throws IOException {
>   connector.close();
>   }
>   public void bulkLoad(String path) {
>   LOGGER.info("begin load data to cassandra " + new 
> Path(path).getName());
>   timer.start();
>   storageBean.bulkLoad(path);
>   timer.end();
>   LOGGER.info("bulk load took " + timer.getTimeTakenMillis() + 
> "ms, path: " + new Path(path).getName());
>   }
> }
> {code}
> bulkload thread
> {code:java} 
> public class BulkThread implements Runnable {
>   private String path;
>   private String jmxHost;
>   private int jmxPort;
>   
>   public BulkThread(String path, String jmxHost, int jmxPort) {
>   super();
>   this.path = path;
>   this.jmxHost = jmxHost;
>   this.jmxPort = jmxPort;
>   }
>   @Override
>   public void run() {
>   JmxBulkLoader bulkLoader = null;
>   try {
>   bulkLoader = new JmxBulkLoader(jmxHost, jmxPort);
>   bulkLoader.bulkLoad(path);
>   } catch (Exception e) {
>   e.printStackTrace();
>   } finally {
>   if (bulkLoader != null)
>   try {
>   bulkLoader.close();
>   bulkLoader = null;
>   } catch (IOException e) {
>   e.printStackTrace();
>   }
>   }
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-10520) Compressed writer and reader should support non-compressed data.

[Branimir Lambov (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885764#comment-15885764
 ] 

Branimir Lambov commented on CASSANDRA-10520:
-

I'm not very comfortable with using different defaults for existing vs new 
tables -- a choice like this looks fragile and can lead to confusion. 
Nevertheless, my personal preference is to not introduce unnecessary risk, and 
my vote would be to keep the option off by default (for both existing and new 
tables).

> Compressed writer and reader should support non-compressed data.
> 
>
> Key: CASSANDRA-10520
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10520
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Local Write-Read Paths
>Reporter: Branimir Lambov
>Assignee: Branimir Lambov
>  Labels: messaging-service-bump-required
> Fix For: 4.x
>
> Attachments: ReadWriteTestCompression.java
>
>
> Compressing uncompressible data, as done, for instance, to write SSTables 
> during stress-tests, results in chunks larger than 64k which are a problem 
> for the buffer pooling mechanisms employed by the 
> {{CompressedRandomAccessReader}}. This results in non-negligible performance 
> issues due to excessive memory allocation.
> To solve this problem and avoid decompression delays in the cases where it 
> does not provide benefits, I think we should allow compressed files to store 
> uncompressed chunks as alternative to compressed data. Such a chunk could be 
> written after compression returns a buffer larger than, for example, 90% of 
> the input, and would not result in additional delays in writing. On reads it 
> could be recognized by size (using a single global threshold constant in the 
> compression metadata) and data could be directly transferred into the 
> decompressed buffer, skipping the decompression step and ensuring a 64k 
> buffer for compressed data always suffices.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13266) Bulk loading sometimes is very slow?

[liangsibin (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885763#comment-15885763
 ] 

liangsibin commented on CASSANDRA-13266:


I can see a cassandra node hold cpu 100%

> Bulk loading sometimes is very slow?
> 
>
> Key: CASSANDRA-13266
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13266
> Project: Cassandra
>  Issue Type: Improvement
>Reporter: liangsibin
>
> When I bulkload sstable created with CQLSSTableWriter, it's sometimes very 
> slow.  
> CQLSSTableWriter withBufferSizeInMB 32MB
> use 2 nodes write SSTable and bulkload
> 1、Use CQLSSTableWriter create SSTable （60 threads）
> 2、When the directory over 10 rows，bulkload the directory （20 threads）
> the normal bulkload speed is about 70M/s per node，and bulkload 141G SStables 
> per node cost 90 minutes but sometimes is very slow，the same data cost 4 
> hours why?
> here is the code bulkload sstable
> {code:java}
> public class JmxBulkLoader {
>   
> static final Logger LOGGER = LoggerFactory.getLogger(JmxBulkLoader.class);
>   private JMXConnector connector;
>   private StorageServiceMBean storageBean;
>   private Timer timer = new Timer();
>   public JmxBulkLoader(String host, int port) throws Exception {
>   connect(host, port);
>   }
>   private void connect(String host, int port) throws IOException, 
> MalformedObjectNameException {
>   JMXServiceURL jmxUrl = new JMXServiceURL(
>   
> String.format("service:jmx:rmi:///jndi/rmi://%s:%d/jmxrmi", host, port));
>   Map env = new HashMap();
>   connector = JMXConnectorFactory.connect(jmxUrl, env);
>   MBeanServerConnection mbeanServerConn = 
> connector.getMBeanServerConnection();
>   ObjectName name = new 
> ObjectName("org.apache.cassandra.db:type=StorageService");
>   storageBean = JMX.newMBeanProxy(mbeanServerConn, name, 
> StorageServiceMBean.class);
>   }
>   public void close() throws IOException {
>   connector.close();
>   }
>   public void bulkLoad(String path) {
>   LOGGER.info("begin load data to cassandra " + new 
> Path(path).getName());
>   timer.start();
>   storageBean.bulkLoad(path);
>   timer.end();
>   LOGGER.info("bulk load took " + timer.getTimeTakenMillis() + 
> "ms, path: " + new Path(path).getName());
>   }
> }
> {code}
> bulkload thread
> {code:java} 
> public class BulkThread implements Runnable {
>   private String path;
>   private String jmxHost;
>   private int jmxPort;
>   
>   public BulkThread(String path, String jmxHost, int jmxPort) {
>   super();
>   this.path = path;
>   this.jmxHost = jmxHost;
>   this.jmxPort = jmxPort;
>   }
>   @Override
>   public void run() {
>   JmxBulkLoader bulkLoader = null;
>   try {
>   bulkLoader = new JmxBulkLoader(jmxHost, jmxPort);
>   bulkLoader.bulkLoad(path);
>   } catch (Exception e) {
>   e.printStackTrace();
>   } finally {
>   if (bulkLoader != null)
>   try {
>   bulkLoader.close();
>   bulkLoader = null;
>   } catch (IOException e) {
>   e.printStackTrace();
>   }
>   }
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13235) All thread blocked and writes pending.

[zhaoyan (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885606#comment-15885606
 ] 

zhaoyan commented on CASSANDRA-13235:
-

I Think it like CASSANDRA-12689

I find many threads are

{code}
"SharedPool-Worker-127" #579 daemon prio=5 os_prio=0 tid=0x00f16800 
nid=0x5a1b waiting for monitor entry [0x7f6a2d1a1000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at sun.misc.Unsafe.monitorEnter(Native Method)
at 
org.apache.cassandra.utils.concurrent.Locks.monitorEnterUnsafe(Locks.java:46)
at 
org.apache.cassandra.db.AtomicBTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:202)
at org.apache.cassandra.db.Memtable.put(Memtable.java:210)
at 
org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1244)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396)
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359)
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
at 
org.apache.cassandra.service.StorageProxy$8.runMayThrow(StorageProxy.java:1086)
at 
org.apache.cassandra.service.StorageProxy$LocalMutationRunnable$1.runMayThrow(StorageProxy.java:2275)
at 
org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:2313)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at 
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
at java.lang.Thread.run(Thread.java:745)

{code}


> All thread blocked and writes pending.
> --
>
> Key: CASSANDRA-13235
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13235
> Project: Cassandra
>  Issue Type: Bug
>  Components: Core
> Environment: jdk8
> cassandra 2.1.15
>Reporter: zhaoyan
>
> I found cassandra many pending  MutationStage task
> {code}
> NFO  [Service Thread] 2017-02-17 16:00:14,440 StatusLogger.java:51 - Pool 
> NameActive   Pending  Completed   Blocked  All Time 
> Blocked
> INFO  [Service Thread] 2017-02-17 16:00:14,440 StatusLogger.java:66 - 
> MutationStage   384  4553 4294213082 0
>  0
> INFO  [Service Thread] 2017-02-17 16:00:14,441 StatusLogger.java:66 - 
> RequestResponseStage  0 0 2172612382 0
>  0
> INFO  [Service Thread] 2017-02-17 16:00:14,441 StatusLogger.java:66 - 
> ReadRepairStage   0 05378852 0
>  0
> INFO  [Service Thread] 2017-02-17 16:00:14,441 StatusLogger.java:66 - 
> CounterMutationStage  0 0  0 0
>  0
> INFO  [Service Thread] 2017-02-17 16:00:14,441 StatusLogger.java:66 - 
> ReadStage 5 0  577242284 0
>  0
> INFO  [Service Thread] 2017-02-17 16:00:14,441 StatusLogger.java:66 - 
> MiscStage 0 0  0 0
>  0
> INFO  [Service Thread] 2017-02-17 16:00:14,441 StatusLogger.java:66 - 
> HintedHandoff 0 0   1480 0
>  0
> INFO  [Service Thread] 2017-02-17 16:00:14,441 StatusLogger.java:66 - 
> GossipStage   0 09342250 0
>  0
> {code}
> And I found there are many blocked thread with jstack
> {code}
> "SharedPool-Worker-28" #416 daemon prio=5 os_prio=0 tid=0x01fb8000 
> nid=0x7459 waiting for monitor entry [0x7fdd83ca]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at sun.misc.Unsafe.monitorEnter(Native Method)
> at 
> org.apache.cassandra.utils.concurrent.Locks.monitorEnterUnsafe(Locks.java:46)
> at 
> org.apache.cassandra.db.AtomicBTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:202)
> at org.apache.cassandra.db.Memtable.put(Memtable.java:210)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1244)
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396)
> at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359)
> at org.apache.cassandra.db.Mutation.apply(Mutation.java:214)
> at 
> org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
> at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at 
> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
> at 

[jira] [Commented] (CASSANDRA-13258) Rethink read-time defragmentation introduced in 1.1 (CASSANDRA-2503)

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885576#comment-15885576
 ] 

Kurt Greaves commented on CASSANDRA-13258:
--

I don't think it's reasonable to expect to hit less than min threshold SSTables 
when using STCS. STCS to me is "this strategy kind of just works but is by no 
means perfect, but that doesn't matter to us". I can definitely see how this 
kind of behaviour would seriously make bad situations worse however, and I 
think the optimisation use case would be pretty rare.

That being said it would be interesting to see some benchmarks associated with 
removing it. If there is a use case where it helps then some kind of 
middle-ground where only hot rows are re-written would be cool and probably 
reduce the impact on compactions.

> Rethink read-time defragmentation introduced in 1.1 (CASSANDRA-2503)
> 
>
> Key: CASSANDRA-13258
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13258
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Nate McCall
>
> tl,dr; we issue a Mutation(!) on a read when using STCS and there are more 
> than minCompactedThreshold SSTables encountered by the iterator. (See 
> org/apache/cassandra/db/SinglePartitionReadCommand.java:782)
> I can see a couple of use cases where this *might* be useful, but from a 
> practical stand point, this is an excellent way to exacerbate compaction 
> falling behind.
> With the introduction of other, purpose built compaction strategies, I would 
> be interested to hear why anyone would consider this still a good idea. Note 
> that we only do it for STCS so at best, we are inconsistent. 
> There are some interesting comments on CASSANDRA-10342 regarding this as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (CASSANDRA-13274) Fix rolling upgrade to 4.0+

[Robert Stupp (JIRA)]

Robert Stupp created CASSANDRA-13274:


 Summary: Fix rolling upgrade to 4.0+
 Key: CASSANDRA-13274
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13274
 Project: Cassandra
  Issue Type: Bug
Reporter: Robert Stupp
Assignee: Robert Stupp
Priority: Critical
 Fix For: 3.0.x, 3.11.x


A rolling upgrade from 3.* to 4.0 (messaging version {{11}}) unveils a 
regression caused by CASSANDRA-11128.

Generally, we store all possible options/attributes including the default 
values in the schema. This causes (expected) schema-version-mismatches during 
rolling upgrades and therefore we prevent schema pulls/pushes in this 
situation, which has been broken by CASSANDRA-11128.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13267) Add new CQL functions

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885531#comment-15885531
 ] 

Sylvain Lebresne commented on CASSANDRA-13267:
--

Not a proper review, but from a quick scanning, I don't think it's a good idea 
for {{toString()}} to rely on the value of the java {{toString()}} method of 
the java object the deserializer return. In particular, this isn't necessary 
(and won't be in a number of case) a valid CQL string representation (and is in 
fact a fairly useless representation in the case of ByteTypes), and so makes 
for a weird CQL function.

Also:
* I find it bothering that we'd start adding more special cases to 
{{Selectable}} like this. If we're having more than one method taking arbitrary 
types, we should devise a slightly more elegant way to support it.
* We'd want an error message like for {{toJson}} in {{FunctionResolver}} for 
user friendliness.
* Some unit tests wouldn't hurt.

> Add new CQL functions
> -
>
> Key: CASSANDRA-13267
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13267
> Project: Cassandra
>  Issue Type: Improvement
>  Components: CQL
>Reporter: vincent royer
>Priority: Trivial
>  Labels: features
> Fix For: 3.0.x
>
> Attachments: 0001-CASSANDRA-13267-Add-CQL-functions.patch
>
>
> Introduce 2 new CQL functions :
> -toString(x) converts a column to its string representation.
> -toJsonArray(x, y, z...) generates a JSON array of JSON string.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13141) test failure in cql_tests.MiscellaneousCQLTester.prepared_statement_invalidation_test

[vishnu prasad (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885528#comment-15885528
 ] 

vishnu prasad commented on CASSANDRA-13141:
---

I am getting error when i try to alter a column from time type to timeuuid as 
follows:
InvalidRequest: Error from server: code=2200 [Invalid query] message="Altering 
of types is not allowed"\

existing structure:
CREATE TABLE products.streaminginfo (
userid text,
transtime time,
appid text,
clientip inet,
gatewayip inet,
geolocation map,
processtime timeuuid,
serverip inet,
sessionid text,
url text,
PRIMARY KEY (userid, transtime)
) ;

Command issued is :
alter table products.streaminginfo ALTER transtime TYPE timeuuid;

> test failure in 
> cql_tests.MiscellaneousCQLTester.prepared_statement_invalidation_test
> -
>
> Key: CASSANDRA-13141
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13141
> Project: Cassandra
>  Issue Type: Bug
>  Components: Testing
>Reporter: Sean McCarthy
>  Labels: dtest, test-failure
> Attachments: node1_debug.log, node1_gc.log, node1.log
>
>
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1471/testReport/cql_tests/MiscellaneousCQLTester/prepared_statement_invalidation_test
> {code}
> Error Message
> Error from server: code=2200 [Invalid query] message="Altering of types is 
> not allowed"
> {code}{code}
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
> testMethod()
>   File "/home/automaton/cassandra-dtest/cql_tests.py", line 633, in 
> prepared_statement_invalidation_test
> session.execute("ALTER TABLE test ALTER d TYPE blob")
>   File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 
> 1998, in execute
> return self.execute_async(query, parameters, trace, custom_payload, 
> timeout, execution_profile, paging_state).result()
>   File "/home/automaton/src/cassandra-driver/cassandra/cluster.py", line 
> 3784, in result
> raise self._final_exception
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-13270) Add function hooks to deliver Elasticsearch as a Cassandra plugin

[Kurt Greaves (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885521#comment-15885521
 ] 

Kurt Greaves commented on CASSANDRA-13270:
--

Just an FYI you'll need to target 4.0 for improvements (that goes for the other 
tickets as well). Great work though, I don't know if it'll all get in for 4.0 
but definitely cool.

> Add function hooks to deliver Elasticsearch as a Cassandra plugin
> -
>
> Key: CASSANDRA-13270
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13270
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: vincent royer
>Priority: Minor
>  Labels: features
> Fix For: 3.0.12, 3.11.0
>
> Attachments: 0001-CASSANDRA-13270-elasticsearch-as-a-plugin.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> With these basic modifications (see the patch) and the following tickets, the 
> Elassandra project (see https://github.com/strapdata/elassandra) could be an 
> Elasticsearch plugin for Cassandra.
> * CASSANDRA-12837 Add multi-threaded support to nodetool rebuild_index.
> * CASSANDRA-13267 Add CQL functions.
> * CASSANDRA-13268 Allow to create custom secondary index on static columns.
> * CASSANDRA-13269 Snapshot support for custom secondary indices



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (CASSANDRA-10520) Compressed writer and reader should support non-compressed data.

[Sylvain Lebresne (JIRA)]

[ 
https://issues.apache.org/jira/browse/CASSANDRA-10520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15885494#comment-15885494
 ] 

Sylvain Lebresne commented on CASSANDRA-10520:
--

bq. should reopen CASSANDRA-11128

We don't re-open issues that have made it in a release, but it's worth opening 
a followup, yes.

bq.  this means that for upgrades from 3.0/3.x to 4.0 users must ensure 
this/11128 is fixed.

Yes, and I wonder if avoiding this isn't a good enough reason to avoid enabling 
this by default, _at least on existing tables_. I mean, in general, I wonder if 
we shouldn't default on being more conservative for existing tables on upgrade. 
That is, what I'd suggest is that we'd default existing table to this being 
disabled (no change from now), but enable it on new table (basically, default 
to false if the table doesn't have the option, but force it to true on new 
tables if not provided).

> Compressed writer and reader should support non-compressed data.
> 
>
> Key: CASSANDRA-10520
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10520
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Local Write-Read Paths
>Reporter: Branimir Lambov
>Assignee: Branimir Lambov
>  Labels: messaging-service-bump-required
> Fix For: 4.x
>
> Attachments: ReadWriteTestCompression.java
>
>
> Compressing uncompressible data, as done, for instance, to write SSTables 
> during stress-tests, results in chunks larger than 64k which are a problem 
> for the buffer pooling mechanisms employed by the 
> {{CompressedRandomAccessReader}}. This results in non-negligible performance 
> issues due to excessive memory allocation.
> To solve this problem and avoid decompression delays in the cases where it 
> does not provide benefits, I think we should allow compressed files to store 
> uncompressed chunks as alternative to compressed data. Such a chunk could be 
> written after compression returns a buffer larger than, for example, 90% of 
> the input, and would not result in additional delays in writing. On reads it 
> could be recognized by size (using a single global threshold constant in the 
> compression metadata) and data could be directly transferred into the 
> decompressed buffer, skipping the decompression step and ensuring a 64k 
> buffer for compressed data always suffices.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (CASSANDRA-13273) Test case : DatabaseDescriptorRefTest failing

[Amitkumar Ghatwal (JIRA)]

Amitkumar Ghatwal created CASSANDRA-13273:
-

 Summary: Test case : DatabaseDescriptorRefTest failing
 Key: CASSANDRA-13273
 URL: https://issues.apache.org/jira/browse/CASSANDRA-13273
 Project: Cassandra
  Issue Type: Test
  Components: Testing
 Environment: amit@p006n03:~/cassandra$ lscpu
Architecture:  ppc64le
Byte Order:Little Endian
CPU(s):160
On-line CPU(s) list:   0-159
Thread(s) per core:8
Core(s) per socket:5
Socket(s): 4
NUMA node(s):  4
Model: 2.1 (pvr 004b 0201)
Model name:POWER8E (raw), altivec supported
CPU max MHz:   3690.
CPU min MHz:   2061.
L1d cache: 64K
L1i cache: 32K
L2 cache:  512K
L3 cache:  8192K
NUMA node0 CPU(s): 0-39
NUMA node1 CPU(s): 40-79
NUMA node16 CPU(s):80-119
NUMA node17 CPU(s):120-159

amit@p006n03:~/cassandra$ cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.1 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.1 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/;
SUPPORT_URL="http://help.ubuntu.com/;
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/;
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

amit@p006n03:~/cassandra$ bin/cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.10-SNAPSHOT | CQL spec 3.4.3 | Native protocol v4]
Use HELP for help.
cqlsh> exit



Reporter: Amitkumar Ghatwal
 Fix For: 3.10


Hi All,

I am getting test case failures for  "DatabaseDescriptorRefTest"


amit@p006n03:~/cassandra$ ant test -Dtest.name=DatabaseDescriptorRefTest
Buildfile: /home/amit/cassandra/build.xml

init:

maven-ant-tasks-localrepo:

maven-ant-tasks-download:

maven-ant-tasks-init:

maven-declare-dependencies:

maven-ant-tasks-retrieve-build:

init-dependencies:
 [echo] Loading dependency paths from file: 
/home/amit/cassandra/build/build-dependencies.xml

init-dependencies:
 [echo] Loading dependency paths from file: 
/home/amit/cassandra/build/build-dependencies-sources.xml
[unzip] Expanding: 
/home/amit/cassandra/build/lib/jars/org.jacoco.agent-0.7.5.201505241946.jar 
into /home/amit/cassandra/build/lib/jars

check-gen-cql3-grammar:

gen-cql3-grammar:

generate-cql-html:

generate-jflex-java:

build-project:
 [echo] apache-cassandra: /home/amit/cassandra/build.xml

createVersionPropFile:
[propertyfile] Updating property file: 
/home/amit/cassandra/src/resources/org/apache/cassandra/config/version.properties
 [copy] Copying 1 file to /home/amit/cassandra/build/classes/main

build:

build-test:

test:
[junit] WARNING: multiple versions of ant detected in path for junit
[junit]  
jar:file:/usr/share/ant/lib/ant.jar!/org/apache/tools/ant/Project.class
[junit]  and 
jar:file:/home/amit/cassandra/build/lib/jars/ant-1.9.4.jar!/org/apache/tools/ant/Project.class
[junit] Testsuite: org.apache.cassandra.config.DatabaseDescriptorRefTest
[junit] Testsuite: org.apache.cassandra.config.DatabaseDescriptorRefTest 
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.006 sec
[junit]
[junit] - Standard Output ---
[junit] ERROR [main] 2017-02-26 09:56:05,169 ?:? - SLF4J: stderr
[junit] -  ---
[junit] - Standard Error -
[junit]
[junit]
[junit] VIOLATION: 
org.apache.cassandra.config.Config$CAPIFlashCommitlogChunkManagerType
[junit] java.lang.Exception
[junit] at 
org.apache.cassandra.config.DatabaseDescriptorRefTest$1.findClass(DatabaseDescriptorRefTest.java:169)
[junit] at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
[junit] at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
[junit] at java.lang.Class.getDeclaredMethods0(Native Method)
[junit] at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
[junit] at java.lang.Class.getDeclaredMethod(Class.java:2128)
[junit] at 
org.apache.cassandra.config.DatabaseDescriptorRefTest.testDatabaseDescriptorRef(DatabaseDescriptorRefTest.java:215)
[junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit] at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit] at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit] at java.lang.reflect.Method.invoke(Method.java:498)
[junit] at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
[junit] at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
[junit] at