Old tombstones not being cleaned up

[Bo Finnerup Madsen]

Hi,

We are running a small 9 node Cassandra v2.1.17 cluster. The cluster
generally runs fine, but we have one table that are causing OOMs because an
enormous amount of tombstones.
Looking at the data in the table (sstable2json), the first of the
tombstones are almost a year old. The table was initially created with a
gc_grace_period of 10 days, but I have now lowered it to 1 hour.
I have run a full repair of the table across all nodes. I have forced
several major compactions of the table by using "nodetool compact", and
also tried to switch from LeveledCompaction to SizeTierCompaction and back.

What could cause cassandra to keep these tombstones?

sstable2json:
{"key": "foo",
 "cells":
[["082f-25ef-4324-bb8a-8cf013c823c1:_","082f-25ef-4324-bb8a-8cf013c823c1:!",1507819135148000,"t",1507819135],

 
["10f3-c05d-4ab9-9b8a-e6ebd8f5818a:_","10f3-c05d-4ab9-9b8a-e6ebd8f5818a:!",1503661731697000,"t",1503661731],

 
["1d7a-ce95-4c74-b67e-f8cdffec4f85:_","1d7a-ce95-4c74-b67e-f8cdffec4f85:!",1509542102909000,"t",1509542102],

 
["1dd3-ae22-4f6e-944a-8cfa147cde68:_","1dd3-ae22-4f6e-944a-8cfa147cde68:!",1512418006838000,"t",1512418006],

 
["22cc-d69c-4596-89e5-3e976c0cb9a8:_","22cc-d69c-4596-89e5-3e976c0cb9a8:!",1497377448737001,"t",1497377448],

 
["2777-4b1a-4267-8efc-c43054e63170:_","2777-4b1a-4267-8efc-c43054e63170:!",1491014691515001,"t",1491014691],

 
["61e8-f48b-4484-96f1-f8b6a3ed8f9f:_","61e8-f48b-4484-96f1-f8b6a3ed8f9f:!",1500820300544000,"t",1500820300],

 
["63da-f165-449b-b65d-2b7869368734:_","63da-f165-449b-b65d-2b7869368734:!",1512806634968000,"t",1512806634],

 
["656f-f8b5-472b-93ed-1a893002f027:_","656f-f8b5-472b-93ed-1a893002f027:!",1514554716141000,"t",1514554716],
...
{"key": "bar",
 "metadata": {"deletionInfo":
{"markedForDeleteAt":1517402198585982,"localDeletionTime":1517402198}},
 "cells":
[["000af8c2-ffe9-4217-9032-61a1cd21781d:_","000af8c2-ffe9-4217-9032-61a1cd21781d:!",1495094965916000,"t",1495094965],

 
["005b96cb-7eb3-4ec3-bfa2-8573e46892f4:_","005b96cb-7eb3-4ec3-bfa2-8573e46892f4:!",1516360186865000,"t",1516360186],

 
["005ec167-aa61-4868-a3ae-a44b00099eb6:_","005ec167-aa61-4868-a3ae-a44b00099eb6:!",1516671840920002,"t",1516671840],


sstablemetadata:
stablemetadata
/data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741-Data.db
SSTable:
/data/cassandra/data/xxx/yyy-9ed502c0734011e6a128fdafd829b1c6/ddp-yyy-ka-2741
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Bloom Filter FP chance: 0.10
Minimum timestamp: 1488976211688000
Maximum timestamp: 1517468644066000
SSTable max local deletion time: 2147483647
Compression ratio: 0.5121956624389545
Estimated droppable tombstones: 18.00161766553587
SSTable Level: 0
Repaired at: 0
ReplayPosition(segmentId=1517168739626, position=22690189)
Estimated tombstone drop times:%n
1488976211: 1
1489906506:  4706
1490174752:  6111
1490449759:  6554
1490735410:  6559
1491016789:  6369
1491347982: 10216
1491680214: 13502
...

desc:
CREATE TABLE xxx.yyy (
ti text,
uuid text,
json_data text,
PRIMARY KEY (ti, uuid)
) WITH CLUSTERING ORDER BY (uuid ASC)
AND bloom_filter_fp_chance = 0.1
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {'sstable_compression':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 3600
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';

jmx props(picture):
[image: image.png]

Re: Nodes show different number of tokens than initially

[kurt greaves]

>
> I don’t know why this is a surprise (maybe because people like to talk
> about multiple rings, but the fact that replication strategy is set per
> keyspace and that you could use SimpleStrategy in a multiple dc cluster
> demonstrates this), but we can chat about that another time

This is actually a point of confusion for a lot of new users. It seems
obvious for people who know the internals or who have been around since
pre-NTS/vnodes, but it's really not. Especially because NTS makes it seem
like there are two separate rings.

> that's a manual thing you did, Cassandra didn't do it for you, right? The
> fact that Cassandra didn't STOP you from doing it could be considered a
> bug, but YOU made that config choice?

This should be fairly easy to reproduce, however Kurt mentioned that there
> supposed to be some sort of protection against that. I'll try again
> tomorrow.

Sorry, the behaviour was expected. I was under the impression that you
couldn't 'steal' a token from another node (thought C* stopped you), and I
misread the code. It actually gives the token up to the new node - not the
other way round. I haven't thought about it long enough to really consider
what the behaviour should be, or whether the current behaviour is right or
wrong though.
​

Re: Security Updates

[kurt greaves]

Regarding security releases, nothing currently exists to notify users when
security related patches are released. At the moment I imagine
announcements would only be made in NEWS.txt or on the user mailing list...
but only if you're lucky.

On 31 January 2018 at 19:18, Michael Shuler  wrote:

> I should also mention the dev@ mailing list - this is where the [VOTE]
> emails are sent and you'd get an advanced heads up on upcoming releases,
> along with the release emails that are sent to both user@ and dev@. The
> dev@ traffic is generally lower than user@, so pretty easy to spot votes
> & releases.
>
> --
> Michael
>
> On 01/31/2018 01:12 PM, Michael Shuler wrote:
> > I usually install cron-apt for Ubuntu & Debian, forward and read root's
> > email to be notified of all system upgrades, including Cassandra.
> >
> > There are likely other utilities for other operating systems, or just a
> > cron script that checks for system update & emails would work, too.
> >
> > Also, it's possible to use something like urlwatch to look for changes
> > on http://cassandra.apache.org/download/ or any site, email out a
> > notification, etc. Maybe http://fetchrss.com/ or similar would work?
> >
> > I think there are a multitude of immediate ways to do this, until there
> > is a site patch submitted to JIRA for RSS addition.
> >
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: Upgrading sstables not using all available compaction slots on version 2.2

[kurt greaves]

Would you be able to create a JIRA ticket for this? Not sure if this is
still a problem in 3.0+ but worth creating a ticket to investigate. It'd be
really helpful if you could try and reproduce on 3.0.15 or 3.11.1 to see if
it's an issue there as well.​

Re: group by select queries

[kurt greaves]

Seems problematic. Would you be able to create a JIRA ticket with the above
information/examples?

On 30 January 2018 at 22:41, Modha, Digant 
wrote:

> It was local quorum.  There’s no difference with CONSISTENCY ALL.
>
>
>
> Consistency level set to LOCAL_QUORUM.
>
> cassandra@cqlsh> select * from wp.position  where account_id = 'user_1';
>
>
>
> account_id | security_id | counter | avg_exec_price | pending_quantity |
> quantity | transaction_id | update_time
>
> +-+-++--
> +--++-
>
>  user_1 |AMZN |   2 | 1239.2 |0
> | 1011 |   null | 2018-01-25 17:18:07.158000+
>
>  user_1 |AMZN |   1 | 1239.2 |0
> | 1010 |   null | 2018-01-25 17:18:07.158000+
>
>
>
> (2 rows)
>
> cassandra@cqlsh> select * from wp.position  where account_id = 'user_1'
> group by security_id;
>
>
>
> account_id | security_id | counter | avg_exec_price | pending_quantity |
> quantity | transaction_id | update_time
>
> +-+-++--
> +--++-
>
>  user_1 |AMZN |   1 | 1239.2 |0
> | 1010 |   null | 2018-01-25 17:18:07.158000+
>
>
>
> (1 rows)
>
> cassandra@cqlsh> select account_id,security_id, counter,
> avg_exec_price,quantity, update_time from wp.position  where account_id =
> 'user_1' group by security_id ;
>
>
>
> account_id | security_id | counter | avg_exec_price | quantity |
> update_time
>
> +-+-++--
> +-
>
>  user_1 |AMZN |   2 | 1239.2 | 1011 |
> 2018-01-25 17:18:07.158000+
>
>
>
> (1 rows)
>
> cassandra@cqlsh>  consistency all;
>
> Consistency level set to ALL.
>
> cassandra@cqlsh> select * from wp.position  where account_id = 'user_1'
> group by security_id;
>
>
>
> account_id | security_id | counter | avg_exec_price | pending_quantity |
> quantity | transaction_id | update_time
>
> +-+-++--
> +--++-
>
>  user_1 |AMZN |   1 | 1239.2 |0
> | 1010 |   null | 2018-01-25 17:18:07.158000+
>
>
>
> (1 rows)
>
> cassandra@cqlsh> select account_id,security_id, counter,
> avg_exec_price,quantity, update_time from wp.position  where account_id =
> 'user_1' group by security_id ;
>
>
>
> account_id | security_id | counter | avg_exec_price | quantity |
> update_time
>
> +-+-++--
> +-
>
>  user_1 |AMZN |   2 | 1239.2 | 1011 |
> 2018-01-25 17:18:07.158000+
>
>
>
>
>
> *From:* kurt greaves [mailto:k...@instaclustr.com]
> *Sent:* Monday, January 29, 2018 11:03 PM
> *To:* User
> *Subject:* Re: group by select queries
>
>
>
> What consistency were you querying at? Can you retry with CONSISTENCY ALL?
>
>
>
> ​
>
>
> TD Securities disclaims any liability or losses either direct or
> consequential caused by the use of this information. This communication is
> for informational purposes only and is not intended as an offer or
> solicitation for the purchase or sale of any financial instrument or as an
> official confirmation of any transaction. TD Securities is neither making
> any investment recommendation nor providing any professional or advisory
> services relating to the activities described herein. All market prices,
> data and other information are not warranted as to completeness or accuracy
> and are subject to change without notice Any products described herein are
> (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or
> guaranteed by, an insured depository institution and (iii) subject to
> investment risks, including possible loss of the principal amount invested.
> The information shall not be further distributed or duplicated in whole or
> in part by any means without the prior written consent of TD Securities. TD
> Securities is a trademark of The Toronto-Dominion Bank and represents TD
> Securities (USA) LLC and certain investment banking activities of The
> Toronto-Dominion Bank and its subsidiaries.
>

Re: Not what I‘ve expected Performance

[kurt greaves]

How are you copying? With CQLSH COPY or your own script? If you've got
spark already it's quite simple to copy between tables and it should be
pretty much as fast as you can get it. (you may even need to throttle).
There's some sample code here (albeit it's copying between clusters but
easily tailored to copy between tables).
https://www.instaclustr.com/support/documentation/apache-spark/using-spark-to-sample-data-from-one-cassandra-cluster-and-write-to-another/

On 30 January 2018 at 21:05, Jürgen Albersdorfer 
wrote:

> Hi, We are using C* 3.11.1 with a 9 Node Cluster built on CentOS Servers
> eac=
> h having 2x Quad Core Xeon, 128GB of RAM and two separate 2TB spinning
> Disks=
> , one for Log one for Data with Spark on Top.
>
> Due to bad Schema (Partitions of about 4 to 8 GB) I need to copy a whole
> Tab=
> le into another having same fields but different partitioning.=20
>
> I expected glowing Iron when I started the copy Job, but instead cannot
> even=
> See some Impact on CPU, mem or disks. - but the Job does copy the Data
> over=
> veeerry slowly at about a MB or two per Minute.
>
> Any suggestion where to start investigation?
>
> Thanks already
>
> Von meinem iPhone gesendet
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: TWCS not deleting expired sstables

[kurt greaves]

Well, that shouldn't happen. Seems like it's possibly not looking in the
correct location for data directories. Try setting CASSANDRA_INCLUDE= prior to running the script?
e.g: CASSANDRA_INCLUDE=/cassandra.in.sh
sstableexpiredblockers ae raw_logs_by_user

On 30 January 2018 at 15:34, Thakrar, Jayesh 
wrote:

> Thanks Kurt and Kenneth.
>
>
>
> Now only if they would work as expected.
>
>
>
> *node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>ls
> -lt | tail *
>
> -rw-r--r--. 1 vchadoop vchadoop286889260 Sep 18 14:14
> mc-1070-big-Index.db
>
> -rw-r--r--. 1 vchadoop vchadoop12236 Sep 13 20:53
> mc-178-big-Statistics.db
>
> -rw-r--r--. 1 vchadoop vchadoop   92 Sep 13 20:53
> mc-178-big-TOC.txt
>
> -rw-r--r--. 1 vchadoop vchadoop  9371211 Sep 13 20:53
> mc-178-big-CompressionInfo.db
>
> -rw-r--r--. 1 vchadoop vchadoop   10 Sep 13 20:53
> mc-178-big-Digest.crc32
>
> -rw-r--r--. 1 vchadoop vchadoop  13609890747 <(360)%20989-0747> Sep 13
> 20:53 mc-178-big-Data.db
>
> -rw-r--r--. 1 vchadoop vchadoop  1394968 Sep 13 20:53
> mc-178-big-Summary.db
>
> -rw-r--r--. 1 vchadoop vchadoop 11172592 Sep 13 20:53
> mc-178-big-Filter.db
>
> -rw-r--r--. 1 vchadoop vchadoop190508739 Sep 13 20:53
> mc-178-big-Index.db
>
> drwxr-xr-x. 2 vchadoop vchadoop   10 Sep 12 21:47 backups
>
>
>
> *node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>sstableexpiredblockers
> ae raw_logs_by_user*
>
> Exception in thread "main" java.lang.IllegalArgumentException: Unknown
> keyspace/table ae.raw_logs_by_user
>
> at org.apache.cassandra.tools.SSTableExpiredBlockers.main(
> SSTableExpiredBlockers.java:66)
>
>
>
> *node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>sstableexpiredblockers
> system peers*
>
> No sstables for system.peers
>
>
>
> *node111.ord.ae.tsg.cnvr.net:/ae/disk1/data/ae/raw_logs_by_user-f58b9960980311e79ac26928246f09c1>ls
> -l ../../system/peers-37f71aca7dc2383ba70672528af04d4f/*
>
> total 308
>
> drwxr-xr-x. 2 vchadoop vchadoop 10 Sep 11 22:59 backups
>
> -rw-rw-r--. 1 vchadoop vchadoop 83 Jan 25 02:11
> mc-137-big-CompressionInfo.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 180369 Jan 25 02:11 mc-137-big-Data.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 10 Jan 25 02:11 mc-137-big-Digest.crc32
>
> -rw-rw-r--. 1 vchadoop vchadoop 64 Jan 25 02:11 mc-137-big-Filter.db
>
> -rw-rw-r--. 1 vchadoop vchadoop386 Jan 25 02:11 mc-137-big-Index.db
>
> -rw-rw-r--. 1 vchadoop vchadoop   5171 Jan 25 02:11
> mc-137-big-Statistics.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 56 Jan 25 02:11 mc-137-big-Summary.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 92 Jan 25 02:11 mc-137-big-TOC.txt
>
> -rw-rw-r--. 1 vchadoop vchadoop 43 Jan 29 21:11
> mc-138-big-CompressionInfo.db
>
> -rw-rw-r--. 1 vchadoop vchadoop   9723 Jan 29 21:11 mc-138-big-Data.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 10 Jan 29 21:11 mc-138-big-Digest.crc32
>
> -rw-rw-r--. 1 vchadoop vchadoop 16 Jan 29 21:11 mc-138-big-Filter.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 17 Jan 29 21:11 mc-138-big-Index.db
>
> -rw-rw-r--. 1 vchadoop vchadoop   5015 Jan 29 21:11
> mc-138-big-Statistics.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 56 Jan 29 21:11 mc-138-big-Summary.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 92 Jan 29 21:11 mc-138-big-TOC.txt
>
> -rw-rw-r--. 1 vchadoop vchadoop 43 Jan 29 21:53
> mc-139-big-CompressionInfo.db
>
> -rw-rw-r--. 1 vchadoop vchadoop  18908 Jan 29 21:53 mc-139-big-Data.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 10 Jan 29 21:53 mc-139-big-Digest.crc32
>
> -rw-rw-r--. 1 vchadoop vchadoop 16 Jan 29 21:53 mc-139-big-Filter.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 36 Jan 29 21:53 mc-139-big-Index.db
>
> -rw-rw-r--. 1 vchadoop vchadoop   5055 Jan 29 21:53
> mc-139-big-Statistics.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 56 Jan 29 21:53 mc-139-big-Summary.db
>
> -rw-rw-r--. 1 vchadoop vchadoop 92 Jan 29 21:53 mc-139-big-TOC.txt
>
>
>
>
>
>
>
> *From: *Kenneth Brotman 
> *Date: *Tuesday, January 30, 2018 at 7:37 AM
> *To: *
> *Subject: *RE: TWCS not deleting expired sstables
>
>
>
> Wow!  It’s in the DataStax documentation: https://docs.datastax.com/en/
> dse/5.1/dse-admin/datastax_enterprise/tools/toolsSStables/
> toolsSStabExpiredBlockers.html
>
>
>
> Other nice tools there as well: https://docs.datastax.com/en/
> dse/5.1/dse-admin/datastax_enterprise/tools/toolsSStables/
> toolsSSTableUtilitiesTOC.html
>
>
>
> Kenneth Brotman
>
>
>
> *From:* kurt greaves [mailto:k...@instaclustr.com]
> *Sent:* Monday, January 29, 2018 8:20 PM
> *To:* User
> *Subject:* Re: TWCS not deleting expired sstables
>
>
>
> Likely a read repair caused old data to be brought into a newer SSTable.
> Try running sstableexpiredblockers to find out if there's a newer SSTable
> 

Re: Nodes show different number of tokens than initially

[Jeff Jirsa]

On Wed, Jan 31, 2018 at 12:08 PM, Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

> On 31 Jan 2018 17:18, "Jeff Jirsa"  wrote:
>
>
> I don’t know why this is a surprise (maybe because people like to talk
> about multiple rings, but the fact that replication strategy is set per
> keyspace and that you could use SimpleStrategy in a multiple dc cluster
> demonstrates this), but we can chat about that another time
>
>
> The reason I find it surprising, is that it makes very little *sense* to
> put a token belonging to a mode from one DC between tokens of nodes from
> another one.
>
>
I don't want to really turn this into an argument over what should and
shouldn't make sense, but I do agree, it doesn't make sense to put a token
on one node in one DC onto another node in another DC. But also being very
clear (I want to make sure I understand what you're saying): that's a
manual thing you did, Cassandra didn't do it for you, right? The fact that
Cassandra didn't STOP you from doing it could be considered a bug, but YOU
made that config choice?


> Having token ranges like that, with ends from nodes in different DCs,
> doesn't convey any *meaning* and have no correspondence to what is being
> modelled here. It also makes it nearly impossible to reason about range
> ownership (unless you're a machine, in which case you probably don't care).
>
> I understand that it works in the end, but it doesn't help to know that.
> It is an implementation detail sticking outside the code guts and it sure
> *is* surprising in all its ugliness. It also opens up the possibility of
> problems just like the one which have started this discussion.
>
> I don't find the argument of using SimpleStrategy for multi-DC
> particularly interesting, lest can I predict what to be expected from such
> an attempt.
>

You can trivially predict what would happen with SimpleStrategy in
multi-DC: run nodetool ring, and the first RF nodes listed after a given
token own that data, regardless of which DC they're in. Because it's all
one big ring.

Re: Cleanup blocking snapshots - Options?

[kurt greaves]

Thanks Thomas. I'll give it a shot myself and see if backporting the patch
fixes the problem. If it does I'll create a new ticket for backporting.

On 30 January 2018 at 09:22, Steinmaurer, Thomas <
thomas.steinmau...@dynatrace.com> wrote:

> Hi Kurt,
>
>
>
> had another try now, and yes, with 2.1.18, this constantly happens.
> Currently running nodetool cleanup on a single node in production with
> disabled hourly snapshots. SSTables with > 100G involved here. Triggering
> nodetool snapshot will result in being blocked. From an operational
> perspective, a bit annoying right now 
>
>
>
> Have asked on https://issues.apache.org/jira/browse/CASSANDRA-13873
> regarding a backport to 2.1, but possibly won’t get attention, cause the
> ticket has been resolved for 2.2+ already.
>
>
>
> Regards,
>
> Thomas
>
>
>
> *From:* kurt greaves [mailto:k...@instaclustr.com]
> *Sent:* Montag, 15. Jänner 2018 06:18
> *To:* User 
> *Subject:* Re: Cleanup blocking snapshots - Options?
>
>
>
> Disabling the snapshots is the best and only real option other than
> upgrading at the moment. Although apparently it was thought that there was
> only a small race condition in 2.1 that triggered this and it wasn't worth
> fixing. If you are triggering it easily maybe it is worth fixing in 2.1 as
> well. Does this happen consistently? Can you provide some more logs on the
> JIRA or better yet a way to reproduce?
>
>
>
> On 14 January 2018 at 16:12, Steinmaurer, Thomas <
> thomas.steinmau...@dynatrace.com> wrote:
>
> Hello,
>
>
>
> we are running 2.1.18 with vnodes in production and due to (
> https://issues.apache.org/jira/browse/CASSANDRA-11155) we can’t run
> cleanup e.g. after extending the cluster without blocking our hourly
> snapshots.
>
>
>
> What options do we have to get rid of partitions a node does not own
> anymore?
>
> · Using a version which has this issue fixed, although upgrading
> to 2.2+, due to various issues, is not an option at the moment
>
> · Temporarily disabling the hourly cron job before starting
> cleanup and re-enable after cleanup has finished
>
> · Any other way to re-write SSTables with data a node owns after
> a cluster scale out
>
>
>
> Thanks,
>
> Thomas
>
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freist
> 
> ädterstra
> 
> ße 313
> 
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> 
>

RE: [announce] Release of Cassandra Prometheus metrics exporter

[ZAIDI, ASAD A]

Hi Romain Gérard - Thanks for sharing Cassandra-exporter.

Since you're   already monitoring tons of Cassandra instances nodes - can you 
please let us know if you can  share Prometheus dashboards/json code along with 
Cassandra-exporter.

Thanks/Asad



-Original Message-
From: Romain Gerard [mailto:romain.ger...@erebe.eu] 
Sent: Wednesday, January 10, 2018 9:06 AM
To: user@cassandra.apache.org
Subject: [announce] Release of Cassandra Prometheus metrics exporter

Hello C*,

A little mail to announce that we released today our internal tool at Criteo to 
monitor Cassandra nodes with Prometheus[1].
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_criteo_cassandra-5Fexporter=DwIDaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=jf3zVfMnJpPJbsjnF3FyFzYuEN0ayq393EwVCYgeEto=OM9PZ3fFgHrar-8ASL4x69WKoY1dSULUUyNu9foTe10=
 

The application is production ready as we use it internally to monitor our > 
100 Cassandra nodes.

I hope it can be useful to you too !
Feel free to send feedbacks/contributions/questions.

[1] 
https://urldefense.proofpoint.com/v2/url?u=https-3A__prometheus.io_=DwIDaQ=LFYZ-o9_HUMeMTSQicvjIg=FsmDztdsVuIKml8IDhdHdg=jf3zVfMnJpPJbsjnF3FyFzYuEN0ayq393EwVCYgeEto=LRtrxtND7d8TLlmTdy-eYyQTOUEHZvnpHguyQ_SRJO4=
 

Regards,
Romain Gérard

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

RE: Slender Cassandra Cluster Project

[Kenneth Brotman]

Thank you Yuri and Michael for the suggestion.  Yes, a Terraform version makes 
sense.  Will do.

Kenneth Brotman

-Original Message-
From: Yuri Subach [mailto:ysub...@gmail.com] 
Sent: Wednesday, January 31, 2018 7:20 AM
To: user@cassandra.apache.org
Subject: Re: Slender Cassandra Cluster Project

Yes, I'd prefer Terraform too.

On 2018-01-31 06:32:21, Michael Mior  wrote:
> While whatever format this comes out in would be helpful, you might 
> want to consider Terraform. 1Password recently published a blog post 
> on their experience with Terraform vs. CloudFormation.
> 
> https://blog.agilebits.com/2018/01/25/terraforming-1password/
> 
> --
> Michael Mior
> mm...@apache.org
> 
> 2018-01-31 2:34 GMT-05:00 Kenneth Brotman :
> 
> > Hi Yuri,
> >
> > If possible I will do everything with AWS Cloudformation.  I'm 
> > working on it now.  Nothing published yet.
> >
> > Kenneth Brotman
> >
> > -Original Message-
> > From: Yuri Subach [mailto:ysub...@gmail.com]
> > Sent: Tuesday, January 30, 2018 7:02 PM
> > To: user@cassandra.apache.org
> > Subject: RE: Slender Cassandra Cluster Project
> >
> > Hi Kenneth,
> >
> > I like this project idea!
> >
> > A couple of questions:
> > - What tools are you going to use for AWS cluster setup?
> > - Do you have anything published already (github)?
> >
> > On 2018-01-22 22:42:11, Kenneth Brotman 
> > 
> > wrote:
> > > Thanks Anthony!  I’ve made a note to include that information in 
> > > the
> > documentation. You’re right.  It won’t work as intended unless that 
> > is configured properly.
> > >
> > >
> > >
> > > I’m also favoring a couple other guidelines for Slender Cassandra:
> > >
> > > 1.   SSD’s only, no spinning disks
> > >
> > > 2.   At least two cores per node
> > >
> > >
> > >
> > > For AWS, I’m favoring the c3.large on Linux.  It’s available in 
> > > these
> > regions: US-East, US-West and US-West2.  The specifications are listed as:
> > >
> > > · Two (2) vCPU’s
> > >
> > > · 3.7 Gib Memory
> > >
> > > · Two (2) 16 GB SSD’s
> > >
> > > · Moderate I/O
> > >
> > >
> > >
> > > It’s going to be hard to beat the inexpensive cost of operating a
> > Slender Cluster on demand in the cloud – and it fits a lot of the 
> > use cases
> > well:
> > >
> > >
> > >
> > > · For under a $100 a month, in current pricing for EC2
> > instances, you can operate an eighteen (18) node Slender Cluster for 
> > five
> > (5) hours a day, ten (10) days a month.  That’s fine for 
> > demonstrations, teaching or experiments that last half a day or less.
> > >
> > > · For under $20, you can have that Slender Cluster up all day
> > long, up to ten (10) hours, for whatever demonstrations or 
> > experiments you want it for.
> > >
> > >
> > >
> > > As always, feedback is encouraged.
> > >
> > >
> > >
> > > Thanks,
> > >
> > >
> > >
> > > Kenneth Brotman
> > >
> > >
> > >
> > > From: Anthony Grasso [mailto:anthony.gra...@gmail.com]
> > > Sent: Sunday, January 21, 2018 3:57 PM
> > > To: user
> > > Subject: Re: Slender Cassandra Cluster Project
> > >
> > >
> > >
> > > Hi Kenneth,
> > >
> > >
> > >
> > > Fantastic idea!
> > >
> > >
> > >
> > > One thing that came to mind from my reading of the proposed setup 
> > > was
> > rack awareness of each node. Given that the proposed setup contains 
> > three DCs, I assume that each node will be made rack aware? If not, 
> > consider defining three racks for each DC and placing two nodes in 
> > each rack. This will ensure that all the nodes in a single rack 
> > contain at most one replica of the data.
> > >
> > >
> > >
> > > Regards,
> > >
> > > Anthony
> > >
> > >
> > >
> > > On 17 January 2018 at 11:24, Kenneth Brotman
> >  wrote:
> > >
> > > Sure.  That takes the project from awesome to 10X awesome.  I 
> > > absolutely
> > would be willing to do that.  Thanks Kurt!
> > >
> > >
> > >
> > > Regarding your comment on the keyspaces, I agree.  There should be 
> > > a few
> > simple examples one way or the other that can be duplicated and 
> > observed, and then an example to duplicate and play with that has a 
> > nice real world mix, with some keyspaces that replicate over only a 
> > subset of DC’s and some that replicate to all DC’s.
> > >
> > >
> > >
> > > Kenneth Brotman
> > >
> > >
> > >
> > > From: kurt greaves [mailto:k...@instaclustr.com]
> > > Sent: Tuesday, January 16, 2018 1:31 PM
> > > To: User
> > > Subject: Re: Slender Cassandra Cluster Project
> > >
> > >
> > >
> > > Sounds like a great idea. Probably would be valuable to add to the
> > official docs as an example set up if you're willing.
> > >
> > >
> > >
> > > Only thing I'd add is that you should have keyspaces that 
> > > replicate over
> > only a subset of DC's, plus one/some replicated to all DC's
> > >
> > >
> > >
> > > On 17 Jan. 2018 03:26, "Kenneth Brotman" 
> > > 
> > 

Re: Nodes show different number of tokens than initially

[Oleksandr Shulgin]

On 31 Jan 2018 17:18, "Jeff Jirsa"  wrote:


I don’t know why this is a surprise (maybe because people like to talk
about multiple rings, but the fact that replication strategy is set per
keyspace and that you could use SimpleStrategy in a multiple dc cluster
demonstrates this), but we can chat about that another time


The reason I find it surprising, is that it makes very little *sense* to
put a token belonging to a mode from one DC between tokens of nodes from
another one.

Having token ranges like that, with ends from nodes in different DCs,
doesn't convey any *meaning* and have no correspondence to what is being
modelled here. It also makes it nearly impossible to reason about range
ownership (unless you're a machine, in which case you probably don't care).

I understand that it works in the end, but it doesn't help to know that. It
is an implementation detail sticking outside the code guts and it sure *is*
surprising in all its ugliness. It also opens up the possibility of
problems just like the one which have started this discussion.

I don't find the argument of using SimpleStrategy for multi-DC particularly
interesting, lest can I predict what to be expected from such an attempt.

If this is deemed invalid config why does the new node *silently* steals
the existing token, badly affecting the ownership of the rest of the
nodes?  It should just refuse to start!

Philosophically, With multiple DCs, it may start up and not see the other
DC for minutes/hours/days before it realizes there’s a token conflict -
what should it do then?


This was not the case for us - the new mode has seen all of the ring and
could detect that there is a conflict. Still it decided to claim the token
ownership, removing it from a longer-lived mode.

This should be fairly easy to reproduce, however Kurt mentioned that there
supposed to be some sort of protection against that. I'll try again
tomorrow.

If your suggestion to resolve that is to make sure we see the whole ring
before starting up, we end up in a situation where we try not to startup
unless we can see all nodes, and create outages during DC separations.


I don't really see a problem here. A newly started node learns topology
from the seed nodes - it doesn't need to *see* all nodes, just learn that
the *exist* and which tokens are assigned to them. A node which is
restarting doesn't even need to do that, because it doesn't need to
reconsider its token ownership.

Cheers,
--
Alex

Re: Security Updates

[Michael Shuler]

I should also mention the dev@ mailing list - this is where the [VOTE]
emails are sent and you'd get an advanced heads up on upcoming releases,
along with the release emails that are sent to both user@ and dev@. The
dev@ traffic is generally lower than user@, so pretty easy to spot votes
& releases.

-- 
Michael

On 01/31/2018 01:12 PM, Michael Shuler wrote:
> I usually install cron-apt for Ubuntu & Debian, forward and read root's
> email to be notified of all system upgrades, including Cassandra.
> 
> There are likely other utilities for other operating systems, or just a
> cron script that checks for system update & emails would work, too.
> 
> Also, it's possible to use something like urlwatch to look for changes
> on http://cassandra.apache.org/download/ or any site, email out a
> notification, etc. Maybe http://fetchrss.com/ or similar would work?
> 
> I think there are a multitude of immediate ways to do this, until there
> is a site patch submitted to JIRA for RSS addition.
> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Security Updates

[Michael Shuler]

I usually install cron-apt for Ubuntu & Debian, forward and read root's
email to be notified of all system upgrades, including Cassandra.

There are likely other utilities for other operating systems, or just a
cron script that checks for system update & emails would work, too.

Also, it's possible to use something like urlwatch to look for changes
on http://cassandra.apache.org/download/ or any site, email out a
notification, etc. Maybe http://fetchrss.com/ or similar would work?

I think there are a multitude of immediate ways to do this, until there
is a site patch submitted to JIRA for RSS addition.

-- 
Michael

On 01/31/2018 12:22 PM, Rob Oxspring wrote:
> Hi,
> 
> As a user of Cassandra I'd like to be able to get notified when there are 
> security releases so that I can get my installation patched ASAP. For feature 
> and patch releases I'm happy to come and look at the web page or trawl 
> through some mail archives, but I'd like for security releases to come and 
> find me.
> 
> Is there a dedicated mailing list for this?
> Or a dedicated list for just announcements without everyday chatter?
> An RSS / atom feed would work just as well.
> 
> Thanks,
> 
> Rob Oxspring
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 


-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Security Updates

[Rob Oxspring]

Hi,

As a user of Cassandra I'd like to be able to get notified when there are 
security releases so that I can get my installation patched ASAP. For feature 
and patch releases I'm happy to come and look at the web page or trawl through 
some mail archives, but I'd like for security releases to come and find me.

Is there a dedicated mailing list for this?
Or a dedicated list for just announcements without everyday chatter?
An RSS / atom feed would work just as well.

Thanks,

Rob Oxspring

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

RE: confirm subscribe to user@cassandra.apache.org

[joe.schwartz]

Subscribe confirmed. 

Joe

-Original Message-
From: Rob Oxspring [mailto:roxspr...@imapmail.org] 
Sent: Wednesday, January 31, 2018 10:16 AM
To: user@cassandra.apache.org
Subject: Re: confirm subscribe to user@cassandra.apache.org



Thanks,

Rob Oxspring

On Wed, Jan 31, 2018, at 6:15 PM, user-h...@cassandra.apache.org wrote:
> Hi! This is the ezmlm program. I'm managing the 
> user@cassandra.apache.org mailing list.
> 
> To confirm that you would like
> 
>roxspr...@imapmail.org
> 
> added to the user mailing list, please send a short reply to this 
> address:
> 
>user-sc.1517422520.mpjkofkajcpakmflbogi-
> roxspring=imapmail@cassandra.apache.org
> 
> Usually, this happens when you just hit the "reply" button.
> If this does not work, simply copy the address and paste it into the 
> "To:" field of a new message.
> 
> This confirmation serves two purposes. First, it verifies that I am 
> able to get mail through to you. Second, it protects you in case 
> someone forges a subscription request in your name.
> 
> Please note that ALL Apache dev- and user- mailing lists are publicly 
> archived.  Do familiarize yourself with Apache's public archive policy 
> at
> 
> http://www.apache.org/foundation/public-archives.html
> 
> prior to subscribing and posting messages to user@cassandra.apache.org.
> If you're not sure whether or not the policy applies to this mailing 
> list, assume it does unless the list name contains the word "private" in it.
> 
> Some mail programs are broken and cannot handle long addresses. If you 
> cannot reply to this request, instead send a message to 
>  and put the entire address listed 
> above into the "Subject:" line.
> 
> 
> --- Administrative commands for the user list ---
> 
> I can handle administrative requests automatically. Please do not send 
> them to the list address! Instead, send your message to the correct 
> command address:
> 
> To subscribe to the list, send a message to:
>
> 
> To remove your address from the list, send a message to:
>
> 
> Send mail to the following for info and FAQ for this list:
>
>
> 
> Similar addresses exist for the digest list:
>
>
> 
> To get messages 123 through 145 (a maximum of 100 per request), mail:
>
> 
> To get an index with subject and author for messages 123-456 , mail:
>
> 
> They are always returned as sets of 100, max 2000 per request, so 
> you'll actually get 100-499.
> 
> To receive all messages with the same subject as message 12345, send a 
> short message to:
>
> 
> The messages should contain one line or word of text to avoid being 
> treated as sp@m, but I will ignore their content.
> Only the ADDRESS you send to is important.
> 
> You can start a subscription for an alternate address, for example 
> "john@host.domain", just add a hyphen and your address (with '=' 
> instead of '@') after the command word:
> 

Re: confirm subscribe to user@cassandra.apache.org

[Rob Oxspring]

Thanks,

Rob Oxspring

On Wed, Jan 31, 2018, at 6:15 PM, user-h...@cassandra.apache.org wrote:
> Hi! This is the ezmlm program. I'm managing the
> user@cassandra.apache.org mailing list.
> 
> To confirm that you would like
> 
>roxspr...@imapmail.org
> 
> added to the user mailing list, please send
> a short reply to this address:
> 
>user-sc.1517422520.mpjkofkajcpakmflbogi-
> roxspring=imapmail@cassandra.apache.org
> 
> Usually, this happens when you just hit the "reply" button.
> If this does not work, simply copy the address and paste it into
> the "To:" field of a new message.
> 
> This confirmation serves two purposes. First, it verifies that I am able
> to get mail through to you. Second, it protects you in case someone
> forges a subscription request in your name.
> 
> Please note that ALL Apache dev- and user- mailing lists are publicly
> archived.  Do familiarize yourself with Apache's public archive policy at
> 
> http://www.apache.org/foundation/public-archives.html
> 
> prior to subscribing and posting messages to user@cassandra.apache.org.
> If you're not sure whether or not the policy applies to this mailing list,
> assume it does unless the list name contains the word "private" in it.
> 
> Some mail programs are broken and cannot handle long addresses. If you
> cannot reply to this request, instead send a message to
>  and put the
> entire address listed above into the "Subject:" line.
> 
> 
> --- Administrative commands for the user list ---
> 
> I can handle administrative requests automatically. Please
> do not send them to the list address! Instead, send
> your message to the correct command address:
> 
> To subscribe to the list, send a message to:
>
> 
> To remove your address from the list, send a message to:
>
> 
> Send mail to the following for info and FAQ for this list:
>
>
> 
> Similar addresses exist for the digest list:
>
>
> 
> To get messages 123 through 145 (a maximum of 100 per request), mail:
>
> 
> To get an index with subject and author for messages 123-456 , mail:
>
> 
> They are always returned as sets of 100, max 2000 per request,
> so you'll actually get 100-499.
> 
> To receive all messages with the same subject as message 12345,
> send a short message to:
>
> 
> The messages should contain one line or word of text to avoid being
> treated as sp@m, but I will ignore their content.
> Only the ADDRESS you send to is important.
> 
> You can start a subscription for an alternate address,
> for example "john@host.domain", just add a hyphen and your
> address (with '=' instead of '@') after the command word:
> 

Re: Nodes show different number of tokens than initially

[Jeff Jirsa]

> On Jan 31, 2018, at 12:35 AM, Oleksandr Shulgin 
>  wrote:
> 
>> On Tue, Jan 30, 2018 at 5:44 PM, Jeff Jirsa  wrote:
>> All DCs in a cluster use the same token space in the DHT,
> 
> I can't believe my bloody eyes, but this seems to be true...

I don’t know why this is a surprise (maybe because people like to talk about 
multiple rings, but the fact that replication strategy is set per keyspace and 
that you could use SimpleStrategy in a multiple dc cluster demonstrates this), 
but we can chat about that another time 


> 
>> so token conflicts across datacenters are invalid config
>  
> If this is deemed invalid config why does the new node *silently* steals the 
> existing token, badly affecting the ownership of the rest of the nodes?  It 
> should just refuse to start!


Philosophically, With multiple DCs, it may start up and not see the other DC 
for minutes/hours/days before it realizes there’s a token conflict - what 
should it do then? Which node gets stopped? If your suggestion to resolve that 
is to make sure we see the whole ring before starting up, we end up in a 
situation where we try not to startup unless we can see all nodes, and create 
outages during DC separations. 

Distributed systems and occasional availability make these decisions harder.

Please open a jira if you think it’s wrong, but I’m not sure I know what the 
“right” answer is either.

Re: CDC usability and future development

[Josh McKenzie]

>
> CDC provides only the mutation as opposed to the full column value, which
> tends to be of limited use for us. Applications might want to know the full
> column value, without having to issue a read back. We also see value in
> being able to publish the full column value both before and after the
> update. This is especially true when deleting a column since this stream
> may be joined with others, or consumers may require other fields to
> properly process the delete.


Philosophically, my first pass at the feature prioritized minimizing impact
to node performance first and usability second, punting a lot of the
de-duplication and RbW implications of having full column values, or
materializing stuff off-heap for consumption from a user and flagging as
persisted to disk etc, for future work on the feature. I don't personally
have any time to devote to moving the feature forward now but as Jeff
indicates, Jay and Simon are both active in the space and taking up the
torch.


On Tue, Jan 30, 2018 at 8:35 PM, Jeff Jirsa  wrote:

> Here's a deck of some proposed additions, discussed at one of the NGCC
> sessions last fall:
>
> https://github.com/ngcc/ngcc2017/blob/master/CassandraDataIngestion.pdf
>
>
>
> On Tue, Jan 30, 2018 at 5:10 PM, Andrew Prudhomme  wrote:
>
> > Hi all,
> >
> > We are currently designing a system that allows our Cassandra clusters to
> > produce a stream of data updates. Naturally, we have been evaluating if
> CDC
> > can aid in this endeavor. We have found several challenges in using CDC
> for
> > this purpose.
> >
> > CDC provides only the mutation as opposed to the full column value, which
> > tends to be of limited use for us. Applications might want to know the
> full
> > column value, without having to issue a read back. We also see value in
> > being able to publish the full column value both before and after the
> > update. This is especially true when deleting a column since this stream
> > may be joined with others, or consumers may require other fields to
> > properly process the delete.
> >
> > Additionally, there is some difficulty with processing CDC itself such
> as:
> > - Updates not being immediately available (addressed by CASSANDRA-12148)
> > - Each node providing an independent streams of updates that must be
> > unified and deduplicated
> >
> > Our question is, what is the vision for CDC development? The current
> > implementation could work for some use cases, but is a ways from a
> general
> > streaming solution. I understand that the nature of Cassandra makes this
> > quite complicated, but are there any thoughts or desires on the future
> > direction of CDC?
> >
> > Thanks
> >
> >
>

Re: Upgrading sstables not using all available compaction slots on version 2.2

[Oleksandr Shulgin]

On Wed, Jan 24, 2018 at 10:40 AM, Oleksandr Shulgin <
oleksandr.shul...@zalando.de> wrote:

> Hello,
>
> In the process of upgrading our cluster from 2.1 to 2.2 we have triggered
> the SSTable rewriting process like this:
>
> $ nodetool upgradesstables -j 4  # concurrent_compactors=5
>
> Then if we would immediately check the compactionstats, we see that 4
> compactions of type 'Upgrade sstables' are running for the same
> keyspace/table.  Fine.
>
> What is surprising, though, is that when any of the tasks which are
> smaller in size complete, no new tasks are added and we are left waiting
> for a smaller number of long running tasks to complete, e.g:
>
> pending tasks: 4
>  idcompaction type   keyspace
>table   completed  totalunit   progress
>d535e962-00de-11e8-9c2c-ade7b0b922e3   Upgrade sstables   cel_data
>  event_history_unbounded32.87 GB   84.12 GB   bytes 39.08%
>d535e960-00de-11e8-9c2c-ade7b0b922e3   Upgrade sstables   cel_data
>  event_history_unbounded33.15 GB   84.79 GB   bytes 39.09%
>
> Periodically, an ordinary compaction would be listed, but no new Upgrade
> tasks appear.
> Only after all of the "current" tasks are finished, the new 4 tasks are
> scheduled and the process repeats.
>
> We would expect that new tasks would be scheduled as soon as a compaction
> slot is freed up, but this doesn't seem to happen.  Is this by design?  I
> do not see any open issue about this in Jira.
>

FYI: Because of this issue it has taken more than 7 days instead of the
estimated 1.5-2 days, based on the available throughput, to migrate ~50 TiB
of data files spread over 12 nodes.

--
Alex

Re: Slender Cassandra Cluster Project

[Yuri Subach]

Yes, I'd prefer Terraform too.

On 2018-01-31 06:32:21, Michael Mior  wrote:
> While whatever format this comes out in would be helpful, you might want to
> consider Terraform. 1Password recently published a blog post on their
> experience with Terraform vs. CloudFormation.
> 
> https://blog.agilebits.com/2018/01/25/terraforming-1password/
> 
> --
> Michael Mior
> mm...@apache.org
> 
> 2018-01-31 2:34 GMT-05:00 Kenneth Brotman :
> 
> > Hi Yuri,
> >
> > If possible I will do everything with AWS Cloudformation.  I'm working on
> > it now.  Nothing published yet.
> >
> > Kenneth Brotman
> >
> > -Original Message-
> > From: Yuri Subach [mailto:ysub...@gmail.com]
> > Sent: Tuesday, January 30, 2018 7:02 PM
> > To: user@cassandra.apache.org
> > Subject: RE: Slender Cassandra Cluster Project
> >
> > Hi Kenneth,
> >
> > I like this project idea!
> >
> > A couple of questions:
> > - What tools are you going to use for AWS cluster setup?
> > - Do you have anything published already (github)?
> >
> > On 2018-01-22 22:42:11, Kenneth Brotman 
> > wrote:
> > > Thanks Anthony!  I’ve made a note to include that information in the
> > documentation. You’re right.  It won’t work as intended unless that is
> > configured properly.
> > >
> > >
> > >
> > > I’m also favoring a couple other guidelines for Slender Cassandra:
> > >
> > > 1.   SSD’s only, no spinning disks
> > >
> > > 2.   At least two cores per node
> > >
> > >
> > >
> > > For AWS, I’m favoring the c3.large on Linux.  It’s available in these
> > regions: US-East, US-West and US-West2.  The specifications are listed as:
> > >
> > > · Two (2) vCPU’s
> > >
> > > · 3.7 Gib Memory
> > >
> > > · Two (2) 16 GB SSD’s
> > >
> > > · Moderate I/O
> > >
> > >
> > >
> > > It’s going to be hard to beat the inexpensive cost of operating a
> > Slender Cluster on demand in the cloud – and it fits a lot of the use cases
> > well:
> > >
> > >
> > >
> > > · For under a $100 a month, in current pricing for EC2
> > instances, you can operate an eighteen (18) node Slender Cluster for five
> > (5) hours a day, ten (10) days a month.  That’s fine for demonstrations,
> > teaching or experiments that last half a day or less.
> > >
> > > · For under $20, you can have that Slender Cluster up all day
> > long, up to ten (10) hours, for whatever demonstrations or experiments you
> > want it for.
> > >
> > >
> > >
> > > As always, feedback is encouraged.
> > >
> > >
> > >
> > > Thanks,
> > >
> > >
> > >
> > > Kenneth Brotman
> > >
> > >
> > >
> > > From: Anthony Grasso [mailto:anthony.gra...@gmail.com]
> > > Sent: Sunday, January 21, 2018 3:57 PM
> > > To: user
> > > Subject: Re: Slender Cassandra Cluster Project
> > >
> > >
> > >
> > > Hi Kenneth,
> > >
> > >
> > >
> > > Fantastic idea!
> > >
> > >
> > >
> > > One thing that came to mind from my reading of the proposed setup was
> > rack awareness of each node. Given that the proposed setup contains three
> > DCs, I assume that each node will be made rack aware? If not, consider
> > defining three racks for each DC and placing two nodes in each rack. This
> > will ensure that all the nodes in a single rack contain at most one replica
> > of the data.
> > >
> > >
> > >
> > > Regards,
> > >
> > > Anthony
> > >
> > >
> > >
> > > On 17 January 2018 at 11:24, Kenneth Brotman
> >  wrote:
> > >
> > > Sure.  That takes the project from awesome to 10X awesome.  I absolutely
> > would be willing to do that.  Thanks Kurt!
> > >
> > >
> > >
> > > Regarding your comment on the keyspaces, I agree.  There should be a few
> > simple examples one way or the other that can be duplicated and observed,
> > and then an example to duplicate and play with that has a nice real world
> > mix, with some keyspaces that replicate over only a subset of DC’s and some
> > that replicate to all DC’s.
> > >
> > >
> > >
> > > Kenneth Brotman
> > >
> > >
> > >
> > > From: kurt greaves [mailto:k...@instaclustr.com]
> > > Sent: Tuesday, January 16, 2018 1:31 PM
> > > To: User
> > > Subject: Re: Slender Cassandra Cluster Project
> > >
> > >
> > >
> > > Sounds like a great idea. Probably would be valuable to add to the
> > official docs as an example set up if you're willing.
> > >
> > >
> > >
> > > Only thing I'd add is that you should have keyspaces that replicate over
> > only a subset of DC's, plus one/some replicated to all DC's
> > >
> > >
> > >
> > > On 17 Jan. 2018 03:26, "Kenneth Brotman" 
> > wrote:
> > >
> > > I’ve begun working on a reference project intended to provide guidance
> > on configuring and operating a modest Cassandra cluster of about 18 nodes
> > suitable for the economic study, demonstration, experimentation and testing
> > of a Cassandra cluster.
> > >
> > >
> > >
> > > The slender cluster would be designed to be as inexpensive as possible
> > while still 

Re: Slender Cassandra Cluster Project

[Michael Mior]

While whatever format this comes out in would be helpful, you might want to
consider Terraform. 1Password recently published a blog post on their
experience with Terraform vs. CloudFormation.

https://blog.agilebits.com/2018/01/25/terraforming-1password/

--
Michael Mior
mm...@apache.org

2018-01-31 2:34 GMT-05:00 Kenneth Brotman :

> Hi Yuri,
>
> If possible I will do everything with AWS Cloudformation.  I'm working on
> it now.  Nothing published yet.
>
> Kenneth Brotman
>
> -Original Message-
> From: Yuri Subach [mailto:ysub...@gmail.com]
> Sent: Tuesday, January 30, 2018 7:02 PM
> To: user@cassandra.apache.org
> Subject: RE: Slender Cassandra Cluster Project
>
> Hi Kenneth,
>
> I like this project idea!
>
> A couple of questions:
> - What tools are you going to use for AWS cluster setup?
> - Do you have anything published already (github)?
>
> On 2018-01-22 22:42:11, Kenneth Brotman 
> wrote:
> > Thanks Anthony!  I’ve made a note to include that information in the
> documentation. You’re right.  It won’t work as intended unless that is
> configured properly.
> >
> >
> >
> > I’m also favoring a couple other guidelines for Slender Cassandra:
> >
> > 1.   SSD’s only, no spinning disks
> >
> > 2.   At least two cores per node
> >
> >
> >
> > For AWS, I’m favoring the c3.large on Linux.  It’s available in these
> regions: US-East, US-West and US-West2.  The specifications are listed as:
> >
> > · Two (2) vCPU’s
> >
> > · 3.7 Gib Memory
> >
> > · Two (2) 16 GB SSD’s
> >
> > · Moderate I/O
> >
> >
> >
> > It’s going to be hard to beat the inexpensive cost of operating a
> Slender Cluster on demand in the cloud – and it fits a lot of the use cases
> well:
> >
> >
> >
> > · For under a $100 a month, in current pricing for EC2
> instances, you can operate an eighteen (18) node Slender Cluster for five
> (5) hours a day, ten (10) days a month.  That’s fine for demonstrations,
> teaching or experiments that last half a day or less.
> >
> > · For under $20, you can have that Slender Cluster up all day
> long, up to ten (10) hours, for whatever demonstrations or experiments you
> want it for.
> >
> >
> >
> > As always, feedback is encouraged.
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Kenneth Brotman
> >
> >
> >
> > From: Anthony Grasso [mailto:anthony.gra...@gmail.com]
> > Sent: Sunday, January 21, 2018 3:57 PM
> > To: user
> > Subject: Re: Slender Cassandra Cluster Project
> >
> >
> >
> > Hi Kenneth,
> >
> >
> >
> > Fantastic idea!
> >
> >
> >
> > One thing that came to mind from my reading of the proposed setup was
> rack awareness of each node. Given that the proposed setup contains three
> DCs, I assume that each node will be made rack aware? If not, consider
> defining three racks for each DC and placing two nodes in each rack. This
> will ensure that all the nodes in a single rack contain at most one replica
> of the data.
> >
> >
> >
> > Regards,
> >
> > Anthony
> >
> >
> >
> > On 17 January 2018 at 11:24, Kenneth Brotman
>  wrote:
> >
> > Sure.  That takes the project from awesome to 10X awesome.  I absolutely
> would be willing to do that.  Thanks Kurt!
> >
> >
> >
> > Regarding your comment on the keyspaces, I agree.  There should be a few
> simple examples one way or the other that can be duplicated and observed,
> and then an example to duplicate and play with that has a nice real world
> mix, with some keyspaces that replicate over only a subset of DC’s and some
> that replicate to all DC’s.
> >
> >
> >
> > Kenneth Brotman
> >
> >
> >
> > From: kurt greaves [mailto:k...@instaclustr.com]
> > Sent: Tuesday, January 16, 2018 1:31 PM
> > To: User
> > Subject: Re: Slender Cassandra Cluster Project
> >
> >
> >
> > Sounds like a great idea. Probably would be valuable to add to the
> official docs as an example set up if you're willing.
> >
> >
> >
> > Only thing I'd add is that you should have keyspaces that replicate over
> only a subset of DC's, plus one/some replicated to all DC's
> >
> >
> >
> > On 17 Jan. 2018 03:26, "Kenneth Brotman" 
> wrote:
> >
> > I’ve begun working on a reference project intended to provide guidance
> on configuring and operating a modest Cassandra cluster of about 18 nodes
> suitable for the economic study, demonstration, experimentation and testing
> of a Cassandra cluster.
> >
> >
> >
> > The slender cluster would be designed to be as inexpensive as possible
> while still using real world hardware in order to lower the cost to those
> with limited initial resources. Sorry no Raspberry Pi’s for this project.
> >
> >
> >
> > There would be an on-premises version and a cloud version.  Guidance
> would be provided on configuring the cluster, on demonstrating key
> Cassandra behaviors, on files sizes, capacity to use with the Slender
> Cassandra Cluster, and so on.
> >
> >
> >
> > Why about eighteen 

Re: Remote host closed connection during handshake SSL Cassandra 3.0.9

[Jean Carlo]

Hi Marcus

>java version ?
We use oracle-java8-jre the version 8u92.


>TLSv1 disabled ?
No it is not disable. We tried also with the version 1.2




Saludos

Jean Carlo

"The best way to predict the future is to invent it" Alan Kay

On Wed, Jan 31, 2018 at 12:11 PM, Marcus Haarmann  wrote:

> java version ? TLSv1 disabled ? (TLSv1 should not be used any more, since
> it is outdated, but should work internally)
>
> Marcus Haarmann
>
> --
> *Von: *"Jean Carlo" 
> *An: *user@cassandra.apache.org
> *Gesendet: *Mittwoch, 31. Januar 2018 11:53:37
> *Betreff: *Remote host closed connection during handshake SSL Cassandra
> 3.0.9
>
> Hello!
>
> I have a problem enabling  inter-node encryption in cassandra 3.0.9
>
> After I set my conf like that:
>
> server_encryption_options:
> internode_encryption: all
> keystore: /etc/certs/node1.keystore
> keystore_password: cassandra
> truststore: /etc/certs/node1.truststore
> truststore_password: cassandra
> # More advanced defaults below:
> protocol: *TLSv1*
> # algorithm: SunX509
> # store_type: JKS
> cipher_suites: [*TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA*]
> # require_client_auth: false
>
> I got this error all the time
>
> ERROR [ACCEPT-/node1] 2018-01-31 11:29:20,358 MessagingService.java:1081 -
> SSL handshake error for inbound connection from 
> a8265dd[SSL_NULL_WITH_NULL_NULL:
> Socket[addr=/node2,port=40352,localport=7001]]
> javax.net.ssl.SSLHandshakeException: Remote host closed connection during
> handshake
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:992)
> ~[na:1.8.0_92]
> at 
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
> ~[na:1.8.0_92]
> at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
> ~[na:1.8.0_92]
> at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
> ~[na:1.8.0_92]
> at sun.security.ssl.AppInputStream.read(AppInputStream.java:71)
> ~[na:1.8.0_92]
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> ~[na:1.8.0_92]
> at org.apache.cassandra.net.MessagingService$SocketThread.
> run(MessagingService.java:1055) ~[apache-cassandra-3.0.9.jar:3.0.9]
> Caused by: java.io.EOFException: SSL peer shut down incorrectly
> at sun.security.ssl.InputRecord.read(InputRecord.java:505)
> ~[na:1.8.0_92]
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
> ~[na:1.8.0_92]
> ... 6 common frames omitted
>
>
> I think I tested the correctness of my certs using the command openssl
> s_client ( or at least I think I did)
>
> user@node1 /home/user $ openssl s_client -connect node2:7001 -tls1
> CONNECTED(0003)
> ...
> SSL-Session:
> Protocol  : *TLSv1*
> Cipher: *ECDHE-RSA-AES256-SHA*
> ...
>
> So it seems I am using the right configuration but still having the 'SSL
> peer shut down incorrectly' error. Anyone have had this error before?
>
> best greetings
>
> Jean Carlo
>
> "The best way to predict the future is to invent it" Alan Kay
>
>

Re: Remote host closed connection during handshake SSL Cassandra 3.0.9

[Marcus Haarmann]

java version ? TLSv1 disabled ? (TLSv1 should not be used any more, since it is 
outdated, but should work internally) 

Marcus Haarmann 


Von: "Jean Carlo"  
An: user@cassandra.apache.org 
Gesendet: Mittwoch, 31. Januar 2018 11:53:37 
Betreff: Remote host closed connection during handshake SSL Cassandra 3.0.9 

Hello! 

I have a problem enabling inter-node encryption in cassandra 3.0.9 

After I set my conf like that: 

server_encryption_options: 
internode_encryption: all 
keystore: /etc/certs/node1.keystore 
keystore_password: cassandra 
truststore: /etc/certs/node1.truststore 
truststore_password: cassandra 
# More advanced defaults below: 
protocol: TLSv1 
# algorithm: SunX509 
# store_type: JKS 
cipher_suites: [ TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA ] 
# require_client_auth: false 

I got this error all the time 

ERROR [ACCEPT-/node1] 2018-01-31 11:29:20,358 MessagingService.java:1081 - SSL 
handshake error for inbound connection from a8265dd[SSL_NULL_WITH_NULL_NULL: 
Socket[addr=/node2,port=40352,localport=7001]] 
javax.net.ssl.SSLHandshakeException: Remote host closed connection during 
handshake 
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:992) 
~[na:1.8.0_92] 
at 
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) 
~[na:1.8.0_92] 
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928) 
~[na:1.8.0_92] 
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[na:1.8.0_92] 
at sun.security.ssl.AppInputStream.read(AppInputStream.java:71) ~[na:1.8.0_92] 
at java.io.DataInputStream.readInt(DataInputStream.java:387) ~[na:1.8.0_92] 
at 
org.apache.cassandra.net.MessagingService$SocketThread.run(MessagingService.java:1055)
 ~[apache-cassandra-3.0.9.jar:3.0.9] 
Caused by: java.io.EOFException: SSL peer shut down incorrectly 
at sun.security.ssl.InputRecord.read(InputRecord.java:505) ~[na:1.8.0_92] 
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) 
~[na:1.8.0_92] 
... 6 common frames omitted 


I think I tested the correctness of my certs using the command openssl s_client 
( or at least I think I did) 

user@node1 /home/user $ openssl s_client -connect node2:7001 -tls1 
CONNECTED(0003) 
... 
SSL-Session: 
Protocol : TLSv1 
Cipher : ECDHE-RSA-AES256-SHA 
... 

So it seems I am using the right configuration but still having the 'SSL peer 
shut down incorrectly' error. Anyone have had this error before? 

best greetings 

Jean Carlo 

"The best way to predict the future is to invent it" Alan Kay 

Remote host closed connection during handshake SSL Cassandra 3.0.9

[Jean Carlo]

Hello!

I have a problem enabling  inter-node encryption in cassandra 3.0.9

After I set my conf like that:

server_encryption_options:
internode_encryption: all
keystore: /etc/certs/node1.keystore
keystore_password: cassandra
truststore: /etc/certs/node1.truststore
truststore_password: cassandra
# More advanced defaults below:
protocol: *TLSv1*
# algorithm: SunX509
# store_type: JKS
cipher_suites: [*TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA*]
# require_client_auth: false

I got this error all the time

ERROR [ACCEPT-/node1] 2018-01-31 11:29:20,358 MessagingService.java:1081 -
SSL handshake error for inbound connection from
a8265dd[SSL_NULL_WITH_NULL_NULL:
Socket[addr=/node2,port=40352,localport=7001]]
javax.net.ssl.SSLHandshakeException: Remote host closed connection during
handshake
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:992)
~[na:1.8.0_92]
at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
~[na:1.8.0_92]
at
sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:928)
~[na:1.8.0_92]
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
~[na:1.8.0_92]
at sun.security.ssl.AppInputStream.read(AppInputStream.java:71)
~[na:1.8.0_92]
at java.io.DataInputStream.readInt(DataInputStream.java:387)
~[na:1.8.0_92]
at
org.apache.cassandra.net.MessagingService$SocketThread.run(MessagingService.java:1055)
~[apache-cassandra-3.0.9.jar:3.0.9]
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at sun.security.ssl.InputRecord.read(InputRecord.java:505)
~[na:1.8.0_92]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
~[na:1.8.0_92]
... 6 common frames omitted


I think I tested the correctness of my certs using the command openssl
s_client ( or at least I think I did)

user@node1 /home/user $ openssl s_client -connect node2:7001 -tls1
CONNECTED(0003)
...
SSL-Session:
Protocol  : *TLSv1*
Cipher: *ECDHE-RSA-AES256-SHA*
...

So it seems I am using the right configuration but still having the 'SSL
peer shut down incorrectly' error. Anyone have had this error before?

best greetings

Jean Carlo

"The best way to predict the future is to invent it" Alan Kay

Re: Nodes show different number of tokens than initially

[kurt greaves]

So the only reason that the new node would "steal" the token is if it
started up earlier - which is based off how many heartbeats have occurred
since entering NORMAL status on each node. I can't see any reason the new
nodes would have higher generation numbers, so sounds likely there's a bug
somewhere there. I'm not really sure why this comparison would be relevant
unless you were starting multiple nodes at the same time, and based off
your example it seems it definitely shouldn't have happened. Can you create
a JIRA ticket with the above information?​

Re: Nodes show different number of tokens than initially

[Oleksandr Shulgin]

On Wed, Jan 31, 2018 at 5:06 AM, Dikang Gu  wrote:

> What's the partitioner you use? We have logic to prevent duplicate tokens.
>

We are using the default Murmur3Partitioner.  The problem arises from the
fact that we manually allocating the tokens as described earlier.

--
Alex

Re: Nodes show different number of tokens than initially

[Oleksandr Shulgin]

On Tue, Jan 30, 2018 at 5:44 PM, Jeff Jirsa  wrote:

> All DCs in a cluster use the same token space in the DHT,
>

I can't believe my bloody eyes, but this seems to be true...

so token conflicts across datacenters are invalid config
>

If this is deemed invalid config why does the new node *silently* steals
the existing token, badly affecting the ownership of the rest of the
nodes?  It should just refuse to start!

--
Alex