Rebuild / removenode with MV is inconsistent

[benjamin roth]

Hi there,

Today I come up with the following thesis:

A rebuild / removenode may break the base-table <> MV contract.
I'd even claim that a rebuild / removenode requires rebuilding all MVs to
guarantee MV consistency.

Reason:
A node can have base tables with MVs. This is no problem. If these are
streamed during rebuild/removenode, underlying MVs are updated by write
path and consistency contract will be fulfilled.
BUT a node may also contain ranges for MVs whose base table reside on a
different node. When these are streamed from a another node, then for
example base table on node A suddenly has the replica from the base table
of node B and this is not consistent any more.

Re: Non-zero nodes are marked as down after restarting cassandra process

[benjamin roth]

You should always drain nodes before stopping the daemon whenever possible.
This avoids commitlog replay on startup. This can take a while. But
according to your description commit log replay seems not to be the cause.

I once had a similar effect. Some nodes appeared down for some other nodes
and up for others. At that time the cluster had overall stability problems
due to some bugs. After those bugs have gone, I haven't seen this effect
any more.

If that happens again to you, you could check your logs or "nodetool
tpstats" for dropped messages, watch out for suspicious network-related
logs and the load of your nodes in general.

2017-03-01 17:36 GMT+01:00 Ben Dalling :

> Hi Andrew,
>
> We were having problems with gossip TCP connections being held open and
> changed our SOP for stopping cassandra to being:
>
> nodetool disablegossip
> nodetool drain
> service cassandra stop
>
> This seemed to close down the gossip cleanly (the nodetool drain is
> advised as well) and meant that the node rejoined the cluster fine after
> issuing "service cassandra start".
>
> *Ben*
>
> On 1 March 2017 at 16:29, Andrew Jorgensen 
> wrote:
>
>> Helllo,
>>
>> I have a cassandra cluster running on cassandra 3.0.3 and am seeing some
>> strange behavior that I cannot explain when restarting cassandra nodes. The
>> cluster is currently setup in a single datacenter and consists of 55 nodes.
>> I am currently in the process of restarting nodes in the cluster but have
>> noticed that after restarting the cassandra process with `service cassandra
>> start; service cassandra stop` when the node comes back and I run `nodetool
>> status` there is usually a non-zero number of nodes in the rest of the
>> cluster that are marked as DN. If I got to another node in the cluster,
>> from its perspective all nodes included the restarted one are marked as UN.
>> It seems to take ~15 to 20 minutes before the restarted node is updated to
>> show all nodes as UN. During the 15 minutes writes and reads . to the
>> cluster appear to be degraded and do not recover unless I stop the
>> cassandra process again or wait for all nodes to be marked as UN. The
>> cluster also has 3 seed nodes which during this process are up and
>> available the whole time.
>>
>> I have also tried doing `gossipinfo` on the restarted node and according
>> to the output all nodes have a status of NORMAL. Has anyone seen this
>> before and is there anything I can do to fix/reduce the impact of running a
>> restart on a cassandra node?
>>
>> Thanks,
>> Andrew Jorgensen
>> @ajorgensen
>>
>
>

Re: AWS NVMe i3 instances performances

[Romain Hardouin]

Thanks for your feedback Daemeon!I'm a disappointed and I hope that some system 
settings will allow to leverage NVMe :-/What i3 instances did you 
benchmarked?Did you have a "preview access" to i3? Or was it available in a 
specific region before the announcement?
Best,Romain 

Le Mercredi 1 mars 2017 17h44, daemeon reiydelle  a 
écrit :
 

 We did. Found that, even with (CentOS, Ubuntu both for application 
compatibility reasons) that there is somewhat less IO and better CPU throughput 
at the price point. At the time my optimization work for that client ended, 
Amazon was looking at the IO issue, as perhaps the frame configurations needed 
further optimization. this was 2 months ago. A very superficial (no kernel 
tuning) done last month seems to indicate the same tradeoffs. Testing was 
performed in both cases with C* stress tool and with CI test suites. Does this 
help?


...

Daemeon C.M. Reiydelle
USA (+1) 415.501.0198
London (+44) (0) 20 8144 9872

On Wed, Mar 1, 2017 at 3:30 AM, Romain Hardouin  wrote:

Hi all,
AWS launched i3 instances few days ago*. NVMe SSDs seem very promising!
Did someone already benchmark an i3 with Cassandra? e.g. i2 vs i3If yes, with 
which OS and kernel version?Did you make any system tuning for NVMe? e.g. PCIe 
IRQ? etc.
We plan to make some benchmarks but Debian is not listed as a supported OS so 
we have to upgrade our kernel and see if it works :PHere is what we have in 
mind for the time being:* OS: Debian* Kernel: v4.9* IRQ: try several 
configurationsAlso I would like to compare performances between our Debian AMI 
and a standard AWS Linux AMI.
Thanks!
[*] https://aws.amazon.com/fr/ blogs/aws/now-available-i3- 
instances-for-demanding-io- intensive-applications/





   

Re: AWS NVMe i3 instances performances

[daemeon reiydelle]

We did. Found that, even with (CentOS, Ubuntu both for application
compatibility reasons) that there is somewhat less IO and better CPU
throughput at the price point. At the time my optimization work for that
client ended, Amazon was looking at the IO issue, as perhaps the frame
configurations needed further optimization. this was 2 months ago. A very
superficial (no kernel tuning) done last month seems to indicate the same
tradeoffs. Testing was performed in both cases with C* stress tool and with
CI test suites. Does this help?


*...*



*Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872*

On Wed, Mar 1, 2017 at 3:30 AM, Romain Hardouin  wrote:

> Hi all,
>
> AWS launched i3 instances few days ago*. NVMe SSDs seem very promising!
>
> Did someone already benchmark an i3 with Cassandra? e.g. i2 vs i3
> If yes, with which OS and kernel version?
> Did you make any system tuning for NVMe? e.g. PCIe IRQ? etc.
>
> We plan to make some benchmarks but Debian is not listed as a supported OS
> so we have to upgrade our kernel and see if it works :P
> Here is what we have in mind for the time being:
> * OS: Debian
> * Kernel: v4.9
> * IRQ: try several configurations
> Also I would like to compare performances between our Debian AMI and a
> standard AWS Linux AMI.
>
> Thanks!
>
> [*] https://aws.amazon.com/fr/blogs/aws/now-available-i3-
> instances-for-demanding-io-intensive-applications/
>
>
>

Re: Non-zero nodes are marked as down after restarting cassandra process

[Ben Dalling]

Hi Andrew,

We were having problems with gossip TCP connections being held open and
changed our SOP for stopping cassandra to being:

nodetool disablegossip
nodetool drain
service cassandra stop

This seemed to close down the gossip cleanly (the nodetool drain is advised
as well) and meant that the node rejoined the cluster fine after issuing
"service cassandra start".

*Ben*

On 1 March 2017 at 16:29, Andrew Jorgensen 
wrote:

> Helllo,
>
> I have a cassandra cluster running on cassandra 3.0.3 and am seeing some
> strange behavior that I cannot explain when restarting cassandra nodes. The
> cluster is currently setup in a single datacenter and consists of 55 nodes.
> I am currently in the process of restarting nodes in the cluster but have
> noticed that after restarting the cassandra process with `service cassandra
> start; service cassandra stop` when the node comes back and I run `nodetool
> status` there is usually a non-zero number of nodes in the rest of the
> cluster that are marked as DN. If I got to another node in the cluster,
> from its perspective all nodes included the restarted one are marked as UN.
> It seems to take ~15 to 20 minutes before the restarted node is updated to
> show all nodes as UN. During the 15 minutes writes and reads . to the
> cluster appear to be degraded and do not recover unless I stop the
> cassandra process again or wait for all nodes to be marked as UN. The
> cluster also has 3 seed nodes which during this process are up and
> available the whole time.
>
> I have also tried doing `gossipinfo` on the restarted node and according
> to the output all nodes have a status of NORMAL. Has anyone seen this
> before and is there anything I can do to fix/reduce the impact of running a
> restart on a cassandra node?
>
> Thanks,
> Andrew Jorgensen
> @ajorgensen
>

Non-zero nodes are marked as down after restarting cassandra process

[Andrew Jorgensen]

Helllo,

I have a cassandra cluster running on cassandra 3.0.3 and am seeing some
strange behavior that I cannot explain when restarting cassandra nodes. The
cluster is currently setup in a single datacenter and consists of 55 nodes.
I am currently in the process of restarting nodes in the cluster but have
noticed that after restarting the cassandra process with `service cassandra
start; service cassandra stop` when the node comes back and I run `nodetool
status` there is usually a non-zero number of nodes in the rest of the
cluster that are marked as DN. If I got to another node in the cluster,
from its perspective all nodes included the restarted one are marked as UN.
It seems to take ~15 to 20 minutes before the restarted node is updated to
show all nodes as UN. During the 15 minutes writes and reads . to the
cluster appear to be degraded and do not recover unless I stop the
cassandra process again or wait for all nodes to be marked as UN. The
cluster also has 3 seed nodes which during this process are up and
available the whole time.

I have also tried doing `gossipinfo` on the restarted node and according to
the output all nodes have a status of NORMAL. Has anyone seen this before
and is there anything I can do to fix/reduce the impact of running a
restart on a cassandra node?

Thanks,
Andrew Jorgensen
@ajorgensen

Re: Resources for fire drills

[Oskar Kjellin]

Throttle your compaction so low that it practically stops and then try so save 
the nodes to simulate not keeping up with compaction 

Sent from my iPhone

> On 1 Mar 2017, at 14:35, Stefan Podkowinski  wrote:
> 
> I've just created a page for this topic that we can use to collect some
> content:
> https://github.com/spodkowinski/cassandra-collab/blob/docs_firedrill/doc/source/operating/failure_scenarios.rst
> 
> I've invited both of you Malte and Benjamin as collaborators in github,
> so you can either push changes or use the github editor for changes.
> 
> Let me know if that would work for you.
> 
>> On 01.03.2017 13:56, benjamin roth wrote:
>> @Doc:
>> http://cassandra.apache.org/doc/latest/ is built from the git repo. So
>> you can add documentation in doc/source and submit a patch.
>> I personally think that is not the very best place or way to build a
>> knowledge DB but thats what we have.
>> 
>> 
>> 2017-03-01 13:39 GMT+01:00 Malte Pickhan > >:
>> 
>>Hi,
>> 
>>really cool that this discussion gets attention.
>> 
>>You are right my question was quite open.
>> 
>>For me it would already be helpful to compile a list like Ben
>>started with scenarios that can happen to a cluster
>>and what actions/strategies you have to take to resolve the incident
>>without loosing data and having a healthy cluster.
>> 
>>Ideally we would add some kind of rating of hard the scenario is to
>>be resolved so that teams can go through a kind of learning curve.
>> 
>>For the beginning I think it would already be sufficient to document
>>the steps how you can get a cluster into the situation which has
>>been described
>>in the scenario.
>> 
>>Hope it’s a bit clearer now what I mean.
>> 
>>Is there some kind of community space where we could start a
>>document for this purpose?
>> 
>>Best,
>> 
>>Malte
>> 
>>> On 1 Mar 2017, at 13:33, Stefan Podkowinski >> wrote:
>>> 
>>> I've been thinking about this for a while, but haven't found a
>>practical
>>> solution yet, although the term "fire drill" leaves a lot of room for
>>> interpretation. The most basic requirements I'd have for these kind of
>>> trainings would start with automated cluster provisioning for each
>>> scenario (either for teams or individuals) and provisioning of
>>test data
>>> for the cluster, with optionally some kind of load generator
>>constantly
>>> running in the background. I started to work on some Ansible scripts
>>> that would do that on AWS a couple of months ago, but it turned out to
>>> be a lot of work with all the details you have to take care of. So I'd
>>> be happy to hear about any existing resources on that as well!
>>> 
>>> 
 On 01.03.2017 10:59, Malte Pickhan wrote:
 Hi Cassandra users,
 
 I am looking for some resources/guides for firedrill scenarios
>>with apache cassandra.
 
 Do you know anything like that?
 
 Best,
 
 Malte
 
>> 
>> 

Re: Resources for fire drills

[Stefan Podkowinski]

I've just created a page for this topic that we can use to collect some
content:
https://github.com/spodkowinski/cassandra-collab/blob/docs_firedrill/doc/source/operating/failure_scenarios.rst

I've invited both of you Malte and Benjamin as collaborators in github,
so you can either push changes or use the github editor for changes.

Let me know if that would work for you.

On 01.03.2017 13:56, benjamin roth wrote:
> @Doc:
> http://cassandra.apache.org/doc/latest/ is built from the git repo. So
> you can add documentation in doc/source and submit a patch.
> I personally think that is not the very best place or way to build a
> knowledge DB but thats what we have.
> 
> 
> 2017-03-01 13:39 GMT+01:00 Malte Pickhan  >:
> 
> Hi,
> 
> really cool that this discussion gets attention.
> 
> You are right my question was quite open.
> 
> For me it would already be helpful to compile a list like Ben
> started with scenarios that can happen to a cluster
> and what actions/strategies you have to take to resolve the incident
> without loosing data and having a healthy cluster.
> 
> Ideally we would add some kind of rating of hard the scenario is to
> be resolved so that teams can go through a kind of learning curve.
> 
> For the beginning I think it would already be sufficient to document
> the steps how you can get a cluster into the situation which has
> been described
> in the scenario.
> 
> Hope it’s a bit clearer now what I mean.
> 
> Is there some kind of community space where we could start a
> document for this purpose?
> 
> Best,
> 
> Malte
> 
> > On 1 Mar 2017, at 13:33, Stefan Podkowinski  > wrote:
> >
> > I've been thinking about this for a while, but haven't found a
> practical
> > solution yet, although the term "fire drill" leaves a lot of room for
> > interpretation. The most basic requirements I'd have for these kind of
> > trainings would start with automated cluster provisioning for each
> > scenario (either for teams or individuals) and provisioning of
> test data
> > for the cluster, with optionally some kind of load generator
> constantly
> > running in the background. I started to work on some Ansible scripts
> > that would do that on AWS a couple of months ago, but it turned out to
> > be a lot of work with all the details you have to take care of. So I'd
> > be happy to hear about any existing resources on that as well!
> >
> >
> > On 01.03.2017 10:59, Malte Pickhan wrote:
> >> Hi Cassandra users,
> >>
> >> I am looking for some resources/guides for firedrill scenarios
> with apache cassandra.
> >>
> >> Do you know anything like that?
> >>
> >> Best,
> >>
> >> Malte
> >>
> 
> 

Re: Resources for fire drills

[benjamin roth]

@Doc:
http://cassandra.apache.org/doc/latest/ is built from the git repo. So you
can add documentation in doc/source and submit a patch.
I personally think that is not the very best place or way to build a
knowledge DB but thats what we have.


2017-03-01 13:39 GMT+01:00 Malte Pickhan :

> Hi,
>
> really cool that this discussion gets attention.
>
> You are right my question was quite open.
>
> For me it would already be helpful to compile a list like Ben started with
> scenarios that can happen to a cluster
> and what actions/strategies you have to take to resolve the incident
> without loosing data and having a healthy cluster.
>
> Ideally we would add some kind of rating of hard the scenario is to be
> resolved so that teams can go through a kind of learning curve.
>
> For the beginning I think it would already be sufficient to document the
> steps how you can get a cluster into the situation which has been described
> in the scenario.
>
> Hope it’s a bit clearer now what I mean.
>
> Is there some kind of community space where we could start a document for
> this purpose?
>
> Best,
>
> Malte
>
> > On 1 Mar 2017, at 13:33, Stefan Podkowinski  wrote:
> >
> > I've been thinking about this for a while, but haven't found a practical
> > solution yet, although the term "fire drill" leaves a lot of room for
> > interpretation. The most basic requirements I'd have for these kind of
> > trainings would start with automated cluster provisioning for each
> > scenario (either for teams or individuals) and provisioning of test data
> > for the cluster, with optionally some kind of load generator constantly
> > running in the background. I started to work on some Ansible scripts
> > that would do that on AWS a couple of months ago, but it turned out to
> > be a lot of work with all the details you have to take care of. So I'd
> > be happy to hear about any existing resources on that as well!
> >
> >
> > On 01.03.2017 10:59, Malte Pickhan wrote:
> >> Hi Cassandra users,
> >>
> >> I am looking for some resources/guides for firedrill scenarios with
> apache cassandra.
> >>
> >> Do you know anything like that?
> >>
> >> Best,
> >>
> >> Malte
> >>
>
>

Re: Resources for fire drills

[Malte Pickhan]

Hi,

really cool that this discussion gets attention.

You are right my question was quite open.

For me it would already be helpful to compile a list like Ben started with 
scenarios that can happen to a cluster
and what actions/strategies you have to take to resolve the incident without 
loosing data and having a healthy cluster.

Ideally we would add some kind of rating of hard the scenario is to be resolved 
so that teams can go through a kind of learning curve.

For the beginning I think it would already be sufficient to document the steps 
how you can get a cluster into the situation which has been described
in the scenario.

Hope it’s a bit clearer now what I mean.

Is there some kind of community space where we could start a document for this 
purpose?

Best,

Malte

> On 1 Mar 2017, at 13:33, Stefan Podkowinski  wrote:
> 
> I've been thinking about this for a while, but haven't found a practical
> solution yet, although the term "fire drill" leaves a lot of room for
> interpretation. The most basic requirements I'd have for these kind of
> trainings would start with automated cluster provisioning for each
> scenario (either for teams or individuals) and provisioning of test data
> for the cluster, with optionally some kind of load generator constantly
> running in the background. I started to work on some Ansible scripts
> that would do that on AWS a couple of months ago, but it turned out to
> be a lot of work with all the details you have to take care of. So I'd
> be happy to hear about any existing resources on that as well!
> 
> 
> On 01.03.2017 10:59, Malte Pickhan wrote:
>> Hi Cassandra users,
>> 
>> I am looking for some resources/guides for firedrill scenarios with apache 
>> cassandra.
>> 
>> Do you know anything like that?
>> 
>> Best,
>> 
>> Malte
>> 

Re: Resources for fire drills

[Stefan Podkowinski]

I've been thinking about this for a while, but haven't found a practical
solution yet, although the term "fire drill" leaves a lot of room for
interpretation. The most basic requirements I'd have for these kind of
trainings would start with automated cluster provisioning for each
scenario (either for teams or individuals) and provisioning of test data
for the cluster, with optionally some kind of load generator constantly
running in the background. I started to work on some Ansible scripts
that would do that on AWS a couple of months ago, but it turned out to
be a lot of work with all the details you have to take care of. So I'd
be happy to hear about any existing resources on that as well!


On 01.03.2017 10:59, Malte Pickhan wrote:
> Hi Cassandra users,
> 
> I am looking for some resources/guides for firedrill scenarios with apache 
> cassandra.
> 
> Do you know anything like that?
> 
> Best,
> 
> Malte
> 

AWS NVMe i3 instances performances

[Romain Hardouin]

Hi all,
AWS launched i3 instances few days ago*. NVMe SSDs seem very promising!
Did someone already benchmark an i3 with Cassandra? e.g. i2 vs i3If yes, with 
which OS and kernel version?Did you make any system tuning for NVMe? e.g. PCIe 
IRQ? etc.
We plan to make some benchmarks but Debian is not listed as a supported OS so 
we have to upgrade our kernel and see if it works :PHere is what we have in 
mind for the time being:* OS: Debian* Kernel: v4.9* IRQ: try several 
configurationsAlso I would like to compare performances between our Debian AMI 
and a standard AWS Linux AMI.
Thanks!
[*] 
https://aws.amazon.com/fr/blogs/aws/now-available-i3-instances-for-demanding-io-intensive-applications/

Re: Resources for fire drills

[benjamin roth]

But if you want to do fire-drills you only have to break things on purpose.

Examples:
- Cut off a commitlog file at a random position and restart CS
- Overwrite some bytes in an SSTables and read all data from it
- Delete some files in /var/lib/cassandra and try to restore them from
backups or different server
- Shut down more server than your RF settings allow
- Do all this while your system is under load

Btw.: Restoring from backups is NOT trivial and can lead to unwanted
resurrected data.

2017-03-01 11:06 GMT+01:00 Malte Pickhan :

> Yeah thats the point.
>
> What I mean are some overview for basic scenarios for firedrills, so that
> you can exercise them with your team.
>
> Best
>
>
> On 1 Mar 2017, at 11:01, benjamin roth  wrote:
>
> Could you specify it a little bit? There are really a lot of things that
> can go wrong.
>
> 2017-03-01 10:59 GMT+01:00 Malte Pickhan :
>
>> Hi Cassandra users,
>>
>> I am looking for some resources/guides for firedrill scenarios with
>> apache cassandra.
>>
>> Do you know anything like that?
>>
>> Best,
>>
>> Malte
>
>
>
>

Re: Resources for fire drills

[benjamin roth]

As far as I know there is no such resource, at least not officially. IMHO
things like this can be improved a lot within the CS community.

I just proposed on the dev-list to move the official docs out of the repo
into an easier to maintain place like a Wiki or sth.
This could help the community to tackle these issues in a better way -
faster (editing + deployment), easier access, better tools.

2017-03-01 11:06 GMT+01:00 Malte Pickhan :

> Yeah thats the point.
>
> What I mean are some overview for basic scenarios for firedrills, so that
> you can exercise them with your team.
>
> Best
>
>
> On 1 Mar 2017, at 11:01, benjamin roth  wrote:
>
> Could you specify it a little bit? There are really a lot of things that
> can go wrong.
>
> 2017-03-01 10:59 GMT+01:00 Malte Pickhan :
>
>> Hi Cassandra users,
>>
>> I am looking for some resources/guides for firedrill scenarios with
>> apache cassandra.
>>
>> Do you know anything like that?
>>
>> Best,
>>
>> Malte
>
>
>
>

Re: Resources for fire drills

[Malte Pickhan]

Yeah thats the point.

What I mean are some overview for basic scenarios for firedrills, so that you 
can exercise them with your team.

Best


> On 1 Mar 2017, at 11:01, benjamin roth  wrote:
> 
> Could you specify it a little bit? There are really a lot of things that can 
> go wrong.
> 
> 2017-03-01 10:59 GMT+01:00 Malte Pickhan  >:
> Hi Cassandra users,
> 
> I am looking for some resources/guides for firedrill scenarios with apache 
> cassandra.
> 
> Do you know anything like that?
> 
> Best,
> 
> Malte
> 

Re: Resources for fire drills

[benjamin roth]

Could you specify it a little bit? There are really a lot of things that
can go wrong.

2017-03-01 10:59 GMT+01:00 Malte Pickhan :

> Hi Cassandra users,
>
> I am looking for some resources/guides for firedrill scenarios with apache
> cassandra.
>
> Do you know anything like that?
>
> Best,
>
> Malte

Resources for fire drills

[Malte Pickhan]

Hi Cassandra users,

I am looking for some resources/guides for firedrill scenarios with apache 
cassandra.

Do you know anything like that?

Best,

Malte