Re: merge two cluster

[Osman YOZGATLIOĞLU]

thank you all.
you saved my time and resource.
regards
osman

Get Outlook for Android<https://aka.ms/ghei36>


From: Jon Haddad 
Sent: Thursday, October 24, 2019 12:13:45 AM
To: user@cassandra.apache.org 
Subject: Re: merge two cluster

Probably not beneficial, I wouldn't do it.  Not a fan of multi-tenancy with 
Cassandra unless the use cases are so small that your noisy neighbor problem is 
not very noisy at all.  For those cases I don't know what you get from 
Cassandra other than a cool resume.

On Wed, Oct 23, 2019 at 12:41 PM Reid Pinchback 
mailto:rpinchb...@tripadvisor.com>> wrote:
I haven’t seen much evidence that larger cluster = more performance, plus or 
minus the statistics of speculative retry.  It horizontally scales for storage 
definitely, and somewhat for connection volume.  If anything, per Sean’s 
observation, you have less ability to have a stable tuning for a particular 
usage pattern.

Try to have a mental picture of what you think is happening in the JVM while 
Cassandra is running.  There are short-lived objects, medium-lived objects, 
long/static-lived objects, and behind the scenes some degree of read I/O and 
write I/O against disk.  Garbage collectors struggle badly with medium-lived 
objects, but Cassandra really depends a great deal on those.  If you merge two 
clusters together, within any one node you still have the JVM size and disk 
architecture you had before, but you are adding competition on fixed resources 
and potentially in the very way they find most difficult to handle.

If those resources were heavily underutilized, like Sean’s point about merging 
small apps together, then sure.  But if those two clusters of yours are already 
showing that they experience significant load, then you are unlikely to improve 
anything, far more likely to end up worse off.  GC overhead and compaction 
flushes to disk are your challenges; merging two clusters doesn’t change the 
physics of those two areas, but could increase the demand on them.

The only caveat to all of the above I can think of is if there was a 
fault-tolerance story motivating the merging.  Like “management wants us in two 
AZs in AWS, but lacks the budget for more instances, and each pool by itself is 
too small for us to come up with a 2 rack organization that makes sense”.

R

From: Osman YOZGATLIOĞLU 
mailto:osman.yozgatlio...@kron.com.tr>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Date: Wednesday, October 23, 2019 at 10:40 AM
To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
mailto:user@cassandra.apache.org>>
Subject: Re: merge two cluster

Message from External Sender

Sorry, missing question;

Actually I'm asking this for performance perspective. At application level both 
cluster used at the same time and approx same level. Inserted data inserted to 
both cluster, different parts of course.

If I merge two cluster, can I gain some performance improvements? Like raid 
stripes, more disk, more stripe, more speed..



Regards
On 23.10.2019 17:30, Durity, Sean R wrote:
Beneficial to whom? The apps, the admins, the developers?

I suggest that app teams have separate clusters per application. This prevents 
the noisy neighbor problem, isolates any security issues, and helps when it is 
time for maintenance, upgrade, performance testing, etc. to not have to 
coordinate multiple app teams at the same time. Also, an individual cluster can 
be tuned for its specific workload. Sometimes, though, costs and data size push 
us towards combining smaller apps owned by the same team onto a single cluster. 
Those are the exceptions.

As a Cassandra admin, I am always trying to scale the ability to admin multiple 
clusters without just adding new admins. That is an on-going task, dependent on 
your operating environment.

Also, because every table has a portion of memory (memtable), there is a 
practical limit to the number of tables that any one cluster should have. I 
have heard it is in the low hundreds of tables. This puts a limit on the number 
of applications that a cluster can safely support.


Sean Durity – Staff Systems Engineer, Cassandra

From: Osman YOZGATLIOĞLU 
<mailto:osman.yozgatlio...@kron.com.tr>
Sent: Wednesday, October 23, 2019 6:23 AM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: [EXTERNAL] merge two cluster


Hello,

I have two cluster and both contains different data sets with different node 
counts.

Would it be beneficial to merge two cluster?



Regards,

Osman



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action

Re: merge two cluster

[Jon Haddad]

Probably not beneficial, I wouldn't do it.  Not a fan of multi-tenancy with
Cassandra unless the use cases are so small that your noisy neighbor
problem is not very noisy at all.  For those cases I don't know what you
get from Cassandra other than a cool resume.

On Wed, Oct 23, 2019 at 12:41 PM Reid Pinchback 
wrote:

> I haven’t seen much evidence that larger cluster = more performance, plus
> or minus the statistics of speculative retry.  It horizontally scales for
> storage definitely, and somewhat for connection volume.  If anything, per
> Sean’s observation, you have less ability to have a stable tuning for a
> particular usage pattern.
>
>
>
> Try to have a mental picture of what you think is happening in the JVM
> while Cassandra is running.  There are short-lived objects, medium-lived
> objects, long/static-lived objects, and behind the scenes some degree of
> read I/O and write I/O against disk.  Garbage collectors struggle badly
> with medium-lived objects, but Cassandra really depends a great deal on
> those.  If you merge two clusters together, within any one node you still
> have the JVM size and disk architecture you had before, but you are adding
> competition on fixed resources and potentially in the very way they find
> most difficult to handle.
>
>
>
> If those resources were heavily underutilized, like Sean’s point about
> merging small apps together, then sure.  But if those two clusters of yours
> are already showing that they experience significant load, then you are
> unlikely to improve anything, far more likely to end up worse off.  GC
> overhead and compaction flushes to disk are your challenges; merging two
> clusters doesn’t change the physics of those two areas, but could increase
> the demand on them.
>
>
>
> The only caveat to all of the above I can think of is if there was a
> fault-tolerance story motivating the merging.  Like “management wants us in
> two AZs in AWS, but lacks the budget for more instances, and each pool by
> itself is too small for us to come up with a 2 rack organization that makes
> sense”.
>
>
>
> R
>
>
>
> *From: *Osman YOZGATLIOĞLU 
> *Reply-To: *"user@cassandra.apache.org" 
> *Date: *Wednesday, October 23, 2019 at 10:40 AM
> *To: *"user@cassandra.apache.org" 
> *Subject: *Re: merge two cluster
>
>
>
> *Message from External Sender*
>
> Sorry, missing question;
>
> Actually I'm asking this for performance perspective. At application level
> both cluster used at the same time and approx same level. Inserted data
> inserted to both cluster, different parts of course.
>
> If I merge two cluster, can I gain some performance improvements? Like
> raid stripes, more disk, more stripe, more speed..
>
>
>
> Regards
>
> On 23.10.2019 17:30, Durity, Sean R wrote:
>
> Beneficial to whom? The apps, the admins, the developers?
>
>
>
> I suggest that app teams have separate clusters per application. This
> prevents the noisy neighbor problem, isolates any security issues, and
> helps when it is time for maintenance, upgrade, performance testing, etc.
> to not have to coordinate multiple app teams at the same time. Also, an
> individual cluster can be tuned for its specific workload. Sometimes,
> though, costs and data size push us towards combining smaller apps owned by
> the same team onto a single cluster. Those are the exceptions.
>
>
>
> As a Cassandra admin, I am always trying to scale the ability to admin
> multiple clusters without just adding new admins. That is an on-going task,
> dependent on your operating environment.
>
>
>
> Also, because every table has a portion of memory (memtable), there is a
> practical limit to the number of tables that any one cluster should have. I
> have heard it is in the low hundreds of tables. This puts a limit on the
> number of applications that a cluster can safely support.
>
>
>
>
>
> Sean Durity – Staff Systems Engineer, Cassandra
>
>
>
> *From:* Osman YOZGATLIOĞLU 
> 
> *Sent:* Wednesday, October 23, 2019 6:23 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] merge two cluster
>
>
>
> Hello,
>
> I have two cluster and both contains different data sets with different
> node counts.
>
> Would it be beneficial to merge two cluster?
>
>
>
> Regards,
>
> Osman
>
>
> --
>
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted t

Re: merge two cluster

[Reid Pinchback]

I haven’t seen much evidence that larger cluster = more performance, plus or 
minus the statistics of speculative retry.  It horizontally scales for storage 
definitely, and somewhat for connection volume.  If anything, per Sean’s 
observation, you have less ability to have a stable tuning for a particular 
usage pattern.

Try to have a mental picture of what you think is happening in the JVM while 
Cassandra is running.  There are short-lived objects, medium-lived objects, 
long/static-lived objects, and behind the scenes some degree of read I/O and 
write I/O against disk.  Garbage collectors struggle badly with medium-lived 
objects, but Cassandra really depends a great deal on those.  If you merge two 
clusters together, within any one node you still have the JVM size and disk 
architecture you had before, but you are adding competition on fixed resources 
and potentially in the very way they find most difficult to handle.

If those resources were heavily underutilized, like Sean’s point about merging 
small apps together, then sure.  But if those two clusters of yours are already 
showing that they experience significant load, then you are unlikely to improve 
anything, far more likely to end up worse off.  GC overhead and compaction 
flushes to disk are your challenges; merging two clusters doesn’t change the 
physics of those two areas, but could increase the demand on them.

The only caveat to all of the above I can think of is if there was a 
fault-tolerance story motivating the merging.  Like “management wants us in two 
AZs in AWS, but lacks the budget for more instances, and each pool by itself is 
too small for us to come up with a 2 rack organization that makes sense”.

R

From: Osman YOZGATLIOĞLU 
Reply-To: "user@cassandra.apache.org" 
Date: Wednesday, October 23, 2019 at 10:40 AM
To: "user@cassandra.apache.org" 
Subject: Re: merge two cluster

Message from External Sender

Sorry, missing question;

Actually I'm asking this for performance perspective. At application level both 
cluster used at the same time and approx same level. Inserted data inserted to 
both cluster, different parts of course.

If I merge two cluster, can I gain some performance improvements? Like raid 
stripes, more disk, more stripe, more speed..



Regards
On 23.10.2019 17:30, Durity, Sean R wrote:
Beneficial to whom? The apps, the admins, the developers?

I suggest that app teams have separate clusters per application. This prevents 
the noisy neighbor problem, isolates any security issues, and helps when it is 
time for maintenance, upgrade, performance testing, etc. to not have to 
coordinate multiple app teams at the same time. Also, an individual cluster can 
be tuned for its specific workload. Sometimes, though, costs and data size push 
us towards combining smaller apps owned by the same team onto a single cluster. 
Those are the exceptions.

As a Cassandra admin, I am always trying to scale the ability to admin multiple 
clusters without just adding new admins. That is an on-going task, dependent on 
your operating environment.

Also, because every table has a portion of memory (memtable), there is a 
practical limit to the number of tables that any one cluster should have. I 
have heard it is in the low hundreds of tables. This puts a limit on the number 
of applications that a cluster can safely support.


Sean Durity – Staff Systems Engineer, Cassandra

From: Osman YOZGATLIOĞLU 
<mailto:osman.yozgatlio...@kron.com.tr>
Sent: Wednesday, October 23, 2019 6:23 AM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: [EXTERNAL] merge two cluster


Hello,

I have two cluster and both contains different data sets with different node 
counts.

Would it be beneficial to merge two cluster?



Regards,

Osman



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

Re: merge two cluster

[Osman YOZGATLIOĞLU]

Sorry, missing question;

Actually I'm asking this for performance perspective. At application level both 
cluster used at the same time and approx same level. Inserted data inserted to 
both cluster, different parts of course.

If I merge two cluster, can I gain some performance improvements? Like raid 
stripes, more disk, more stripe, more speed..


Regards

On 23.10.2019 17:30, Durity, Sean R wrote:
Beneficial to whom? The apps, the admins, the developers?

I suggest that app teams have separate clusters per application. This prevents 
the noisy neighbor problem, isolates any security issues, and helps when it is 
time for maintenance, upgrade, performance testing, etc. to not have to 
coordinate multiple app teams at the same time. Also, an individual cluster can 
be tuned for its specific workload. Sometimes, though, costs and data size push 
us towards combining smaller apps owned by the same team onto a single cluster. 
Those are the exceptions.

As a Cassandra admin, I am always trying to scale the ability to admin multiple 
clusters without just adding new admins. That is an on-going task, dependent on 
your operating environment.

Also, because every table has a portion of memory (memtable), there is a 
practical limit to the number of tables that any one cluster should have. I 
have heard it is in the low hundreds of tables. This puts a limit on the number 
of applications that a cluster can safely support.


Sean Durity – Staff Systems Engineer, Cassandra

From: Osman YOZGATLIOĞLU 

Sent: Wednesday, October 23, 2019 6:23 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] merge two cluster


Hello,

I have two cluster and both contains different data sets with different node 
counts.

Would it be beneficial to merge two cluster?



Regards,

Osman



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.

RE: merge two cluster

[Durity, Sean R]

Beneficial to whom? The apps, the admins, the developers?

I suggest that app teams have separate clusters per application. This prevents 
the noisy neighbor problem, isolates any security issues, and helps when it is 
time for maintenance, upgrade, performance testing, etc. to not have to 
coordinate multiple app teams at the same time. Also, an individual cluster can 
be tuned for its specific workload. Sometimes, though, costs and data size push 
us towards combining smaller apps owned by the same team onto a single cluster. 
Those are the exceptions.

As a Cassandra admin, I am always trying to scale the ability to admin multiple 
clusters without just adding new admins. That is an on-going task, dependent on 
your operating environment.

Also, because every table has a portion of memory (memtable), there is a 
practical limit to the number of tables that any one cluster should have. I 
have heard it is in the low hundreds of tables. This puts a limit on the number 
of applications that a cluster can safely support.


Sean Durity – Staff Systems Engineer, Cassandra

From: Osman YOZGATLIOĞLU 
Sent: Wednesday, October 23, 2019 6:23 AM
To: user@cassandra.apache.org
Subject: [EXTERNAL] merge two cluster


Hello,

I have two cluster and both contains different data sets with different node 
counts.

Would it be beneficial to merge two cluster?



Regards,

Osman



The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email by 
anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be taken in 
reliance on it, is prohibited and may be unlawful. When addressed to our 
clients any opinions or advice contained in this Email are subject to the terms 
and conditions expressed in any applicable governing The Home Depot terms of 
business or client engagement letter. The Home Depot disclaims all 
responsibility and liability for the accuracy and content of this attachment 
and for any damages or losses arising from any inaccuracies, errors, viruses, 
e.g., worms, trojan horses, etc., or other items of a destructive nature, which 
may be contained in this attachment and shall not be liable for direct, 
indirect, consequential or special damages in connection with this e-mail 
message or its attachment.