Re: Mesos masters and zookeeper running together?

[Rogier Dikkes]

It all depends on your use of zookeeper and the amount of services you 
have communicating with zookeeper. In a typical deployment with some 
marathon jobs and a couple of servers an ensemble of 5-7 voting members 
is good enough.


If zookeeper is used by thousands of processes to read states then it 
might be worth to investigate adding non voting members to your 
zookeeper cluster called observers. They can reduce the load on your 
voting members.


You can always decide to setup seperate zookeeper clusters as Adam stated.

Information i gathered about SSD's is that they can have latency issues, 
there are articles from 2012 where the SSD write cliff is mentioned. 
This can introduce high latency spikes when garbage collection kicks in, 
resulting in zookeeper session timeouts. The current generation of SSD 
vendors however over-provision their drives to ensure smoother garbage 
collection, you can free up more space yourself by simply not 
partitioning a part of your SSD.


On 12/25/15 8:32 AM, Adam Bordelon wrote:

If you're only using ZK for Mesos, then it should be fine to have it
on the same nodes as the Mesos masters, since Mesos only uses ZK for
leader election, and not any other metadata storage. Marathon and
other services, however, use ZK more heavily, and if you're seeing
significant ZK load, it would be better to put ZK on a separate
cluster. Alternatively, you could have 1 ZK cluster just for Mesos,
co-located with the masters, and a separate ZK cluster (or multiple)
for your other services.

On Thu, Dec 24, 2015 at 9:39 PM, Rodrick Brown  wrote:

3x - r3.xlarge  @ 4x vCPU 30GB Mem 1x80GB SSD

On Dec 24 2015, at 6:01 pm, craig w  wrote:

What specs do you have for the zookeeper servers? Ram, cpu, disk, OS.

On Dec 24, 2015 5:13 PM, "Dick Davies"  wrote:

zookeeper really wants a dedicated cluster IMO; preferably with SSD
under it - if zookeeper
starts to run slow then everything else will start to bog down. I've
co-hosted it with mesos masters
before now for demo purposes etc. but for production it's probably
worth choosing dedicated hosts.

On 24 December 2015 at 20:36, Rodrick Brown 
wrote:

With our design we end up building out a stand alone zookeeper cluster 3
nodes.  Zookeeper seems to be the default dumping ground for many Apache
based products these days. You will eventually see many services and
frameworks require a zk instance for leader election, coordination, Kv
store
etc.. I've seen situations where the masters can become extremely busy
and
cause performance problem with Zk which can be huge issue for mesos.

Sent from Outlook Mobile




On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke" 
wrote:


Hello, I've been working on setting up a mesos cluster for eventual
production use and I have a question on configuring zookeeper alongside
the mesos masters.
Is it best practice to run zookeeper/exhibitor as a separate cluster
(in
our case, three nodes) or on the same machines as the mesos masters?  I
understand the drawbacks of increased cost for compute resources that
will just be running a single service and most of the reference docs
have them running together, but just wondering if it's beneficial to
have them uncoupled.

Thanks in advance for any input.

Ron Lipke
@neverminding


NOTICE TO RECIPIENTS: This communication is confidential and intended
for
the use of the addressee only. If you are not an intended recipient of
this
communication, please delete it immediately and notify the sender by
return
email. Unauthorized reading, dissemination, distribution or copying of
this
communication is prohibited. This communication does not constitute an
offer
to sell or a solicitation of an indication of interest to purchase any
loan,
security or any other financial product or instrument, nor is it an
offer to
sell or a solicitation of an indication of interest to purchase any
products
or services to any persons who are prohibited from receiving such
information under applicable law. The contents of this communication may
not
be accurate or complete and are subject to change without notice. As
such,
Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard")
makes no representation regarding the accuracy or completeness of the
information contained herein. The intended recipient is advised to
consult
its own professional advisors, including those specializing in legal,
tax
and accounting matters. Orchard does not provide legal, tax or
accounting
advice.


NOTICE TO RECIPIENTS: This communication is confidential and intended for
the use of the addressee only. If you are not an intended recipient of this
communication, please delete it immediately and notify the sender by return
email. Unauthorized reading, dissemination, distribution or copying of this
communication is prohibited. This communication does not constitute an offer
to sell or a solicitation of an indication of interest to purchase any loan,
security or any other financial product or instrument, nor is it an off

Re: Mesos masters and zookeeper running together?

[Adam Bordelon]

If you're only using ZK for Mesos, then it should be fine to have it
on the same nodes as the Mesos masters, since Mesos only uses ZK for
leader election, and not any other metadata storage. Marathon and
other services, however, use ZK more heavily, and if you're seeing
significant ZK load, it would be better to put ZK on a separate
cluster. Alternatively, you could have 1 ZK cluster just for Mesos,
co-located with the masters, and a separate ZK cluster (or multiple)
for your other services.

On Thu, Dec 24, 2015 at 9:39 PM, Rodrick Brown  wrote:
> 3x - r3.xlarge  @ 4x vCPU 30GB Mem 1x80GB SSD
>>
>> On Dec 24 2015, at 6:01 pm, craig w  wrote:
>>
>> What specs do you have for the zookeeper servers? Ram, cpu, disk, OS.
>>
>> On Dec 24, 2015 5:13 PM, "Dick Davies"  wrote:
>>
>> zookeeper really wants a dedicated cluster IMO; preferably with SSD
>> under it - if zookeeper
>> starts to run slow then everything else will start to bog down. I've
>> co-hosted it with mesos masters
>> before now for demo purposes etc. but for production it's probably
>> worth choosing dedicated hosts.
>>
>> On 24 December 2015 at 20:36, Rodrick Brown 
>> wrote:
>> > With our design we end up building out a stand alone zookeeper cluster 3
>> > nodes.  Zookeeper seems to be the default dumping ground for many Apache
>> > based products these days. You will eventually see many services and
>> > frameworks require a zk instance for leader election, coordination, Kv
>> > store
>> > etc.. I've seen situations where the masters can become extremely busy
>> > and
>> > cause performance problem with Zk which can be huge issue for mesos.
>> >
>> > Sent from Outlook Mobile
>> >
>> >
>> >
>> >
>> > On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke" 
>> > wrote:
>> >
>> >> Hello, I've been working on setting up a mesos cluster for eventual
>> >> production use and I have a question on configuring zookeeper alongside
>> >> the mesos masters.
>> >> Is it best practice to run zookeeper/exhibitor as a separate cluster
>> >> (in
>> >> our case, three nodes) or on the same machines as the mesos masters?  I
>> >> understand the drawbacks of increased cost for compute resources that
>> >> will just be running a single service and most of the reference docs
>> >> have them running together, but just wondering if it's beneficial to
>> >> have them uncoupled.
>> >>
>> >> Thanks in advance for any input.
>> >>
>> >> Ron Lipke
>> >> @neverminding
>> >
>> >
>> > NOTICE TO RECIPIENTS: This communication is confidential and intended
>> > for
>> > the use of the addressee only. If you are not an intended recipient of
>> > this
>> > communication, please delete it immediately and notify the sender by
>> > return
>> > email. Unauthorized reading, dissemination, distribution or copying of
>> > this
>> > communication is prohibited. This communication does not constitute an
>> > offer
>> > to sell or a solicitation of an indication of interest to purchase any
>> > loan,
>> > security or any other financial product or instrument, nor is it an
>> > offer to
>> > sell or a solicitation of an indication of interest to purchase any
>> > products
>> > or services to any persons who are prohibited from receiving such
>> > information under applicable law. The contents of this communication may
>> > not
>> > be accurate or complete and are subject to change without notice. As
>> > such,
>> > Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard")
>> > makes no representation regarding the accuracy or completeness of the
>> > information contained herein. The intended recipient is advised to
>> > consult
>> > its own professional advisors, including those specializing in legal,
>> > tax
>> > and accounting matters. Orchard does not provide legal, tax or
>> > accounting
>> > advice.
>
>
> NOTICE TO RECIPIENTS: This communication is confidential and intended for
> the use of the addressee only. If you are not an intended recipient of this
> communication, please delete it immediately and notify the sender by return
> email. Unauthorized reading, dissemination, distribution or copying of this
> communication is prohibited. This communication does not constitute an offer
> to sell or a solicitation of an indication of interest to purchase any loan,
> security or any other financial product or instrument, nor is it an offer to
> sell or a solicitation of an indication of interest to purchase any products
> or services to any persons who are prohibited from receiving such
> information under applicable law. The contents of this communication may not
> be accurate or complete and are subject to change without notice. As such,
> Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard")
> makes no representation regarding the accuracy or completeness of the
> information contained herein. The intended recipient is advised to consult
> its own professional advisors, including those specializing in legal, tax
> and accounting matters. Orchard does not p

Re: Mesos masters and zookeeper running together?

[Rodrick Brown]

3x - r3.xlarge  @ 4x vCPU 30GB Mem 1x80GB SSD

> On Dec 24 2015, at 6:01 pm, craig w  wrote:  

>

> What specs do you have for the zookeeper servers? Ram, cpu, disk, OS.

>

> On Dec 24, 2015 5:13 PM, "Dick Davies"
<[d...@hellooperator.net](mailto:d...@hellooperator.net)> wrote:  

>

>> zookeeper really wants a dedicated cluster IMO; preferably with SSD  
under it - if zookeeper  
starts to run slow then everything else will start to bog down. I've  
co-hosted it with mesos masters  
before now for demo purposes etc. but for production it's probably  
worth choosing dedicated hosts.  
  
On 24 December 2015 at 20:36, Rodrick Brown
<[rodr...@orchardplatform.com](mailto:rodr...@orchardplatform.com)>
wrote:  
> With our design we end up building out a stand alone zookeeper cluster 3  
> nodes.  Zookeeper seems to be the default dumping ground for many Apache  
> based products these days. You will eventually see many services and  
> frameworks require a zk instance for leader election, coordination, Kv
store  
> etc.. I've seen situations where the masters can become extremely busy
and  
> cause performance problem with Zk which can be huge issue for mesos.  
>  
> Sent from Outlook Mobile  
>  
>  
>  
>  
> On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke"
<[m...@ronlipke.com](mailto:m...@ronlipke.com)> wrote:  
>  
>> Hello, I've been working on setting up a mesos cluster for eventual  
>> production use and I have a question on configuring zookeeper
alongside  
>> the mesos masters.  
>> Is it best practice to run zookeeper/exhibitor as a separate cluster
(in  
>> our case, three nodes) or on the same machines as the mesos masters?
I  
>> understand the drawbacks of increased cost for compute resources that  
>> will just be running a single service and most of the reference docs  
>> have them running together, but just wondering if it's beneficial to  
>> have them uncoupled.  
>>  
>> Thanks in advance for any input.  
>>  
>> Ron Lipke  
>> @neverminding  
>  
>  
> NOTICE TO RECIPIENTS: This communication is confidential and intended for  
> the use of the addressee only. If you are not an intended recipient of
this  
> communication, please delete it immediately and notify the sender by
return  
> email. Unauthorized reading, dissemination, distribution or copying of
this  
> communication is prohibited. This communication does not constitute an
offer  
> to sell or a solicitation of an indication of interest to purchase any
loan,  
> security or any other financial product or instrument, nor is it an offer
to  
> sell or a solicitation of an indication of interest to purchase any
products  
> or services to any persons who are prohibited from receiving such  
> information under applicable law. The contents of this communication may
not  
> be accurate or complete and are subject to change without notice. As
such,  
> Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard")  
> makes no representation regarding the accuracy or completeness of the  
> information contained herein. The intended recipient is advised to
consult  
> its own professional advisors, including those specializing in legal, tax  
> and accounting matters. Orchard does not provide legal, tax or accounting  
> advice.  


-- 
*NOTICE TO RECIPIENTS*: This communication is confidential and intended for 
the use of the addressee only. If you are not an intended recipient of this 
communication, please delete it immediately and notify the sender by return 
email. Unauthorized reading, dissemination, distribution or copying of this 
communication is prohibited. This communication does not constitute an 
offer to sell or a solicitation of an indication of interest to purchase 
any loan, security or any other financial product or instrument, nor is it 
an offer to sell or a solicitation of an indication of interest to purchase 
any products or services to any persons who are prohibited from receiving 
such information under applicable law. The contents of this communication 
may not be accurate or complete and are subject to change without notice. 
As such, Orchard App, Inc. (including its subsidiaries and affiliates, 
"Orchard") makes no representation regarding the accuracy or completeness 
of the information contained herein. The intended recipient is advised to 
consult its own professional advisors, including those specializing in 
legal, tax and accounting matters. Orchard does not provide legal, tax or 
accounting advice.

Re: Mesos masters and zookeeper running together?

[Ron Lipke]

Thanks, this makes a lot of sense.  

It also allows us to do rolling upgrades to the zookeeper EC2 instances as we 
build out the cluster.  

Thanks everyone for the help!  Happy Holidays.

Ron Lipke
@neverminding
> On Dec 24, 2015, at 5:13 PM, Dick Davies  wrote:
> 
> zookeeper really wants a dedicated cluster IMO; preferably with SSD
> under it - if zookeeper
> starts to run slow then everything else will start to bog down. I've
> co-hosted it with mesos masters
> before now for demo purposes etc. but for production it's probably
> worth choosing dedicated hosts.
> 
> On 24 December 2015 at 20:36, Rodrick Brown  
> wrote:
>> With our design we end up building out a stand alone zookeeper cluster 3
>> nodes.  Zookeeper seems to be the default dumping ground for many Apache
>> based products these days. You will eventually see many services and
>> frameworks require a zk instance for leader election, coordination, Kv store
>> etc.. I've seen situations where the masters can become extremely busy and
>> cause performance problem with Zk which can be huge issue for mesos.
>> 
>> Sent from Outlook Mobile
>> 
>> 
>> 
>> 
>> On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke"  wrote:
>> 
>>> Hello, I've been working on setting up a mesos cluster for eventual
>>> production use and I have a question on configuring zookeeper alongside
>>> the mesos masters.
>>> Is it best practice to run zookeeper/exhibitor as a separate cluster (in
>>> our case, three nodes) or on the same machines as the mesos masters?  I
>>> understand the drawbacks of increased cost for compute resources that
>>> will just be running a single service and most of the reference docs
>>> have them running together, but just wondering if it's beneficial to
>>> have them uncoupled.
>>> 
>>> Thanks in advance for any input.
>>> 
>>> Ron Lipke
>>> @neverminding
>> 
>> 
>> NOTICE TO RECIPIENTS: This communication is confidential and intended for
>> the use of the addressee only. If you are not an intended recipient of this
>> communication, please delete it immediately and notify the sender by return
>> email. Unauthorized reading, dissemination, distribution or copying of this
>> communication is prohibited. This communication does not constitute an offer
>> to sell or a solicitation of an indication of interest to purchase any loan,
>> security or any other financial product or instrument, nor is it an offer to
>> sell or a solicitation of an indication of interest to purchase any products
>> or services to any persons who are prohibited from receiving such
>> information under applicable law. The contents of this communication may not
>> be accurate or complete and are subject to change without notice. As such,
>> Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard")
>> makes no representation regarding the accuracy or completeness of the
>> information contained herein. The intended recipient is advised to consult
>> its own professional advisors, including those specializing in legal, tax
>> and accounting matters. Orchard does not provide legal, tax or accounting
>> advice.

Re: Mesos masters and zookeeper running together?

[craig w]

What specs do you have for the zookeeper servers? Ram, cpu, disk, OS.
On Dec 24, 2015 5:13 PM, "Dick Davies"  wrote:

> zookeeper really wants a dedicated cluster IMO; preferably with SSD
> under it - if zookeeper
> starts to run slow then everything else will start to bog down. I've
> co-hosted it with mesos masters
> before now for demo purposes etc. but for production it's probably
> worth choosing dedicated hosts.
>
> On 24 December 2015 at 20:36, Rodrick Brown 
> wrote:
> > With our design we end up building out a stand alone zookeeper cluster 3
> > nodes.  Zookeeper seems to be the default dumping ground for many Apache
> > based products these days. You will eventually see many services and
> > frameworks require a zk instance for leader election, coordination, Kv
> store
> > etc.. I've seen situations where the masters can become extremely busy
> and
> > cause performance problem with Zk which can be huge issue for mesos.
> >
> > Sent from Outlook Mobile
> >
> >
> >
> >
> > On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke" 
> wrote:
> >
> >> Hello, I've been working on setting up a mesos cluster for eventual
> >> production use and I have a question on configuring zookeeper alongside
> >> the mesos masters.
> >> Is it best practice to run zookeeper/exhibitor as a separate cluster (in
> >> our case, three nodes) or on the same machines as the mesos masters?  I
> >> understand the drawbacks of increased cost for compute resources that
> >> will just be running a single service and most of the reference docs
> >> have them running together, but just wondering if it's beneficial to
> >> have them uncoupled.
> >>
> >> Thanks in advance for any input.
> >>
> >> Ron Lipke
> >> @neverminding
> >
> >
> > NOTICE TO RECIPIENTS: This communication is confidential and intended for
> > the use of the addressee only. If you are not an intended recipient of
> this
> > communication, please delete it immediately and notify the sender by
> return
> > email. Unauthorized reading, dissemination, distribution or copying of
> this
> > communication is prohibited. This communication does not constitute an
> offer
> > to sell or a solicitation of an indication of interest to purchase any
> loan,
> > security or any other financial product or instrument, nor is it an
> offer to
> > sell or a solicitation of an indication of interest to purchase any
> products
> > or services to any persons who are prohibited from receiving such
> > information under applicable law. The contents of this communication may
> not
> > be accurate or complete and are subject to change without notice. As
> such,
> > Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard")
> > makes no representation regarding the accuracy or completeness of the
> > information contained herein. The intended recipient is advised to
> consult
> > its own professional advisors, including those specializing in legal, tax
> > and accounting matters. Orchard does not provide legal, tax or accounting
> > advice.
>

Re: Mesos masters and zookeeper running together?

[Dick Davies]

zookeeper really wants a dedicated cluster IMO; preferably with SSD
under it - if zookeeper
starts to run slow then everything else will start to bog down. I've
co-hosted it with mesos masters
before now for demo purposes etc. but for production it's probably
worth choosing dedicated hosts.

On 24 December 2015 at 20:36, Rodrick Brown  wrote:
> With our design we end up building out a stand alone zookeeper cluster 3
> nodes.  Zookeeper seems to be the default dumping ground for many Apache
> based products these days. You will eventually see many services and
> frameworks require a zk instance for leader election, coordination, Kv store
> etc.. I've seen situations where the masters can become extremely busy and
> cause performance problem with Zk which can be huge issue for mesos.
>
> Sent from Outlook Mobile
>
>
>
>
> On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke"  wrote:
>
>> Hello, I've been working on setting up a mesos cluster for eventual
>> production use and I have a question on configuring zookeeper alongside
>> the mesos masters.
>> Is it best practice to run zookeeper/exhibitor as a separate cluster (in
>> our case, three nodes) or on the same machines as the mesos masters?  I
>> understand the drawbacks of increased cost for compute resources that
>> will just be running a single service and most of the reference docs
>> have them running together, but just wondering if it's beneficial to
>> have them uncoupled.
>>
>> Thanks in advance for any input.
>>
>> Ron Lipke
>> @neverminding
>
>
> NOTICE TO RECIPIENTS: This communication is confidential and intended for
> the use of the addressee only. If you are not an intended recipient of this
> communication, please delete it immediately and notify the sender by return
> email. Unauthorized reading, dissemination, distribution or copying of this
> communication is prohibited. This communication does not constitute an offer
> to sell or a solicitation of an indication of interest to purchase any loan,
> security or any other financial product or instrument, nor is it an offer to
> sell or a solicitation of an indication of interest to purchase any products
> or services to any persons who are prohibited from receiving such
> information under applicable law. The contents of this communication may not
> be accurate or complete and are subject to change without notice. As such,
> Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard")
> makes no representation regarding the accuracy or completeness of the
> information contained herein. The intended recipient is advised to consult
> its own professional advisors, including those specializing in legal, tax
> and accounting matters. Orchard does not provide legal, tax or accounting
> advice.

Re: Mesos masters and zookeeper running together?

[Rodrick Brown]

With our design we end up building out a stand alone zookeeper cluster 3 nodes. 
 Zookeeper seems to be the default dumping ground for many Apache based 
products these days. You will eventually see many services and frameworks 
require a zk instance for leader election, coordination, Kv store etc.. I've 
seen situations where the masters can become extremely busy and cause 
performance problem with Zk which can be huge issue for mesos. 
Sent from Outlook Mobile




On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke"  wrote:










Hello, I've been working on setting up a mesos cluster for eventual
production use and I have a question on configuring zookeeper alongside
the mesos masters.  
Is it best practice to run zookeeper/exhibitor as a separate cluster (in
our case, three nodes) or on the same machines as the mesos masters?  I
understand the drawbacks of increased cost for compute resources that
will just be running a single service and most of the reference docs
have them running together, but just wondering if it's beneficial to
have them uncoupled.

Thanks in advance for any input.

Ron Lipke
@neverminding






-- 
*NOTICE TO RECIPIENTS*: This communication is confidential and intended for 
the use of the addressee only. If you are not an intended recipient of this 
communication, please delete it immediately and notify the sender by return 
email. Unauthorized reading, dissemination, distribution or copying of this 
communication is prohibited. This communication does not constitute an 
offer to sell or a solicitation of an indication of interest to purchase 
any loan, security or any other financial product or instrument, nor is it 
an offer to sell or a solicitation of an indication of interest to purchase 
any products or services to any persons who are prohibited from receiving 
such information under applicable law. The contents of this communication 
may not be accurate or complete and are subject to change without notice. 
As such, Orchard App, Inc. (including its subsidiaries and affiliates, 
"Orchard") makes no representation regarding the accuracy or completeness 
of the information contained herein. The intended recipient is advised to 
consult its own professional advisors, including those specializing in 
legal, tax and accounting matters. Orchard does not provide legal, tax or 
accounting advice.

Re: Mesos masters and zookeeper running together?

[craig w]

We've run the mesos-master and marathon processes in the same servers as
zookeeper.
On Dec 24, 2015 11:01 AM, "Ron Lipke"  wrote:

> Hello, I've been working on setting up a mesos cluster for eventual
> production use and I have a question on configuring zookeeper alongside
> the mesos masters.
> Is it best practice to run zookeeper/exhibitor as a separate cluster (in
> our case, three nodes) or on the same machines as the mesos masters?  I
> understand the drawbacks of increased cost for compute resources that
> will just be running a single service and most of the reference docs
> have them running together, but just wondering if it's beneficial to
> have them uncoupled.
>
> Thanks in advance for any input.
>
> Ron Lipke
> @neverminding
>

Mesos masters and zookeeper running together?

[Ron Lipke]

Hello, I've been working on setting up a mesos cluster for eventual
production use and I have a question on configuring zookeeper alongside
the mesos masters.  
Is it best practice to run zookeeper/exhibitor as a separate cluster (in
our case, three nodes) or on the same machines as the mesos masters?  I
understand the drawbacks of increased cost for compute resources that
will just be running a single service and most of the reference docs
have them running together, but just wondering if it's beneficial to
have them uncoupled.

Thanks in advance for any input.

Ron Lipke
@neverminding