Mesos masters and zookeeper running together?
Hello, I've been working on setting up a mesos cluster for eventual production use and I have a question on configuring zookeeper alongside the mesos masters. Is it best practice to run zookeeper/exhibitor as a separate cluster (in our case, three nodes) or on the same machines as the mesos masters? I understand the drawbacks of increased cost for compute resources that will just be running a single service and most of the reference docs have them running together, but just wondering if it's beneficial to have them uncoupled. Thanks in advance for any input. Ron Lipke @neverminding
Re: Mesos fetcher in dockerized slave
Hi Marica/Shuai, Sorry haven't been able to spend the time to repro, but looks like Shuai confirmed it. Can one of you file a JIRA? Thanks! Tim On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin wrote: > Hi Marica, > > I can reproduce the problem exactly as you described in the first email of > this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment variable set, > the fetcher works just fine; With it, the fetcher steps seems skipped. This > looks like a bug to me. > > Regards, > Shuai > > On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci < > marica.antona...@ba.infn.it> wrote: > >> Dear all, >> >> I have not solved this issue yet. Please, can anyone run the same test >> and let me know if the fetcher is correctly invoked? >> The test is really simple, just try to start a dockerized app (see json >> definition file below) through marathon on a mesos slave running in a >> docker container started with the option —docker_mesos_image=> image>. >> I would appreciate very much any feedback. >> >> Sample Marathon app: >> { >> "id": "test-app", >> "container": { >>"type": "DOCKER", >>"docker": { >> "image": "libmesos/ubuntu" >>} >> }, >> "cpus": 1, >> "mem": 512, >> "uris": [ " >> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv"; >> ], >> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; >> done" >> } >> >> Docker run command to start dockerized mesos slave: >> >> # docker run -d MESOS_HOSTNAME= -e MESOS_IP= -e >> MESOS_MASTER=zk://:2181,:2181,:2181/mesos -e >> MESOS_CONTAINERIZERS=docker,mesos >> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e >> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v >> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos >> --name slave --net host --privileged --pid host mesos-slave >> >> Thank you very much in advance! >> Best regards, >> Marica >> >> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci < >> marica.antona...@ba.infn.it> ha scritto: >> >> Dear Tim, >> >> I have collected some information from my test environment, starting the >> slave container with and without the —docker_mesos_image startup flag. >> Please let me know if you need further input. Thank you very much for your >> support! >> >> Using the flag —docker_mesos_image: >> >> root@mesos-slave:~# docker ps >> CONTAINER IDIMAGE COMMAND CREATED >> STATUS PORTS NAMES >> b30cea22a07clibmesos/ubuntu "/bin/sh -c 'cd $MESO" 2 >> minutes ago Up 2 minutes >> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d >> da9c78ec5727mesos-slave "/bin/sh -c '/usr/lib" 2 >> minutes ago Up 2 minutes >> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor >> 150f78fbf327mesos-slave "/entrypoint.sh /usr/" 3 >> minutes ago Up 3 minutesslave >> >> *root@mesos-slave:~# docker logs slave* >> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task >> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework >> 246b272b-d649-47c0-88ca-6b1ff35f437a- >> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task >> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework >> 246b272b-d649-47c0-88ca-6b1ff35f437a- >> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown >> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' >> to user 'root' >> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor >> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >> 246b272b-d649-47c0-88ca-6b1ff35f437a- with resources cpus(*):0.1; >> mem(*):32 in work directory >> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' >> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task >> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor >> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework >> 246b272b-d649-47c0-88ca-6b1ff35f437a- >> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container >> 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task >> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor >> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework >> '246b272b-d649-47c0-88ca-6b1ff35f437a-' >> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590 to >> '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/p
Re: Mesos masters and zookeeper running together?
We've run the mesos-master and marathon processes in the same servers as zookeeper. On Dec 24, 2015 11:01 AM, "Ron Lipke" wrote: > Hello, I've been working on setting up a mesos cluster for eventual > production use and I have a question on configuring zookeeper alongside > the mesos masters. > Is it best practice to run zookeeper/exhibitor as a separate cluster (in > our case, three nodes) or on the same machines as the mesos masters? I > understand the drawbacks of increased cost for compute resources that > will just be running a single service and most of the reference docs > have them running together, but just wondering if it's beneficial to > have them uncoupled. > > Thanks in advance for any input. > > Ron Lipke > @neverminding >
Re: Mesos masters and zookeeper running together?
With our design we end up building out a stand alone zookeeper cluster 3 nodes. Zookeeper seems to be the default dumping ground for many Apache based products these days. You will eventually see many services and frameworks require a zk instance for leader election, coordination, Kv store etc.. I've seen situations where the masters can become extremely busy and cause performance problem with Zk which can be huge issue for mesos. Sent from Outlook Mobile On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke" wrote: Hello, I've been working on setting up a mesos cluster for eventual production use and I have a question on configuring zookeeper alongside the mesos masters. Is it best practice to run zookeeper/exhibitor as a separate cluster (in our case, three nodes) or on the same machines as the mesos masters? I understand the drawbacks of increased cost for compute resources that will just be running a single service and most of the reference docs have them running together, but just wondering if it's beneficial to have them uncoupled. Thanks in advance for any input. Ron Lipke @neverminding -- *NOTICE TO RECIPIENTS*: This communication is confidential and intended for the use of the addressee only. If you are not an intended recipient of this communication, please delete it immediately and notify the sender by return email. Unauthorized reading, dissemination, distribution or copying of this communication is prohibited. This communication does not constitute an offer to sell or a solicitation of an indication of interest to purchase any loan, security or any other financial product or instrument, nor is it an offer to sell or a solicitation of an indication of interest to purchase any products or services to any persons who are prohibited from receiving such information under applicable law. The contents of this communication may not be accurate or complete and are subject to change without notice. As such, Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard") makes no representation regarding the accuracy or completeness of the information contained herein. The intended recipient is advised to consult its own professional advisors, including those specializing in legal, tax and accounting matters. Orchard does not provide legal, tax or accounting advice.
Re: Mesos masters and zookeeper running together?
zookeeper really wants a dedicated cluster IMO; preferably with SSD under it - if zookeeper starts to run slow then everything else will start to bog down. I've co-hosted it with mesos masters before now for demo purposes etc. but for production it's probably worth choosing dedicated hosts. On 24 December 2015 at 20:36, Rodrick Brown wrote: > With our design we end up building out a stand alone zookeeper cluster 3 > nodes. Zookeeper seems to be the default dumping ground for many Apache > based products these days. You will eventually see many services and > frameworks require a zk instance for leader election, coordination, Kv store > etc.. I've seen situations where the masters can become extremely busy and > cause performance problem with Zk which can be huge issue for mesos. > > Sent from Outlook Mobile > > > > > On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke" wrote: > >> Hello, I've been working on setting up a mesos cluster for eventual >> production use and I have a question on configuring zookeeper alongside >> the mesos masters. >> Is it best practice to run zookeeper/exhibitor as a separate cluster (in >> our case, three nodes) or on the same machines as the mesos masters? I >> understand the drawbacks of increased cost for compute resources that >> will just be running a single service and most of the reference docs >> have them running together, but just wondering if it's beneficial to >> have them uncoupled. >> >> Thanks in advance for any input. >> >> Ron Lipke >> @neverminding > > > NOTICE TO RECIPIENTS: This communication is confidential and intended for > the use of the addressee only. If you are not an intended recipient of this > communication, please delete it immediately and notify the sender by return > email. Unauthorized reading, dissemination, distribution or copying of this > communication is prohibited. This communication does not constitute an offer > to sell or a solicitation of an indication of interest to purchase any loan, > security or any other financial product or instrument, nor is it an offer to > sell or a solicitation of an indication of interest to purchase any products > or services to any persons who are prohibited from receiving such > information under applicable law. The contents of this communication may not > be accurate or complete and are subject to change without notice. As such, > Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard") > makes no representation regarding the accuracy or completeness of the > information contained herein. The intended recipient is advised to consult > its own professional advisors, including those specializing in legal, tax > and accounting matters. Orchard does not provide legal, tax or accounting > advice.
Re: Mesos masters and zookeeper running together?
What specs do you have for the zookeeper servers? Ram, cpu, disk, OS. On Dec 24, 2015 5:13 PM, "Dick Davies" wrote: > zookeeper really wants a dedicated cluster IMO; preferably with SSD > under it - if zookeeper > starts to run slow then everything else will start to bog down. I've > co-hosted it with mesos masters > before now for demo purposes etc. but for production it's probably > worth choosing dedicated hosts. > > On 24 December 2015 at 20:36, Rodrick Brown > wrote: > > With our design we end up building out a stand alone zookeeper cluster 3 > > nodes. Zookeeper seems to be the default dumping ground for many Apache > > based products these days. You will eventually see many services and > > frameworks require a zk instance for leader election, coordination, Kv > store > > etc.. I've seen situations where the masters can become extremely busy > and > > cause performance problem with Zk which can be huge issue for mesos. > > > > Sent from Outlook Mobile > > > > > > > > > > On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke" > wrote: > > > >> Hello, I've been working on setting up a mesos cluster for eventual > >> production use and I have a question on configuring zookeeper alongside > >> the mesos masters. > >> Is it best practice to run zookeeper/exhibitor as a separate cluster (in > >> our case, three nodes) or on the same machines as the mesos masters? I > >> understand the drawbacks of increased cost for compute resources that > >> will just be running a single service and most of the reference docs > >> have them running together, but just wondering if it's beneficial to > >> have them uncoupled. > >> > >> Thanks in advance for any input. > >> > >> Ron Lipke > >> @neverminding > > > > > > NOTICE TO RECIPIENTS: This communication is confidential and intended for > > the use of the addressee only. If you are not an intended recipient of > this > > communication, please delete it immediately and notify the sender by > return > > email. Unauthorized reading, dissemination, distribution or copying of > this > > communication is prohibited. This communication does not constitute an > offer > > to sell or a solicitation of an indication of interest to purchase any > loan, > > security or any other financial product or instrument, nor is it an > offer to > > sell or a solicitation of an indication of interest to purchase any > products > > or services to any persons who are prohibited from receiving such > > information under applicable law. The contents of this communication may > not > > be accurate or complete and are subject to change without notice. As > such, > > Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard") > > makes no representation regarding the accuracy or completeness of the > > information contained herein. The intended recipient is advised to > consult > > its own professional advisors, including those specializing in legal, tax > > and accounting matters. Orchard does not provide legal, tax or accounting > > advice. >
Re: Mesos masters and zookeeper running together?
Thanks, this makes a lot of sense. It also allows us to do rolling upgrades to the zookeeper EC2 instances as we build out the cluster. Thanks everyone for the help! Happy Holidays. Ron Lipke @neverminding > On Dec 24, 2015, at 5:13 PM, Dick Davies wrote: > > zookeeper really wants a dedicated cluster IMO; preferably with SSD > under it - if zookeeper > starts to run slow then everything else will start to bog down. I've > co-hosted it with mesos masters > before now for demo purposes etc. but for production it's probably > worth choosing dedicated hosts. > > On 24 December 2015 at 20:36, Rodrick Brown > wrote: >> With our design we end up building out a stand alone zookeeper cluster 3 >> nodes. Zookeeper seems to be the default dumping ground for many Apache >> based products these days. You will eventually see many services and >> frameworks require a zk instance for leader election, coordination, Kv store >> etc.. I've seen situations where the masters can become extremely busy and >> cause performance problem with Zk which can be huge issue for mesos. >> >> Sent from Outlook Mobile >> >> >> >> >> On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke" wrote: >> >>> Hello, I've been working on setting up a mesos cluster for eventual >>> production use and I have a question on configuring zookeeper alongside >>> the mesos masters. >>> Is it best practice to run zookeeper/exhibitor as a separate cluster (in >>> our case, three nodes) or on the same machines as the mesos masters? I >>> understand the drawbacks of increased cost for compute resources that >>> will just be running a single service and most of the reference docs >>> have them running together, but just wondering if it's beneficial to >>> have them uncoupled. >>> >>> Thanks in advance for any input. >>> >>> Ron Lipke >>> @neverminding >> >> >> NOTICE TO RECIPIENTS: This communication is confidential and intended for >> the use of the addressee only. If you are not an intended recipient of this >> communication, please delete it immediately and notify the sender by return >> email. Unauthorized reading, dissemination, distribution or copying of this >> communication is prohibited. This communication does not constitute an offer >> to sell or a solicitation of an indication of interest to purchase any loan, >> security or any other financial product or instrument, nor is it an offer to >> sell or a solicitation of an indication of interest to purchase any products >> or services to any persons who are prohibited from receiving such >> information under applicable law. The contents of this communication may not >> be accurate or complete and are subject to change without notice. As such, >> Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard") >> makes no representation regarding the accuracy or completeness of the >> information contained herein. The intended recipient is advised to consult >> its own professional advisors, including those specializing in legal, tax >> and accounting matters. Orchard does not provide legal, tax or accounting >> advice.
Re: Mesos masters and zookeeper running together?
3x - r3.xlarge @ 4x vCPU 30GB Mem 1x80GB SSD > On Dec 24 2015, at 6:01 pm, craig wwrote: > > What specs do you have for the zookeeper servers? Ram, cpu, disk, OS. > > On Dec 24, 2015 5:13 PM, "Dick Davies" <[d...@hellooperator.net](mailto:d...@hellooperator.net)> wrote: > >> zookeeper really wants a dedicated cluster IMO; preferably with SSD under it - if zookeeper starts to run slow then everything else will start to bog down. I've co-hosted it with mesos masters before now for demo purposes etc. but for production it's probably worth choosing dedicated hosts. On 24 December 2015 at 20:36, Rodrick Brown <[rodr...@orchardplatform.com](mailto:rodr...@orchardplatform.com)> wrote: > With our design we end up building out a stand alone zookeeper cluster 3 > nodes. Zookeeper seems to be the default dumping ground for many Apache > based products these days. You will eventually see many services and > frameworks require a zk instance for leader election, coordination, Kv store > etc.. I've seen situations where the masters can become extremely busy and > cause performance problem with Zk which can be huge issue for mesos. > > Sent from Outlook Mobile > > > > > On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke" <[m...@ronlipke.com](mailto:m...@ronlipke.com)> wrote: > >> Hello, I've been working on setting up a mesos cluster for eventual >> production use and I have a question on configuring zookeeper alongside >> the mesos masters. >> Is it best practice to run zookeeper/exhibitor as a separate cluster (in >> our case, three nodes) or on the same machines as the mesos masters? I >> understand the drawbacks of increased cost for compute resources that >> will just be running a single service and most of the reference docs >> have them running together, but just wondering if it's beneficial to >> have them uncoupled. >> >> Thanks in advance for any input. >> >> Ron Lipke >> @neverminding > > > NOTICE TO RECIPIENTS: This communication is confidential and intended for > the use of the addressee only. If you are not an intended recipient of this > communication, please delete it immediately and notify the sender by return > email. Unauthorized reading, dissemination, distribution or copying of this > communication is prohibited. This communication does not constitute an offer > to sell or a solicitation of an indication of interest to purchase any loan, > security or any other financial product or instrument, nor is it an offer to > sell or a solicitation of an indication of interest to purchase any products > or services to any persons who are prohibited from receiving such > information under applicable law. The contents of this communication may not > be accurate or complete and are subject to change without notice. As such, > Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard") > makes no representation regarding the accuracy or completeness of the > information contained herein. The intended recipient is advised to consult > its own professional advisors, including those specializing in legal, tax > and accounting matters. Orchard does not provide legal, tax or accounting > advice. -- *NOTICE TO RECIPIENTS*: This communication is confidential and intended for the use of the addressee only. If you are not an intended recipient of this communication, please delete it immediately and notify the sender by return email. Unauthorized reading, dissemination, distribution or copying of this communication is prohibited. This communication does not constitute an offer to sell or a solicitation of an indication of interest to purchase any loan, security or any other financial product or instrument, nor is it an offer to sell or a solicitation of an indication of interest to purchase any products or services to any persons who are prohibited from receiving such information under applicable law. The contents of this communication may not be accurate or complete and are subject to change without notice. As such, Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard") makes no representation regarding the accuracy or completeness of the information contained herein. The intended recipient is advised to consult its own professional advisors, including those specializing in legal, tax and accounting matters. Orchard does not provide legal, tax or accounting advice.
Re: Mesos masters and zookeeper running together?
If you're only using ZK for Mesos, then it should be fine to have it on the same nodes as the Mesos masters, since Mesos only uses ZK for leader election, and not any other metadata storage. Marathon and other services, however, use ZK more heavily, and if you're seeing significant ZK load, it would be better to put ZK on a separate cluster. Alternatively, you could have 1 ZK cluster just for Mesos, co-located with the masters, and a separate ZK cluster (or multiple) for your other services. On Thu, Dec 24, 2015 at 9:39 PM, Rodrick Brown wrote: > 3x - r3.xlarge @ 4x vCPU 30GB Mem 1x80GB SSD >> >> On Dec 24 2015, at 6:01 pm, craig w wrote: >> >> What specs do you have for the zookeeper servers? Ram, cpu, disk, OS. >> >> On Dec 24, 2015 5:13 PM, "Dick Davies" wrote: >> >> zookeeper really wants a dedicated cluster IMO; preferably with SSD >> under it - if zookeeper >> starts to run slow then everything else will start to bog down. I've >> co-hosted it with mesos masters >> before now for demo purposes etc. but for production it's probably >> worth choosing dedicated hosts. >> >> On 24 December 2015 at 20:36, Rodrick Brown >> wrote: >> > With our design we end up building out a stand alone zookeeper cluster 3 >> > nodes. Zookeeper seems to be the default dumping ground for many Apache >> > based products these days. You will eventually see many services and >> > frameworks require a zk instance for leader election, coordination, Kv >> > store >> > etc.. I've seen situations where the masters can become extremely busy >> > and >> > cause performance problem with Zk which can be huge issue for mesos. >> > >> > Sent from Outlook Mobile >> > >> > >> > >> > >> > On Thu, Dec 24, 2015 at 8:01 AM -0800, "Ron Lipke" >> > wrote: >> > >> >> Hello, I've been working on setting up a mesos cluster for eventual >> >> production use and I have a question on configuring zookeeper alongside >> >> the mesos masters. >> >> Is it best practice to run zookeeper/exhibitor as a separate cluster >> >> (in >> >> our case, three nodes) or on the same machines as the mesos masters? I >> >> understand the drawbacks of increased cost for compute resources that >> >> will just be running a single service and most of the reference docs >> >> have them running together, but just wondering if it's beneficial to >> >> have them uncoupled. >> >> >> >> Thanks in advance for any input. >> >> >> >> Ron Lipke >> >> @neverminding >> > >> > >> > NOTICE TO RECIPIENTS: This communication is confidential and intended >> > for >> > the use of the addressee only. If you are not an intended recipient of >> > this >> > communication, please delete it immediately and notify the sender by >> > return >> > email. Unauthorized reading, dissemination, distribution or copying of >> > this >> > communication is prohibited. This communication does not constitute an >> > offer >> > to sell or a solicitation of an indication of interest to purchase any >> > loan, >> > security or any other financial product or instrument, nor is it an >> > offer to >> > sell or a solicitation of an indication of interest to purchase any >> > products >> > or services to any persons who are prohibited from receiving such >> > information under applicable law. The contents of this communication may >> > not >> > be accurate or complete and are subject to change without notice. As >> > such, >> > Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard") >> > makes no representation regarding the accuracy or completeness of the >> > information contained herein. The intended recipient is advised to >> > consult >> > its own professional advisors, including those specializing in legal, >> > tax >> > and accounting matters. Orchard does not provide legal, tax or >> > accounting >> > advice. > > > NOTICE TO RECIPIENTS: This communication is confidential and intended for > the use of the addressee only. If you are not an intended recipient of this > communication, please delete it immediately and notify the sender by return > email. Unauthorized reading, dissemination, distribution or copying of this > communication is prohibited. This communication does not constitute an offer > to sell or a solicitation of an indication of interest to purchase any loan, > security or any other financial product or instrument, nor is it an offer to > sell or a solicitation of an indication of interest to purchase any products > or services to any persons who are prohibited from receiving such > information under applicable law. The contents of this communication may not > be accurate or complete and are subject to change without notice. As such, > Orchard App, Inc. (including its subsidiaries and affiliates, "Orchard") > makes no representation regarding the accuracy or completeness of the > information contained herein. The intended recipient is advised to consult > its own professional advisors, including those specializing in legal, tax > and accounting matters. Orchard does not p