Re: Zookeeper integration for Mesos-DNS
Aaron, It depends on what you mean however, Mesos-DNS works outside the cluster IMO. It is a bridge for things in the cluster (services launched by mesos)... But at that point it is DNS. Any client in or out of the cluster that can query DNS that leverage the service. Sent from my iPhone On Mar 23, 2015, at 4:25 AM, Aaron Carey aca...@ilm.com wrote: Hey, I don't suppose there is anything like Mesos-DNS but for services/users outside the mesos cluster? So having a service which updates a DNS provider with task port/ips running inside the cluster so that external users are able to find those services? Am I correct in thinking Mesos-DNS only works inside the cluster? Currently we're using consul for this, but I'd be interested if there was some sort of magical plug and play solution? Thanks, Aaron From: Christos Kozyrakis [kozyr...@gmail.com] Sent: 21 March 2015 00:18 To: user@mesos.apache.org Subject: Zookeeper integration for Mesos-DNS Hi everybody, we have updated Mesos-DNS to integrate directly with Zookeeper. Instead of providing Mesos-DNS with a list of masters, you point it to the Zookeeper instances. Meson-DNS will watch Zookeeper to detect the current leading master. So, while the list of Zookeeper instances is configured in a static manner, Mesos masters can be added or removed freely without restarting Mesos-DNS. The integration with Zookeeper forced to switch from -v and -vv as the flags to control verbosity to -v=0 (default), -v=1 (verbose), and -v=2 (very verbose). To reduce complications because of dependencies to other packages, we have also started using godep. Please take a look at the branch https://github.com/mesosphere/mesos-dns/tree/zk and provide us with any feedback on the code or the documentation. Thanks -- Christos
Re: Unable to install subversion-devel 1.8+ on CentOS 7
https://issues.apache.org/jira/browse/MESOS-2115 On Mon, Mar 23, 2015 at 8:30 AM, Aaron Carey aca...@ilm.com wrote: ah interesting.. what causes this difference? I think this probably makes sense for our setup currently.. -- *From:* craig w [codecr...@gmail.com] *Sent:* 23 March 2015 12:20 *To:* user@mesos.apache.org *Subject:* Re: Unable to install subversion-devel 1.8+ on CentOS 7 I had considered running Mesos in docker containers, however with Mesos 0.21 if the slaves are running in a container and you have tasks running in containers, if the slave container were to exit/die, the tasks running in containers would also exit. If running mesos-slave on the host, any tasks that it had running will remain running even if the process dies. That's why I hadn't gone that route. Have you considered that? On Mon, Mar 23, 2015 at 8:15 AM, craig w codecr...@gmail.com wrote: What OS is your docker image based on? On Mon, Mar 23, 2015 at 8:11 AM, Aaron Carey aca...@ilm.com wrote: Not sure if this helps, but we've been using docker to run Mesos on Centos 7 hosts. -- *From:* craig w [codecr...@gmail.com] *Sent:* 23 March 2015 12:06 *To:* user@mesos.apache.org *Subject:* Unable to install subversion-devel 1.8+ on CentOS 7 Mesos 0.21.0+ requires subversion-devel 1.8+, which can be installed by adding the Wandisco yum repo. However, it appears that subversion-devel 1.8+ requires libsasl2.so.2, which is not available on CentOS7. I've seen one person try to create a symlink to libsasl2.so.3 and it worked [1], while another person found it did not work [2]. I created a CentOS 7 droplet on DigitalOcean, added the Wandisco repo and tried to install subversion-devel and it failed b/c of the libsasl2.so.2 missing. I tried creating a symlink (ln -s /usr/lib64/libsasl2.so.3 /usr/lib64/libsasl2.so.2), restarting the server and installing subversion-devel still failed b/c of the libsasl issue: Error: Package: subversion-1.8.11-1.x86_64 (WandiscoSVN) Requires: libsasl2.so.2()(64bit) Anyone had any success on CentOS 7 with Mesos 0.21+? [1] - did work: http://unix.stackexchange.com/a/178408 [2] - did not work: http://www.wandisco.com/svnforum/forum/smartsvn-community/smartsvn-help-and-support/69834-installing-subversion-command-line-client-on-centos-7 -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links
Re: Is launchTasks() with multiple offers limited to a single slave?
Thanks Michael! On Mon, Mar 23, 2015 at 7:59 AM, Michael Park mcyp...@gmail.com wrote: Hi Itamar, Thanks for the patch! It looks like Niklas and Jie has looked at the patch and I'm sure they'll commit it soon, if not I'll nudge them :) Great :-) 2. I would imagine there could be a cleaner way (from a framework author perspective) to do that, by setting a policy or a filter or something, that communicates to the master that the scheduler would like to receive only offers that meet some criteria (e.g. min_cpu, min_mem, etc.). effectively, moving the complexity of holding on to offers from the framework to the master. I don't think the ability to request for only offer me resources that contains: [(cpus, 8), (mem, 1048)] can actually replace the current ability of the frameworks, since the framework can currently build up the resources while holding onto the ones it was given. For example, suppose [(cpus, 2), (mem, 1048)] became available but the mesos master doesn't offer it to the framework because it does not meet the requirement, and instead offers it to another framework. Then [(cpus, 6)] becomes available and it still doesn't meet the requirement, so it gets offered to yet another framework. In order to avoid this situation, the framework would have to request for something like offer me at least [(cpus, 8), (mem, 1048)] once you have built it up. I think we could support a mechanism like this via some form of resource request. I think your example is an excellent one to explain why I think the policy approach can work better! If framework A indeed requires [(cpus,8),(mem,1048)], and framework B is happy with [(cpus,2),(mem,1048)], then with the current approach framework A might want to hold on to [(cpus,2],(mem,1048)] until it gets another [(cpus,6)] from the same slave. But the master may see that framework A is not responding (because it's holding on), and offer the same [(cpus,2),(mem,1048)] to framework B. Since framework B may immediately accept, the offer will be rescinded from framework A. This may make it difficult for framework A to get the resources it needs, unless it has a way to tell the master that it wants [(cpus,8),(mem,1048)]. (unless I misunderstood how the master handles offering multiple resources to multiple frameworks) It may also be satisfied by a reservation of some form. Some of these mechanisms such as offer reservations are described in MESOS-1791 https://issues.apache.org/jira/browse/MESOS-1791. Is such a thing possible in mesos? Currently no, but it will be coming! Great! Was it an explicit design decision to keep such logic at framework level? You may have noticed that there already exists a *Request* message in *mesos.proto* which currently does nothing. So while the logic lives at the framework-level right now, I don't think it was an explicit design decision to keep it there in the long run. MPark.
RE: Unable to install subversion-devel 1.8+ on CentOS 7
Not sure if this helps, but we've been using docker to run Mesos on Centos 7 hosts. From: craig w [codecr...@gmail.com] Sent: 23 March 2015 12:06 To: user@mesos.apache.org Subject: Unable to install subversion-devel 1.8+ on CentOS 7 Mesos 0.21.0+ requires subversion-devel 1.8+, which can be installed by adding the Wandisco yum repo. However, it appears that subversion-devel 1.8+ requires libsasl2.so.2, which is not available on CentOS7. I've seen one person try to create a symlink to libsasl2.so.3 and it worked [1], while another person found it did not work [2]. I created a CentOS 7 droplet on DigitalOcean, added the Wandisco repo and tried to install subversion-devel and it failed b/c of the libsasl2.so.2 missing. I tried creating a symlink (ln -s /usr/lib64/libsasl2.so.3 /usr/lib64/libsasl2.so.2), restarting the server and installing subversion-devel still failed b/c of the libsasl issue: Error: Package: subversion-1.8.11-1.x86_64 (WandiscoSVN) Requires: libsasl2.so.2()(64bit) Anyone had any success on CentOS 7 with Mesos 0.21+? [1] - did work: http://unix.stackexchange.com/a/178408 [2] - did not work:http://www.wandisco.com/svnforum/forum/smartsvn-community/smartsvn-help-and-support/69834-installing-subversion-command-line-client-on-centos-7
Unable to install subversion-devel 1.8+ on CentOS 7
Mesos 0.21.0+ requires subversion-devel 1.8+, which can be installed by adding the Wandisco yum repo. However, it appears that subversion-devel 1.8+ requires libsasl2.so.2, which is not available on CentOS7. I've seen one person try to create a symlink to libsasl2.so.3 and it worked [1], while another person found it did not work [2]. I created a CentOS 7 droplet on DigitalOcean, added the Wandisco repo and tried to install subversion-devel and it failed b/c of the libsasl2.so.2 missing. I tried creating a symlink (ln -s /usr/lib64/libsasl2.so.3 /usr/lib64/libsasl2.so.2), restarting the server and installing subversion-devel still failed b/c of the libsasl issue: Error: Package: subversion-1.8.11-1.x86_64 (WandiscoSVN) Requires: libsasl2.so.2()(64bit) Anyone had any success on CentOS 7 with Mesos 0.21+? [1] - did work: http://unix.stackexchange.com/a/178408 [2] - did not work: http://www.wandisco.com/svnforum/forum/smartsvn-community/smartsvn-help-and-support/69834-installing-subversion-command-line-client-on-centos-7
Re: HAProxy for Hive Thrift Server on Mesos
I have a similar error when launch jenkins in mesos cluster. And I use HA-Proxy to route jenkins URL(and JNLP port) to a fixed one. But from time to time, connection between jenkins master and jenkins slave will lost, while will cause jobs fail. I will try to set timeout value for HAProxy 2015-03-23 4:18 GMT+08:00 John Omernik j...@omernik.com: Yep all three are going through HAProxy and seem to be working well. I should do some testing to determine if I run a thrift server that connects to metastore in HAProxy to determine if that's the broken link, but I believe it's between the thrift server. I will check out the link you posted to get a better idea out of how it works. I am running this all on a MapR cluster which is Great for Docker/Mesos as I used the MapR FS, mounted locally on each node's NFS to put the data for MySQL, the logs for the metastore/thift server, and my world files for Minecraft. So far it's all really slick. On Sun, Mar 22, 2015 at 10:44 AM, Ben Whitehead ben.whiteh...@mesosphere.io wrote: It sounds like you may be running into the connection timeout that is configured for the HAProxy instance. You can see an instance of this in Marathon's HAProxy-marathon-bridge[1]. You may need to raise the timeout values. Are you accessing MySql, Hive metastore and minecraft through HAProxy as well? I know that Minecraft running in docker on mesos works well, though I've always used a direct connection not a proxied one. [1] https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119 On Sat, Mar 21, 2015 at 1:50 PM, John Omernik j...@omernik.com wrote: I have a nice setup with a Hive thrift server running in a docker container on Mesos. It works pretty well, but something, I believe in how HAProxy works with the connection, is causing the thrift server connection to die after a time. Basically, I can run a few queries, but after 2 or 3, or specifically after a longer query, I get the error below indicating End of File on the connection. Then no more connections work until I reestablish the connection to the thrift server. I've tried looking in logs. the thrift server std err logs show no issues. I am guess I need to dig into haproxy logs, but I am not seeing any issues in syslog so far. I'd love any pointers on how to trouble shoot this. By the way, I have MySQl, hive metastore, and a minecraft server all running on Mesos/Docker with no issues, not sure why the thrift server is so sensitive. :) pyhs2 connection string: hs2 = pyhs2.connect(host='marathonmaster',port=1,authMechanism='PLAIN',user='bestuser',password='removed',database='default') Error /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc in read(self, sz) 118 if len(buff) == 0: 119 raise TTransportException(type=TTransportException.END_OF_FILE, -- 120 message='TSocket read 0 bytes') 121 return buff 122 TTransportException: TSocket read 0 bytes
RE: Unable to install subversion-devel 1.8+ on CentOS 7
ah interesting.. what causes this difference? I think this probably makes sense for our setup currently.. From: craig w [codecr...@gmail.com] Sent: 23 March 2015 12:20 To: user@mesos.apache.org Subject: Re: Unable to install subversion-devel 1.8+ on CentOS 7 I had considered running Mesos in docker containers, however with Mesos 0.21 if the slaves are running in a container and you have tasks running in containers, if the slave container were to exit/die, the tasks running in containers would also exit. If running mesos-slave on the host, any tasks that it had running will remain running even if the process dies. That's why I hadn't gone that route. Have you considered that? On Mon, Mar 23, 2015 at 8:15 AM, craig w codecr...@gmail.commailto:codecr...@gmail.com wrote: What OS is your docker image based on? On Mon, Mar 23, 2015 at 8:11 AM, Aaron Carey aca...@ilm.commailto:aca...@ilm.com wrote: Not sure if this helps, but we've been using docker to run Mesos on Centos 7 hosts. From: craig w [codecr...@gmail.commailto:codecr...@gmail.com] Sent: 23 March 2015 12:06 To: user@mesos.apache.orgmailto:user@mesos.apache.org Subject: Unable to install subversion-devel 1.8+ on CentOS 7 Mesos 0.21.0+ requires subversion-devel 1.8+, which can be installed by adding the Wandisco yum repo. However, it appears that subversion-devel 1.8+ requires libsasl2.so.2, which is not available on CentOS7. I've seen one person try to create a symlink to libsasl2.so.3 and it worked [1], while another person found it did not work [2]. I created a CentOS 7 droplet on DigitalOcean, added the Wandisco repo and tried to install subversion-devel and it failed b/c of the libsasl2.so.2 missing. I tried creating a symlink (ln -s /usr/lib64/libsasl2.so.3 /usr/lib64/libsasl2.so.2), restarting the server and installing subversion-devel still failed b/c of the libsasl issue: Error: Package: subversion-1.8.11-1.x86_64 (WandiscoSVN) Requires: libsasl2.so.2()(64bit) Anyone had any success on CentOS 7 with Mesos 0.21+? [1] - did work: http://unix.stackexchange.com/a/178408 [2] - did not work:http://www.wandisco.com/svnforum/forum/smartsvn-community/smartsvn-help-and-support/69834-installing-subversion-command-line-client-on-centos-7 -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links
Re: Unable to install subversion-devel 1.8+ on CentOS 7
What OS is your docker image based on? On Mon, Mar 23, 2015 at 8:11 AM, Aaron Carey aca...@ilm.com wrote: Not sure if this helps, but we've been using docker to run Mesos on Centos 7 hosts. -- *From:* craig w [codecr...@gmail.com] *Sent:* 23 March 2015 12:06 *To:* user@mesos.apache.org *Subject:* Unable to install subversion-devel 1.8+ on CentOS 7 Mesos 0.21.0+ requires subversion-devel 1.8+, which can be installed by adding the Wandisco yum repo. However, it appears that subversion-devel 1.8+ requires libsasl2.so.2, which is not available on CentOS7. I've seen one person try to create a symlink to libsasl2.so.3 and it worked [1], while another person found it did not work [2]. I created a CentOS 7 droplet on DigitalOcean, added the Wandisco repo and tried to install subversion-devel and it failed b/c of the libsasl2.so.2 missing. I tried creating a symlink (ln -s /usr/lib64/libsasl2.so.3 /usr/lib64/libsasl2.so.2), restarting the server and installing subversion-devel still failed b/c of the libsasl issue: Error: Package: subversion-1.8.11-1.x86_64 (WandiscoSVN) Requires: libsasl2.so.2()(64bit) Anyone had any success on CentOS 7 with Mesos 0.21+? [1] - did work: http://unix.stackexchange.com/a/178408 [2] - did not work: http://www.wandisco.com/svnforum/forum/smartsvn-community/smartsvn-help-and-support/69834-installing-subversion-command-line-client-on-centos-7 -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links
Re: Unable to install subversion-devel 1.8+ on CentOS 7
I had considered running Mesos in docker containers, however with Mesos 0.21 if the slaves are running in a container and you have tasks running in containers, if the slave container were to exit/die, the tasks running in containers would also exit. If running mesos-slave on the host, any tasks that it had running will remain running even if the process dies. That's why I hadn't gone that route. Have you considered that? On Mon, Mar 23, 2015 at 8:15 AM, craig w codecr...@gmail.com wrote: What OS is your docker image based on? On Mon, Mar 23, 2015 at 8:11 AM, Aaron Carey aca...@ilm.com wrote: Not sure if this helps, but we've been using docker to run Mesos on Centos 7 hosts. -- *From:* craig w [codecr...@gmail.com] *Sent:* 23 March 2015 12:06 *To:* user@mesos.apache.org *Subject:* Unable to install subversion-devel 1.8+ on CentOS 7 Mesos 0.21.0+ requires subversion-devel 1.8+, which can be installed by adding the Wandisco yum repo. However, it appears that subversion-devel 1.8+ requires libsasl2.so.2, which is not available on CentOS7. I've seen one person try to create a symlink to libsasl2.so.3 and it worked [1], while another person found it did not work [2]. I created a CentOS 7 droplet on DigitalOcean, added the Wandisco repo and tried to install subversion-devel and it failed b/c of the libsasl2.so.2 missing. I tried creating a symlink (ln -s /usr/lib64/libsasl2.so.3 /usr/lib64/libsasl2.so.2), restarting the server and installing subversion-devel still failed b/c of the libsasl issue: Error: Package: subversion-1.8.11-1.x86_64 (WandiscoSVN) Requires: libsasl2.so.2()(64bit) Anyone had any success on CentOS 7 with Mesos 0.21+? [1] - did work: http://unix.stackexchange.com/a/178408 [2] - did not work: http://www.wandisco.com/svnforum/forum/smartsvn-community/smartsvn-help-and-support/69834-installing-subversion-command-line-client-on-centos-7 -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links
Re: Zookeeper integration for Mesos-DNS
roger that On Mar 23, 2015, at 9:22 AM, Aaron Carey aca...@ilm.com wrote: Thanks Ken, So basically we just need to add mesos-dns to our /etc/resolv.conf on every machine and hey presto auto-service discovery (using DNS)? (Here I mean service discovery to be: hey where is rabbitmq? DNS says: 172.20.121.292:8393 or whatever) Aaron From: Ken Sipe [kens...@gmail.com] Sent: 23 March 2015 14:29 To: user@mesos.apache.org Subject: Re: Zookeeper integration for Mesos-DNS Aaron, Mesos-DNS is a DNS name server + a monitor of mesos-masters. It listens to the mesos-master. If a service is launched by mesos then mesos-dns conjures a service name (app_id + framework_id +.mesos) and associates it to the IP and PORT of the service. Since Mesos-DNS is a name service, it needs to be in your list of name services for service discovery. From a service discovery stand point there is no need to be in the cluster and there is no need to have a dependency on Mesos. Mesos-DNS is not a proxy. It doesn’t provide any special services to clients or services inside the cluster. more detail below. On Mar 23, 2015, at 7:52 AM, Aaron Carey aca...@ilm.com mailto:aca...@ilm.com wrote: As I understood it, it provides a service for containers within the cluster to automatically find each other as it handles their dns calls? The way this is stated this doesn’t seem true.Mesos-DNS is a DNS name server.From a service discovery stand point, It doesn’t do anything different than a standard DNS naming server. However clients outside the cluster will not use the mesos-dns service by default, so won't have knowledge of anything running inside the cluster? This is all dependent on how /etc/resolv.conf is setup. If mesos-dns is in the list… then this is not true. Is there an easy way to set this up to (for example) add records to AWS Route 53 when services get started in the cluster, so other clients can see them? This is outside of Mesos-DNS Good Luck!! Thanks! Aaron From: Ken Sipe [kens...@gmail.com mailto:kens...@gmail.com] Sent: 23 March 2015 13:31 To: user@mesos.apache.org mailto:user@mesos.apache.org Subject: Re: Zookeeper integration for Mesos-DNS Aaron, It depends on what you mean however, Mesos-DNS works outside the cluster IMO. It is a bridge for things in the cluster (services launched by mesos)... But at that point it is DNS. Any client in or out of the cluster that can query DNS that leverage the service. Sent from my iPhone On Mar 23, 2015, at 4:25 AM, Aaron Carey aca...@ilm.com mailto:aca...@ilm.com wrote: Hey, I don't suppose there is anything like Mesos-DNS but for services/users outside the mesos cluster? So having a service which updates a DNS provider with task port/ips running inside the cluster so that external users are able to find those services? Am I correct in thinking Mesos-DNS only works inside the cluster? Currently we're using consul for this, but I'd be interested if there was some sort of magical plug and play solution? Thanks, Aaron From: Christos Kozyrakis [kozyr...@gmail.com mailto:kozyr...@gmail.com] Sent: 21 March 2015 00:18 To: user@mesos.apache.org mailto:user@mesos.apache.org Subject: Zookeeper integration for Mesos-DNS Hi everybody, we have updated Mesos-DNS to integrate directly with Zookeeper. Instead of providing Mesos-DNS with a list of masters, you point it to the Zookeeper instances. Meson-DNS will watch Zookeeper to detect the current leading master. So, while the list of Zookeeper instances is configured in a static manner, Mesos masters can be added or removed freely without restarting Mesos-DNS. The integration with Zookeeper forced to switch from -v and -vv as the flags to control verbosity to -v=0 (default), -v=1 (verbose), and -v=2 (very verbose). To reduce complications because of dependencies to other packages, we have also started using godep. Please take a look at the branch https://github.com/mesosphere/mesos-dns/tree/zk https://github.com/mesosphere/mesos-dns/tree/zk and provide us with any feedback on the code or the documentation. Thanks -- Christos
Re: Zookeeper integration for Mesos-DNS
Aaron, Mesos-DNS is a DNS name server + a monitor of mesos-masters. It listens to the mesos-master. If a service is launched by mesos then mesos-dns conjures a service name (app_id + framework_id +.mesos) and associates it to the IP and PORT of the service. Since Mesos-DNS is a name service, it needs to be in your list of name services for service discovery. From a service discovery stand point there is no need to be in the cluster and there is no need to have a dependency on Mesos. Mesos-DNS is not a proxy. It doesn’t provide any special services to clients or services inside the cluster. more detail below. On Mar 23, 2015, at 7:52 AM, Aaron Carey aca...@ilm.com wrote: As I understood it, it provides a service for containers within the cluster to automatically find each other as it handles their dns calls? The way this is stated this doesn’t seem true.Mesos-DNS is a DNS name server.From a service discovery stand point, It doesn’t do anything different than a standard DNS naming server. However clients outside the cluster will not use the mesos-dns service by default, so won't have knowledge of anything running inside the cluster? This is all dependent on how /etc/resolv.conf is setup. If mesos-dns is in the list… then this is not true. Is there an easy way to set this up to (for example) add records to AWS Route 53 when services get started in the cluster, so other clients can see them? This is outside of Mesos-DNS Good Luck!! Thanks! Aaron From: Ken Sipe [kens...@gmail.com mailto:kens...@gmail.com] Sent: 23 March 2015 13:31 To: user@mesos.apache.org mailto:user@mesos.apache.org Subject: Re: Zookeeper integration for Mesos-DNS Aaron, It depends on what you mean however, Mesos-DNS works outside the cluster IMO. It is a bridge for things in the cluster (services launched by mesos)... But at that point it is DNS. Any client in or out of the cluster that can query DNS that leverage the service. Sent from my iPhone On Mar 23, 2015, at 4:25 AM, Aaron Carey aca...@ilm.com mailto:aca...@ilm.com wrote: Hey, I don't suppose there is anything like Mesos-DNS but for services/users outside the mesos cluster? So having a service which updates a DNS provider with task port/ips running inside the cluster so that external users are able to find those services? Am I correct in thinking Mesos-DNS only works inside the cluster? Currently we're using consul for this, but I'd be interested if there was some sort of magical plug and play solution? Thanks, Aaron From: Christos Kozyrakis [kozyr...@gmail.com mailto:kozyr...@gmail.com] Sent: 21 March 2015 00:18 To: user@mesos.apache.org mailto:user@mesos.apache.org Subject: Zookeeper integration for Mesos-DNS Hi everybody, we have updated Mesos-DNS to integrate directly with Zookeeper. Instead of providing Mesos-DNS with a list of masters, you point it to the Zookeeper instances. Meson-DNS will watch Zookeeper to detect the current leading master. So, while the list of Zookeeper instances is configured in a static manner, Mesos masters can be added or removed freely without restarting Mesos-DNS. The integration with Zookeeper forced to switch from -v and -vv as the flags to control verbosity to -v=0 (default), -v=1 (verbose), and -v=2 (very verbose). To reduce complications because of dependencies to other packages, we have also started using godep. Please take a look at the branch https://github.com/mesosphere/mesos-dns/tree/zk https://github.com/mesosphere/mesos-dns/tree/zk and provide us with any feedback on the code or the documentation. Thanks -- Christos
RE: Unable to install subversion-devel 1.8+ on CentOS 7
Thanks, that's very useful to know! From: craig w [codecr...@gmail.com] Sent: 23 March 2015 12:41 To: user@mesos.apache.org Subject: Re: Unable to install subversion-devel 1.8+ on CentOS 7 https://issues.apache.org/jira/browse/MESOS-2115 On Mon, Mar 23, 2015 at 8:30 AM, Aaron Carey aca...@ilm.commailto:aca...@ilm.com wrote: ah interesting.. what causes this difference? I think this probably makes sense for our setup currently.. From: craig w [codecr...@gmail.commailto:codecr...@gmail.com] Sent: 23 March 2015 12:20 To: user@mesos.apache.orgmailto:user@mesos.apache.org Subject: Re: Unable to install subversion-devel 1.8+ on CentOS 7 I had considered running Mesos in docker containers, however with Mesos 0.21 if the slaves are running in a container and you have tasks running in containers, if the slave container were to exit/die, the tasks running in containers would also exit. If running mesos-slave on the host, any tasks that it had running will remain running even if the process dies. That's why I hadn't gone that route. Have you considered that? On Mon, Mar 23, 2015 at 8:15 AM, craig w codecr...@gmail.commailto:codecr...@gmail.com wrote: What OS is your docker image based on? On Mon, Mar 23, 2015 at 8:11 AM, Aaron Carey aca...@ilm.commailto:aca...@ilm.com wrote: Not sure if this helps, but we've been using docker to run Mesos on Centos 7 hosts. From: craig w [codecr...@gmail.commailto:codecr...@gmail.com] Sent: 23 March 2015 12:06 To: user@mesos.apache.orgmailto:user@mesos.apache.org Subject: Unable to install subversion-devel 1.8+ on CentOS 7 Mesos 0.21.0+ requires subversion-devel 1.8+, which can be installed by adding the Wandisco yum repo. However, it appears that subversion-devel 1.8+ requires libsasl2.so.2, which is not available on CentOS7. I've seen one person try to create a symlink to libsasl2.so.3 and it worked [1], while another person found it did not work [2]. I created a CentOS 7 droplet on DigitalOcean, added the Wandisco repo and tried to install subversion-devel and it failed b/c of the libsasl2.so.2 missing. I tried creating a symlink (ln -s /usr/lib64/libsasl2.so.3 /usr/lib64/libsasl2.so.2), restarting the server and installing subversion-devel still failed b/c of the libsasl issue: Error: Package: subversion-1.8.11-1.x86_64 (WandiscoSVN) Requires: libsasl2.so.2()(64bit) Anyone had any success on CentOS 7 with Mesos 0.21+? [1] - did work: http://unix.stackexchange.com/a/178408 [2] - did not work:http://www.wandisco.com/svnforum/forum/smartsvn-community/smartsvn-help-and-support/69834-installing-subversion-command-line-client-on-centos-7 -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links
RE: Zookeeper integration for Mesos-DNS
As I understood it, it provides a service for containers within the cluster to automatically find each other as it handles their dns calls? However clients outside the cluster will not use the mesos-dns service by default, so won't have knowledge of anything running inside the cluster? Is there an easy way to set this up to (for example) add records to AWS Route 53 when services get started in the cluster, so other clients can see them? Thanks! Aaron From: Ken Sipe [kens...@gmail.com] Sent: 23 March 2015 13:31 To: user@mesos.apache.org Subject: Re: Zookeeper integration for Mesos-DNS Aaron, It depends on what you mean however, Mesos-DNS works outside the cluster IMO. It is a bridge for things in the cluster (services launched by mesos)... But at that point it is DNS. Any client in or out of the cluster that can query DNS that leverage the service. Sent from my iPhone On Mar 23, 2015, at 4:25 AM, Aaron Carey aca...@ilm.commailto:aca...@ilm.com wrote: Hey, I don't suppose there is anything like Mesos-DNS but for services/users outside the mesos cluster? So having a service which updates a DNS provider with task port/ips running inside the cluster so that external users are able to find those services? Am I correct in thinking Mesos-DNS only works inside the cluster? Currently we're using consul for this, but I'd be interested if there was some sort of magical plug and play solution? Thanks, Aaron From: Christos Kozyrakis [kozyr...@gmail.commailto:kozyr...@gmail.com] Sent: 21 March 2015 00:18 To: user@mesos.apache.orgmailto:user@mesos.apache.org Subject: Zookeeper integration for Mesos-DNS Hi everybody, we have updated Mesos-DNS to integrate directly with Zookeeper. Instead of providing Mesos-DNS with a list of masters, you point it to the Zookeeper instances. Meson-DNS will watch Zookeeper to detect the current leading master. So, while the list of Zookeeper instances is configured in a static manner, Mesos masters can be added or removed freely without restarting Mesos-DNS. The integration with Zookeeper forced to switch from -v and -vv as the flags to control verbosity to -v=0 (default), -v=1 (verbose), and -v=2 (very verbose). To reduce complications because of dependencies to other packages, we have also started using godep. Please take a look at the branch https://github.com/mesosphere/mesos-dns/tree/zk and provide us with any feedback on the code or the documentation. Thanks -- Christos
Re: [VOTE] Release Apache Mesos 0.22.0 (rc4)
I would like to steer this conversation/thread back to the release vote. Can you restart the distcc discussion in another thread? Cheers, Niklas On 23 March 2015 at 21:31, CCAAT cc...@tampabay.rr.com wrote: Excellent start! Nice links I was not aware of (thanks). Folks can use distcc now to test new rollouts of mesos? That was the quest of the thread was/is to establish some codes for testing new rollouts of mesos. It'll need to be 'extended' for cross compiling too for my needs. I'd like to follow up with anyone that get's distcc working with cross compiling for different arches. arm64 would be really cool. (LLVM?) My small cluster needs some work, much of it not related to mesos, but my efforts to run mesos and spark without the HDFS and use Cephfs, btrfs and supporting codes. So I'm not sure when I'll get this mesos-distcc installed and running in the near future; but I am most interested in follow the issues other encounter on compiling and cross compiling on a mesos cluster. What would be really cool is to run this distcc-mesos on top of spark and cephfs for some real fast compile times of large codes. Thanks, James On 03/23/2015 10:22 PM, Adam Bordelon wrote: I know it's over a year old and hasn't been updated, but bmahler already created a distcc framework example for Mesos. https://github.com/mesos/mesos-distcc On Mon, Mar 23, 2015 at 7:56 PM, CCAAT cc...@tampabay.rr.com mailto:cc...@tampabay.rr.com wrote: On 03/23/2015 09:02 PM, Adam Bordelon wrote: Integration tests are definitely desired/recommended. Some of us devs just do make [dist]check, but others test integrations with their favourite frameworks, or push it to their internal testing clusters. We're open to any additional testing you would like to propose and/or perform. Thanks, -Adam- Distcc is a distributed compiling program that has been a long staple on Gentoo linux and many other distros. I work on Gentoo and I think that setting up distcc to run on a mesos cluster is a fabulous idea. Not only can you compile native binaries for the inherent arch, but cross compiling should work too. Everyone has to recompile (optimized) kernels frequently with the release cycle of the linux kernel team. With the current roll out of all sorts of 64 bit arm systems, there's going to be a great opportunity to cross compile arm64 bit codes on CISC (X86_64) bit clusters. This also starts the process of heterogeneous mesos clusters, surely inevitable. https://code.google.com/p/__distcc/ https://code.google.com/p/ distcc/ https://code.google.com/p/__distcc/downloads/list https://code.google.com/p/distcc/downloads/list With LLvm, gnu and other projects, compiling and cross compiling on a mesos cluster is sure to be a very, very popoular idea. Any CI endeavor will necessitate lots of compiling too. hope this helps, James
RE: Zookeeper integration for Mesos-DNS
lovely, thanks! From: craig w [codecr...@gmail.com] Sent: 23 March 2015 15:35 To: user@mesos.apache.org Subject: Re: Zookeeper integration for Mesos-DNS Keep in mind DNS will give you the ipaddress of the host, so rabbitmq.marathon.mesos will resolve to some IP address. Do get port information you have to query mesos-dns for its SRV records. On Mon, Mar 23, 2015 at 11:29 AM, Ken Sipe kens...@gmail.commailto:kens...@gmail.com wrote: roger that On Mar 23, 2015, at 9:22 AM, Aaron Carey aca...@ilm.commailto:aca...@ilm.com wrote: Thanks Ken, So basically we just need to add mesos-dns to our /etc/resolv.conf on every machine and hey presto auto-service discovery (using DNS)? (Here I mean service discovery to be: hey where is rabbitmq? DNS says: 172.20.121.292:8393 or whatever) Aaron From: Ken Sipe [kens...@gmail.commailto:kens...@gmail.com] Sent: 23 March 2015 14:29 To: user@mesos.apache.orgmailto:user@mesos.apache.org Subject: Re: Zookeeper integration for Mesos-DNS Aaron, Mesos-DNS is a DNS name server + a monitor of mesos-masters. It listens to the mesos-master. If a service is launched by mesos then mesos-dns conjures a service name (app_id + framework_id +.mesos) and associates it to the IP and PORT of the service. Since Mesos-DNS is a name service, it needs to be in your list of name services for service discovery. From a service discovery stand point there is no need to be in the cluster and there is no need to have a dependency on Mesos. Mesos-DNS is not a proxy. It doesn’t provide any special services to clients or services inside the cluster. more detail below. On Mar 23, 2015, at 7:52 AM, Aaron Carey aca...@ilm.commailto:aca...@ilm.com wrote: As I understood it, it provides a service for containers within the cluster to automatically find each other as it handles their dns calls? The way this is stated this doesn’t seem true.Mesos-DNS is a DNS name server.From a service discovery stand point, It doesn’t do anything different than a standard DNS naming server. However clients outside the cluster will not use the mesos-dns service by default, so won't have knowledge of anything running inside the cluster? This is all dependent on how /etc/resolv.conf is setup. If mesos-dns is in the list… then this is not true. Is there an easy way to set this up to (for example) add records to AWS Route 53 when services get started in the cluster, so other clients can see them? This is outside of Mesos-DNS Good Luck!! Thanks! Aaron From: Ken Sipe [kens...@gmail.commailto:kens...@gmail.com] Sent: 23 March 2015 13:31 To: user@mesos.apache.orgmailto:user@mesos.apache.org Subject: Re: Zookeeper integration for Mesos-DNS Aaron, It depends on what you mean however, Mesos-DNS works outside the cluster IMO. It is a bridge for things in the cluster (services launched by mesos)... But at that point it is DNS. Any client in or out of the cluster that can query DNS that leverage the service. Sent from my iPhone On Mar 23, 2015, at 4:25 AM, Aaron Carey aca...@ilm.commailto:aca...@ilm.com wrote: Hey, I don't suppose there is anything like Mesos-DNS but for services/users outside the mesos cluster? So having a service which updates a DNS provider with task port/ips running inside the cluster so that external users are able to find those services? Am I correct in thinking Mesos-DNS only works inside the cluster? Currently we're using consul for this, but I'd be interested if there was some sort of magical plug and play solution? Thanks, Aaron From: Christos Kozyrakis [kozyr...@gmail.commailto:kozyr...@gmail.com] Sent: 21 March 2015 00:18 To: user@mesos.apache.orgmailto:user@mesos.apache.org Subject: Zookeeper integration for Mesos-DNS Hi everybody, we have updated Mesos-DNS to integrate directly with Zookeeper. Instead of providing Mesos-DNS with a list of masters, you point it to the Zookeeper instances. Meson-DNS will watch Zookeeper to detect the current leading master. So, while the list of Zookeeper instances is configured in a static manner, Mesos masters can be added or removed freely without restarting Mesos-DNS. The integration with Zookeeper forced to switch from -v and -vv as the flags to control verbosity to -v=0 (default), -v=1 (verbose), and -v=2 (very verbose). To reduce complications because of dependencies to other packages, we have also started using godep. Please take a look at the branch https://github.com/mesosphere/mesos-dns/tree/zk and provide us with any feedback on the code or the documentation. Thanks -- Christos -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links
Re: Mesos slaves connecting but not active.
How many containers are you running, and what is your system like? Also are you able to capture through perf or strace what docker rm is blocked on? Tim On Mon, Mar 23, 2015 at 10:12 AM, Giulio Eulisse giulio.euli...@cern.ch wrote: I suspect my problem is that docker rm takes forever in my case. I'm not running docker in docker though. On 23 Mar 2015, at 18:01, haosdent wrote: Are your issue relevant to this? https://issues.apache.org/jira/browse/MESOS-2115 On Tue, Mar 24, 2015 at 12:52 AM, Giulio Eulisse giulio.euli...@cern.ch wrote: Hi, I'm running using 0.20.1 and I seem to have troubles due to the fact a mesos slave is not able to recover the docker containers after a restart, resulting in a very long wait. Is this some known issue? -- Ciao, Giulio -- Best Regards, Haosdent Huang
RE: Zookeeper integration for Mesos-DNS
Hey, I don't suppose there is anything like Mesos-DNS but for services/users outside the mesos cluster? So having a service which updates a DNS provider with task port/ips running inside the cluster so that external users are able to find those services? Am I correct in thinking Mesos-DNS only works inside the cluster? Currently we're using consul for this, but I'd be interested if there was some sort of magical plug and play solution? Thanks, Aaron From: Christos Kozyrakis [kozyr...@gmail.com] Sent: 21 March 2015 00:18 To: user@mesos.apache.org Subject: Zookeeper integration for Mesos-DNS Hi everybody, we have updated Mesos-DNS to integrate directly with Zookeeper. Instead of providing Mesos-DNS with a list of masters, you point it to the Zookeeper instances. Meson-DNS will watch Zookeeper to detect the current leading master. So, while the list of Zookeeper instances is configured in a static manner, Mesos masters can be added or removed freely without restarting Mesos-DNS. The integration with Zookeeper forced to switch from -v and -vv as the flags to control verbosity to -v=0 (default), -v=1 (verbose), and -v=2 (very verbose). To reduce complications because of dependencies to other packages, we have also started using godep. Please take a look at the branch https://github.com/mesosphere/mesos-dns/tree/zk and provide us with any feedback on the code or the documentation. Thanks -- Christos
Re: Mesos slaves connecting but not active.
I suspect my problem is that docker rm takes forever in my case. I'm not running docker in docker though. On 23 Mar 2015, at 18:01, haosdent wrote: Are your issue relevant to this? https://issues.apache.org/jira/browse/MESOS-2115 On Tue, Mar 24, 2015 at 12:52 AM, Giulio Eulisse giulio.euli...@cern.ch wrote: Hi, I'm running using 0.20.1 and I seem to have troubles due to the fact a mesos slave is not able to recover the docker containers after a restart, resulting in a very long wait. Is this some known issue? -- Ciao, Giulio -- Best Regards, Haosdent Huang
Re: Mesos slaves connecting but not active.
Ciao, How many containers are you running, and what is your system like? I've something like a dozen of slaves a 2 / 3 containers per slave. I'm running on a Centos6 derived distribution (Scientific Linux CERN). On the specific slave I do not have any running container: ``` [root@cmsbuild11 ~]# docker ps -q | wc 0 0 0 ``` but I do have a bunch of dead one: ``` [root@cmsbuild11 ~]# docker ps -qa | wc 999 999 12987 ``` due to some runaway process. By attaching via gdb to the docker daemon I get: ``` #0 0x005b0ad4 in syscall.Syscall () #1 0x0084f91b in github.com/docker/docker/pkg/devicemapper.ioctlBlkDiscard () #2 0x0010 in ?? () #3 0x000b in ?? () #4 0x1277 in ?? () #5 0x7f06d004e128 in ?? () #6 0x00c209341e68 in ?? () #7 0x7f06d004e140 in ?? () #8 0x0018 in ?? () #9 0x00c209341e40 in ?? () #10 0x in ?? () ``` for a few of the running threads (the other ones are blocked in some futex). Notice I'm running on a CEPH volume. -- Ciao, Giulio Also are you able to capture through perf or strace what docker rm is blocked on? Tim On Mon, Mar 23, 2015 at 10:12 AM, Giulio Eulisse giulio.euli...@cern.ch wrote: I suspect my problem is that docker rm takes forever in my case. I'm not running docker in docker though. On 23 Mar 2015, at 18:01, haosdent wrote: Are your issue relevant to this? https://issues.apache.org/jira/browse/MESOS-2115 On Tue, Mar 24, 2015 at 12:52 AM, Giulio Eulisse giulio.euli...@cern.ch wrote: Hi, I'm running using 0.20.1 and I seem to have troubles due to the fact a mesos slave is not able to recover the docker containers after a restart, resulting in a very long wait. Is this some known issue? -- Ciao, Giulio -- Best Regards, Haosdent Huang
Re: [VOTE] Release Apache Mesos 0.22.0 (rc4)
+1 (Tested with Chronos) -Elizabeth On Mon, Mar 23, 2015 at 3:40 PM, Ben Whitehead ben.whiteh...@mesosphere.io wrote: +1 (non-binding) openSUSE 13.2 Linux 3.16.7 / gcc-4.8.3 Tested running Marathon 0.8.1 and Cassandra on Mesos 0.1.0-SNAPSHOT. On Thu, Mar 19, 2015 at 2:18 PM, Alex Rukletsov a...@mesosphere.io wrote: +1 (non-binding) Mac OS 10.9.5 + clang CentOS 7 + gcc 4.4.7 [cgroups tests disabled] On Wed, Mar 18, 2015 at 4:04 PM, Brenden Matthews bren...@diddyinc.com wrote: +1 Tested with internal testing cluster. On Wed, Mar 18, 2015 at 1:25 PM, craig w codecr...@gmail.com wrote: +1 On Wed, Mar 18, 2015 at 3:52 PM, Niklas Nielsen nik...@mesosphere.io wrote: Hi all, Please vote on releasing the following candidate as Apache Mesos 0.22.0. 0.22.0 includes the following: * Support for explicitly sending status updates acknowledgements from schedulers; refer to the upgrades document for upgrading schedulers. * Rate limiting slave removal, to safeguard against unforeseen bugs leading to widespread slave removal. * Disk quota isolation in Mesos containerizer; refer to the containerization documentation to enable disk quota monitoring and enforcement. * Support for module hooks in task launch sequence. Refer to the modules documentation for more information. * Anonymous modules: a new kind of module that does not receive any callbacks but coexists with its parent process. * New service discovery info in task info allows framework users to specify discoverability of tasks for external service discovery systems. Refer to the framework development guide for more information. * New '--external_log_file' flag to serve external logs through the Mesos web UI. * New '--gc_disk_headroom' flag to control maxmimum executor sandbox age. The CHANGELOG for the release is available at: https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.22.0-rc4 The candidate for Mesos 0.22.0 release is available at: https://dist.apache.org/repos/dist/dev/mesos/0.22.0-rc4/mesos-0.22.0.tar.gz The tag to be voted on is 0.22.0-rc4: https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.22.0-rc4 The MD5 checksum of the tarball can be found at: https://dist.apache.org/repos/dist/dev/mesos/0.22.0-rc4/mesos-0.22.0.tar.gz.md5 The signature of the tarball can be found at: https://dist.apache.org/repos/dist/dev/mesos/0.22.0-rc4/mesos-0.22.0.tar.gz.asc The PGP key used to sign the release is here: https://dist.apache.org/repos/dist/release/mesos/KEYS The JAR is up in Maven in a staging repository here: https://repository.apache.org/content/repositories/orgapachemesos-1048 Please vote on releasing this package as Apache Mesos 0.22.0! The vote is open until Sat Mar 21 12:49:56 PDT 2015 and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Mesos 0.22.0 [ ] -1 Do not release this package because ... Thanks, Niklas -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links
Re: [VOTE] Release Apache Mesos 0.22.0 (rc4)
+1 (non-binding) tested with Marathon On 24 Mar 2015, at 00:03, Adam Bordelon a...@mesosphere.io wrote: +1 (binding) make check passes on Ubuntu 14.04 and Mac OS X (buildbot) On Mon, Mar 23, 2015 at 3:44 PM, Elizabeth Lingg elizab...@mesosphere.io mailto:elizab...@mesosphere.io wrote: +1 (Tested with Chronos) -Elizabeth On Mon, Mar 23, 2015 at 3:40 PM, Ben Whitehead ben.whiteh...@mesosphere.io mailto:ben.whiteh...@mesosphere.io wrote: +1 (non-binding) openSUSE 13.2 Linux 3.16.7 / gcc-4.8.3 Tested running Marathon 0.8.1 and Cassandra on Mesos 0.1.0-SNAPSHOT. On Thu, Mar 19, 2015 at 2:18 PM, Alex Rukletsov a...@mesosphere.io mailto:a...@mesosphere.io wrote: +1 (non-binding) Mac OS 10.9.5 + clang CentOS 7 + gcc 4.4.7 [cgroups tests disabled] On Wed, Mar 18, 2015 at 4:04 PM, Brenden Matthews bren...@diddyinc.com mailto:bren...@diddyinc.com wrote: +1 Tested with internal testing cluster. On Wed, Mar 18, 2015 at 1:25 PM, craig w codecr...@gmail.com mailto:codecr...@gmail.com wrote: +1 On Wed, Mar 18, 2015 at 3:52 PM, Niklas Nielsen nik...@mesosphere.io mailto:nik...@mesosphere.io wrote: Hi all, Please vote on releasing the following candidate as Apache Mesos 0.22.0. 0.22.0 includes the following: * Support for explicitly sending status updates acknowledgements from schedulers; refer to the upgrades document for upgrading schedulers. * Rate limiting slave removal, to safeguard against unforeseen bugs leading to widespread slave removal. * Disk quota isolation in Mesos containerizer; refer to the containerization documentation to enable disk quota monitoring and enforcement. * Support for module hooks in task launch sequence. Refer to the modules documentation for more information. * Anonymous modules: a new kind of module that does not receive any callbacks but coexists with its parent process. * New service discovery info in task info allows framework users to specify discoverability of tasks for external service discovery systems. Refer to the framework development guide for more information. * New '--external_log_file' flag to serve external logs through the Mesos web UI. * New '--gc_disk_headroom' flag to control maxmimum executor sandbox age. The CHANGELOG for the release is available at: https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.22.0-rc4 https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.22.0-rc4 The candidate for Mesos 0.22.0 release is available at: https://dist.apache.org/repos/dist/dev/mesos/0.22.0-rc4/mesos-0.22.0.tar.gz https://dist.apache.org/repos/dist/dev/mesos/0.22.0-rc4/mesos-0.22.0.tar.gz The tag to be voted on is 0.22.0-rc4: https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.22.0-rc4 https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.22.0-rc4 The MD5 checksum of the tarball can be found at: https://dist.apache.org/repos/dist/dev/mesos/0.22.0-rc4/mesos-0.22.0.tar.gz.md5 https://dist.apache.org/repos/dist/dev/mesos/0.22.0-rc4/mesos-0.22.0.tar.gz.md5 The signature of the tarball can be found at: https://dist.apache.org/repos/dist/dev/mesos/0.22.0-rc4/mesos-0.22.0.tar.gz.asc https://dist.apache.org/repos/dist/dev/mesos/0.22.0-rc4/mesos-0.22.0.tar.gz.asc The PGP key used to sign the release is here: https://dist.apache.org/repos/dist/release/mesos/KEYS https://dist.apache.org/repos/dist/release/mesos/KEYS The JAR is up in Maven in a staging repository here: https://repository.apache.org/content/repositories/orgapachemesos-1048 https://repository.apache.org/content/repositories/orgapachemesos-1048 Please vote on releasing this package as Apache Mesos 0.22.0! The vote is open until Sat Mar 21 12:49:56 PDT 2015 and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Mesos 0.22.0 [ ] -1 Do not release this package because ... Thanks, Niklas -- https://github.com/mindscratch https://github.com/mindscratch https://www.google.com/+CraigWickesser https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/mind_scratch https://twitter.com/craig_links https://twitter.com/craig_links
Re: [VOTE] Release Apache Mesos 0.22.0 (rc4)
+1 (binding) make check passes on Ubuntu 14.04 and Mac OS X (buildbot) On Mon, Mar 23, 2015 at 3:44 PM, Elizabeth Lingg elizab...@mesosphere.io wrote: +1 (Tested with Chronos) -Elizabeth On Mon, Mar 23, 2015 at 3:40 PM, Ben Whitehead ben.whiteh...@mesosphere.io wrote: +1 (non-binding) openSUSE 13.2 Linux 3.16.7 / gcc-4.8.3 Tested running Marathon 0.8.1 and Cassandra on Mesos 0.1.0-SNAPSHOT. On Thu, Mar 19, 2015 at 2:18 PM, Alex Rukletsov a...@mesosphere.io wrote: +1 (non-binding) Mac OS 10.9.5 + clang CentOS 7 + gcc 4.4.7 [cgroups tests disabled] On Wed, Mar 18, 2015 at 4:04 PM, Brenden Matthews bren...@diddyinc.com wrote: +1 Tested with internal testing cluster. On Wed, Mar 18, 2015 at 1:25 PM, craig w codecr...@gmail.com wrote: +1 On Wed, Mar 18, 2015 at 3:52 PM, Niklas Nielsen nik...@mesosphere.io wrote: Hi all, Please vote on releasing the following candidate as Apache Mesos 0.22.0. 0.22.0 includes the following: * Support for explicitly sending status updates acknowledgements from schedulers; refer to the upgrades document for upgrading schedulers. * Rate limiting slave removal, to safeguard against unforeseen bugs leading to widespread slave removal. * Disk quota isolation in Mesos containerizer; refer to the containerization documentation to enable disk quota monitoring and enforcement. * Support for module hooks in task launch sequence. Refer to the modules documentation for more information. * Anonymous modules: a new kind of module that does not receive any callbacks but coexists with its parent process. * New service discovery info in task info allows framework users to specify discoverability of tasks for external service discovery systems. Refer to the framework development guide for more information. * New '--external_log_file' flag to serve external logs through the Mesos web UI. * New '--gc_disk_headroom' flag to control maxmimum executor sandbox age. The CHANGELOG for the release is available at: https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.22.0-rc4 The candidate for Mesos 0.22.0 release is available at: https://dist.apache.org/repos/dist/dev/mesos/0.22.0-rc4/mesos-0.22.0.tar.gz The tag to be voted on is 0.22.0-rc4: https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.22.0-rc4 The MD5 checksum of the tarball can be found at: https://dist.apache.org/repos/dist/dev/mesos/0.22.0-rc4/mesos-0.22.0.tar.gz.md5 The signature of the tarball can be found at: https://dist.apache.org/repos/dist/dev/mesos/0.22.0-rc4/mesos-0.22.0.tar.gz.asc The PGP key used to sign the release is here: https://dist.apache.org/repos/dist/release/mesos/KEYS The JAR is up in Maven in a staging repository here: https://repository.apache.org/content/repositories/orgapachemesos-1048 Please vote on releasing this package as Apache Mesos 0.22.0! The vote is open until Sat Mar 21 12:49:56 PDT 2015 and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Mesos 0.22.0 [ ] -1 Do not release this package because ... Thanks, Niklas -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links
Re: [VOTE] Release Apache Mesos 0.22.0 (rc4)
Hi Jeff, Integration tests are definitely desired/recommended. Some of us devs just do make [dist]check, but others test integrations with their favourite frameworks, or push it to their internal testing clusters. We're open to any additional testing you would like to propose and/or perform. Thanks, -Adam- On Mon, Mar 23, 2015 at 6:58 PM, Jeff Schroeder jeffschroe...@computer.org wrote: How does everyone verify each release works? Obviously make check is important, but are there any sets of automated integration tests that anyone runs for a new release of mesos? If not, should there be? On Monday, March 23, 2015, Dario Rexin da...@mesosphere.io wrote: +1 (non-binding) tested with Marathon On 24 Mar 2015, at 00:03, Adam Bordelon a...@mesosphere.io wrote: +1 (binding) make check passes on Ubuntu 14.04 and Mac OS X (buildbot) On Mon, Mar 23, 2015 at 3:44 PM, Elizabeth Lingg elizab...@mesosphere.io wrote: +1 (Tested with Chronos) -Elizabeth On Mon, Mar 23, 2015 at 3:40 PM, Ben Whitehead ben.whiteh...@mesosphere.io wrote: +1 (non-binding) openSUSE 13.2 Linux 3.16.7 / gcc-4.8.3 Tested running Marathon 0.8.1 and Cassandra on Mesos 0.1.0-SNAPSHOT. On Thu, Mar 19, 2015 at 2:18 PM, Alex Rukletsov a...@mesosphere.io wrote: +1 (non-binding) Mac OS 10.9.5 + clang CentOS 7 + gcc 4.4.7 [cgroups tests disabled] On Wed, Mar 18, 2015 at 4:04 PM, Brenden Matthews bren...@diddyinc.com wrote: +1 Tested with internal testing cluster. On Wed, Mar 18, 2015 at 1:25 PM, craig w codecr...@gmail.com wrote: +1 On Wed, Mar 18, 2015 at 3:52 PM, Niklas Nielsen nik...@mesosphere.io wrote: Hi all, Please vote on releasing the following candidate as Apache Mesos 0.22.0. 0.22.0 includes the following: * Support for explicitly sending status updates acknowledgements from schedulers; refer to the upgrades document for upgrading schedulers. * Rate limiting slave removal, to safeguard against unforeseen bugs leading to widespread slave removal. * Disk quota isolation in Mesos containerizer; refer to the containerization documentation to enable disk quota monitoring and enforcement. * Support for module hooks in task launch sequence. Refer to the modules documentation for more information. * Anonymous modules: a new kind of module that does not receive any callbacks but coexists with its parent process. * New service discovery info in task info allows framework users to specify discoverability of tasks for external service discovery systems. Refer to the framework development guide for more information. * New '--external_log_file' flag to serve external logs through the Mesos web UI. * New '--gc_disk_headroom' flag to control maxmimum executor sandbox age. The CHANGELOG for the release is available at: https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.22.0-rc4 The candidate for Mesos 0.22.0 release is available at: https://dist.apache.org/repos/dist/dev/mesos/0.22.0-rc4/mesos-0.22.0.tar.gz The tag to be voted on is 0.22.0-rc4: https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.22.0-rc4 The MD5 checksum of the tarball can be found at: https://dist.apache.org/repos/dist/dev/mesos/0.22.0-rc4/mesos-0.22.0.tar.gz.md5 The signature of the tarball can be found at: https://dist.apache.org/repos/dist/dev/mesos/0.22.0-rc4/mesos-0.22.0.tar.gz.asc The PGP key used to sign the release is here: https://dist.apache.org/repos/dist/release/mesos/KEYS The JAR is up in Maven in a staging repository here: https://repository.apache.org/content/repositories/orgapachemesos-1048 Please vote on releasing this package as Apache Mesos 0.22.0! The vote is open until Sat Mar 21 12:49:56 PDT 2015 and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Mesos 0.22.0 [ ] -1 Do not release this package because ... Thanks, Niklas -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links -- Text by Jeff, typos by iPhone
Re: [VOTE] Release Apache Mesos 0.22.0 (rc4)
How does everyone verify each release works? Obviously make check is important, but are there any sets of automated integration tests that anyone runs for a new release of mesos? If not, should there be? On Monday, March 23, 2015, Dario Rexin da...@mesosphere.io wrote: +1 (non-binding) tested with Marathon On 24 Mar 2015, at 00:03, Adam Bordelon a...@mesosphere.io javascript:_e(%7B%7D,'cvml','a...@mesosphere.io'); wrote: +1 (binding) make check passes on Ubuntu 14.04 and Mac OS X (buildbot) On Mon, Mar 23, 2015 at 3:44 PM, Elizabeth Lingg elizab...@mesosphere.io javascript:_e(%7B%7D,'cvml','elizab...@mesosphere.io'); wrote: +1 (Tested with Chronos) -Elizabeth On Mon, Mar 23, 2015 at 3:40 PM, Ben Whitehead ben.whiteh...@mesosphere.io javascript:_e(%7B%7D,'cvml','ben.whiteh...@mesosphere.io'); wrote: +1 (non-binding) openSUSE 13.2 Linux 3.16.7 / gcc-4.8.3 Tested running Marathon 0.8.1 and Cassandra on Mesos 0.1.0-SNAPSHOT. On Thu, Mar 19, 2015 at 2:18 PM, Alex Rukletsov a...@mesosphere.io javascript:_e(%7B%7D,'cvml','a...@mesosphere.io'); wrote: +1 (non-binding) Mac OS 10.9.5 + clang CentOS 7 + gcc 4.4.7 [cgroups tests disabled] On Wed, Mar 18, 2015 at 4:04 PM, Brenden Matthews bren...@diddyinc.com javascript:_e(%7B%7D,'cvml','bren...@diddyinc.com'); wrote: +1 Tested with internal testing cluster. On Wed, Mar 18, 2015 at 1:25 PM, craig w codecr...@gmail.com javascript:_e(%7B%7D,'cvml','codecr...@gmail.com'); wrote: +1 On Wed, Mar 18, 2015 at 3:52 PM, Niklas Nielsen nik...@mesosphere.io javascript:_e(%7B%7D,'cvml','nik...@mesosphere.io'); wrote: Hi all, Please vote on releasing the following candidate as Apache Mesos 0.22.0. 0.22.0 includes the following: * Support for explicitly sending status updates acknowledgements from schedulers; refer to the upgrades document for upgrading schedulers. * Rate limiting slave removal, to safeguard against unforeseen bugs leading to widespread slave removal. * Disk quota isolation in Mesos containerizer; refer to the containerization documentation to enable disk quota monitoring and enforcement. * Support for module hooks in task launch sequence. Refer to the modules documentation for more information. * Anonymous modules: a new kind of module that does not receive any callbacks but coexists with its parent process. * New service discovery info in task info allows framework users to specify discoverability of tasks for external service discovery systems. Refer to the framework development guide for more information. * New '--external_log_file' flag to serve external logs through the Mesos web UI. * New '--gc_disk_headroom' flag to control maxmimum executor sandbox age. The CHANGELOG for the release is available at: https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.22.0-rc4 The candidate for Mesos 0.22.0 release is available at: https://dist.apache.org/repos/dist/dev/mesos/0.22.0-rc4/mesos-0.22.0.tar.gz The tag to be voted on is 0.22.0-rc4: https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.22.0-rc4 The MD5 checksum of the tarball can be found at: https://dist.apache.org/repos/dist/dev/mesos/0.22.0-rc4/mesos-0.22.0.tar.gz.md5 The signature of the tarball can be found at: https://dist.apache.org/repos/dist/dev/mesos/0.22.0-rc4/mesos-0.22.0.tar.gz.asc The PGP key used to sign the release is here: https://dist.apache.org/repos/dist/release/mesos/KEYS The JAR is up in Maven in a staging repository here: https://repository.apache.org/content/repositories/orgapachemesos-1048 Please vote on releasing this package as Apache Mesos 0.22.0! The vote is open until Sat Mar 21 12:49:56 PDT 2015 and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Mesos 0.22.0 [ ] -1 Do not release this package because ... Thanks, Niklas -- https://github.com/mindscratch https://www.google.com/+CraigWickesser https://twitter.com/mind_scratch https://twitter.com/craig_links -- Text by Jeff, typos by iPhone
Re: [VOTE] Release Apache Mesos 0.22.0 (rc4)
I know it's over a year old and hasn't been updated, but bmahler already created a distcc framework example for Mesos. https://github.com/mesos/mesos-distcc On Mon, Mar 23, 2015 at 7:56 PM, CCAAT cc...@tampabay.rr.com wrote: On 03/23/2015 09:02 PM, Adam Bordelon wrote: Integration tests are definitely desired/recommended. Some of us devs just do make [dist]check, but others test integrations with their favourite frameworks, or push it to their internal testing clusters. We're open to any additional testing you would like to propose and/or perform. Thanks, -Adam- Distcc is a distributed compiling program that has been a long staple on Gentoo linux and many other distros. I work on Gentoo and I think that setting up distcc to run on a mesos cluster is a fabulous idea. Not only can you compile native binaries for the inherent arch, but cross compiling should work too. Everyone has to recompile (optimized) kernels frequently with the release cycle of the linux kernel team. With the current roll out of all sorts of 64 bit arm systems, there's going to be a great opportunity to cross compile arm64 bit codes on CISC (X86_64) bit clusters. This also starts the process of heterogeneous mesos clusters, surely inevitable. https://code.google.com/p/distcc/ https://code.google.com/p/distcc/downloads/list With LLvm, gnu and other projects, compiling and cross compiling on a mesos cluster is sure to be a very, very popoular idea. Any CI endeavor will necessitate lots of compiling too. hope this helps, James
Re: [VOTE] Release Apache Mesos 0.22.0 (rc4)
Excellent start! Nice links I was not aware of (thanks). Folks can use distcc now to test new rollouts of mesos? That was the quest of the thread was/is to establish some codes for testing new rollouts of mesos. It'll need to be 'extended' for cross compiling too for my needs. I'd like to follow up with anyone that get's distcc working with cross compiling for different arches. arm64 would be really cool. (LLVM?) My small cluster needs some work, much of it not related to mesos, but my efforts to run mesos and spark without the HDFS and use Cephfs, btrfs and supporting codes. So I'm not sure when I'll get this mesos-distcc installed and running in the near future; but I am most interested in follow the issues other encounter on compiling and cross compiling on a mesos cluster. What would be really cool is to run this distcc-mesos on top of spark and cephfs for some real fast compile times of large codes. Thanks, James On 03/23/2015 10:22 PM, Adam Bordelon wrote: I know it's over a year old and hasn't been updated, but bmahler already created a distcc framework example for Mesos. https://github.com/mesos/mesos-distcc On Mon, Mar 23, 2015 at 7:56 PM, CCAAT cc...@tampabay.rr.com mailto:cc...@tampabay.rr.com wrote: On 03/23/2015 09:02 PM, Adam Bordelon wrote: Integration tests are definitely desired/recommended. Some of us devs just do make [dist]check, but others test integrations with their favourite frameworks, or push it to their internal testing clusters. We're open to any additional testing you would like to propose and/or perform. Thanks, -Adam- Distcc is a distributed compiling program that has been a long staple on Gentoo linux and many other distros. I work on Gentoo and I think that setting up distcc to run on a mesos cluster is a fabulous idea. Not only can you compile native binaries for the inherent arch, but cross compiling should work too. Everyone has to recompile (optimized) kernels frequently with the release cycle of the linux kernel team. With the current roll out of all sorts of 64 bit arm systems, there's going to be a great opportunity to cross compile arm64 bit codes on CISC (X86_64) bit clusters. This also starts the process of heterogeneous mesos clusters, surely inevitable. https://code.google.com/p/__distcc/ https://code.google.com/p/distcc/ https://code.google.com/p/__distcc/downloads/list https://code.google.com/p/distcc/downloads/list With LLvm, gnu and other projects, compiling and cross compiling on a mesos cluster is sure to be a very, very popoular idea. Any CI endeavor will necessitate lots of compiling too. hope this helps, James
Re: [VOTE] Release Apache Mesos 0.22.0 (rc4)
On 03/23/2015 09:02 PM, Adam Bordelon wrote: Integration tests are definitely desired/recommended. Some of us devs just do make [dist]check, but others test integrations with their favourite frameworks, or push it to their internal testing clusters. We're open to any additional testing you would like to propose and/or perform. Thanks, -Adam- Distcc is a distributed compiling program that has been a long staple on Gentoo linux and many other distros. I work on Gentoo and I think that setting up distcc to run on a mesos cluster is a fabulous idea. Not only can you compile native binaries for the inherent arch, but cross compiling should work too. Everyone has to recompile (optimized) kernels frequently with the release cycle of the linux kernel team. With the current roll out of all sorts of 64 bit arm systems, there's going to be a great opportunity to cross compile arm64 bit codes on CISC (X86_64) bit clusters. This also starts the process of heterogeneous mesos clusters, surely inevitable. https://code.google.com/p/distcc/ https://code.google.com/p/distcc/downloads/list With LLvm, gnu and other projects, compiling and cross compiling on a mesos cluster is sure to be a very, very popoular idea. Any CI endeavor will necessitate lots of compiling too. hope this helps, James
