Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
Justin Clift jus...@gluster.org wrote: If the DNS problem does turn out to be the dodgy iWeb hardware firewall, then this fixes the DNS issue. (if not... well damn!) The DNS problem was worked around by installing a /etc/hosts, but jenkins does not realize it is there. It should probably be restarted, but nobody dare to try. -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz m...@netbsd.org ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
On Thu, Jun 18, 2015 at 12:57:05AM +0100, Justin Clift wrote: On 17 Jun 2015, at 20:14, Niels de Vos nde...@redhat.com wrote: On Wed, Jun 17, 2015 at 03:14:31PM +0200, Michael Scherer wrote: Le mercredi 17 juin 2015 à 11:58 +0100, Justin Clift a écrit : On 17 Jun 2015, at 10:53, Michael Scherer msche...@redhat.com wrote: Le mercredi 17 juin 2015 à 11:48 +0200, Michael Scherer a écrit : Le mercredi 17 juin 2015 à 08:20 +0200, Emmanuel Dreyfus a écrit : Venky Shankar yknev.shan...@gmail.com wrote: If that's the case, then I'll vote for this even if it takes some time to get things in workable state. See my other mail about this: you enter a new slave VM in the DNS and it does not resolve, or somethimes you get 20s delays. I am convinced this is the reason why Jenkins bugs. But cloud.gluster.org is handled by rackspace, not sure how much control we have for it ( not sure even where to start there ). So I cannot change the DNS destination. What I can do is to create a new dns zone, and then, we can delegate as we want. And migrate some slaves and not others, and see how it goes ? slaves.gluster.org would be ok for everybody ? Try it out, and see if it works. :) On the scaling the infrastructure side of things, are the two OSAS servers for Gluster still available? They are online. $ ssh r...@ci.gluster.org uptime 09:13:37 up 33 days, 16:34, 0 users, load average: 0,00, 0,01, 0,05 Can it run some Jenkins Slave VMs too? There are two boxes. A pretty beefy one for running Jenkins slave VM's (probably about 40 VM's simultaneously), and a slightly less beefy one for running Jenkins/Gerrit/whatever. Good to know, but it would be much more helpful if someone could install VMs there and add them to the Jenkins instance... Who can do that, or who can guide someone else to get it done? Thanks, Niels ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
On Wed, Jun 17, 2015 at 12:13:46PM +, Emmanuel Dreyfus wrote: On Wed, Jun 17, 2015 at 07:44:14AM -0400, Vijay Bellur wrote: Do we still have the NFS crash that was causing tests to hang? Do we still have it on rebased patchsets? Yes, the fixes depend on the refcounting change which does not seem as trivial as I hoped. http://review.gluster.org/11022 for the interested. http://review.gluster.org/11023 is the fix that should solve the segfaults in the NFS-server. Niels ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
On Wed, Jun 17, 2015 at 11:48:46AM +0200, Michael Scherer wrote: Le mercredi 17 juin 2015 à 08:20 +0200, Emmanuel Dreyfus a écrit : Venky Shankar yknev.shan...@gmail.com wrote: If that's the case, then I'll vote for this even if it takes some time to get things in workable state. See my other mail about this: you enter a new slave VM in the DNS and it does not resolve, or somethimes you get 20s delays. I am convinced this is the reason why Jenkins bugs. But cloud.gluster.org is handled by rackspace, not sure how much control we have for it ( not sure even where to start there ). On build.gluster.org there now is a /usr/local/bin/get-hosts.py script (needs to be executed through sude). This pulls down the DNS records from our cloud.gluster.org domain in Rackspace and proves a /etc/hosts formatted output. /etc/hosts on build.gluster.org contains all the current entries. We could automatically update it with a cron job or something, if needed. New VMs should get added to /etc/hosts too, either manually or by executing the script (sudo vim /etc/hosts, :r!/usr/local/bin/get-hosts.py). And I think the DNS issues are just a symptom of a bigger network issue, having local DNS might just mask the problem and which would then be non DNS related ( like tcp connexion not working ). Maybe, but I hope those issues stay masked when resolving the hostnames is more stable. When we have the other servers up and running, we would have a better understanding and options to investigate issues like this. HTH, Niels ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
Niels de Vos nde...@redhat.com wrote: Maybe, but I hope those issues stay masked when resolving the hostnames is more stable. When we have the other servers up and running, we would have a better understanding and options to investigate issues like this. But Jenkins is still unable to launch an agent on e.g. nbslave75. Perhaps it needs to be restarted? -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz m...@netbsd.org ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
On Wed, Jun 17, 2015 at 03:14:31PM +0200, Michael Scherer wrote: Le mercredi 17 juin 2015 à 11:58 +0100, Justin Clift a écrit : On 17 Jun 2015, at 10:53, Michael Scherer msche...@redhat.com wrote: Le mercredi 17 juin 2015 à 11:48 +0200, Michael Scherer a écrit : Le mercredi 17 juin 2015 à 08:20 +0200, Emmanuel Dreyfus a écrit : Venky Shankar yknev.shan...@gmail.com wrote: If that's the case, then I'll vote for this even if it takes some time to get things in workable state. See my other mail about this: you enter a new slave VM in the DNS and it does not resolve, or somethimes you get 20s delays. I am convinced this is the reason why Jenkins bugs. But cloud.gluster.org is handled by rackspace, not sure how much control we have for it ( not sure even where to start there ). So I cannot change the DNS destination. What I can do is to create a new dns zone, and then, we can delegate as we want. And migrate some slaves and not others, and see how it goes ? slaves.gluster.org would be ok for everybody ? Try it out, and see if it works. :) On the scaling the infrastructure side of things, are the two OSAS servers for Gluster still available? They are online. $ ssh r...@ci.gluster.org uptime 09:13:37 up 33 days, 16:34, 0 users, load average: 0,00, 0,01, 0,05 Can it run some Jenkins Slave VMs too? Thanks, Niels ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
On Wed, Jun 17, 2015 at 09:56:32PM +0200, Emmanuel Dreyfus wrote: Niels de Vos nde...@redhat.com wrote: Maybe, but I hope those issues stay masked when resolving the hostnames is more stable. When we have the other servers up and running, we would have a better understanding and options to investigate issues like this. But Jenkins is still unable to launch an agent on e.g. nbslave75. Perhaps it needs to be restarted? Yes, a Jenkins restart might be good. But, I do not know how it gets stopped safely, or started. Niels ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
On 17 Jun 2015, at 20:14, Niels de Vos nde...@redhat.com wrote: On Wed, Jun 17, 2015 at 03:14:31PM +0200, Michael Scherer wrote: Le mercredi 17 juin 2015 à 11:58 +0100, Justin Clift a écrit : On 17 Jun 2015, at 10:53, Michael Scherer msche...@redhat.com wrote: Le mercredi 17 juin 2015 à 11:48 +0200, Michael Scherer a écrit : Le mercredi 17 juin 2015 à 08:20 +0200, Emmanuel Dreyfus a écrit : Venky Shankar yknev.shan...@gmail.com wrote: If that's the case, then I'll vote for this even if it takes some time to get things in workable state. See my other mail about this: you enter a new slave VM in the DNS and it does not resolve, or somethimes you get 20s delays. I am convinced this is the reason why Jenkins bugs. But cloud.gluster.org is handled by rackspace, not sure how much control we have for it ( not sure even where to start there ). So I cannot change the DNS destination. What I can do is to create a new dns zone, and then, we can delegate as we want. And migrate some slaves and not others, and see how it goes ? slaves.gluster.org would be ok for everybody ? Try it out, and see if it works. :) On the scaling the infrastructure side of things, are the two OSAS servers for Gluster still available? They are online. $ ssh r...@ci.gluster.org uptime 09:13:37 up 33 days, 16:34, 0 users, load average: 0,00, 0,01, 0,05 Can it run some Jenkins Slave VMs too? There are two boxes. A pretty beefy one for running Jenkins slave VM's (probably about 40 VM's simultaneously), and a slightly less beefy one for running Jenkins/Gerrit/whatever. + Justin -- GlusterFS - http://www.gluster.org An open source, distributed file system scaling to several petabytes, and handling thousands of clients. My personal twitter: twitter.com/realjustinclift ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
On Wednesday 17 June 2015 04:23 PM, Niels de Vos wrote: On Wed, Jun 17, 2015 at 09:56:32PM +0200, Emmanuel Dreyfus wrote: Niels de Vos nde...@redhat.com wrote: Maybe, but I hope those issues stay masked when resolving the hostnames is more stable. When we have the other servers up and running, we would have a better understanding and options to investigate issues like this. But Jenkins is still unable to launch an agent on e.g. nbslave75. Perhaps it needs to be restarted? Yes, a Jenkins restart might be good. But, I do not know how it gets stopped safely, or started. The only downside of a Jenkins restart is that we would need to manually re-trigger all existing jobs. Shall we just do that now? -Vijay ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
On Wed, Jun 17, 2015 at 07:44:14AM -0400, Vijay Bellur wrote: Do we still have the NFS crash that was causing tests to hang? Do we still have it on rebased patchsets? -- Emmanuel Dreyfus m...@netbsd.org ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
On Wednesday 17 June 2015 08:13 AM, Emmanuel Dreyfus wrote: On Wed, Jun 17, 2015 at 07:44:14AM -0400, Vijay Bellur wrote: Do we still have the NFS crash that was causing tests to hang? Do we still have it on rebased patchsets? I am not certain. I am still trying to come to terms with my email backlog and hence seeking a quick opinion here to see if we need to address it asap. -Vijay ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
cloud.gluster.org is served by Rackspace Cloud DNS. AFAICT, there is no readily available option to do zone transfers from it. We might have to contact the Rackspace support to find out if they can do it as a special request. On Wed, Jun 17, 2015 at 11:50 AM, Emmanuel Dreyfus m...@netbsd.org wrote: Venky Shankar yknev.shan...@gmail.com wrote: If that's the case, then I'll vote for this even if it takes some time to get things in workable state. See my other mail about this: you enter a new slave VM in the DNS and it does not resolve, or somethimes you get 20s delays. I am convinced this is the reason why Jenkins bugs. -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz m...@netbsd.org ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
Venky Shankar yknev.shan...@gmail.com wrote: If that's the case, then I'll vote for this even if it takes some time to get things in workable state. See my other mail about this: you enter a new slave VM in the DNS and it does not resolve, or somethimes you get 20s delays. I am convinced this is the reason why Jenkins bugs. -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz m...@netbsd.org ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
- Original Message - From: Kaushal M kshlms...@gmail.com To: Emmanuel Dreyfus m...@netbsd.org Cc: Gluster Devel gluster-de...@gluster.org, gluster-infra gluster-infra@gluster.org Sent: Wednesday, 17 June, 2015 11:59:22 AM Subject: Re: [Gluster-devel] [Gluster-infra] NetBSD regressions not being triggered for patches cloud.gluster.org is served by Rackspace Cloud DNS. AFAICT, there is no readily available option to do zone transfers from it. We might have to contact the Rackspace support to find out if they can do it as a special request. If this is going to take time then I prefer not to block patches for NetBSD. We can address any NetBSD regression caused by patches as a separate bug. Otherwise our regression queue will continue to grow. On Wed, Jun 17, 2015 at 11:50 AM, Emmanuel Dreyfus m...@netbsd.org wrote: Venky Shankar yknev.shan...@gmail.com wrote: If that's the case, then I'll vote for this even if it takes some time to get things in workable state. See my other mail about this: you enter a new slave VM in the DNS and it does not resolve, or somethimes you get 20s delays. I am convinced this is the reason why Jenkins bugs. -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz m...@netbsd.org ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra ___ Gluster-devel mailing list gluster-de...@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
- Original Message - From: Avra Sengupta aseng...@redhat.com To: Rajesh Joseph rjos...@redhat.com, Kaushal M kshlms...@gmail.com Cc: Gluster Devel gluster-de...@gluster.org, gluster-infra gluster-infra@gluster.org Sent: Wednesday, June 17, 2015 1:42:25 PM Subject: Re: [Gluster-devel] [Gluster-infra] NetBSD regressions not being triggered for patches On 06/17/2015 12:12 PM, Rajesh Joseph wrote: - Original Message - From: Kaushal M kshlms...@gmail.com To: Emmanuel Dreyfus m...@netbsd.org Cc: Gluster Devel gluster-de...@gluster.org, gluster-infra gluster-infra@gluster.org Sent: Wednesday, 17 June, 2015 11:59:22 AM Subject: Re: [Gluster-devel] [Gluster-infra] NetBSD regressions not being triggered for patches cloud.gluster.org is served by Rackspace Cloud DNS. AFAICT, there is no readily available option to do zone transfers from it. We might have to contact the Rackspace support to find out if they can do it as a special request. If this is going to take time then I prefer not to block patches for NetBSD. We can address any NetBSD regression caused by patches as a separate bug. Otherwise our regression queue will continue to grow. +1 for this. We shouldn't be blocking patches for NetBSD regression till the infra scales enough to handle the kind of load we are throwing at it. Once the regression framework is scalable enough, we can fix any regressions (if any) introduced. This will bring down the turnaround time, for the patch acceptance. +1 On Wed, Jun 17, 2015 at 11:50 AM, Emmanuel Dreyfus m...@netbsd.org wrote: Venky Shankar yknev.shan...@gmail.com wrote: If that's the case, then I'll vote for this even if it takes some time to get things in workable state. See my other mail about this: you enter a new slave VM in the DNS and it does not resolve, or somethimes you get 20s delays. I am convinced this is the reason why Jenkins bugs. -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz m...@netbsd.org ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra ___ Gluster-devel mailing list gluster-de...@gluster.org http://www.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
On Wed, Jun 17, 2015 at 11:05:38AM +0200, Niels de Vos wrote: I've already scripted the reboot-vm job to use Rackspace API, the DNS requesting and formatting the results into some file can't be that difficult. Let me know if a /etc/hosts format would do, or if you expect something else. Perhaps a /etc/hosts would do it: jenkins launches the ssh command, and ssh should use /etc/hosts before the DNS. -- Emmanuel Dreyfus m...@netbsd.org ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
Le mercredi 17 juin 2015 à 11:48 +0200, Michael Scherer a écrit : Le mercredi 17 juin 2015 à 08:20 +0200, Emmanuel Dreyfus a écrit : Venky Shankar yknev.shan...@gmail.com wrote: If that's the case, then I'll vote for this even if it takes some time to get things in workable state. See my other mail about this: you enter a new slave VM in the DNS and it does not resolve, or somethimes you get 20s delays. I am convinced this is the reason why Jenkins bugs. But cloud.gluster.org is handled by rackspace, not sure how much control we have for it ( not sure even where to start there ). So I cannot change the DNS destination. What I can do is to create a new dns zone, and then, we can delegate as we want. And migrate some slaves and not others, and see how it goes ? slaves.gluster.org would be ok for everybody ? -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
On 06/11/2015 08:04 PM, Emmanuel Dreyfus wrote: On Thu, Jun 11, 2015 at 04:04:44PM +0200, Niels de Vos wrote: Michael installed and configured dnsmasq on build.gluster.org yesterday. If that does not help today, we need other ideas... Just to confirm the problem: [manu@build ~]$ time nslookup nbslave7i.cloud.gluster.org ;; connection timed out; trying next origin ;; connection timed out; no servers could be reached real0m20.013s user0m0.002s sys 0m0.012s Having a local cache does not help because upstream DNS service is weak. Without the local cache, individual processes crave for a reply, and with the local server, the local server crave itself crave for a reply. And here upstream DNS is really at fault: at mine I get a reply in 0.29s. We need to configure a local authoritative secondary DNS for the zone, so that the answer is always available locally wihtout having to rely on outside's infrastructure. I am not sure whether we have any improvements on this front. I still see patches are waiting for ages to get their turn for the regression run and hence delaying merges and effecting the release process. I still feel we don't need to wait for NetBSD's vote for merging patches on a temporary basis till we fix the infrastructure problem. This is the only quick solution which I can think of now. Thoughts? ~Atin -- ~Atin ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
Le jeudi 11 juin 2015 à 14:34 +, Emmanuel Dreyfus a écrit : On Thu, Jun 11, 2015 at 04:04:44PM +0200, Niels de Vos wrote: Michael installed and configured dnsmasq on build.gluster.org yesterday. If that does not help today, we need other ideas... Just to confirm the problem: [manu@build ~]$ time nslookup nbslave7i.cloud.gluster.org ;; connection timed out; trying next origin ;; connection timed out; no servers could be reached real0m20.013s user0m0.002s sys 0m0.012s Having a local cache does not help because upstream DNS service is weak. Without the local cache, individual processes crave for a reply, and with the local server, the local server crave itself crave for a reply. And here upstream DNS is really at fault: at mine I get a reply in 0.29s. We need to configure a local authoritative secondary DNS for the zone, so that the answer is always available locally wihtout having to rely on outside's infrastructure. So, I am not sure I fully follow the issue. Here, the time nslookup is fast (for now). And the dnsmsq cache use google dns rather than iweb provided one. So when you say upstream dns server is bad, which one do you mean exactly ? ( cloud.gluster.org is handled by rackspace, we could move to named managed locally, but then, this will not be self service anymore ) -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
Le vendredi 12 juin 2015 à 12:02 +0200, Michael Scherer a écrit : Le jeudi 11 juin 2015 à 14:34 +, Emmanuel Dreyfus a écrit : On Thu, Jun 11, 2015 at 04:04:44PM +0200, Niels de Vos wrote: Michael installed and configured dnsmasq on build.gluster.org yesterday. If that does not help today, we need other ideas... Just to confirm the problem: [manu@build ~]$ time nslookup nbslave7i.cloud.gluster.org ;; connection timed out; trying next origin ;; connection timed out; no servers could be reached real0m20.013s user0m0.002s sys 0m0.012s Having a local cache does not help because upstream DNS service is weak. Without the local cache, individual processes crave for a reply, and with the local server, the local server crave itself crave for a reply. And here upstream DNS is really at fault: at mine I get a reply in 0.29s. We need to configure a local authoritative secondary DNS for the zone, so that the answer is always available locally wihtout having to rely on outside's infrastructure. So, I am not sure I fully follow the issue. Here, the time nslookup is fast (for now). And the dnsmsq cache use google dns rather than iweb provided one. So when you say upstream dns server is bad, which one do you mean exactly ? ( cloud.gluster.org is handled by rackspace, we could move to named managed locally, but then, this will not be self service anymore ) So, I suspect some device between build.gluster;org and the DNS, because what I see is indeed that on a regular basis, jenkins dns resolution fail without a obvious pattern. IMHO, the long term solution is to move out of iweb, or to fix the device/network. Short term, I guess having a local dump of cloud.gluster.org would work; This requires scripting I guess. http://docs.rackspace.com/cdns/api/v1.0/cdns-getting-started/content/DNS_Overview.html -- Michael Scherer Sysadmin, Community Infrastructure and Platform, OSAS signature.asc Description: This is a digitally signed message part ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
Re: [Gluster-infra] [Gluster-devel] NetBSD regressions not being triggered for patches
On Thu, Jun 11, 2015 at 12:57:58PM +0530, Kaushal M wrote: The problem was nbslave71. It used to be picked first for all changes and would fail instantly. I've disabled it now. The other slaves are working correctly. Saddly the Jenkins upgrade did not help here. Last time I investigated the failure was caused by the master breaking connexion, but I was not able to understand why. I w once able to receover a VM by fiddeling the jenkins configuration in web UI, but experimenting is not easy, as a miss will drain all the queue into complete failures. -- Emmanuel Dreyfus m...@netbsd.org ___ Gluster-infra mailing list Gluster-infra@gluster.org http://www.gluster.org/mailman/listinfo/gluster-infra
