I have a similar error when launch jenkins in mesos cluster. And I use HA-Proxy to route jenkins URL(and JNLP port) to a fixed one. But from time to time, connection between jenkins master and jenkins slave will lost, while will cause jobs fail.
I will try to set timeout value for HAProxy 2015-03-23 4:18 GMT+08:00 John Omernik <j...@omernik.com>: > Yep all three are going through HAProxy and seem to be working well. > I should do some testing to determine if I run a thrift server that > connects to metastore in HAProxy to determine if that's the broken > link, but I believe it's between the thrift server. I will check out > the link you posted to get a better idea out of how it works. I am > running this all on a MapR cluster which is Great for Docker/Mesos as > I used the MapR FS, mounted locally on each node's NFS to put the data > for MySQL, the logs for the metastore/thift server, and my world files > for Minecraft. So far it's all really slick. > > On Sun, Mar 22, 2015 at 10:44 AM, Ben Whitehead > <ben.whiteh...@mesosphere.io> wrote: > > It sounds like you may be running into the connection timeout that is > > configured for the HAProxy instance. > > > > You can see an instance of this in Marathon's HAProxy-marathon-bridge[1]. > > You may need to raise the timeout values. > > > > Are you accessing MySql, Hive metastore and minecraft through HAProxy as > > well? I know that Minecraft running in docker on mesos works well, though > > I've always used a direct connection not a proxied one. > > > > > > [1] > > > https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119 > > > > On Sat, Mar 21, 2015 at 1:50 PM, John Omernik <j...@omernik.com> wrote: > >> > >> I have a nice setup with a Hive thrift server running in a docker > >> container on Mesos. It works pretty well, but something, I believe in > >> how HAProxy works with the connection, is causing the thrift server > >> connection to die after a time. Basically, I can run a few queries, > >> but after 2 or 3, or specifically after a longer query, I get the > >> error below indicating End of File on the connection. Then no more > >> connections work until I reestablish the connection to the thrift > >> server. I've tried looking in logs. the thrift server std err logs > >> show no issues. I am guess I need to dig into haproxy logs, but I am > >> not seeing any issues in syslog so far. I'd love any pointers on how > >> to trouble shoot this. By the way, I have MySQl, hive metastore, and > >> a minecraft server all running on Mesos/Docker with no issues, not > >> sure why the thrift server is so sensitive. > >> > >> :) > >> > >> > >> pyhs2 connection string: > >> > >> hs2 = > >> > pyhs2.connect(host='marathonmaster',port=10000,authMechanism='PLAIN',user='bestuser',password='removed',database='default') > >> > >> > >> Error > >> > >> /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc in > >> read(self, sz) > >> 118 if len(buff) == 0: > >> 119 raise > >> TTransportException(type=TTransportException.END_OF_FILE, > >> --> 120 message='TSocket read 0 bytes') > >> 121 return buff > >> 122 > >> > >> TTransportException: TSocket read 0 bytes > > > > >
