After set the timeout value, the tcp request become stable. 2015-03-23 23:26 GMT+08:00 John Omernik <j...@omernik.com>:
> So per my HA-Proxy logs, I am getting a condition where the client > side time expired while waiting for the client to send or receive > data, and this is occurring during the DATA phase. (the two letter > HAProxy code is cD . To test, I expanded the timeout (on all things) > to 100000. > > This was interesting in that it showed me the problem a bit clearer. > Now, as long as I submit and results for a query in the 100 second > range, my connection stays open and happy. If I let it sit for 100 > seconds, and then try a query, then it breaks again. > > It appears HA proxy gets irritable when things aren't being sent over > a connection for a long period of time. For services like Minecraft, > this is not an issue, when a client is connected, there is always data > there, and no issues occur. For Thrift, there may be multiple long > pauses as an analyst thinks through the next query etc. I wonder what > would be the best way to handle that on a per container level. (not > all containers need this sort of customization...) > > > > On Mon, Mar 23, 2015 at 5:44 AM, xiaokun <xiaokun...@gmail.com> wrote: > > I have a similar error when launch jenkins in mesos cluster. And I use > > HA-Proxy to route jenkins URL(and JNLP port) to a fixed one. But from > time > > to time, connection between jenkins master and jenkins slave will lost, > > while will cause jobs fail. > > > > I will try to set timeout value for HAProxy > > > > 2015-03-23 4:18 GMT+08:00 John Omernik <j...@omernik.com>: > >> > >> Yep all three are going through HAProxy and seem to be working well. > >> I should do some testing to determine if I run a thrift server that > >> connects to metastore in HAProxy to determine if that's the broken > >> link, but I believe it's between the thrift server. I will check out > >> the link you posted to get a better idea out of how it works. I am > >> running this all on a MapR cluster which is Great for Docker/Mesos as > >> I used the MapR FS, mounted locally on each node's NFS to put the data > >> for MySQL, the logs for the metastore/thift server, and my world files > >> for Minecraft. So far it's all really slick. > >> > >> On Sun, Mar 22, 2015 at 10:44 AM, Ben Whitehead > >> <ben.whiteh...@mesosphere.io> wrote: > >> > It sounds like you may be running into the connection timeout that is > >> > configured for the HAProxy instance. > >> > > >> > You can see an instance of this in Marathon's > >> > HAProxy-marathon-bridge[1]. > >> > You may need to raise the timeout values. > >> > > >> > Are you accessing MySql, Hive metastore and minecraft through HAProxy > as > >> > well? I know that Minecraft running in docker on mesos works well, > >> > though > >> > I've always used a direct connection not a proxied one. > >> > > >> > > >> > [1] > >> > > >> > > https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119 > >> > > >> > On Sat, Mar 21, 2015 at 1:50 PM, John Omernik <j...@omernik.com> > wrote: > >> >> > >> >> I have a nice setup with a Hive thrift server running in a docker > >> >> container on Mesos. It works pretty well, but something, I believe in > >> >> how HAProxy works with the connection, is causing the thrift server > >> >> connection to die after a time. Basically, I can run a few queries, > >> >> but after 2 or 3, or specifically after a longer query, I get the > >> >> error below indicating End of File on the connection. Then no more > >> >> connections work until I reestablish the connection to the thrift > >> >> server. I've tried looking in logs. the thrift server std err logs > >> >> show no issues. I am guess I need to dig into haproxy logs, but I am > >> >> not seeing any issues in syslog so far. I'd love any pointers on how > >> >> to trouble shoot this. By the way, I have MySQl, hive metastore, and > >> >> a minecraft server all running on Mesos/Docker with no issues, not > >> >> sure why the thrift server is so sensitive. > >> >> > >> >> :) > >> >> > >> >> > >> >> pyhs2 connection string: > >> >> > >> >> hs2 = > >> >> > >> >> > pyhs2.connect(host='marathonmaster',port=10000,authMechanism='PLAIN',user='bestuser',password='removed',database='default') > >> >> > >> >> > >> >> Error > >> >> > >> >> /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc > in > >> >> read(self, sz) > >> >> 118 if len(buff) == 0: > >> >> 119 raise > >> >> TTransportException(type=TTransportException.END_OF_FILE, > >> >> --> 120 message='TSocket read 0 > bytes') > >> >> 121 return buff > >> >> 122 > >> >> > >> >> TTransportException: TSocket read 0 bytes > >> > > >> > > > > > >
