Awesome glad to hear it. On Tue, Mar 24, 2015 at 10:00 PM, xiaokun <xiaokun...@gmail.com> wrote:
> After set the timeout value, the tcp request become stable. > > 2015-03-23 23:26 GMT+08:00 John Omernik <j...@omernik.com>: > >> So per my HA-Proxy logs, I am getting a condition where the client >> side time expired while waiting for the client to send or receive >> data, and this is occurring during the DATA phase. (the two letter >> HAProxy code is cD . To test, I expanded the timeout (on all things) >> to 100000. >> >> This was interesting in that it showed me the problem a bit clearer. >> Now, as long as I submit and results for a query in the 100 second >> range, my connection stays open and happy. If I let it sit for 100 >> seconds, and then try a query, then it breaks again. >> >> It appears HA proxy gets irritable when things aren't being sent over >> a connection for a long period of time. For services like Minecraft, >> this is not an issue, when a client is connected, there is always data >> there, and no issues occur. For Thrift, there may be multiple long >> pauses as an analyst thinks through the next query etc. I wonder what >> would be the best way to handle that on a per container level. (not >> all containers need this sort of customization...) >> >> >> >> On Mon, Mar 23, 2015 at 5:44 AM, xiaokun <xiaokun...@gmail.com> wrote: >> > I have a similar error when launch jenkins in mesos cluster. And I use >> > HA-Proxy to route jenkins URL(and JNLP port) to a fixed one. But from >> time >> > to time, connection between jenkins master and jenkins slave will lost, >> > while will cause jobs fail. >> > >> > I will try to set timeout value for HAProxy >> > >> > 2015-03-23 4:18 GMT+08:00 John Omernik <j...@omernik.com>: >> >> >> >> Yep all three are going through HAProxy and seem to be working well. >> >> I should do some testing to determine if I run a thrift server that >> >> connects to metastore in HAProxy to determine if that's the broken >> >> link, but I believe it's between the thrift server. I will check out >> >> the link you posted to get a better idea out of how it works. I am >> >> running this all on a MapR cluster which is Great for Docker/Mesos as >> >> I used the MapR FS, mounted locally on each node's NFS to put the data >> >> for MySQL, the logs for the metastore/thift server, and my world files >> >> for Minecraft. So far it's all really slick. >> >> >> >> On Sun, Mar 22, 2015 at 10:44 AM, Ben Whitehead >> >> <ben.whiteh...@mesosphere.io> wrote: >> >> > It sounds like you may be running into the connection timeout that is >> >> > configured for the HAProxy instance. >> >> > >> >> > You can see an instance of this in Marathon's >> >> > HAProxy-marathon-bridge[1]. >> >> > You may need to raise the timeout values. >> >> > >> >> > Are you accessing MySql, Hive metastore and minecraft through >> HAProxy as >> >> > well? I know that Minecraft running in docker on mesos works well, >> >> > though >> >> > I've always used a direct connection not a proxied one. >> >> > >> >> > >> >> > [1] >> >> > >> >> > >> https://github.com/mesosphere/marathon/blob/master/bin/haproxy-marathon-bridge#L113-L119 >> >> > >> >> > On Sat, Mar 21, 2015 at 1:50 PM, John Omernik <j...@omernik.com> >> wrote: >> >> >> >> >> >> I have a nice setup with a Hive thrift server running in a docker >> >> >> container on Mesos. It works pretty well, but something, I believe >> in >> >> >> how HAProxy works with the connection, is causing the thrift server >> >> >> connection to die after a time. Basically, I can run a few >> queries, >> >> >> but after 2 or 3, or specifically after a longer query, I get the >> >> >> error below indicating End of File on the connection. Then no more >> >> >> connections work until I reestablish the connection to the thrift >> >> >> server. I've tried looking in logs. the thrift server std err logs >> >> >> show no issues. I am guess I need to dig into haproxy logs, but I am >> >> >> not seeing any issues in syslog so far. I'd love any pointers on >> how >> >> >> to trouble shoot this. By the way, I have MySQl, hive metastore, >> and >> >> >> a minecraft server all running on Mesos/Docker with no issues, not >> >> >> sure why the thrift server is so sensitive. >> >> >> >> >> >> :) >> >> >> >> >> >> >> >> >> pyhs2 connection string: >> >> >> >> >> >> hs2 = >> >> >> >> >> >> >> pyhs2.connect(host='marathonmaster',port=10000,authMechanism='PLAIN',user='bestuser',password='removed',database='default') >> >> >> >> >> >> >> >> >> Error >> >> >> >> >> >> /usr/local/lib/python2.7/dist-packages/thrift/transport/TSocket.pyc >> in >> >> >> read(self, sz) >> >> >> 118 if len(buff) == 0: >> >> >> 119 raise >> >> >> TTransportException(type=TTransportException.END_OF_FILE, >> >> >> --> 120 message='TSocket read 0 >> bytes') >> >> >> 121 return buff >> >> >> 122 >> >> >> >> >> >> TTransportException: TSocket read 0 bytes >> >> > >> >> > >> > >> > >> > >
