On Wed, Sep 7, 2016 at 11:22 PM, Diego Castro <diego.cas...@getupcloud.com> wrote:
> Hello, list. > We have been running Origin since last November and i'd like to share some > experiences, pains and thoughts. > > Our origin cluster has about 25 servers including masters,nodes and > routers. We have roughly 500 applications exposing services and a bunch of > HPA firing up containers all the time. > > 1) Resource consumption: i noticed during the day a increase of memory > consumption due multiple reloads, a lot of process keep running until the > connections is finished or OOM kill. Other issue regarding restarts is that > due to TCP SYN DROP iptables we are facing some high latencies. What can > we do to reduce restart overhead ? > You seem to have several questions intertwined here, and I am by no means an expert on this, but on the "lots of processes keep running" topic, you may be hitting https://bugzilla.redhat.com/show_bug.cgi?id=1364870 (though this manifests as more of a CPU consumption issue than a memory issue). In short, what we've seen is cases where haproxy connections are "orphaned", so the old processes never exit -- they continuously think they have one or two "jobs" left, but they never actually handle them. I think this is fixed in the latest 1.5.x release of haproxy, but have not had a chance to test yet. > > 2) Metrics: Would be nice to pull some metrics from the routers, something > like general network i/o and per endpoint traffic, i found a prometheus > export but due to process restart the endpoint states are cleaned. HAProxy > 1.6 have a fix for that (http://blog.haproxy.com/2015/ > 10/14/whats-new-in-haproxy-1-6/). Do we have plans to upgrade to 1.6 ? > What kind of metrics do we have available today? > > > --- > Diego Castro / The CloudFather > GetupCloud.com - Eliminamos a Gravidade > > _______________________________________________ > users mailing list > users@lists.openshift.redhat.com > http://lists.openshift.redhat.com/openshiftmm/listinfo/users > >
_______________________________________________ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
