Feature Request for log stdout ...
Hi. how difficult is it to be able to add "log stdout;" to haproxy? I ask because in some PaaS environment is it difficult to setup a dedicated user yust for haproxy. It fits also a little bit better to http://12factor.net/logs BR Aleks
Re: Feature Request for log stdout ...
Hi Aleks, On Wed, Feb 17, 2016 at 04:30:06PM +0100, Aleksandar Lazic wrote: > Hi. > > how difficult is it to be able to add "log stdout;" to haproxy? > > I ask because in some PaaS environment is it difficult to setup a > dedicated user yust for haproxy. > > It fits also a little bit better to http://12factor.net/logs It's been discussed a few times in the past. The response is "no". It's totally insane to emit logs to a blocking destination. Your whole haproxy process will run at the speed of the logs consumer and the log processing will incure its latency to the process. If one day we implement an synchronous stream logging task, this could change, but for now we send immediate logs as datagrams in order never to block. To get an idea about what it can look like with blocking logs, simply run "haproxy -d 2>&1 | more" and don't press any key. You'll quickly see that the system continues to accept new connections and that they will randomly freeze at various steps. Regards, Willy
Re: Feature Request for log stdout ...
On Thu, Feb 18, 2016 at 10:57 AM, Willy Tarreau wrote: > Hi Aleks, > > On Wed, Feb 17, 2016 at 04:30:06PM +0100, Aleksandar Lazic wrote: >> Hi. >> >> how difficult is it to be able to add "log stdout;" to haproxy? >> >> I ask because in some PaaS environment is it difficult to setup a >> dedicated user yust for haproxy. >> >> It fits also a little bit better to http://12factor.net/logs > > It's been discussed a few times in the past. The response is "no". > It's totally insane to emit logs to a blocking destination. Your > whole haproxy process will run at the speed of the logs consumer > and the log processing will incure its latency to the process. > > If one day we implement an synchronous stream logging task, this > could change, but for now we send immediate logs as datagrams in > order never to block. > > To get an idea about what it can look like with blocking logs, > simply run "haproxy -d 2>&1 | more" and don't press any key. > You'll quickly see that the system continues to accept new > connections and that they will randomly freeze at various steps. > > Regards, > Willy > > My 2 cents: Some tools may be used for this purpose: Configure HAProxy to send logs to port 2000, then use: - socat: socat -u UDP-RECV:2000 - <133>Feb 18 11:27:02 haproxy[4134]: Proxy f started. <133>Feb 18 11:27:02 haproxy[4134]: Proxy b started. <133>Feb 18 11:27:02 haproxy[4134]: Proxy stats started. <129>Feb 18 11:27:02 haproxy[4134]: Server b/s is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. <128>Feb 18 11:27:02 haproxy[4134]: backend b has no server available! - netcat: netcat -l -k -u 2000 <133>Feb 18 11:28:17 haproxy[4303]: Proxy f started. <133>Feb 18 11:28:17 haproxy[4303]: Proxy b started. <133>Feb 18 11:28:17 haproxy[4303]: Proxy stats started. <129>Feb 18 11:28:17 haproxy[4303]: Server b/s is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. <128>Feb 18 11:28:17 haproxy[4303]: backend b has no server available! Baptiste
Re: Feature Request for log stdout ...
Two more cents from my side: socklog [1] also works pretty well... [1] http://smarden.org/socklog/ Conrad On 02/18/2016 11:28 AM, Baptiste wrote: > On Thu, Feb 18, 2016 at 10:57 AM, Willy Tarreau wrote: >> Hi Aleks, >> >> On Wed, Feb 17, 2016 at 04:30:06PM +0100, Aleksandar Lazic wrote: >>> Hi. >>> >>> how difficult is it to be able to add "log stdout;" to haproxy? >>> >>> I ask because in some PaaS environment is it difficult to setup a >>> dedicated user yust for haproxy. >>> >>> It fits also a little bit better to http://12factor.net/logs >> >> It's been discussed a few times in the past. The response is "no". >> It's totally insane to emit logs to a blocking destination. Your >> whole haproxy process will run at the speed of the logs consumer >> and the log processing will incure its latency to the process. >> >> If one day we implement an synchronous stream logging task, this >> could change, but for now we send immediate logs as datagrams in >> order never to block. >> >> To get an idea about what it can look like with blocking logs, >> simply run "haproxy -d 2>&1 | more" and don't press any key. >> You'll quickly see that the system continues to accept new >> connections and that they will randomly freeze at various steps. >> >> Regards, >> Willy >> >> > > My 2 cents: Some tools may be used for this purpose: > > Configure HAProxy to send logs to port 2000, then use: > > - socat: > socat -u UDP-RECV:2000 - > <133>Feb 18 11:27:02 haproxy[4134]: Proxy f started. > <133>Feb 18 11:27:02 haproxy[4134]: Proxy b started. > <133>Feb 18 11:27:02 haproxy[4134]: Proxy stats started. > <129>Feb 18 11:27:02 haproxy[4134]: Server b/s is DOWN, reason: Layer4 > connection problem, info: "Connection refused", check duration: 0ms. 0 > active and 0 backup servers left. 0 sessions active, 0 requeued, 0 > remaining in queue. > <128>Feb 18 11:27:02 haproxy[4134]: backend b has no server available! > > - netcat: > netcat -l -k -u 2000 > <133>Feb 18 11:28:17 haproxy[4303]: Proxy f started. > <133>Feb 18 11:28:17 haproxy[4303]: Proxy b started. > <133>Feb 18 11:28:17 haproxy[4303]: Proxy stats started. > <129>Feb 18 11:28:17 haproxy[4303]: Server b/s is DOWN, reason: Layer4 > connection problem, info: "Connection refused", check duration: 0ms. 0 > active and 0 backup servers left. 0 sessions active, 0 requeued, 0 > remaining in queue. > <128>Feb 18 11:28:17 haproxy[4303]: backend b has no server available! > > > > Baptiste > -- Conrad Hoffmann Traffic Engineer SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany Managing Director: Alexander Ljung | Incorporated in England & Wales with Company No. 6343600 | Local Branch Office | AG Charlottenburg | HRB 110657B
Re: Feature Request for log stdout ...
On Thu, Feb 18, 2016 at 11:47:03AM +0100, Conrad Hoffmann wrote: > Two more cents from my side: > > socklog [1] also works pretty well... > > [1] http://smarden.org/socklog/ Thanks for the links guys. These should probably be added to the management doc so that people find them more easily. willy
Re: Feature Request for log stdout ...
Hi. Am 18-02-2016 11:47, schrieb Conrad Hoffmann: Two more cents from my side: socklog [1] also works pretty well... [1] http://smarden.org/socklog/ Conrad On 02/18/2016 11:28 AM, Baptiste wrote: On Thu, Feb 18, 2016 at 10:57 AM, Willy Tarreau wrote: Hi Aleks, On Wed, Feb 17, 2016 at 04:30:06PM +0100, Aleksandar Lazic wrote: Hi. how difficult is it to be able to add "log stdout;" to haproxy? [snipp] It's been discussed a few times in the past. The response is "no". It's totally insane to emit logs to a blocking destination. Your whole haproxy process will run at the speed of the logs consumer and the log processing will incure its latency to the process. [snipp] My 2 cents: Some tools may be used for this purpose: Configure HAProxy to send logs to port 2000, then use: - socat: socat -u UDP-RECV:2000 - [snipp] - netcat: netcat -l -k -u 2000 [snipp] Thanks for answers and suggestions. But this moves just the stdout handling to other tools and does not solve the problem with blocking handling of std*, as far as I have understood right. It also 'violates' the best practice of docker. https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#run-only-one-process-per-container Okay this could be solved with the linking as described in the link. For openshift I will try to use 2 container in 1 pod. If there any interests I can write here if this works ;-) BR Aleks
Re: Feature Request for log stdout ...
Aleksandar, Shouldn’t your setup allow for one central syslog type container that collects all logs sent to it from the network? I think it beats solutions that will handle the write to file after HAProxy and it certainly respects the one process per container best practice. Pedro. > On 18 Feb 2016, at 13:53, Aleksandar Lazic wrote: > > Hi. > > Am 18-02-2016 11:47, schrieb Conrad Hoffmann: >> Two more cents from my side: >> socklog [1] also works pretty well... >> [1] http://smarden.org/socklog/ >> Conrad >> On 02/18/2016 11:28 AM, Baptiste wrote: >>> On Thu, Feb 18, 2016 at 10:57 AM, Willy Tarreau wrote: Hi Aleks, On Wed, Feb 17, 2016 at 04:30:06PM +0100, Aleksandar Lazic wrote: > Hi. > how difficult is it to be able to add "log stdout;" to haproxy? > > [snipp] > It's been discussed a few times in the past. The response is "no". It's totally insane to emit logs to a blocking destination. Your whole haproxy process will run at the speed of the logs consumer and the log processing will incure its latency to the process. > > > [snipp] > >>> My 2 cents: Some tools may be used for this purpose: >>> Configure HAProxy to send logs to port 2000, then use: >>> - socat: >>> socat -u UDP-RECV:2000 - > > [snipp] > >>> - netcat: >>> netcat -l -k -u 2000 > > [snipp] > > Thanks for answers and suggestions. > > But this moves just the stdout handling to other tools and does not solve the > problem with blocking handling of std*, as far as I have understood right. > > It also 'violates' the best practice of docker. > > https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#run-only-one-process-per-container > > Okay this could be solved with the linking as described in the link. > > For openshift I will try to use 2 container in 1 pod. > > If there any interests I can write here if this works ;-) > > BR Aleks >
Re: Feature Request for log stdout ...
On 02/18/2016 07:53 AM, Aleksandar Lazic wrote: Thanks for answers and suggestions. But this moves just the stdout handling to other tools and does not solve the problem with blocking handling of std*, as far as I have understood right. haproxy will not block. If your logging system hangs it will drop the UDP packets. I've seen it and had discussions before when people use TCP for logging systems. A hiccup in the logging systems brings the whole infrastructure down. It also 'violates' the best practice of docker. Your application should be more important than docker's best practices. These guides are suggestions not mandates. Having said that, you should run your logging system in its own container. Process in this context is a micro-service, not a single kernel process (ie. pid) Hope this helps, Alberto
Re: Feature Request for log stdout ...
Hi Aleks, On Thu, Feb 18, 2016 at 02:53:29PM +0100, Aleksandar Lazic wrote: > But this moves just the stdout handling to other tools and does not > solve the problem with blocking handling of std*, as far as I have > understood right. Yes it does because if the logging daemon blocks, logs are simply lost on the UDP socket between haproxy and the daemon without blocking haproxy. > It also 'violates' the best practice of docker. > > https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#run-only-one-process-per-container Well it's written "in almost all cases". Otherwise you would not even be allowed to use nbproc or the systemd wrapper. If you consider your deamon as the log-dedicated process, it's OK :-) > Okay this could be solved with the linking as described in the link. > > For openshift I will try to use 2 container in 1 pod. > > If there any interests I can write here if this works ;-) Sure, please report anyway. Cheers, Willy
Re: Feature Request for log stdout ...
Sorry I'm a bit late to this party but when running in a container it's also easy to configure haproxy to log to a unix socket and bind mount that socket to the host. in haproxy.cnf log /dev/log local2 Then when launching the container an option like "-v /var/log:/var/log" works quite well to get container syslogs to the host. -Bryan On Thu, Feb 18, 2016 at 6:22 AM, Willy Tarreau wrote: > Hi Aleks, > > On Thu, Feb 18, 2016 at 02:53:29PM +0100, Aleksandar Lazic wrote: > > But this moves just the stdout handling to other tools and does not > > solve the problem with blocking handling of std*, as far as I have > > understood right. > > Yes it does because if the logging daemon blocks, logs are simply lost > on the UDP socket between haproxy and the daemon without blocking > haproxy. > > > It also 'violates' the best practice of docker. > > > > > https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#run-only-one-process-per-container > > Well it's written "in almost all cases". Otherwise you would not even > be allowed to use nbproc or the systemd wrapper. If you consider your > deamon as the log-dedicated process, it's OK :-) > > > Okay this could be solved with the linking as described in the link. > > > > For openshift I will try to use 2 container in 1 pod. > > > > If there any interests I can write here if this works ;-) > > Sure, please report anyway. > > Cheers, > Willy > > >
Re: Feature Request for log stdout ...
Hi Bryan. Am 18-02-2016 21:18, schrieb Bryan Talbot: Sorry I'm a bit late to this party but when running in a container it's also easy to configure haproxy to log to a unix socket and bind mount that socket to the host. in haproxy.cnf log /dev/log local2 Then when launching the container an option like "-v /var/log:/var/log" works quite well to get container syslogs to the host. Well this way is not possible in openshift due to the fact that the pods are not running as root! -Bryan On Thu, Feb 18, 2016 at 6:22 AM, Willy Tarreau wrote: Hi Aleks, On Thu, Feb 18, 2016 at 02:53:29PM +0100, Aleksandar Lazic wrote: But this moves just the stdout handling to other tools and does not solve the problem with blocking handling of std*, as far as I have understood right. Yes it does because if the logging daemon blocks, logs are simply lost on the UDP socket between haproxy and the daemon without blocking haproxy. It also 'violates' the best practice of docker. https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/#run-only-one-process-per-container Well it's written "in almost all cases". Otherwise you would not even be allowed to use nbproc or the systemd wrapper. If you consider your deamon as the log-dedicated process, it's OK :-) Okay this could be solved with the linking as described in the link. For openshift I will try to use 2 container in 1 pod. If there any interests I can write here if this works ;-) Sure, please report anyway. Cheers, Willy
Re: Feature Request for log stdout ...
Hi. Am 18-02-2016 15:22, schrieb Willy Tarreau: Hi Aleks, On Thu, Feb 18, 2016 at 02:53:29PM +0100, Aleksandar Lazic wrote: [snipp] For openshift I will try to use 2 container in 1 pod. If there any interests I can write here if this works ;-) Sure, please report anyway. You can find my solution at this repo https://github.com/git001/haproxy I use socklog ( http://smarden.org/socklog/ ) instead of socat. socklog is in debian but not in centos/rhel so I just build it in the docker file. BR Aleks