Thanks for responding Jarno.
So yes, as I stated Rsyslog is looking at the UNIX socket for haproxy in the
/etc/rsyslog.d/49-haproxy.conf file, and rsyslog.conf has the default option
enabled to load /etc/rsyslog.d/*.conf contents.
And Haproxy is logging, I just feel.. or no I wanted to make sure I was logging
in a proper manner to not miss anything I guess.
Since sending this, I did discover the tcplog option, as well as the
dontlognull. Removing the latter actually helped me discover a Chinese source
trying to hit one of these servers, resulting in Networking find out there was
an issue with the Block rule in the firewall! So that was cool. I did re-add
the dontlognull just because the Route53 healthc hecks were very annoying and
filling the log.
Anyway so as of right now, I'm getting better HTTP logs. The LDAPS ones
though, IDK. I've got the SNI/requested header server name so that's a big
bonus. But I wonder if I'm still not getting everything.
Attached is a revised haproxy.cfg, I look forward to your advice and
suggestions.
Thanks!
On 11/23/21, 1:57 AM, "Jarno Huuskonen" <[email protected]> wrote:
Hi,
On 11/22/21 16:33, Ben Hart wrote:
> Hey there! I’ve got a handful of Haproxy servers that are serving LDAPS
> and HTTPS front/back ends. I am new to this, so I built these and
> reused the config from the older Haproxy servers we had.
>
> Anyway I mention that because I likely have little idea what I should be
> done here. So far everything is working.. we are able to bind and
> perform lookups successfully. What’s not working like I think it should
> is logging. I have Firewalld setup that is blocking all traffic inbound
> from the same internal subnet as the server, and allowing 0.0.0.0/0 in
> from all other sources for ports 636 and 443.
>
> Rsyslog is matching on program name ‘haproxy’ and the default UNIX
> socket /dev/log and forwarding all info to /var/log/haproxy.log
>
> Rsyslog is matching on program name ‘firewalld’ and sending all info to
> /var/log/firewalld.log
>
> If I tail both files, I see many inbound connections allowed to port
> 636, but no corresponding events in the haproxy.log file. So I’m hoping
Do you get any logs in haproxy.log ? (Any logs from "frontend
ecorp_https" ?)
> that maybe I have something on the Haproxy side that’s not quite what it
> should be. The thought is, Maybe the connection attempts are coming in,
> but Haproxy is not fulfilling them for some reason. And I don’t have the
> appropriate log options or formats setup to determine that.
>
> Attached is my sanitized haproxy.cfg
> global
> log /dev/log local0
> log /dev/log local1 notice
> # log 127.0.0.1 local1
> chroot /var/lib/haproxy
You're using chroot, is rsyslog configured to listen to
/var/lib/haproxy/dev/log ? (And if this is centos/rhel based system
selinux allows rsyslog to create the socket and haproxy to connect to it).
Have you checked that haproxy sends logs for example with
enable logging to 127.0.0.1 and use tcpdump -nn -XX -i lo port 514 or
something similar ?
> frontend ecorp_https
> option tcplog
You probably don't want to use
tcplog(https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#4-option%20tcplog)
with mode http. AFAIK it overrides your custom log-format.
-Jarno
--
Jarno Huuskonen
global
log /dev/log local0
log /dev/log local1 notice
# log 127.0.0.1 local1
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
maxconn 2048
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
# An alternative list with additional directives can be obtained from
#
https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
ssl-default-bind-ciphers
ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
tune.ssl.default-dh-param 2048
defaults
log global
mode http
# option httplog
log-format "%{+Q}o client_ip=%ci client_port=%cp
datetime_of_request=[%tr] frontend_name_transport=%ft backend_name=%b
server_name=%s time_to_receive_full_request=%TR Tw=%Tw Tc=%Tc response_time=%Tr
active_time_of_request=%Ta status_code=%ST bytes_read=%B
captured_request_cookie=%CC captured_response_cookie=%CS
termination_state_with_cookie_status=%tsc actconn=%ac feconn=%fc beconn=%bc
srv_conn=%sc retries=%rc srv_queue=%sq backend_queue=%bq
captured_request_headers_default_style=%hr
captured_response_headers_default_style=%hs server_ip=%si server_port=%sp
frontend_name=%f http_method=%HM http_request_uri_without_query=%HP
http_request_query_string=%HQ http_request_uri=%HU bytes_uploaded=%U
ssl_ciphers=%sslc ssl_version=%sslv"
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
# Enabling HAProxy Stats
listen stats
bind *:8404
mode http
log global
maxconn 10
stats enable
stats refresh 30s
stats show-node
stats auth admin:password
stats uri /stats
# LDAPS
frontend ldaps
mode tcp
# log global
bind *:636 ssl crt /etc/ssl/private/hap/ldaps.net.pem crt
/etc/ssl/private/hap/wild.ecorp.com.pem
log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts
%ac/%fc/%bc/%sc/%rc %sq/%bq capture0: %[capture.req.hdr(0)]"
description LDAPS Service
# option tcplog
# option logasap
option socket-stats
# option tcpka
timeout client 60s
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
use_backend ecorp_ldaps if { ssl_fc_sni_end ecorp.com }
default_backend ldaps
# ECorp Stuff
frontend ecorp_https
bind *:443 ssl crt /etc/ssl/private/hap/wild.ecorp.com.pem
log global
mode http
description ECorpTech
option socket-stats
default_backend ecorp_https
# option tcplog
capture request header Host len <len>
capture request header Content-Type len <len>
capture request header User-Agent len <len>
capture request header Referer len <len>
capture request header X-Forwarded-For len <len>
capture response header Content-Type len <len>
capture cookie Cookie_2 len <len>
# LDAPS backend
backend ldaps
mode http
log global
balance leastconn
server rodc01 x.x.x.x:636 maxconn 100 check ssl fall 3 rise 1
inter 2s verify none check check-ssl ca-file /etc/ssl/private/hap/ldaps.pem
timeout server 60s
timeout connect 60s
# option tcpka
# option ldap-check
option tcp-check
tcp-check connect port 636 ssl
tcp-check send-binary 300c0201 # LDAP bind request "<ROOT>" simple
tcp-check send-binary 01 # message ID
tcp-check send-binary 6007 # protocol Op
tcp-check send-binary 0201 # bind request
tcp-check send-binary 03 # LDAP v3
tcp-check send-binary 04008000 # name, simple authentication
tcp-check expect binary 0a0100 # bind response + result code: success
tcp-check send-binary 30050201034200 # unbind request
# ECorp backend LDAPS
backend ecorp_ldaps
mode http
log global
server dc02 x.x.x.x:636 maxconn 20 check ssl fall 3 rise 1
inter 10s verify none check check-ssl ca-file
/etc/ssl/private/hap/wild.ecorp.com.pem
timeout server 60s
timeout connect 60s
option tcpka
option tcp-check
tcp-check connect port 636 ssl
tcp-check send-binary 300c0201 # LDAP bind request "<ROOT>" simple
tcp-check send-binary 01 # message ID
tcp-check send-binary 6007 # protocol Op
tcp-check send-binary 0201 # bind request
tcp-check send-binary 03 # LDAP v3
tcp-check send-binary 04008000 # name, simple authentication
tcp-check expect binary 0a0100 # bind response + result code: success
tcp-check send-binary 30050201034200 # unbind request
# Ecorp backend https
backend ecorp_https
mode http
Log global
server subca02.ecorp.com x.x.x.x:443 maxconn 20 fall 3 rise 1
inter 10s verify none check check-ssl
option httpchk OPTIONS / HTTP/1.0
