Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)
Von: lxc-users [mailto:lxc-users-boun...@lists.linuxcontainers.org] Im Auftrag von Fajar A. Nugraha On Wed, Mar 11, 2015 at 8:03 PM, Fiedler Roman roman.fied...@ait.ac.at wrote: But the current issue is different: The guest can snoop on the NFLOG messages generated on host and destined for the host and hence can get knowledge of ANY NFLOGed connection of host or any guest, no matter if on same bridge or another one. Ah, sorry I misunderstood your problem. All I can say is that it works for me on my simple test. I have ulogd2 on both host and guest, and if you look at my iptables command on the host and guest, they are almost identical (including nflog group) except for chain names (forward/input/output). The logged packets are from the correct one (the one inside the container has in/out=eth0, while the one on the host has in/out=br0). That was on Ubuntu 14.10 (kernel 3.16) with lxc-1.1 from daily ppa, so you might want to try that before filing a bug report to ubuntu. OK, I think I understand it now: On my setup, the packets captured by iptables firewall use the correct NFLOG group and make it to the ulogd on the host BUT the log lines end up in kernel message ring buffer (dump with dmesg) and those messages can be read/manipulated by both host AND guest. As there is a rsyslog running in guest, extracting messages from kernel message buffer from time to time, this will consume those messages and write it to the guest logs. With that information, I found report [1] suggesting to use echo 1 /proc/sys/kernel/dmesg_restrict, but that somehow fails. At least we can now change the subject to dmesg insecure with default settings Roman [1] http://unix.stackexchange.com/questions/103576/container-lockdown smime.p7s Description: S/MIME cryptographic signature ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)
On Wed, Mar 11, 2015 at 8:03 PM, Fiedler Roman roman.fied...@ait.ac.at wrote: But the current issue is different: The guest can snoop on the NFLOG messages generated on host and destined for the host and hence can get knowledge of ANY NFLOGed connection of host or any guest, no matter if on same bridge or another one. Ah, sorry I misunderstood your problem. All I can say is that it works for me on my simple test. I have ulogd2 on both host and guest, and if you look at my iptables command on the host and guest, they are almost identical (including nflog group) except for chain names (forward/input/output). The logged packets are from the correct one (the one inside the container has in/out=eth0, while the one on the host has in/out=br0). That was on Ubuntu 14.10 (kernel 3.16) with lxc-1.1 from daily ppa, so you might want to try that before filing a bug report to ubuntu. -- Fajar ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)
On Wed, Mar 11, 2015 at 5:48 PM, Fiedler Roman roman.fied...@ait.ac.at wrote: Hello list, Has someone managed to get reliable network traffic auditing with LXC up and running? That means, that it is possible to write a protocol of e.g. every new connection from and to host. On my setup (Ubuntu Trusty), both host and guest may have different iptables rulesets. But the guest NFLOG messages are lost completely, those from host are sometimes sent to the ulogd in the guest (time-race), so the host log is not trustworthy also. What could be the best solution to get trustworthy logs with LXC? Try something like this on the host: echo 1 /proc/sys/net/bridge/bridge-nf-call-iptables iptables -I FORWARD 1 -d 192.168.124.173 -j NFLOG --nflog-group 0 --nflog-prefix lxc-v iptables -I FORWARD 1 -s 192.168.124.173 -j NFLOG --nflog-group 0 --nflog-prefix lxc-v with the default ulogd2 setup on ubuntu 14.10 (which already includes rules for nflog-group 0 logging to a file) you should then be able to get something like this when the container (192.168.124.173) pings another container (192.168.124.134) # tail -f /var/log/ulog/syslogemu.log Mar 11 18:40:49 utopic lxc-v IN=br0 OUT=br0 MAC=00:16:3e:2e:d2:6d:00:16:3e:f5:cd:94:08:00 SRC=192.168.124.173 DST=192.168.124.134 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=10868 DF PROTO=ICMP TYPE=8 CODE=0 ID=1916 SEQ=2 MARK=0 Mar 11 18:40:49 utopic lxc-v IN=br0 OUT=br0 MAC=00:16:3e:f5:cd:94:00:16:3e:2e:d2:6d:08:00 SRC=192.168.124.134 DST=192.168.124.173 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=46525 PROTO=ICMP TYPE=0 CODE=0 ID=1916 SEQ=2 MARK=0 You might only be missing the bridge-nf-call-iptables part. Note that you shouldn't need it IF you use a custom lxc network setup which doesn't use bridges: https://www.mail-archive.com/lxc-users@lists.linuxcontainers.org/msg02587.html -- Fajar ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)
Von: lxc-users [mailto:lxc-users-boun...@lists.linuxcontainers.org] Im Auftrag On Wed, Mar 11, 2015 at 5:48 PM, Fiedler Roman roman.fied...@ait.ac.at wrote: Hello list, Has someone managed to get reliable network traffic auditing with LXC up and running? That means, that it is possible to write a protocol of e.g. every new connection from and to host. On my setup (Ubuntu Trusty), both host and guest may have different iptables rulesets. But the guest NFLOG messages are lost completely, those from host are sometimes sent to the ulogd in the guest (time-race), so the host log is not trustworthy also. What could be the best solution to get trustworthy logs with LXC? Try something like this on the host: echo 1 /proc/sys/net/bridge/bridge-nf-call-iptables iptables -I FORWARD 1 -d 192.168.124.173 -j NFLOG --nflog-group 0 --nflog-prefix lxc-v iptables -I FORWARD 1 -s 192.168.124.173 -j NFLOG --nflog-group 0 --nflog-prefix lxc-v with the default ulogd2 setup on ubuntu 14.10 (which already includes rules for nflog-group 0 logging to a file) you should then be able to get something like this when the container (192.168.124.173) pings another container (192.168.124.134) This should be exactly the configuration I have tested so far. But that did not yet solve my problem ... * If some process in guest registers for the same NFLOG queue, he can steal the messages from the host queue, thus removing traces of his activity from host logging. SECURITY-ASPECT: apart from log corruption, the guest can get knowledge about any other connection to/from other containers and the host and as they include sequence numbers, may be able to inject spoofed data into any other unencrypted TCP connection or at least interrupt the connection using another helper machine. * If a guest should be protected with iptables also, e.g. to avoid Apache or Tomcat to connect to the local SSH port, those error logs - which are useful to detect ongoing guest intrusions - do not make it to any log-file. [...snip...] You might only be missing the bridge-nf-call-iptables part. Note that you shouldn't need it IF you use a custom lxc network setup which doesn't use bridges: https://www.mail-archive.com/lxc-users@lists.linuxcontainers.org/msg02587.html I've tried various network configurations also. I fear that effort here is quite futile since I do not yet understand the core kernel namespace concepts, e.g. how nflink and user namespaces work together. Hence everything I could do is (inefficient) trial and error instead of controlled engineering. And in the end I cannot be sure that there are not reliability/security-relevant holes left open with trial/error. Without any other clues from the mailing list, I'll still try out this procedure also and see if it would change the nflog behavior. Roman smime.p7s Description: S/MIME cryptographic signature ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
