Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)
> Von: lxc-users [mailto:lxc-users-boun...@lists.linuxcontainers.org] Im > Auftrag von Fajar A. Nugraha > > On Wed, Mar 11, 2015 at 8:03 PM, Fiedler Roman > wrote: > > But the current issue is different: The guest can snoop on the NFLOG > messages > > generated on host and destined for the host and hence can get knowledge > of ANY > > NFLOGed connection of host or any guest, no matter if on same bridge or > > another one. > > Ah, sorry I misunderstood your problem. > > All I can say is that it works for me on my simple test. I have ulogd2 > on both host and guest, and if you look at my iptables command on the > host and guest, they are almost identical (including nflog group) > except for chain names (forward/input/output). The logged packets are > from the correct one (the one inside the container has in/out=eth0, > while the one on the host has in/out=br0). > > That was on Ubuntu 14.10 (kernel 3.16) with lxc-1.1 from daily ppa, so > you might want to try that before filing a bug report to ubuntu. OK, I think I understand it now: On my setup, the packets captured by iptables firewall use the correct NFLOG group and make it to the ulogd on the host BUT the log lines end up in kernel message ring buffer (dump with dmesg) and those messages can be read/manipulated by both host AND guest. As there is a rsyslog running in guest, extracting messages from kernel message buffer from time to time, this will consume those messages and write it to the guest logs. With that information, I found report [1] suggesting to use "echo 1 > /proc/sys/kernel/dmesg_restrict", but that somehow fails. At least we can now change the subject to "dmesg insecure with default settings" Roman [1] http://unix.stackexchange.com/questions/103576/container-lockdown smime.p7s Description: S/MIME cryptographic signature ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)
> Von: lxc-users [mailto:lxc-users-boun...@lists.linuxcontainers.org] Im > Auftrag > > On Wed, Mar 11, 2015 at 7:02 PM, Fiedler Roman > wrote: > > This should be exactly the configuration I have tested so far. But that > > did > > not yet solve my problem ... > > > > * If some process in guest registers for the same NFLOG queue, he can > "steal" > > the messages from the host queue, thus removing traces of his activity > > from > > host logging. SECURITY-ASPECT: apart from log corruption, the guest can > > get > > knowledge about any other connection to/from other containers and the > host and > > as they include sequence numbers, may be able to inject spoofed data into > any > > other unencrypted TCP connection or at least interrupt the connection > using > > another helper machine. > > No. What makes you believe that? Well, for the NFLOG message leak to guest I have a running demo on a test machine. It is still possible (and likely), that I did some kind of configuration error that causes the logging misbehaviour. I guess with that current behaviour, TCP connection manipulation using packet spoofing could be left as exercise to the experienced reader (or hacker). > Host and containers does not share iptables rules. Their entire > network stack is separated thru network namespace. There's no such > thing as "stealing the message". How could we sort that out? On my test setup the stealing rate is 11.8% for ~1400 NFLOG messages, so there is definitely something wrong. With some CPU-scheduling optimization exploit code or registering of more hooks, a malicious guest might be able steal even far more. > A test would probably prove my statement faster. Try this on your > container, while keeping the same rules on the host side: > > iptables -I INPUT 1 -d 192.168.124.173 -j NFLOG --nflog-group 0 > --nflog-prefix lxc-v > iptables -I OUTPUT 1 -s 192.168.124.173 -j NFLOG --nflog-group 0 > --nflog-prefix lxc-v No luck with that: I've switched both guest/host to the loggroup 0. After that change, no messages from guest are logged but host messages (which are now clearly distinguishable) continue to be logged within the guest. Did you try to install ulogd2 on host and guest and put it to the same NFLOG group? What about kernel versions? I'm using "Linux version 3.13.0-46-generic (buildd@tipua) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #77-Ubuntu SMP Mon Mar 2 18:23:39 UTC 2015". Could it be, that network-ns isolation was not completed back then? Could it be, that the CPU-scheduler interferes with NFLOG queuing/dequeuing so that with some number of CPU cores or other scheduling influences, the probability for stealing is nearly zero, while on my setup (vbox with single CPU) the rate is nearly 12%? > Note that on the container the chain has to be INPUT on OUTPUT instead > of FORWARD. Then test it. The container will logs this > > ... while at the same time, the host also logs it > [..snip ..] > > (note that my host and container uses different timezone, so the times > look 7-hours apart) I see. but still fail to reproduce here. > > * If a guest should be protected with iptables also, e.g. to avoid Apache > > or > > Tomcat to connect to the local SSH port, those error logs - which are > > useful > > to detect ongoing guest intrusions - do not make it to any log-file. > > No. > > Whatever the container does inside it does not matter to the host. The > host simply capture the packets passing thru the bridge (with > bridge-nf-call-iptables on), and doesn't really care what the > container does with it (block it, drop it, accept it, whatever). > > On the host side, nflog does NOT stop iptables processing, so the > chains further down (e.g. the ones doing ACCEPT, or REJECT if you > want) will still see the packets. You can verify it with "iptables -nL > -v", and see that the hit counters for both the nflog rule AND > ACCEPT/REJECT rule increase. Right. From the documentation I understood it exactly in that way, but my test system fails to behave according to that. > >> You might only be missing the "bridge-nf-call-iptables" part. Note > >> that you shouldn't need it IF you use a custom lxc network setup which > >> doesn't use bridges: > >> https://www.mail-archive.com/lxc- > us...@lists.linuxcontainers.org/msg02587.html > > > > I've tried various network configurations also. I fear that effort here is > > quite futile since I do not yet understand the core kernel namespace > concepts, > > That will be your biggest problem. > > Short version: when using containers installed using templates > (preferably download template), you need to treat container's > networking stack as separate (e.g. on a different "server") with the > one on the host. I do that. I used the same firewall/logging setup scripts on host and guest and hoped that they are separate. The are in all points (connection tracking, iptables rules, connection lists, ...) but EXCEPT for NFLOG when using the same loggi
Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)
On Wed, Mar 11, 2015 at 8:03 PM, Fiedler Roman wrote: > But the current issue is different: The guest can snoop on the NFLOG messages > generated on host and destined for the host and hence can get knowledge of ANY > NFLOGed connection of host or any guest, no matter if on same bridge or > another one. Ah, sorry I misunderstood your problem. All I can say is that it works for me on my simple test. I have ulogd2 on both host and guest, and if you look at my iptables command on the host and guest, they are almost identical (including nflog group) except for chain names (forward/input/output). The logged packets are from the correct one (the one inside the container has in/out=eth0, while the one on the host has in/out=br0). That was on Ubuntu 14.10 (kernel 3.16) with lxc-1.1 from daily ppa, so you might want to try that before filing a bug report to ubuntu. -- Fajar ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)
> Von: lxc-users [mailto:lxc-users-boun...@lists.linuxcontainers.org] Im > Auftrag > > On Wed, Mar 11, 2015 at 7:22 PM, Fajar A. Nugraha wrote: > > On Wed, Mar 11, 2015 at 7:02 PM, Fiedler Roman > wrote: > >> This should be exactly the configuration I have tested so far. But that > >> did > >> not yet solve my problem ... > >> > >> * If some process in guest registers for the same NFLOG queue, he can > "steal" > >> the messages from the host queue, thus removing traces of his activity > from > >> host logging. SECURITY-ASPECT: apart from log corruption, the guest can > get > >> knowledge about any other connection to/from other containers and the > host and > >> as they include sequence numbers, may be able to inject spoofed data > into any > >> other unencrypted TCP connection or at least interrupt the connection > using > >> another helper machine. > > > > No. What makes you believe that? > > > > Host and containers does not share iptables rules. Their entire > > network stack is separated thru network namespace. There's no such > > thing as "stealing the message".> > To further clarify: > > The default lxc networking setup (veth with bridge) MAY allow a > container to snoop/hijack traffic to/from other containers. This is > similar to how a computers on the same LAN, connected to a dumb > switch, can potentially snoop/hijack traffic to/from other computers. > This is ethernet bridge issue, not iptables issue, nor lxc issue. > > To prevent that issue, there are some options you can do. One option > is to create a separate bridge for each container. The other option > would be to use my alternative setup which I linked to earlier, which > does NOT use bridge. Yes, I'm completely aware of that property of bridge. But the current issue is different: The guest can snoop on the NFLOG messages generated on host and destined for the host and hence can get knowledge of ANY NFLOGed connection of host or any guest, no matter if on same bridge or another one. smime.p7s Description: S/MIME cryptographic signature ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)
On Wed, Mar 11, 2015 at 7:22 PM, Fajar A. Nugraha wrote: > On Wed, Mar 11, 2015 at 7:02 PM, Fiedler Roman > wrote: >> This should be exactly the configuration I have tested so far. But that did >> not yet solve my problem ... >> >> * If some process in guest registers for the same NFLOG queue, he can "steal" >> the messages from the host queue, thus removing traces of his activity from >> host logging. SECURITY-ASPECT: apart from log corruption, the guest can get >> knowledge about any other connection to/from other containers and the host >> and >> as they include sequence numbers, may be able to inject spoofed data into >> any >> other unencrypted TCP connection or at least interrupt the connection using >> another helper machine. > > No. What makes you believe that? > > Host and containers does not share iptables rules. Their entire > network stack is separated thru network namespace. There's no such > thing as "stealing the message". To further clarify: The default lxc networking setup (veth with bridge) MAY allow a container to snoop/hijack traffic to/from other containers. This is similar to how a computers on the same LAN, connected to a dumb switch, can potentially snoop/hijack traffic to/from other computers. This is ethernet bridge issue, not iptables issue, nor lxc issue. To prevent that issue, there are some options you can do. One option is to create a separate bridge for each container. The other option would be to use my alternative setup which I linked to earlier, which does NOT use bridge. -- Fajar ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)
On Wed, Mar 11, 2015 at 7:02 PM, Fiedler Roman wrote: > This should be exactly the configuration I have tested so far. But that did > not yet solve my problem ... > > * If some process in guest registers for the same NFLOG queue, he can "steal" > the messages from the host queue, thus removing traces of his activity from > host logging. SECURITY-ASPECT: apart from log corruption, the guest can get > knowledge about any other connection to/from other containers and the host and > as they include sequence numbers, may be able to inject spoofed data into any > other unencrypted TCP connection or at least interrupt the connection using > another helper machine. No. What makes you believe that? Host and containers does not share iptables rules. Their entire network stack is separated thru network namespace. There's no such thing as "stealing the message". A test would probably prove my statement faster. Try this on your container, while keeping the same rules on the host side: iptables -I INPUT 1 -d 192.168.124.173 -j NFLOG --nflog-group 0 --nflog-prefix lxc-v iptables -I OUTPUT 1 -s 192.168.124.173 -j NFLOG --nflog-group 0 --nflog-prefix lxc-v Note that on the container the chain has to be INPUT on OUTPUT instead of FORWARD. Then test it. The container will logs this Mar 11 12:10:56 v lxc-v IN= OUT=eth0 MAC= SRC=192.168.124.173 DST=192.168.124.134 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=26016 DF PROTO=ICMP TYPE=8 CODE=0 ID=2087 SEQ=1 UID=0 GID=0 MARK=0 Mar 11 12:10:55 v lxc-v IN=eth0 OUT= MAC=00:16:3e:f5:cd:94:00:16:3e:2e:d2:6d:08:00 SRC=192.168.124.134 DST=192.168.124.173 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=29087 PROTO=ICMP TYPE=0 CODE=0 ID=2087 SEQ=1 MARK=0 Mar 11 12:10:57 v lxc-v IN= OUT=eth0 MAC= SRC=192.168.124.173 DST=192.168.124.134 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=26020 DF PROTO=ICMP TYPE=8 CODE=0 ID=2087 SEQ=2 UID=0 GID=0 MARK=0 Mar 11 12:10:56 v lxc-v IN=eth0 OUT= MAC=00:16:3e:f5:cd:94:00:16:3e:2e:d2:6d:08:00 SRC=192.168.124.134 DST=192.168.124.173 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=29287 PROTO=ICMP TYPE=0 CODE=0 ID=2087 SEQ=2 MARK=0 ... while at the same time, the host also logs it Mar 11 19:10:55 utopic lxc-v IN=br0 OUT=br0 MAC=00:16:3e:2e:d2:6d:00:16:3e:f5:cd:94:08:00 SRC=192.168.124.173 DST=192.168.124.134 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=26016 DF PROTO=ICMP TYPE=8 CODE=0 ID=2087 SEQ=1 MARK=0 Mar 11 19:10:55 utopic lxc-v IN=br0 OUT=br0 MAC=00:16:3e:f5:cd:94:00:16:3e:2e:d2:6d:08:00 SRC=192.168.124.134 DST=192.168.124.173 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=29087 PROTO=ICMP TYPE=0 CODE=0 ID=2087 SEQ=1 MARK=0 Mar 11 19:10:56 utopic lxc-v IN=br0 OUT=br0 MAC=00:16:3e:2e:d2:6d:00:16:3e:f5:cd:94:08:00 SRC=192.168.124.173 DST=192.168.124.134 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=26020 DF PROTO=ICMP TYPE=8 CODE=0 ID=2087 SEQ=2 MARK=0 Mar 11 19:10:56 utopic lxc-v IN=br0 OUT=br0 MAC=00:16:3e:f5:cd:94:00:16:3e:2e:d2:6d:08:00 SRC=192.168.124.134 DST=192.168.124.173 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=29287 PROTO=ICMP TYPE=0 CODE=0 ID=2087 SEQ=2 MARK=0 (note that my host and container uses different timezone, so the times look 7-hours apart) > > * If a guest should be protected with iptables also, e.g. to avoid Apache or > Tomcat to connect to the local SSH port, those error logs - which are useful > to detect ongoing guest intrusions - do not make it to any log-file. No. Whatever the container does inside it does not matter to the host. The host simply capture the packets passing thru the bridge (with bridge-nf-call-iptables on), and doesn't really care what the container does with it (block it, drop it, accept it, whatever). On the host side, nflog does NOT stop iptables processing, so the chains further down (e.g. the ones doing ACCEPT, or REJECT if you want) will still see the packets. You can verify it with "iptables -nL -v", and see that the hit counters for both the nflog rule AND ACCEPT/REJECT rule increase. > > [...snip...] > >> You might only be missing the "bridge-nf-call-iptables" part. Note >> that you shouldn't need it IF you use a custom lxc network setup which >> doesn't use bridges: >> https://www.mail-archive.com/lxc-users@lists.linuxcontainers.org/msg02587.html > > I've tried various network configurations also. I fear that effort here is > quite futile since I do not yet understand the core kernel namespace concepts, That will be your biggest problem. Short version: when using containers installed using templates (preferably download template), you need to treat container's networking stack as separate (e.g. on a different "server") with the one on the host. The only "link" the host has to the container is the veth pair (veth by default on host side, usually eth0 on the container side). Think of lxcbr0 (or whatever bridge you use) as a "switch", with veth as the switch port that is connected to the container. > e.g. how nflink and user namespaces work together. Hence everything I could do > is (inefficient) trial and error instead of controlled engineering. And in the > e
Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)
> Von: lxc-users [mailto:lxc-users-boun...@lists.linuxcontainers.org] Im > Auftrag > > On Wed, Mar 11, 2015 at 5:48 PM, Fiedler Roman > wrote: > > Hello list, > > > > Has someone managed to get reliable network traffic auditing with LXC up > and > > running? That means, that it is possible to write a protocol of e.g. every > > new connection from and to host. > > > > On my setup (Ubuntu Trusty), both host and guest may have different > iptables > > rulesets. But the guest NFLOG messages are lost completely, those from > > host > > are sometimes sent to the ulogd in the guest (time-race), so the host log > > is > > not trustworthy also. > > > > What could be the best solution to get trustworthy logs with LXC? > > Try something like this on the host: > > echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables > iptables -I FORWARD 1 -d 192.168.124.173 -j NFLOG --nflog-group 0 > --nflog-prefix lxc-v > iptables -I FORWARD 1 -s 192.168.124.173 -j NFLOG --nflog-group 0 > --nflog-prefix lxc-v > > with the default ulogd2 setup on ubuntu 14.10 (which already includes > rules for nflog-group 0 logging to a file) you should then be able to > get something like this when the container (192.168.124.173) pings > another container (192.168.124.134) This should be exactly the configuration I have tested so far. But that did not yet solve my problem ... * If some process in guest registers for the same NFLOG queue, he can "steal" the messages from the host queue, thus removing traces of his activity from host logging. SECURITY-ASPECT: apart from log corruption, the guest can get knowledge about any other connection to/from other containers and the host and as they include sequence numbers, may be able to inject spoofed data into any other unencrypted TCP connection or at least interrupt the connection using another helper machine. * If a guest should be protected with iptables also, e.g. to avoid Apache or Tomcat to connect to the local SSH port, those error logs - which are useful to detect ongoing guest intrusions - do not make it to any log-file. [...snip...] > You might only be missing the "bridge-nf-call-iptables" part. Note > that you shouldn't need it IF you use a custom lxc network setup which > doesn't use bridges: > https://www.mail-archive.com/lxc-users@lists.linuxcontainers.org/msg02587.html I've tried various network configurations also. I fear that effort here is quite futile since I do not yet understand the core kernel namespace concepts, e.g. how nflink and user namespaces work together. Hence everything I could do is (inefficient) trial and error instead of controlled engineering. And in the end I cannot be sure that there are not reliability/security-relevant holes left open with trial/error. Without any other clues from the mailing list, I'll still try out this procedure also and see if it would change the nflog behavior. Roman smime.p7s Description: S/MIME cryptographic signature ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)
On Wed, Mar 11, 2015 at 5:48 PM, Fiedler Roman wrote: > Hello list, > > Has someone managed to get reliable network traffic auditing with LXC up and > running? That means, that it is possible to write a protocol of e.g. every > new connection from and to host. > > On my setup (Ubuntu Trusty), both host and guest may have different iptables > rulesets. But the guest NFLOG messages are lost completely, those from host > are sometimes sent to the ulogd in the guest (time-race), so the host log is > not trustworthy also. > > What could be the best solution to get trustworthy logs with LXC? Try something like this on the host: echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables iptables -I FORWARD 1 -d 192.168.124.173 -j NFLOG --nflog-group 0 --nflog-prefix lxc-v iptables -I FORWARD 1 -s 192.168.124.173 -j NFLOG --nflog-group 0 --nflog-prefix lxc-v with the default ulogd2 setup on ubuntu 14.10 (which already includes rules for nflog-group 0 logging to a file) you should then be able to get something like this when the container (192.168.124.173) pings another container (192.168.124.134) # tail -f /var/log/ulog/syslogemu.log Mar 11 18:40:49 utopic lxc-v IN=br0 OUT=br0 MAC=00:16:3e:2e:d2:6d:00:16:3e:f5:cd:94:08:00 SRC=192.168.124.173 DST=192.168.124.134 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=10868 DF PROTO=ICMP TYPE=8 CODE=0 ID=1916 SEQ=2 MARK=0 Mar 11 18:40:49 utopic lxc-v IN=br0 OUT=br0 MAC=00:16:3e:f5:cd:94:00:16:3e:2e:d2:6d:08:00 SRC=192.168.124.134 DST=192.168.124.173 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=46525 PROTO=ICMP TYPE=0 CODE=0 ID=1916 SEQ=2 MARK=0 You might only be missing the "bridge-nf-call-iptables" part. Note that you shouldn't need it IF you use a custom lxc network setup which doesn't use bridges: https://www.mail-archive.com/lxc-users@lists.linuxcontainers.org/msg02587.html -- Fajar ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
[lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)
Hello list, Has someone managed to get reliable network traffic auditing with LXC up and running? That means, that it is possible to write a protocol of e.g. every new connection from and to host. On my setup (Ubuntu Trusty), both host and guest may have different iptables rulesets. But the guest NFLOG messages are lost completely, those from host are sometimes sent to the ulogd in the guest (time-race), so the host log is not trustworthy also. What could be the best solution to get trustworthy logs with LXC? Kind regards, Roma DI Roman Fiedler Scientist Digital Safety & Security Department Assistive Healthcare Information Technology AIT Austrian Institute of Technology GmbH Reininghausstraße 13/1 | 8020 Graz | Austria T +43(0) 50550 2957 | M +43(0) 664 8561599 | F +43(0) 50550 2950 roman.fied...@ait.ac.at | http://www.ait.ac.at/ FN: 115980 i HG Wien | UID: ATU14703506 http://www.ait.ac.at/Email-Disclaimer smime.p7s Description: S/MIME cryptographic signature ___ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users