Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)

2015-03-18 Thread Fiedler Roman
> Von: lxc-users [mailto:lxc-users-boun...@lists.linuxcontainers.org] Im 
> Auftrag von Fajar A. Nugraha
>
> On Wed, Mar 11, 2015 at 8:03 PM, Fiedler Roman 
> wrote:
> > But the current issue is different: The guest can snoop on the NFLOG
> messages
> > generated on host and destined for the host and hence can get knowledge
> of ANY
> > NFLOGed connection of host or any guest, no matter if on same bridge or
> > another one.
>
> Ah, sorry I misunderstood your problem.
>
> All I can say is that it works for me on my simple test. I have ulogd2
> on both host and guest, and if you look at my iptables command on the
> host and guest, they are almost identical (including nflog group)
> except for chain names (forward/input/output). The logged packets are
> from the correct one (the one inside the container has in/out=eth0,
> while the one on the host has in/out=br0).
>
> That was on Ubuntu 14.10 (kernel 3.16) with lxc-1.1 from daily ppa, so
> you might want to try that before filing a bug report to ubuntu.

OK, I think I understand it now:

On my setup, the packets captured by iptables firewall use the correct NFLOG 
group and make it to the ulogd on the host BUT the log lines end up in kernel 
message ring buffer (dump with dmesg) and those messages can be 
read/manipulated by both host AND guest.

As there is a rsyslog running in guest, extracting messages from kernel 
message buffer from time to time, this will consume those messages and write 
it to the guest logs.

With that information, I found report [1] suggesting to use "echo 1 > 
/proc/sys/kernel/dmesg_restrict", but that somehow fails.

At least we can now change the subject to "dmesg insecure with default 
settings"

Roman


[1] http://unix.stackexchange.com/questions/103576/container-lockdown


smime.p7s
Description: S/MIME cryptographic signature
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)

2015-03-12 Thread Fiedler Roman
> Von: lxc-users [mailto:lxc-users-boun...@lists.linuxcontainers.org] Im 
> Auftrag
>
> On Wed, Mar 11, 2015 at 7:02 PM, Fiedler Roman 
> wrote:
> > This should be exactly the configuration I have tested  so far. But that 
> > did
> > not yet solve my problem ...
> >
> > * If some process in guest registers for the same NFLOG queue, he can
> "steal"
> > the messages from the host queue, thus removing traces of his activity 
> > from
> > host logging. SECURITY-ASPECT: apart from log corruption, the guest can 
> > get
> > knowledge about any other connection to/from other containers and the
> host and
> > as they include  sequence numbers, may be able to inject spoofed data into
> any
> > other unencrypted TCP connection or at least interrupt the connection
> using
> > another helper machine.
>
> No. What makes you believe that?

Well, for the NFLOG message leak to guest I have a running demo on a test 
machine. It is still possible (and likely), that I did some kind of 
configuration error that causes the logging misbehaviour. I guess with that 
current behaviour, TCP connection manipulation using packet spoofing could be 
left as exercise to the experienced reader (or hacker).

> Host and containers does not share iptables rules. Their entire
> network stack is separated thru network namespace. There's no such
> thing as "stealing the message".

How could we sort  that out? On my test  setup the stealing rate is 11.8% for 
~1400 NFLOG messages, so there is definitely something wrong. With some 
CPU-scheduling optimization exploit code or registering of more hooks, a 
malicious guest might be able steal even far more.

> A test would probably prove my statement faster. Try this on your
> container, while keeping the same rules on the host side:
>
> iptables -I INPUT 1 -d 192.168.124.173 -j NFLOG --nflog-group 0
> --nflog-prefix lxc-v
> iptables -I OUTPUT 1 -s 192.168.124.173 -j NFLOG --nflog-group 0
> --nflog-prefix lxc-v

No luck with that: I've switched both guest/host to the loggroup 0. After that 
change, no messages from guest are logged but host messages (which are now 
clearly distinguishable) continue to be logged within the guest.

Did you try to install ulogd2 on host and guest and put it to the same NFLOG 
group?

What about kernel versions? I'm using "Linux version 3.13.0-46-generic 
(buildd@tipua) (gcc version 4.8.2 (Ubuntu 4.8.2-19ubuntu1) ) #77-Ubuntu SMP 
Mon Mar 2 18:23:39 UTC 2015". Could it be,  that network-ns isolation was not 
completed back then?

Could it be, that the CPU-scheduler interferes with NFLOG queuing/dequeuing so 
that with some number of CPU cores or other scheduling influences, the 
probability for stealing is nearly zero, while on my setup (vbox with single 
CPU) the rate is nearly 12%?

> Note that on the container the chain has to be INPUT on OUTPUT instead
> of FORWARD. Then test it. The container will logs this
>
> ... while at the same time, the host also logs it
> [..snip ..]
>
> (note that my host and container uses different timezone, so the times
> look 7-hours apart)

I see. but  still fail to reproduce here.

> > * If a guest should be protected with iptables also, e.g. to avoid Apache 
> > or
> > Tomcat to connect to the local SSH port, those error logs - which are 
> > useful
> > to detect ongoing guest intrusions - do not make it to any log-file.
>
> No.
>
> Whatever the container does inside it does not matter to the host. The
> host simply capture the packets passing thru the bridge (with
> bridge-nf-call-iptables on), and doesn't really care what the
> container does with it (block it, drop it, accept it, whatever).
>
> On the host side, nflog does NOT stop iptables processing, so the
> chains further down (e.g. the ones doing ACCEPT, or REJECT if you
> want) will still see the packets. You can verify it with "iptables -nL
> -v", and see that the hit counters for both the nflog rule AND
> ACCEPT/REJECT rule increase.

Right. From the documentation I understood it exactly in that way, but my test 
system fails to behave according to that.

> >> You might only be missing the "bridge-nf-call-iptables" part. Note
> >> that you shouldn't need it IF you use a custom lxc network setup which
> >> doesn't use bridges:
> >> https://www.mail-archive.com/lxc-
> us...@lists.linuxcontainers.org/msg02587.html
> >
> > I've tried various network configurations also. I fear that effort here is
> > quite futile since I do not yet understand the core kernel namespace
> concepts,
>
> That will be your biggest problem.
>
> Short version: when using containers installed using templates
> (preferably download template), you need to treat container's
> networking stack as separate (e.g. on a different "server") with the
> one on the host.

I do that. I used the same firewall/logging setup scripts on host and guest 
and hoped that they are separate. The are in all points (connection tracking, 
iptables rules, connection lists, ...) but EXCEPT for NFLOG when using the 
same loggi

Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)

2015-03-11 Thread Fajar A. Nugraha
On Wed, Mar 11, 2015 at 8:03 PM, Fiedler Roman  wrote:
> But the current issue is different: The guest can snoop on the NFLOG messages
> generated on host and destined for the host and hence can get knowledge of ANY
> NFLOGed connection of host or any guest, no matter if on same bridge or
> another one.

Ah, sorry I misunderstood your problem.

All I can say is that it works for me on my simple test. I have ulogd2
on both host and guest, and if you look at my iptables command on the
host and guest, they are almost identical (including nflog group)
except for chain names (forward/input/output). The logged packets are
from the correct one (the one inside the container has in/out=eth0,
while the one on the host has in/out=br0).

That was on Ubuntu 14.10 (kernel 3.16) with lxc-1.1 from daily ppa, so
you might want to try that before filing a bug report to ubuntu.

-- 
Fajar
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)

2015-03-11 Thread Fiedler Roman
> Von: lxc-users [mailto:lxc-users-boun...@lists.linuxcontainers.org] Im 
> Auftrag
>
> On Wed, Mar 11, 2015 at 7:22 PM, Fajar A. Nugraha  wrote:
> > On Wed, Mar 11, 2015 at 7:02 PM, Fiedler Roman
>  wrote:
> >> This should be exactly the configuration I have tested  so far. But that 
> >> did
> >> not yet solve my problem ...
> >>
> >> * If some process in guest registers for the same NFLOG queue, he can
> "steal"
> >> the messages from the host queue, thus removing traces of his activity
> from
> >> host logging. SECURITY-ASPECT: apart from log corruption, the guest can
> get
> >> knowledge about any other connection to/from other containers and the
> host and
> >> as they include  sequence numbers, may be able to inject spoofed data
> into any
> >> other unencrypted TCP connection or at least interrupt the connection
> using
> >> another helper machine.
> >
> > No. What makes you believe that?
> >
> > Host and containers does not share iptables rules. Their entire
> > network stack is separated thru network namespace. There's no such
> > thing as "stealing the message".>
> To further clarify:
>
> The default lxc networking setup (veth with bridge) MAY allow a
> container to snoop/hijack traffic to/from other containers. This is
> similar to how a computers on the same LAN, connected to a dumb
> switch, can potentially snoop/hijack traffic to/from other computers.
> This is ethernet bridge issue, not iptables issue, nor lxc issue.
>
> To prevent that issue, there are some options you can do. One option
> is to create a separate bridge for each container. The other option
> would be to use my alternative setup which I linked to earlier, which
> does NOT use bridge.

Yes, I'm completely aware of that property of bridge.

But the current issue is different: The guest can snoop on the NFLOG messages 
generated on host and destined for the host and hence can get knowledge of ANY 
NFLOGed connection of host or any guest, no matter if on same bridge or 
another one.


smime.p7s
Description: S/MIME cryptographic signature
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)

2015-03-11 Thread Fajar A. Nugraha
On Wed, Mar 11, 2015 at 7:22 PM, Fajar A. Nugraha  wrote:
> On Wed, Mar 11, 2015 at 7:02 PM, Fiedler Roman  
> wrote:
>> This should be exactly the configuration I have tested  so far. But that did
>> not yet solve my problem ...
>>
>> * If some process in guest registers for the same NFLOG queue, he can "steal"
>> the messages from the host queue, thus removing traces of his activity from
>> host logging. SECURITY-ASPECT: apart from log corruption, the guest can get
>> knowledge about any other connection to/from other containers and the host 
>> and
>> as they include  sequence numbers, may be able to inject spoofed data into 
>> any
>> other unencrypted TCP connection or at least interrupt the connection using
>> another helper machine.
>
> No. What makes you believe that?
>
> Host and containers does not share iptables rules. Their entire
> network stack is separated thru network namespace. There's no such
> thing as "stealing the message".


To further clarify:

The default lxc networking setup (veth with bridge) MAY allow a
container to snoop/hijack traffic to/from other containers. This is
similar to how a computers on the same LAN, connected to a dumb
switch, can potentially snoop/hijack traffic to/from other computers.
This is ethernet bridge issue, not iptables issue, nor lxc issue.

To prevent that issue, there are some options you can do. One option
is to create a separate bridge for each container. The other option
would be to use my alternative setup which I linked to earlier, which
does NOT use bridge.

-- 
Fajar
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)

2015-03-11 Thread Fajar A. Nugraha
On Wed, Mar 11, 2015 at 7:02 PM, Fiedler Roman  wrote:
> This should be exactly the configuration I have tested  so far. But that did
> not yet solve my problem ...
>
> * If some process in guest registers for the same NFLOG queue, he can "steal"
> the messages from the host queue, thus removing traces of his activity from
> host logging. SECURITY-ASPECT: apart from log corruption, the guest can get
> knowledge about any other connection to/from other containers and the host and
> as they include  sequence numbers, may be able to inject spoofed data into any
> other unencrypted TCP connection or at least interrupt the connection using
> another helper machine.

No. What makes you believe that?

Host and containers does not share iptables rules. Their entire
network stack is separated thru network namespace. There's no such
thing as "stealing the message".

A test would probably prove my statement faster. Try this on your
container, while keeping the same rules on the host side:

iptables -I INPUT 1 -d 192.168.124.173 -j NFLOG --nflog-group 0
--nflog-prefix lxc-v
iptables -I OUTPUT 1 -s 192.168.124.173 -j NFLOG --nflog-group 0
--nflog-prefix lxc-v


Note that on the container the chain has to be INPUT on OUTPUT instead
of FORWARD. Then test it. The container will logs this

Mar 11 12:10:56 v lxc-v IN= OUT=eth0 MAC= SRC=192.168.124.173
DST=192.168.124.134 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=26016 DF
PROTO=ICMP TYPE=8 CODE=0 ID=2087 SEQ=1 UID=0 GID=0 MARK=0
Mar 11 12:10:55 v lxc-v IN=eth0 OUT=
MAC=00:16:3e:f5:cd:94:00:16:3e:2e:d2:6d:08:00 SRC=192.168.124.134
DST=192.168.124.173 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=29087 PROTO=ICMP
TYPE=0 CODE=0 ID=2087 SEQ=1 MARK=0
Mar 11 12:10:57 v lxc-v IN= OUT=eth0 MAC= SRC=192.168.124.173
DST=192.168.124.134 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=26020 DF
PROTO=ICMP TYPE=8 CODE=0 ID=2087 SEQ=2 UID=0 GID=0 MARK=0
Mar 11 12:10:56 v lxc-v IN=eth0 OUT=
MAC=00:16:3e:f5:cd:94:00:16:3e:2e:d2:6d:08:00 SRC=192.168.124.134
DST=192.168.124.173 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=29287 PROTO=ICMP
TYPE=0 CODE=0 ID=2087 SEQ=2 MARK=0


... while at the same time, the host also logs it
Mar 11 19:10:55 utopic lxc-v IN=br0 OUT=br0
MAC=00:16:3e:2e:d2:6d:00:16:3e:f5:cd:94:08:00 SRC=192.168.124.173
DST=192.168.124.134 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=26016 DF
PROTO=ICMP TYPE=8 CODE=0 ID=2087 SEQ=1 MARK=0
Mar 11 19:10:55 utopic lxc-v IN=br0 OUT=br0
MAC=00:16:3e:f5:cd:94:00:16:3e:2e:d2:6d:08:00 SRC=192.168.124.134
DST=192.168.124.173 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=29087 PROTO=ICMP
TYPE=0 CODE=0 ID=2087 SEQ=1 MARK=0
Mar 11 19:10:56 utopic lxc-v IN=br0 OUT=br0
MAC=00:16:3e:2e:d2:6d:00:16:3e:f5:cd:94:08:00 SRC=192.168.124.173
DST=192.168.124.134 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=26020 DF
PROTO=ICMP TYPE=8 CODE=0 ID=2087 SEQ=2 MARK=0
Mar 11 19:10:56 utopic lxc-v IN=br0 OUT=br0
MAC=00:16:3e:f5:cd:94:00:16:3e:2e:d2:6d:08:00 SRC=192.168.124.134
DST=192.168.124.173 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=29287 PROTO=ICMP
TYPE=0 CODE=0 ID=2087 SEQ=2 MARK=0


(note that my host and container uses different timezone, so the times
look 7-hours apart)


>
> * If a guest should be protected with iptables also, e.g. to avoid Apache or
> Tomcat to connect to the local SSH port, those error logs - which are useful
> to detect ongoing guest intrusions - do not make it to any log-file.

No.

Whatever the container does inside it does not matter to the host. The
host simply capture the packets passing thru the bridge (with
bridge-nf-call-iptables on), and doesn't really care what the
container does with it (block it, drop it, accept it, whatever).

On the host side, nflog does NOT stop iptables processing, so the
chains further down (e.g. the ones doing ACCEPT, or REJECT if you
want) will still see the packets. You can verify it with "iptables -nL
-v", and see that the hit counters for both the nflog rule AND
ACCEPT/REJECT rule increase.


>
> [...snip...]
>
>> You might only be missing the "bridge-nf-call-iptables" part. Note
>> that you shouldn't need it IF you use a custom lxc network setup which
>> doesn't use bridges:
>> https://www.mail-archive.com/lxc-users@lists.linuxcontainers.org/msg02587.html
>
> I've tried various network configurations also. I fear that effort here is
> quite futile since I do not yet understand the core kernel namespace concepts,

That will be your biggest problem.

Short version: when using containers installed using templates
(preferably download template), you need to treat container's
networking stack as separate (e.g. on a different "server") with the
one on the host.

The only "link" the host has to the container is the veth pair
(veth by default on host side, usually eth0 on the container
side). Think of lxcbr0 (or whatever bridge you use) as a "switch",
with veth as the switch port that is connected to the container.


> e.g. how nflink and user namespaces work together. Hence everything I could do
> is (inefficient) trial and error instead of controlled engineering. And in the
> e

Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)

2015-03-11 Thread Fiedler Roman
> Von: lxc-users [mailto:lxc-users-boun...@lists.linuxcontainers.org] Im 
> Auftrag
>
> On Wed, Mar 11, 2015 at 5:48 PM, Fiedler Roman 
> wrote:
> > Hello list,
> >
> > Has someone managed to get reliable network traffic auditing with LXC up
> and
> > running? That means, that it is possible to write a protocol of e.g. every
> > new connection from and to host.
> >
> > On my setup (Ubuntu Trusty), both host and guest may have different
> iptables
> > rulesets. But the guest NFLOG messages are lost completely, those from 
> > host
> > are sometimes sent to the ulogd in the guest (time-race), so the host log 
> > is
> > not trustworthy also.
> >
> > What could be the best solution to get trustworthy logs with LXC?
>
> Try something like this on the host:
>
> echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
> iptables -I FORWARD 1 -d 192.168.124.173 -j NFLOG --nflog-group 0
> --nflog-prefix lxc-v
> iptables -I FORWARD 1 -s 192.168.124.173 -j NFLOG --nflog-group 0
> --nflog-prefix lxc-v
>
> with the default ulogd2 setup on ubuntu 14.10 (which already includes
> rules for nflog-group 0 logging to a file)  you should then be able to
> get something like this when the container (192.168.124.173) pings
> another container (192.168.124.134)

This should be exactly the configuration I have tested  so far. But that did 
not yet solve my problem ...

* If some process in guest registers for the same NFLOG queue, he can "steal" 
the messages from the host queue, thus removing traces of his activity from 
host logging. SECURITY-ASPECT: apart from log corruption, the guest can get 
knowledge about any other connection to/from other containers and the host and 
as they include  sequence numbers, may be able to inject spoofed data into any 
other unencrypted TCP connection or at least interrupt the connection using 
another helper machine.

* If a guest should be protected with iptables also, e.g. to avoid Apache or 
Tomcat to connect to the local SSH port, those error logs - which are useful 
to detect ongoing guest intrusions - do not make it to any log-file.

[...snip...]

> You might only be missing the "bridge-nf-call-iptables" part. Note
> that you shouldn't need it IF you use a custom lxc network setup which
> doesn't use bridges:
> https://www.mail-archive.com/lxc-users@lists.linuxcontainers.org/msg02587.html

I've tried various network configurations also. I fear that effort here is 
quite futile since I do not yet understand the core kernel namespace concepts, 
e.g. how nflink and user namespaces work together. Hence everything I could do 
is (inefficient) trial and error instead of controlled engineering. And in the 
end I cannot be sure that there are not reliability/security-relevant holes 
left open with trial/error.

Without any other clues from the mailing list, I'll still try out this 
procedure also and see if it would change the nflog behavior.

Roman


smime.p7s
Description: S/MIME cryptographic signature
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

Re: [lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)

2015-03-11 Thread Fajar A. Nugraha
On Wed, Mar 11, 2015 at 5:48 PM, Fiedler Roman  wrote:
> Hello list,
>
> Has someone managed to get reliable network traffic auditing with LXC up and
> running? That means, that it is possible to write a protocol of e.g. every
> new connection from and to host.
>
> On my setup (Ubuntu Trusty), both host and guest may have different iptables
> rulesets. But the guest NFLOG messages are lost completely, those from host
> are sometimes sent to the ulogd in the guest (time-race), so the host log is
> not trustworthy also.
>
> What could be the best solution to get trustworthy logs with LXC?

Try something like this on the host:

echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
iptables -I FORWARD 1 -d 192.168.124.173 -j NFLOG --nflog-group 0
--nflog-prefix lxc-v
iptables -I FORWARD 1 -s 192.168.124.173 -j NFLOG --nflog-group 0
--nflog-prefix lxc-v

with the default ulogd2 setup on ubuntu 14.10 (which already includes
rules for nflog-group 0 logging to a file)  you should then be able to
get something like this when the container (192.168.124.173) pings
another container (192.168.124.134)

# tail -f /var/log/ulog/syslogemu.log
Mar 11 18:40:49 utopic lxc-v IN=br0 OUT=br0
MAC=00:16:3e:2e:d2:6d:00:16:3e:f5:cd:94:08:00 SRC=192.168.124.173
DST=192.168.124.134 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=10868 DF
PROTO=ICMP TYPE=8 CODE=0 ID=1916 SEQ=2 MARK=0
Mar 11 18:40:49 utopic lxc-v IN=br0 OUT=br0
MAC=00:16:3e:f5:cd:94:00:16:3e:2e:d2:6d:08:00 SRC=192.168.124.134
DST=192.168.124.173 LEN=84 TOS=00 PREC=0x00 TTL=64 ID=46525 PROTO=ICMP
TYPE=0 CODE=0 ID=1916 SEQ=2 MARK=0


You might only be missing the "bridge-nf-call-iptables" part. Note
that you shouldn't need it IF you use a custom lxc network setup which
doesn't use bridges:
https://www.mail-archive.com/lxc-users@lists.linuxcontainers.org/msg02587.html

-- 
Fajar
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

[lxc-users] lxc-security: iptables audit with nflog not working with default settings (insecure)

2015-03-11 Thread Fiedler Roman
Hello list,

Has someone managed to get reliable network traffic auditing with LXC up and
running? That means, that it is possible to write a protocol of e.g. every
new connection from and to host.

On my setup (Ubuntu Trusty), both host and guest may have different iptables
rulesets. But the guest NFLOG messages are lost completely, those from host
are sometimes sent to the ulogd in the guest (time-race), so the host log is
not trustworthy also.

What could be the best solution to get trustworthy logs with LXC?

Kind regards,
Roma

DI Roman Fiedler
Scientist
Digital Safety & Security Department
Assistive Healthcare Information Technology

AIT Austrian Institute of Technology GmbH
Reininghausstraße 13/1 | 8020 Graz | Austria
T +43(0) 50550 2957 | M +43(0) 664 8561599 | F +43(0) 50550 2950
roman.fied...@ait.ac.at | http://www.ait.ac.at/

FN: 115980 i HG Wien  |  UID: ATU14703506
http://www.ait.ac.at/Email-Disclaimer



smime.p7s
Description: S/MIME cryptographic signature
___
lxc-users mailing list
lxc-users@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users