[Lxc-users] Network interface isolation
Hi guys, I'm admittedly quite new to LXC therefore forgive me if this is expected behaviour and/or has been documented/discussed here before. I've been toying with using lxc as a sandboxing mechanism (process and filesystem mainly) in my open source (GPLv3) project eBrainPool. I created a barebone debian container as a test bed using the following command: lxc-create -n myfirstcontainer -t debian My host system has two network cards - eth0 and eth2. I have not setup any bridge device atm. My container simply has an eth0 device which gets configured via DHCP. There is no other device mentioned in /etc/network/interfaces. While booting the container, it tries to get eth0 configured via DHCP and fails as expected. However ifconfig shows me another eth2 device in the container. This has the same ip address as the eth2 on the host. Furthermore, I can bring down the eth2 from the container and have it effect the eth2 on the host. Similarly bringing down the interface in the host affects the container. I can also ping to the internet (via eth2) in the container. All of this is done without any explicit network settings in the config file created by lxc-create. I'll try and copy paste contents of my config file just as soon as the internet on my device with the container becomes functional. I may have missed something however from what I can see there are no network related options and it seems to be a bare bone config file created by lxc-create, though of course as I said I may have missed something. Also, lxc-checkconfig does show that my kernel (debian testing with kernel 3.1.x) does have the network namespace enabled. I would appreciate if someone could shed light as to if this is normal and expected behaviour and if so how could I bring about network isolation within my container. Thank you so much :) Regards, Jeetu ebrain.in | Beehive Computing Discover and run software from devices around you - share your software and computing resources. A GPLv3 licensed project. -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Network interface isolation
Thanks so much Matthijs :)truly appreciate the help :)will try this out :) Regards, Jeetu ebrain.in | Beehive Computing Discover and run software from devices around you - share your software and computing resources. A GPLv3 licensed project. On Mon, May 14, 2012 at 8:07 PM, Matthijs Kooijman wrote: > Hi Jeetu, > >> I would appreciate if someone could shed light as to if this is normal >> and expected behaviour and if so how could I bring about network >> isolation within my container. > AFAIU, this is normal: If you don't configure any networks within the > lxc config file, no network isolation happens and the container shares > the same network stack as the host. > > So it should be sufficient to just add network configuration. For > example, to give the container access to (just) the eth0 device: > > lxc.network.type = phys > lxc.network.link = eth0 > > I think these should be sufficient (not using this configuration myself, > though). > > Gr. > > Matthijs > > -BEGIN PGP SIGNATURE- > Version: GnuPG v1.4.9 (GNU/Linux) > > iEYEARECAAYFAk+xGMIACgkQz0nQ5oovr7xBHwCfad342fvu/73nrI69xIYtSYui > cLUAoLy+AHcT7rCejAFpthUZfcyIlft7 > =zWU7 > -END PGP SIGNATURE- > -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Network interface isolation
Just to add to this discussion for the benefit of someone else that runs into a similar issue. Tried the following : >> lxc.network.type = phys >> lxc.network.link = eth0 This resulted in the container failing to start with : lxc-start: failed to move 'eth0' to the container : Message too long lxc-start: failed to create the configured network lxc-start: failed to spawn 'test1' lxc-start: Device or resource busy - failed to remove cgroup '/var/local/cgroup/test1' As per a previous thread here http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg00249.html changed to lxc.network.type = macvlan it all works well now :)and as Matthijs suggests this now provides network isolation and a single network interface eth0 in the container. Am sure there is a very good reason though I'm still not clear as to why the default action for the container is to share the network stack of the host in the absence of explicit specification in the config file. Could someone please point me to a discussion on this just so I can get a better understanding of lxc design decisions. Are there other similar instances where I should make specific mention in the config file in order to prevent accidental and inadvertent sharing of resources between host and container? Thanks again Matthijs and everyone here for all your help :) Bye for now Jeetu ebrain.in | Beehive Computing Discover and run software from devices around you - share your software and computing resources. A GPLv3 licensed project. On Mon, May 14, 2012 at 8:44 PM, jeetu.gol...@gmail.com wrote: > Thanks so much Matthijs :)truly appreciate the help :)will try > this out :) > > Regards, > Jeetu > ebrain.in | Beehive Computing > Discover and run software from devices around you - share your > software and computing resources. A GPLv3 licensed project. > > > On Mon, May 14, 2012 at 8:07 PM, Matthijs Kooijman wrote: >> Hi Jeetu, >> >>> I would appreciate if someone could shed light as to if this is normal >>> and expected behaviour and if so how could I bring about network >>> isolation within my container. >> AFAIU, this is normal: If you don't configure any networks within the >> lxc config file, no network isolation happens and the container shares >> the same network stack as the host. >> >> So it should be sufficient to just add network configuration. For >> example, to give the container access to (just) the eth0 device: >> >> lxc.network.type = phys >> lxc.network.link = eth0 >> >> I think these should be sufficient (not using this configuration myself, >> though). >> >> Gr. >> >> Matthijs >> >> -BEGIN PGP SIGNATURE- >> Version: GnuPG v1.4.9 (GNU/Linux) >> >> iEYEARECAAYFAk+xGMIACgkQz0nQ5oovr7xBHwCfad342fvu/73nrI69xIYtSYui >> cLUAoLy+AHcT7rCejAFpthUZfcyIlft7 >> =zWU7 >> -END PGP SIGNATURE- >> -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Network interface isolation
Hi Fajar, Thanks for your response. I used the debian template and the config file does not (as far as I can tell) have any network related stanzas. Unfortunately this default behaviour lead to the network stack being shared between the host and the container as pointed out by Matthijs. Therefore, I was wondering if there are any other namespaces I should explicitly isolate so as to prevent them being inadvertently shared between host and container. Thanks again for all your help :) Bye for now Jeetu ebrain.in | Beehive Computing Discover and run software from devices around you - share your software and computing resources. A GPLv3 licensed project. On Tue, May 15, 2012 at 2:25 AM, Fajar A. Nugraha wrote: > On Mon, May 14, 2012 at 11:48 PM, jeetu.gol...@gmail.com > wrote: > >> Are there other similar instances where I should make specific mention >> in the config file in order to prevent accidental and inadvertent >> sharing of resources between host and container? > > Try creating a container using templates. e.g: > > lxc-create -n test1 -t busybox > > then look at the resulting config file. It provides a good starting point. > > -- > Fajar -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Network interface isolation
Hi Fajar,
Thanks again for taking the time to reply :)
I'm using Debian testing with lxc version 0.8.0~rc1-4. It's possible
I'm using an outdated version of lxc.
> In Ubuntu host at least, AFAIK using lxc templates will include
> network configuration from /etc/lxc/lxc.conf, which includes these
> lines
I just verified and on my system I don't have a lxc.conf file. I'm not
sure if I have to manually create this.
The default config template I have when I run a lxc-create -n test1 -t
debian is as follows:
lxc.tty = 4
lxc.pts = 1024
lxc.rootfs = /var/lib/lxc/test1/rootfs
lxc.cgroup.devices.deny = a
# /dev/null and zero
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
# consoles
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 4:0 rwm
lxc.cgroup.devices.allow = c 4:1 rwm
# /dev/{,u}random
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
# rtc
lxc.cgroup.devices.allow = c 254:0 rwm
# mounts point
lxc.mount.entry=proc /var/lib/lxc/test1/rootfs/proc proc nodev,noexec,nosuid 0 0
lxc.mount.entry=devpts /var/lib/lxc/test1/rootfs/dev/pts devpts defaults 0 0
lxc.mount.entry=sysfs /var/lib/lxc/test1/rootfs/sys sysfs defaults 0 0
I'm not sure of the specifics in the config file however my first
reaction to lxc.cgroup.devices.deny = a was to assume that this would
deny all devices just as you have mentioned. However this hasn't
isolated the network instance in my case.
Anything in the above config seems out of ordinary? Or is it just that
I'm using an outdated lxc? Of course it's entirely possible that I
haven't rtfm ;) ;) - sorry if that's the case :)
Thanks again for everything guys :)
Bye for now
On Tue, May 15, 2012 at 10:13 AM, Fajar A. Nugraha wrote:
> On Tue, May 15, 2012 at 10:22 AM, jeetu.gol...@gmail.com
> wrote:
>> Hi Fajar,
>>
>> Thanks for your response.
>>
>> I used the debian template and the config file does not (as far as I
>> can tell) have any network related stanzas. Unfortunately this default
>> behaviour lead to the network stack being shared between the host and
>> the container as pointed out by Matthijs.
>
> In Ubuntu host at least, AFAIK using lxc templates will include
> network configuration from /etc/lxc/lxc.conf, which includes these
> lines
>
> lxc.network.type=veth
> lxc.network.link=lxcbr0
> lxc.network.flags=up
>
>>
>> Therefore, I was wondering if there are any other namespaces I should
>> explicitly isolate so as to prevent them being inadvertently shared
>> between host and container.
>
> Again, the resulting config file provides a good starting point. On
> ubuntu precise host, using ubuntu template denies ALL devices (which
> should include host's eth) from the guest container except from ones
> specifically allowed
>
> #==
> lxc.cgroup.devices.deny = a
> # Allow any mknod (but not using the node)
> lxc.cgroup.devices.allow = c *:* m
> lxc.cgroup.devices.allow = b *:* m
> # /dev/null and zero
> lxc.cgroup.devices.allow = c 1:3 rwm
> lxc.cgroup.devices.allow = c 1:5 rwm
> # consoles
> lxc.cgroup.devices.allow = c 5:1 rwm
> lxc.cgroup.devices.allow = c 5:0 rwm
> #lxc.cgroup.devices.allow = c 4:0 rwm
> #lxc.cgroup.devices.allow = c 4:1 rwm
> # /dev/{,u}random
> lxc.cgroup.devices.allow = c 1:9 rwm
> lxc.cgroup.devices.allow = c 1:8 rwm
> lxc.cgroup.devices.allow = c 136:* rwm
> lxc.cgroup.devices.allow = c 5:2 rwm
> # rtc
> lxc.cgroup.devices.allow = c 254:0 rwm
> #fuse
> lxc.cgroup.devices.allow = c 10:229 rwm
> #tun
> lxc.cgroup.devices.allow = c 10:200 rwm
> #full
> lxc.cgroup.devices.allow = c 1:7 rwm
> #hpet
> lxc.cgroup.devices.allow = c 10:228 rwm
> #kvm
> lxc.cgroup.devices.allow = c 10:232 rwm
> #==
>
> --
> Fajar
--
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
___
Lxc-users mailing list
Lxc-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Network interface isolation
Hi Serge,
Thanks for taking the time :)
>
> Note you can of course just add the network lines to this file by
> yourself, you don't have to create a whole new container right now :)
>
> No, the automatic use of a system lxc.conf is just an ubuntu thing. Can't
> really go upstream because it's pretty distro-specific.
That explains that :)
>From my limited knowledge though it seems that lxc.cgroup.devices.deny
= a would deny access to all devices and shouldn't this therefore
isolate network interfaces in the host from the container? As I
mentioned in spite of this setting my container can see and operate on
interfaces in the host. Explicitly adding the network stanza to config
as recommended solves that however I'm wondering if this is deliberate
by design and if so the rationale behind this - just trying to get a
deeper understanding of design considerations of lxc.
I'm also concerned that similarly there could be other devices /
resources not automatically isolated and that require explicity
configuration.
Would be great if I could be pointed in the right direction :)
Thanks so much again :)
Bye for now
Jeetu
ebrain.in | Beehive Computing
Discover and run software from devices around you - share your
software and computing resources. A GPLv3 licensed project.
On Tue, May 15, 2012 at 8:55 PM, Serge Hallyn
wrote:
> Quoting jeetu.gol...@gmail.com (jeetu.gol...@gmail.com):
>> Hi Fajar,
>>
>> Thanks again for taking the time to reply :)
>>
>> I'm using Debian testing with lxc version 0.8.0~rc1-4. It's possible
>> I'm using an outdated version of lxc.
>>
>> > In Ubuntu host at least, AFAIK using lxc templates will include
>> > network configuration from /etc/lxc/lxc.conf, which includes these
>> > lines
>>
>> I just verified and on my system I don't have a lxc.conf file. I'm not
>> sure if I have to manually create this.
>
> Right, just create one yourself, i.e.
>
> cat > lxc.conf << EOF
> lxc.network.type = phys
> lxc.network.link = eth0
> EOF
>
> and then add '-f lxc.conf' to the lxc-create arguments.
>
>> The default config template I have when I run a lxc-create -n test1 -t
>> debian is as follows:
>>
>> lxc.tty = 4
>
> Note you can of course just add the network lines to this file by
> yourself, you don't have to create a whole new container right now :)
>
>> lxc.pts = 1024
>> lxc.rootfs = /var/lib/lxc/test1/rootfs
>> lxc.cgroup.devices.deny = a
>> # /dev/null and zero
>> lxc.cgroup.devices.allow = c 1:3 rwm
>> lxc.cgroup.devices.allow = c 1:5 rwm
>> # consoles
>> lxc.cgroup.devices.allow = c 5:1 rwm
>> lxc.cgroup.devices.allow = c 5:0 rwm
>> lxc.cgroup.devices.allow = c 4:0 rwm
>> lxc.cgroup.devices.allow = c 4:1 rwm
>> # /dev/{,u}random
>> lxc.cgroup.devices.allow = c 1:9 rwm
>> lxc.cgroup.devices.allow = c 1:8 rwm
>> lxc.cgroup.devices.allow = c 136:* rwm
>> lxc.cgroup.devices.allow = c 5:2 rwm
>> # rtc
>> lxc.cgroup.devices.allow = c 254:0 rwm
>>
>> # mounts point
>> lxc.mount.entry=proc /var/lib/lxc/test1/rootfs/proc proc nodev,noexec,nosuid
>> 0 0
>> lxc.mount.entry=devpts /var/lib/lxc/test1/rootfs/dev/pts devpts defaults 0 0
>> lxc.mount.entry=sysfs /var/lib/lxc/test1/rootfs/sys sysfs defaults 0 0
>>
>>
>> I'm not sure of the specifics in the config file however my first
>> reaction to lxc.cgroup.devices.deny = a was to assume that this would
>> deny all devices just as you have mentioned. However this hasn't
>> isolated the network instance in my case.
>>
>> Anything in the above config seems out of ordinary? Or is it just that
>> I'm using an outdated lxc? Of course it's entirely possible that I
>
> No, the automatic use of a system lxc.conf is just an ubuntu thing. Can't
> really go upstream because it's pretty distro-specific.
>
>> haven't rtfm ;) ;) - sorry if that's the case :)
>>
>> Thanks again for everything guys :)
>>
>> Bye for now
>>
>>
>>
>>
>> On Tue, May 15, 2012 at 10:13 AM, Fajar A. Nugraha wrote:
>> > On Tue, May 15, 2012 at 10:22 AM, jeetu.gol...@gmail.com
>> > wrote:
>> >> Hi Fajar,
>> >>
>> >> Thanks for your response.
>> >>
>> >> I used the debian template and the config file does not (as far as I
>> >> can tell) have any network related stanzas. Unfortunately this default
>> >> behaviour lead to the network stack being shared between the host and
>> >> the container as pointed o
Re: [Lxc-users] Network interface isolation
Hi Serge, > > the devices cgroup only prevents access to block and character device > nodes in the filesystem. (i.e. /dev/loop0 which is block maj 7 minor 0) > > > Plenty. Containers are not root-secure. See > https://wiki.ubuntu.com/LxcSecurity for starters. > Awesome :)thanks so much :) Bye for now Jeetu ebrain.in | Beehive Computing Discover and run software from devices around you - share your software and computing resources. A GPLv3 licensed project. On Tue, May 15, 2012 at 10:31 PM, Serge Hallyn wrote: > Quoting jeetu.gol...@gmail.com (jeetu.gol...@gmail.com): >> Hi Serge, >> >> Thanks for taking the time :) >> >> > >> > Note you can of course just add the network lines to this file by >> > yourself, you don't have to create a whole new container right now :) >> > >> >> > No, the automatic use of a system lxc.conf is just an ubuntu thing. Can't >> > really go upstream because it's pretty distro-specific. >> >> That explains that :) >> >> >From my limited knowledge though it seems that lxc.cgroup.devices.deny >> = a would deny access to all devices and shouldn't this therefore >> isolate network interfaces in the host from the container? As I > > the devices cgroup only prevents access to block and character device > nodes in the filesystem. (i.e. /dev/loop0 which is block maj 7 minor 0) > >> mentioned in spite of this setting my container can see and operate on >> interfaces in the host. Explicitly adding the network stanza to config >> as recommended solves that however I'm wondering if this is deliberate >> by design and if so the rationale behind this - just trying to get a >> deeper understanding of design considerations of lxc. >> >> I'm also concerned that similarly there could be other devices / >> resources not automatically isolated and that require explicity >> configuration. > > Plenty. Containers are not root-secure. See > https://wiki.ubuntu.com/LxcSecurity for starters. > > -serge -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
