Re: NPF on domU - more clarity required
On Sun, Dec 28, 2014 at 2:40 AM, Greg Troxel wrote: > > John Nemeth writes: > >> On Dec 27, 10:56am, Greg Troxel wrote: >> One option would be to turn /boot into something that works >> like pvgrub. This shouldn't actually be that hard. This is >> something that I added to the project list a while ago: >> http://wiki.netbsd.org/projects/project/xenboot/ Of course, this >> would require convincing the VPS operator to use it. > > True, but convincing someone to write it and use it is far harder... I think just extending the ufs code in pygrub to understand changes since Solaris might be relatively straightforward. http://xenbits.xensource.com/hg/xen-unstable.hg/file/bca284f67702/tools/libfsimage/ufs/fsys_ufs.c (There are also plans that pygrub should run on NetBSD rump kernel, at which point using the actual NetBSD ffs driver might be possible, but thats away off). Justin
Re: NPF on domU - more clarity required
jnem...@cue.bc.ca (John Nemeth) writes: > I understood what you meant, I was just thinking about the >complexity of dealing with everything. BTW, pvgrub doesn't use PV >ops. It sits in dom0 and uses regular filesystem ops to extract >stuff from the domU's "disk". That sounds like a mixup. pygrub - copies kernel/initrd from the domU image using regular file operations to a temporary place, and then just starts a VM from that. pvgrub - a VM is started with pvgrub as kernel which then chainloads kernel/initrd inside the VM. It is actually a version of grub that uses PV operations.
Re: NPF on domU - more clarity required
On Dec 27, 9:40pm, Greg Troxel wrote: } John Nemeth writes: } > On Dec 27, 10:56am, Greg Troxel wrote: } > } > On Dec 26, 11:32pm, Gerard Lally wrote: } > } } > } > } As a sidenote, if there's a way of eliminating the grub cruft and using } > } > } NetBSD's boot manager instead I'd be glad to hear it. } > } > } > } > No, there isn't a way. This is something that is controlled } > } > by the service provider, and not the domU. } > } } > } But if we made /boot be able to use PV ops, then we could ask people to } > } let it be a kernel choice :-) } > } > Sounds like you would be turning /boot into a mini-kernel. } } I meant to be like pvgrub. I understood what you meant, I was just thinking about the complexity of dealing with everything. BTW, pvgrub doesn't use PV ops. It sits in dom0 and uses regular filesystem ops to extract stuff from the domU's "disk". } > As ext2fs is based on FFS, I'm told that pvgrub (pygrub?) can } > read from FFS, but that the filesystem setup is highly restricted } > (i.e. only works with certain block sizes). I don't seem to have } > the details at hand. } } It would be good to put those in the VPS section. A restricted / } (ffsv1, specific block sizes) may be less annoying for some than a /boot } partition. Unfortunately I don't seem to have the details on hand, so somebody else will have to supply them. } > One option would be to turn /boot into something that works } > like pvgrub. This shouldn't actually be that hard. This is } > something that I added to the project list a while ago: } > http://wiki.netbsd.org/projects/project/xenboot/ Of course, this } > would require convincing the VPS operator to use it. } } True, but convincing someone to write it and use it is far harder... Convincing a VPS to use it would probably be the harder part. }-- End of excerpt from Greg Troxel
Re: NPF on domU - more clarity required
John Nemeth writes: > On Dec 27, 10:56am, Greg Troxel wrote: > } > On Dec 26, 11:32pm, Gerard Lally wrote: > } > } > } As a sidenote, if there's a way of eliminating the grub cruft and using > } > } NetBSD's boot manager instead I'd be glad to hear it. > } > > } > No, there isn't a way. This is something that is controlled > } > by the service provider, and not the domU. > } > } But if we made /boot be able to use PV ops, then we could ask people to > } let it be a kernel choice :-) > > Sounds like you would be turning /boot into a mini-kernel. I meant to be like pvgrub. > As ext2fs is based on FFS, I'm told that pvgrub (pygrub?) can > read from FFS, but that the filesystem setup is highly restricted > (i.e. only works with certain block sizes). I don't seem to have > the details at hand. It would be good to put those in the VPS section. A restricted / (ffsv1, specific block sizes) may be less annoying for some than a /boot partition. > One option would be to turn /boot into something that works > like pvgrub. This shouldn't actually be that hard. This is > something that I added to the project list a while ago: > http://wiki.netbsd.org/projects/project/xenboot/ Of course, this > would require convincing the VPS operator to use it. True, but convincing someone to write it and use it is far harder... pgpvufDrLdzTJ.pgp Description: PGP signature
Re: NPF on domU - more clarity required
On Dec 27, 10:56am, Greg Troxel wrote: } > On Dec 26, 11:32pm, Gerard Lally wrote: } } > } As a sidenote, if there's a way of eliminating the grub cruft and using } > } NetBSD's boot manager instead I'd be glad to hear it. } > } > No, there isn't a way. This is something that is controlled } > by the service provider, and not the domU. } } But if we made /boot be able to use PV ops, then we could ask people to } let it be a kernel choice :-) Sounds like you would be turning /boot into a mini-kernel. } Or, if pvgrub handled FFSv1 and FFSv2 well, we'd still have grub, but at } least not an extra faux root filesystem. As ext2fs is based on FFS, I'm told that pvgrub (pygrub?) can read from FFS, but that the filesystem setup is highly restricted (i.e. only works with certain block sizes). I don't seem to have the details at hand. One option would be to turn /boot into something that works like pvgrub. This shouldn't actually be that hard. This is something that I added to the project list a while ago: http://wiki.netbsd.org/projects/project/xenboot/ Of course, this would require convincing the VPS operator to use it. }-- End of excerpt from Greg Troxel
Re: NPF on domU - more clarity required
At date and time Sat, 27 Dec 2014 14:49:03 +1300, Chris Bannister wrote: > On Fri, Dec 26, 2014 at 11:32:26PM +, Gerard Lally wrote: > > > > Thank you Michael, and thank you to all the other senior NetBSD devs who > > stooped to help out this perpetual newbie, here and in private! > > It would be nice if people posted to the thread so as to help other > users in the future. Point taken, but on this occasion it was just to let me know my question had been posted elsewhere for increased exposure. -- Gerard Lally
Re: NPF on domU - more clarity required
John Nemeth writes: > Even with a "normal" Xen setup, /boot.cfg in a domU is ignored > as the kernel is loaded by the Xen hypervisor and not the NetBSD > bootloader. I clarified this in the HOWTO. > On Dec 26, 11:32pm, Gerard Lally wrote: > } As a sidenote, if there's a way of eliminating the grub cruft and using > } NetBSD's boot manager instead I'd be glad to hear it. > > No, there isn't a way. This is something that is controlled > by the service provider, and not the domU. But if we made /boot be able to use PV ops, then we could ask people to let it be a kernel choice :-) Or, if pvgrub handled FFSv1 and FFSv2 well, we'd still have grub, but at least not an extra faux root filesystem. pgpLuAFe6PP0t.pgp Description: PGP signature
Re: NPF on domU - more clarity required
On Dec 26, 11:32pm, Gerard Lally wrote: } At date and time Fri, 26 Dec 2014 22:38:05 + (UTC), Michael van Elst wrote: } } > lists+netbsd.us...@netmail.ie (Gerard Lally) writes: } > } > >compiling the kernel as a normal user instead of root? I've just noticed } > >the owner and group on /usr/src/sys/arch/amd64/compile/custom-20141226/ } > >are gerard:wsrc. Should that be root:wsrc instead? } > } > It doesn't matter who is the owner of the build directory, but did } > you actually boot this kernel? } } Oh dear. Problem solved. I've made a very silly mistake. With prgmr I } should have placed the custom kernel in /ext2fs/boot/ instead of / } } The domU was not using my custom /netbsd kernel at all. It was still } using the domU kernel installed by sysinst. The kernel specified in } /boot.cfg, which I mistakenly assumed was the booting kernel, is Even with a "normal" Xen setup, /boot.cfg in a domU is ignored as the kernel is loaded by the Xen hypervisor and not the NetBSD bootloader. } irrelevant. NetBSD as a prgmr domU uses a grub setup with the domU } kernel in a small ext2 partition /ext2fs/boot/ and the boot } configuration file /ext2/boot/grub/menu.lst } } Well I am happy this problem is now solved, and I apologise for my } cantankerous first post! Mea culpa. } } Thank you Michael, and thank you to all the other senior NetBSD devs who } stooped to help out this perpetual newbie, here and in private! } } As a sidenote, if there's a way of eliminating the grub cruft and using } NetBSD's boot manager instead I'd be glad to hear it. No, there isn't a way. This is something that is controlled by the service provider, and not the domU. }-- End of excerpt from Gerard Lally
Re: NPF on domU - more clarity required
On Sat, Dec 27, 2014 at 02:49:03PM +1300, Chris Bannister wrote: > On Fri, Dec 26, 2014 at 11:32:26PM +, Gerard Lally wrote: > > > > Thank you Michael, and thank you to all the other senior NetBSD devs who > > stooped to help out this perpetual newbie, here and in private! > > It would be nice if people posted to the thread so as to help other > users in the future. Michael did post the correct solution to the problem to the thread. What more could you want? Thor
Re: NPF on domU - more clarity required
On Fri, Dec 26, 2014 at 11:32:26PM +, Gerard Lally wrote: > > Thank you Michael, and thank you to all the other senior NetBSD devs who > stooped to help out this perpetual newbie, here and in private! It would be nice if people posted to the thread so as to help other users in the future. -- "If you're not careful, the newspapers will have you hating the people who are being oppressed, and loving the people who are doing the oppressing." --- Malcolm X
Re: NPF on domU - more clarity required
In article <20141226214013.8a21.280fc...@netmail.ie>, Gerard Lally wrote: >-=-=-=-=-=- > >late than never: > >NetBSD xx.xen.prgmr.com 7.0_BETA NetBSD 7.0_BETA >(XEN3_DOMU.201412251110Z) amd64 > >System installed using ftp, from nyftp.netbsd.org, with all sets. > >I used the following config to compile the kernel with npf built-in, >using syssrc.tgz from NetBSD 7.0_BETA 201412251110Z: > >/usr/src/sys/arch/amd64/conf/XEN3_DOMU > >Perhaps I caused myself a problem by extracting syssrc.tgz and >compiling the kernel as a normal user instead of root? I've just noticed >the owner and group on /usr/src/sys/arch/amd64/compile/custom-20141226/ >are gerard:wsrc. Should that be root:wsrc instead? (I am in the wsrc >group.) I seem to remember reading it's permissible to compile a kernel >as a normal user once you're in the wsrc group. Sorry, nothing obvious. I'll setup some xen stuff here and give it a try. christos
Re: NPF on domU - more clarity required
At date and time Fri, 26 Dec 2014 22:38:05 + (UTC), Michael van Elst wrote: > lists+netbsd.us...@netmail.ie (Gerard Lally) writes: > > >compiling the kernel as a normal user instead of root? I've just noticed > >the owner and group on /usr/src/sys/arch/amd64/compile/custom-20141226/ > >are gerard:wsrc. Should that be root:wsrc instead? > > It doesn't matter who is the owner of the build directory, but did > you actually boot this kernel? Oh dear. Problem solved. I've made a very silly mistake. With prgmr I should have placed the custom kernel in /ext2fs/boot/ instead of / The domU was not using my custom /netbsd kernel at all. It was still using the domU kernel installed by sysinst. The kernel specified in /boot.cfg, which I mistakenly assumed was the booting kernel, is irrelevant. NetBSD as a prgmr domU uses a grub setup with the domU kernel in a small ext2 partition /ext2fs/boot/ and the boot configuration file /ext2/boot/grub/menu.lst Well I am happy this problem is now solved, and I apologise for my cantankerous first post! Mea culpa. Thank you Michael, and thank you to all the other senior NetBSD devs who stooped to help out this perpetual newbie, here and in private! As a sidenote, if there's a way of eliminating the grub cruft and using NetBSD's boot manager instead I'd be glad to hear it. -- Gerard Lally
Re: NPF on domU - more clarity required
lists+netbsd.us...@netmail.ie (Gerard Lally) writes: >compiling the kernel as a normal user instead of root? I've just noticed >the owner and group on /usr/src/sys/arch/amd64/compile/custom-20141226/ >are gerard:wsrc. Should that be root:wsrc instead? It doesn't matter who is the owner of the build directory, but did you actually boot this kernel?
Re: NPF on domU - more clarity required
At date and time Fri, 26 Dec 2014 20:10:35 + (UTC), Christos Zoulas wrote: > In article <20141226020448.ee93.280fc...@netmail.ie>, > Gerard Lally wrote: > >I have been struggling to get NPF up and running on a NetBSD VPS, > >specifically a Xen domU. I really think for security reasons NPF should > >be nearly ready to go, so that we don't have to spend hours researching > >and pulling our hair out trying to fix what should be a straightforward > >issue, which leaves a machine vulnerable when it probably needs > >protection most. It appears this problem came up some years ago, but > >Googling provides me with no fix. > > > >I understand that NetBSD as a Xen domU does not support kernel modules. > >So the recommendation in the NPF documentation to "modload" npf_ext_log > >does not apply here. Fine, I took a wild guess and compiled a new Xen > >domU kernel with the following two lines added to make sure NPF logging > >and normalisation functionality was compiled into the kernel instead: > > > >options NPF_EXT_LOG > >options NPF_EXT_NORMALISE > > > >Needless to say I also made sure pseudo-device npf was enabled as well. > > > >I also made sure /dev/npf existed, and I created /etc/ifconfig.npflog0 > >with just the word "create". > > > >I kept the contents of npf.conf to a minimum for troubleshooting, but > >NPF just refuses to load. This is the error I get at boot: > > > >npfctl: cannot open '/dev/npf': Device not configured > >npfctl: cannot open '/dev/npf': Device not configured > >/etc/rc.d/npf exited with code 1 > > See if the device driver for npf is registered with the kernel correctly: > > $ sysctl kern.drivers | tr , '\n' | grep npf > [198 -1 npf] Thank you Christos. [root]# sysctl kern.drivers | tr , '\n' | grep npf [198 -1 npf] > Make sure that the device numbers are correct: > > $ ls -l /dev/npf > crw--- 1 root wheel 198, 0 Oct 13 2013 /dev/npf [root]# ls -la /dev/npf crw--- 1 root wheel 198, 0 Dec 26 00:38 /dev/npf > Look at the ktrace output and see what operation fails: > > $ ktrace /sbin/npfctl start > $ kdump | less [root]# ktrace /sbin/npfctl start npfctl: cannot open '/dev/npf': Device not configured [root]# kdump | less kdump.txt attached. I should have added extra information in my last post as well. Better late than never: NetBSD xx.xen.prgmr.com 7.0_BETA NetBSD 7.0_BETA (XEN3_DOMU.201412251110Z) amd64 System installed using ftp, from nyftp.netbsd.org, with all sets. I used the following config to compile the kernel with npf built-in, using syssrc.tgz from NetBSD 7.0_BETA 201412251110Z: /usr/src/sys/arch/amd64/conf/XEN3_DOMU Perhaps I caused myself a problem by extracting syssrc.tgz and compiling the kernel as a normal user instead of root? I've just noticed the owner and group on /usr/src/sys/arch/amd64/compile/custom-20141226/ are gerard:wsrc. Should that be root:wsrc instead? (I am in the wsrc group.) I seem to remember reading it's permissible to compile a kernel as a normal user once you're in the wsrc group. -- Gerard Lally kdump.txt Description: Binary data
Re: NPF on domU - more clarity required
In article <20141226020448.ee93.280fc...@netmail.ie>, Gerard Lally wrote: >I have been struggling to get NPF up and running on a NetBSD VPS, >specifically a Xen domU. I really think for security reasons NPF should >be nearly ready to go, so that we don't have to spend hours researching >and pulling our hair out trying to fix what should be a straightforward >issue, which leaves a machine vulnerable when it probably needs >protection most. It appears this problem came up some years ago, but >Googling provides me with no fix. > >I understand that NetBSD as a Xen domU does not support kernel modules. >So the recommendation in the NPF documentation to "modload" npf_ext_log >does not apply here. Fine, I took a wild guess and compiled a new Xen >domU kernel with the following two lines added to make sure NPF logging >and normalisation functionality was compiled into the kernel instead: > >options NPF_EXT_LOG >options NPF_EXT_NORMALISE > >Needless to say I also made sure pseudo-device npf was enabled as well. > >I also made sure /dev/npf existed, and I created /etc/ifconfig.npflog0 >with just the word "create". > >I kept the contents of npf.conf to a minimum for troubleshooting, but >NPF just refuses to load. This is the error I get at boot: > >npfctl: cannot open '/dev/npf': Device not configured >npfctl: cannot open '/dev/npf': Device not configured >/etc/rc.d/npf exited with code 1 See if the device driver for npf is registered with the kernel correctly: $ sysctl kern.drivers | tr , '\n' | grep npf [198 -1 npf] Make sure that the device numbers are correct: $ ls -l /dev/npf crw--- 1 root wheel 198, 0 Oct 13 2013 /dev/npf Look at the ktrace output and see what operation fails: $ ktrace /sbin/npfctl start $ kdump | less christos
NPF on domU - more clarity required
I have been struggling to get NPF up and running on a NetBSD VPS,
specifically a Xen domU. I really think for security reasons NPF should
be nearly ready to go, so that we don't have to spend hours researching
and pulling our hair out trying to fix what should be a straightforward
issue, which leaves a machine vulnerable when it probably needs
protection most. It appears this problem came up some years ago, but
Googling provides me with no fix.
I understand that NetBSD as a Xen domU does not support kernel modules.
So the recommendation in the NPF documentation to "modload" npf_ext_log
does not apply here. Fine, I took a wild guess and compiled a new Xen
domU kernel with the following two lines added to make sure NPF logging
and normalisation functionality was compiled into the kernel instead:
options NPF_EXT_LOG
options NPF_EXT_NORMALISE
Needless to say I also made sure pseudo-device npf was enabled as well.
I also made sure /dev/npf existed, and I created /etc/ifconfig.npflog0
with just the word "create".
I kept the contents of npf.conf to a minimum for troubleshooting, but
NPF just refuses to load. This is the error I get at boot:
npfctl: cannot open '/dev/npf': Device not configured
npfctl: cannot open '/dev/npf': Device not configured
/etc/rc.d/npf exited with code 1
I have /usr on a separate partition which might cause this error at boot
but should not cause the error when I do
/etc/rc.d/npf reload ; /etc/rc.d/npf start
after the system is up and running.
Here are the contents of npf.conf:
===
# /etc/npf.conf
$wired_v4 = { inet4(xennet0) }
procedure "log" {
log: npflog0
}
group "wired" on $wired_v4 {
# disable 80 until we are sure this is running properly
# pass in final family inet4 proto tcp to $wired_v4 port 80
pass in final family inet4 proto tcp to $wired_v4 port 22022
pass stateful out final family inet4 proto tcp flags S/SA \
from $wired_v4
pass out final family inet4 proto tcp from $wired_v4
pass stateful out final family inet4 from $wired_v4
}
group default {
pass final on lo0 all
block all apply "log"
}
===
I have faced this issue on several occasions now and it is most
frustrating. I would like to be able to have a basic firewall up and
running within five minutes of setting up a machine. I'd been looking
forward to trying NPF but it feels as though I'm in the seven circles
of Hell trying to get it to run.
--
Gerard Lally
