OK
I run a jails (have for the past 18 months in deployment and a year
before that in testing) with the following setup, currently on 6.1-
RELEASE. This works fine and I have no issues. I am trying to work
on a new set up that is giving me issues.
Here is the set up that works:
I create a master jail that I do not "boot".
I create an md(4) memory backed disk using a regular disk file (vnode
mode) as backings tore. I install the basic directories found in /
in this md device (no files, just the directories). I then populate
the /etc and /var as well as some special ones (/local and /stubs for
example) that are used for jail-specific stuff (/local for all the
"customer" stuff and /stubs for jail specific system stuff like a
place for /usr/local to link in to). I then use nullfs mounts to
mount from the master jail, the /lib /bin /libexec /usr and /sbin
directories in read only fashion. I can then boot the jail and it
runs find. I also have a /usr/public which is read only place for
ports to install in to. I have appropriate links out of the read
only directories into local per jail (per md(4) space) directories.
What I just described all works fine and I can upgrade jails really
easily as I just have to upgrade the master jail installation, watch
for any important etc changes which have to be done individually, and
update my one set of ports [and apps built from source without ports]
in the /usr/public and all the jails get all the changes.
What I want to do is use my Solaris 10 server with 1.7TB ZFS file
system exported through NFS as the root for each jail, with the same
nullfs mounts as used above in the md(4) version. This actually
works in my test jail (but I have not tried to run any applications
inside -- just boot it and log in and do basic shell things). I
tried last night to move an existing jail that runs apache2 for my
dad and a few of his family genealogy sites into such a nfs backed
jail. Apache through fits and I saw that you have to have some local
space for LockFile, SSLMutex, and a few other mod specific things.
So what I did is set up a local directory on the FBSD system with the
normal / directories as I do above in the md(4) way of doing things
and left a local directory for the apache stuff. I then used nullfs
to mount the same dirs as above and then nfs to mount the rest from
the ZFS system. It looks like this
solaris-i3:/local/jails/leigh/etc 2.0G 439M
1.6G 21% /local/jails/leigh/etc
solaris-i3:/local/jails/leigh/home 2.0G 439M
1.6G 21% /local/jails/leigh/home
solaris-i3:/local/jails/leigh/local 2.0G 439M
1.6G 21% /local/jails/leigh/local
solaris-i3:/local/jails/leigh/log 2.0G 439M
1.6G 21% /local/jails/leigh/log
solaris-i3:/local/jails/leigh/root 2.0G 439M
1.6G 21% /local/jails/leigh/root
solaris-i3:/local/jails/leigh/space 2.0G 439M
1.6G 21% /local/jails/leigh/space
solaris-i3:/local/jails/leigh/stubs 2.0G 439M
1.6G 21% /local/jails/leigh/stubs
solaris-i3:/local/jails/leigh/var 2.0G 439M
1.6G 21% /local/jails/leigh/var
/local/jails/master/bin 66G 59G
2.4G 96% /local/jails/leigh/bin
/local/jails/master/lib 66G 59G
2.4G 96% /local/jails/leigh/lib
/local/jails/master/libexec 66G 59G
2.4G 96% /local/jails/leigh/libexec
/local/jails/master/sbin 66G 59G
2.4G 96% /local/jails/leigh/sbin
/local/jails/master/usr 66G 59G
2.4G 96% /local/jails/leigh/usr
procfs 4.0K 4.0K
0B 100% /local/jails/leigh/proc
devfs 1.0K 1.0K
0B 100% /local/jails/leigh/dev
This boots fine, and apache no longer gives a fit (nothing in
error_log) as I tell it to put its lock files and stuff in /tmp/
scratch which is a local space not nullfs mounted nor nfs mounted.
I can log in to the jail and do shell things. I can change over to
my dad's shell account and do things (read files and write new files
or change files) as his user and group inside of the nfs mounted /
local space. There are NO permission problems (the nfs is mounted
with maproot=root [or the equivalent solaris way of exporting it]
etc) either root or the user shell account to do things inside the
nfs mounted space.
However, apache, which runs as the user and group of my dad's login
account since he is the sole user of this jail and sole user of
apacge, cannot read any of the website files. The exact same apache
config file when using the md(4) backed space with all the same files
and permissions, has no problems.
Here are some examples from the virtual host error files:
[Wed Oct 04 12:53:19 2006] [error] [client 67.171.127.191] (13)
Permission denied: file permissions deny server access: /local/web/
leigh.org/www.leigh.org/index.html
[Wed Oct 04 12:53:19 2006] [error] [client 67.171.127.191] File does
not exist: /local/web/leigh.org/www.leigh.org/favicon.ico
[Wed Oct 04 12:53:21 2006] [error] [client 67.171.127.191] (13)
Permission denied: file permissions deny server access: /local/web/
leigh.org/www.leigh.org/index.html
[Wed Oct 04 12:53:21 2006] [error] [client 67.171.127.191] File does
not exist: /local/web/leigh.org/www.leigh.org/favicon.ico
[Wed Oct 04 12:54:55 2006] [crit] [client 68.114.59.6] (13)Permission
denied: /local/web/leigh.org/www.leigh.org/_derived/.htaccess
pcfg_openfile: unable to check htaccess file, ensure it is readable,
referer: http://www.leigh.org/running/podcast.html
[Wed Oct 04 12:54:56 2006] [crit] [client 68.114.59.6] (13)Permission
denied: /local/web/leigh.org/www.leigh.org/_themes/.htaccess
pcfg_openfile: unable to check htaccess file, ensure it is readable,
referer: http://www.leigh.org/running/podcast.html
[Wed Oct 04 12:54:56 2006] [crit] [client 68.114.59.6] (13)Permission
denied: /local/web/leigh.org/www.leigh.org/_themes/.htaccess
pcfg_openfile: unable to check htaccess file, ensure it is readable,
referer: http://www.leigh.org/running/podcast.html
[Wed Oct 04 12:54:58 2006] [crit] [client 68.114.59.6] (13)Permission
denied: /local/web/leigh.org/www.leigh.org/_derived/.htaccess
pcfg_openfile: unable to check htaccess file, ensure it is readable,
referer: http://www.leigh.org/running/podcast.html
[Wed Oct 04 12:55:15 2006] [crit] [client 74.6.74.61] (13)Permission
denied: /local/web/leigh.org/www.leigh.org/genealogy/.htaccess
pcfg_openfile: unable to check htaccess file, ensure it is readable
Again, the user that apache is running as can access the files R/W no
problem. I have confirmed this by running a shell as the same user
(and the same apache when running on the md(4) back jail with the
same local directorys tructure with same user and permissions etc
runs fine). I also confirmed the permissions of everything
visually. user and group are r(+x as appropriate) including all dirs.
I am at a loss on why, when the apache virtual host document roots
are living on an nfs mounted space this would happen. (Again, the
LockFile and other things like that [SSLMutex, mod)rewrite stuff] is
set to a local space that is not nfs mounted and apache is not giving
and crazy errors to indicate otherwise in the apache wide error_log
like it did when I first started my attempts).
Any ideas or help would be appreciated.
In case anyone wants to know why I want to do this jail on nfs thing:
1) I want to get rid of the md(4) backed devices as they seem to not
be completely stable -- once in a while I get a lost I/O that leads
to hanging the server. I cannot prove it is related to md(4) but it
always starts in relation to a jail on the md(4) device (and when I
try and do an ls of the file for example, once it starts, the ls of
that file will hang while other ls won't).
2) More importantly, I don't want a jail to be tied to a specific HW
server. I want to be able to move a jail easily around HW servers as
needed. For example, if a specific server were to develop HW
problems, I could easily shut it down and bring up its allotment of
jails on another server or set of servers with very minimal downtime
to my customers. Right now, with the md(4) backed devices, the jails
live where their storage backing is and each server has their own set
of disks/storage. If a server dies, I cannot easily bring up all my
jails on another server without HW hacking. I know I have a single
point of failure with the Solaris server and its large disk array but
it is easier to throw money at a single file server and have a lot of
smaller minimal front end servers than have to buy larger beefier
front end servers in multiples. The solaris server has 2 areca raid
6 arrays (1 installed and 1 to be installed later this Fall) mirrored
together using ZFS, has a 2+1 redundant power supply that will be
spread across 2 circuits, battery backed raid arrays, etc. is on
hospital grade power with UPS etc. with extra spare parts to quickly
replace things that fail.
Thanks
Chad
---
Chad Leigh -- Shire.Net LLC
Your Web App and Email hosting provider
chad at shire.net
