Re: Jail to jail network performance?
On Sun, 25 Sep 2005, Brandon Fosdick wrote: Robert Watson wrote: There are several ways you can do it, but they generally fall into two classes of activies: (1) Modifying the name space exclusion assumption for jails, so that the file system name spaces overlap. One way to do this is with nullfs. (2) Having a daemon or tool that runs outside of the jail and brokers communication between the jails. One example might be a daemon that inserts a UNIX domain socket into both jails and then provides references to shared IPC objects between the two by request. Another example might be a daemon or tool that responds to a request and creates a hard link from a socket/fifo endpoint visible in one jail to a name visible in another jail, perhaps when setting up the jail. The former requires more infrastructure, but the latter is less flexible. The jail(8) man page says that if the MIB security.jail.sysvipc_allowed=1 processes inside a jail can use IPC to talk to stuff in other jails. How does that affect mysql in a jail? Do I need this enabled to run mysql? Last I checked, MySQL used solely TCP and UNIX domain sockets for communication, and not System V IPC. I believe PostgreSQL, however, used System V IPC. Robert N M Watson ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Jail to jail network performance?
Robert Watson wrote: Last I checked, MySQL used solely TCP and UNIX domain sockets for communication, and not System V IPC. I believe PostgreSQL, however, used System V IPC. For some reason I was thinking that domain sockets and System V IPC were the same thing. Now I know better. Thanks. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Jail to jail network performance?
Ok, here's what I've decided. But first, thanks for all of the help. Currently the MySQL databases are on a seperate RAID volume, and I'd like to keep it that way for performance purposes. In general I want to avoid putting anything on the raid that isn't a database. I don't want the raid volme to get hit for both database access and regular httpd/system use. In order to make a hardlink to the mysql socket it would have to be on the same volume as the httpd jail, which would then mean both jails have to be on the raid volume, which is what I want to avoid. However, if mysql isn't jailed it has access to both system and raid volumes and can put it's socket someplace accessible to the httpd jail. Since I've already set mysql to only accept domain socket connections I can probably get away without putting it in a jail. So I've decided to jail httpd and friends and leave mysql running in the host environment. I would have liked to use Robert Watson's suggested socket brokering daemon, and I stumbled on socat which appears to do the trick. But as far as I can tell socat just copies between two sockets, which seems like it might have performance issues. Albeit less so than using TCP sockets. Ideally I would like a daemon like socat that can connect/merge two sockets into one, effectively creating a direct connection and eliminating a copy. But AFAICT that isn't possible with the current interface. I don't know enough about the kernel to know if such a thing is even possible, but intuitively it seems like it should be. If somebody where to make socket merging possible, or tell me how to do it, I would be happy to write the daemon to make use of it. Until then I'm using an unjailed mysql. Thanks for the help. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Jail to jail network performance?
On Mon, Sep 26, 2005 at 12:06:39PM -0700, Brandon Fosdick wrote: Ideally I would like a daemon like socat that can connect/merge two sockets into one, effectively creating a direct connection and eliminating a copy. But AFAICT that isn't possible with the current interface. It depends how dirty you want your hands to get. Such a thing can be achieved. Not so much the merging, but it is possible to pass a file descriptor over a UNIX domain socket, so in theory a small daemon which was able to access both file systems should be able to do a handoff. It would likely mean modifying the MySQL client library, however. See the sendmsg(2) and recvmsg(2) functions, specifically the SOL_SOCKET flag in the recvmsg man page. Craig ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Jail to jail network performance?
Robert Watson wrote: There are several ways you can do it, but they generally fall into two classes of activies: (1) Modifying the name space exclusion assumption for jails, so that the file system name spaces overlap. One way to do this is with nullfs. (2) Having a daemon or tool that runs outside of the jail and brokers communication between the jails. One example might be a daemon that inserts a UNIX domain socket into both jails and then provides references to shared IPC objects between the two by request. Another example might be a daemon or tool that responds to a request and creates a hard link from a socket/fifo endpoint visible in one jail to a name visible in another jail, perhaps when setting up the jail. The former requires more infrastructure, but the latter is less flexible. The jail(8) man page says that if the MIB security.jail.sysvipc_allowed=1 processes inside a jail can use IPC to talk to stuff in other jails. How does that affect mysql in a jail? Do I need this enabled to run mysql? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Jail to jail network performance?
[Sorry, this is a late reply, but might be helpful.] Daniel Gerzo [EMAIL PROTECTED] wrote: Hello Brandon, Thursday, September 15, 2005, 5:17:57 AM, you wrote: [...] nullfs looks interesting. I was thinking about sharing files between jails using NFS, but it looks like nullfs would do the trick with better performance. Although the bugs section of the man page for mount_nullfs is rather scary. Does anyone have any experience with it? Does it actually work? btw unionfs is interesting as well, but the BUGS section is pretty the same :) Another possibility is to use union mounts (i.e. using the -o union mount flag with a regular mount). This works without problems and is very stable, but it is a little less flexible than UNIONFS (or NULLFS) because it merges only the directory entries at the mount point. If the point here is to make /tmp/mysql.sock show up in another jail's file space, can I use a symlink instead? Can a jailed process see the target of the symlink? I read that using such a symlinks has security impacts. Symlinks within a jail cannot point to targets outside of that jail. Best regards Oliver -- Oliver Fromme, secnetix GmbH Co. KG, Marktplatz 29, 85567 Grafing Dienstleistungen mit Schwerpunkt FreeBSD: http://www.secnetix.de/bsd Any opinions expressed in this message may be personal to the author and may not necessarily reflect the opinions of secnetix in any way. When your hammer is C++, everything begins to look like a thumb. -- Steve Haflich, in comp.lang.c++ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Jail to jail network performance?
On 14/09/2005, at 4:59 PM, Uwe Doering wrote: So in case of MySQL you would have to use TCP sockets to communicate between jails. This socket type typically consumes more CPU because of TCP's protocol overhead. However, whether you would actually notice any difference in speed basically depends on how much excess CPU power there is available on that server. The result of running these sort of connections over TCP rather than a UNIX domain connection are far more dramatic than this would indicate. When I wrote mSQL and added UNIX domain sockets as a connection option I was able to produce a 10 fold increase in query processing rates for basic queries. If there's a high and constant rate of queries then doing it over a TCP connection is not going to win you any friends. David ... ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Jail to jail network performance?
Brandon Fosdick wrote: Robert Watson wrote: (1) Modifying the name space exclusion assumption for jails, so that the file system name spaces overlap. One way to do this is with nullfs. nullfs looks interesting. I was thinking about sharing files between jails using NFS, but it looks like nullfs would do the trick with better performance. Although the bugs section of the man page for mount_nullfs is rather scary. Does anyone have any experience with it? Does it actually work? If the point here is to make /tmp/mysql.sock show up in another jail's file space, can I use a symlink instead? Can a jailed process see the target of the symlink? Symlinks are just a path mapping mechanism performed by the kernel at lookup time, that is, before the actual access. In a jail only those parts of a filesystem are visible that are at or below the jail's root directory. The same goes for normal chroots. So if the symlink points to a location outside this scope you cannot access the object. Hardlinks would work, but only if the jails concerned live in the same filesystem. Though they can of course be confined in separate, non-overlapping parts of that filesystem. Uwe -- Uwe Doering | EscapeBox - Managed On-Demand UNIX Servers [EMAIL PROTECTED] | http://www.escapebox.net ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Jail to jail network performance?
Brandon Fosdick wrote: I have a 5.4-S box running apache2 that's serving data from mysql running on the same box. I'm thinking about putting both in seperate jails, partly for security and partly for practice. Would this impact network performance between the two? Currently the mysql connection is using localhost which I understand to be faster than a network socket. Does jail-to-jail traffic use the same mechanism? or something else? In MySQL 'localhost' is a hard-wired shortcut that uses domain sockets instead of TCP sockets. Since domain sockets live in the namespace of a filesystem this requires that both server and client have access to the same filesystem. Now, for security reasons jails normally are confined in separate filesystems, or at least in separate parts of a common one. So in case of MySQL you would have to use TCP sockets to communicate between jails. This socket type typically consumes more CPU because of TCP's protocol overhead. However, whether you would actually notice any difference in speed basically depends on how much excess CPU power there is available on that server. Uwe -- Uwe Doering | EscapeBox - Managed On-Demand UNIX Servers [EMAIL PROTECTED] | http://www.escapebox.net ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Jail to jail network performance?
On Sep 13, 2005, at 11:59 PM, Uwe Doering wrote: Now, for security reasons jails normally are confined in separate filesystems, or at least in separate parts of a common one. So in case of MySQL you would have to use TCP sockets to communicate between jails. This socket type typically consumes more CPU because of TCP's protocol overhead. However, whether you would actually notice any difference in speed basically depends on how much excess CPU power there is available on that server. Ignoring security (or filesystem namespace issues) I will just note that using named sockets for local IPC is a Good Thing. When I worked at Messaging Direct I taught sendmail to speak LMTP over named sockets, and our local delivery rate (to our IMAP server) went up by a factor of 10. It would be really cool if we could figure out a way to do AF_UNIX between jails, but I confess to not having thought about any of the implications ... (Maybe netgraph can help here?) --lyndon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Jail to jail network performance?
On Wed, 14 Sep 2005, Lyndon Nerenberg wrote: On Sep 13, 2005, at 11:59 PM, Uwe Doering wrote: Now, for security reasons jails normally are confined in separate filesystems, or at least in separate parts of a common one. So in case of MySQL you would have to use TCP sockets to communicate between jails. This socket type typically consumes more CPU because of TCP's protocol overhead. However, whether you would actually notice any difference in speed basically depends on how much excess CPU power there is available on that server. Ignoring security (or filesystem namespace issues) I will just note that using named sockets for local IPC is a Good Thing. When I worked at Messaging Direct I taught sendmail to speak LMTP over named sockets, and our local delivery rate (to our IMAP server) went up by a factor of 10. It would be really cool if we could figure out a way to do AF_UNIX between jails, but I confess to not having thought about any of the implications ... (Maybe netgraph can help here?) There are several ways you can do it, but they generally fall into two classes of activies: (1) Modifying the name space exclusion assumption for jails, so that the file system name spaces overlap. One way to do this is with nullfs. (2) Having a daemon or tool that runs outside of the jail and brokers communication between the jails. One example might be a daemon that inserts a UNIX domain socket into both jails and then provides references to shared IPC objects between the two by request. Another example might be a daemon or tool that responds to a request and creates a hard link from a socket/fifo endpoint visible in one jail to a name visible in another jail, perhaps when setting up the jail. The former requires more infrastructure, but the latter is less flexible. Robert N M Watson ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Jail to jail network performance?
Robert Watson wrote: On Wed, 14 Sep 2005, Lyndon Nerenberg wrote: On Sep 13, 2005, at 11:59 PM, Uwe Doering wrote: Now, for security reasons jails normally are confined in separate filesystems, or at least in separate parts of a common one. So in case of MySQL you would have to use TCP sockets to communicate between jails. This socket type typically consumes more CPU because of TCP's protocol overhead. However, whether you would actually notice any difference in speed basically depends on how much excess CPU power there is available on that server. Ignoring security (or filesystem namespace issues) I will just note that using named sockets for local IPC is a Good Thing. When I worked at Messaging Direct I taught sendmail to speak LMTP over named sockets, and our local delivery rate (to our IMAP server) went up by a factor of 10. It would be really cool if we could figure out a way to do AF_UNIX between jails, but I confess to not having thought about any of the implications ... (Maybe netgraph can help here?) There are several ways you can do it, but they generally fall into two classes of activies: (1) Modifying the name space exclusion assumption for jails, so that the file system name spaces overlap. One way to do this is with nullfs. (2) Having a daemon or tool that runs outside of the jail and brokers communication between the jails. One example might be a daemon that inserts a UNIX domain socket into both jails and then provides references to shared IPC objects between the two by request. Another example might be a daemon or tool that responds to a request and creates a hard link from a socket/fifo endpoint visible in one jail to a name visible in another jail, perhaps when setting up the jail. The former requires more infrastructure, but the latter is less flexible. Just a kind reminder to those interested in implementing the daemon approach: Never ever create or write to an object from outside a jail that is located in a part of the filesystem that a live jail can access and modify. Otherwise you may easily fall victim to a symlink attack or similar. Remember that jails set up for security reasons generally are to be considered enemy territory. The correct approach would be to create or open such objects from a chrooted child process. There is only one exception: In the pre-boot phase of a jail you can get away with checking the file path component by component before you touch the object. But as soon as the jail runs the window between checking the path and accessing the object can be exploited from inside the jail. Hope to have helped prevent some rude awakening for some. ;-) Uwe -- Uwe Doering | EscapeBox - Managed On-Demand UNIX Servers [EMAIL PROTECTED] | http://www.escapebox.net ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Jail to jail network performance?
Robert Watson wrote: (1) Modifying the name space exclusion assumption for jails, so that the file system name spaces overlap. One way to do this is with nullfs. nullfs looks interesting. I was thinking about sharing files between jails using NFS, but it looks like nullfs would do the trick with better performance. Although the bugs section of the man page for mount_nullfs is rather scary. Does anyone have any experience with it? Does it actually work? If the point here is to make /tmp/mysql.sock show up in another jail's file space, can I use a symlink instead? Can a jailed process see the target of the symlink? ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
