Re: Long Format Directory Listing 15x Slower in FreeBSD 5.x
If you are using NIS and have any compat options in /etc/nsswitch.conf your performance will really suck in situations like this. IIRC, the compat code is worse than O(n^2) if you look up each user and the non-compat code is close to O(n). I'd really like to stop generating nsswitch.conf entries that use compat in 7.0. I'm pretty sure there's a problem with the database hash parameters used by pwd_mkdb on systems with large numbers of users..this is what I was alluding to above. That wouldn't suprised me. For that matter it could be both problems. Looks like exact problem is described here: http://www.freebsd.org/cgi/query-pr.cgi?pr=75855 I'm not using NIS, so I will try the change /etc/nsswitch.conf group and passwd lines from 'compat' to 'files' and report back. Last night I tried to migration over to 5.4 after making the changes in /etc/nsswitch.conf and the load average reduced from 50 to 0.5-1.5. This made a *huge* difference, however the load average is still noticeable higher than on FBSD 4. Using top you can see the ipop3d and imapd processors are using more CPU than on FBSD 4. On FBSD 4 at the same time (3am relatively light load) the load average is between 0.15 - 0.4. This may not seem like a huge difference, but the hardware specs between the two system are huge. We are going from a P4 2.4Ghz, Raid 1 IDE 7200RPM Drives to Dual Xeon 3.4Ghz Six Drive RAID 10 SATA 10K RPM Raptors /w 3ware 9000 Controller. We have also tried increasing the hash parameters in pwd_mkdb.c, but this made no difference whatsoever. As a result we are going to run the system on FBSD 4.11 now; it seems to me that FBSD 5 has some serious performance issues for certain operations. -- Nikolai. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Long Format Directory Listing 15x Slower in FreeBSD 5.x
Hello, We have been trying to migrate to FreeBSD 5.4 from an older 4.x release for one of our busy mail servers. But we have encountered problems with directory listings on 5.4. Our /var/mail directory contains approximately 8,000 files doing a long directory listing (ls -l) takes approximately 5 min and during this time the CPU is running near 100%, on a FSBD 4 box the same directory contents takes less than 20 seconds to list. Even on a directory with 200 files, each file with a different owner, it will still take at least 4-5 seconds to list. The problems only seems to occur when the directory contains files from many different users. (as in /var/mail). If I chown all the files in the /var/mail directory to a single user the directory listing is near instant. It appears it has something to do with 'ls' looking up the id's in the password database, because if I instruct ls to display numeric IDs (ls -ln), rather than converting to user and group names, the directory with 8,000 files, with 8,000 different owners will list instantly. The reason this concerns me so much is we are also having a problem with our Washington IMAP / POP3 server on FBSD 5 using a lot of CPU while operating on small and even 0 byte mailboxes, when there are approx five or more concurrent POP3 or IMAP sessions. And I can't help thinking that the two problems are related. Does anyone have any ideas? Has anyone else noticed this problem? If I can't resolve it I'm most likely going to revert to using 4.11, but I'd really like to know what is going on. (and yes I'm using UFS_DIRHASH) -- Nikolai ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Long Format Directory Listing 15x Slower in FreeBSD 5.x
On Tue, Sep 13, 2005 at 11:55:55AM +1200, Nikolai Schupbach wrote: Hello, We have been trying to migrate to FreeBSD 5.4 from an older 4.x release for one of our busy mail servers. But we have encountered problems with directory listings on 5.4. Our /var/mail directory contains approximately 8,000 files doing a long directory listing (ls -l) takes approximately 5 min and during this time the CPU is running near 100%, on a FSBD 4 box the same directory contents takes less than 20 seconds to list. Even on a directory with 200 files, each file with a different owner, it will still take at least 4-5 seconds to list. The problems only seems to occur when the directory contains files from many different users. (as in /var/mail). If I chown all the files in the /var/mail directory to a single user the directory listing is near instant. It appears it has something to do with 'ls' looking up the id's in the password database, because if I instruct ls to display numeric IDs (ls -ln), rather than converting to user and group names, the directory with 8,000 files, with 8,000 different owners will list instantly. The reason this concerns me so much is we are also having a problem with our Washington IMAP / POP3 server on FBSD 5 using a lot of CPU while operating on small and even 0 byte mailboxes, when there are approx five or more concurrent POP3 or IMAP sessions. And I can't help thinking that the two problems are related. Does anyone have any ideas? Has anyone else noticed this problem? If I can't resolve it I'm most likely going to revert to using 4.11, but I'd really like to know what is going on. (and yes I'm using UFS_DIRHASH) Sounds like the real problem is that you have 8000 users on your machine and lookups are taking a long time. There has been discussion of this problem and how to solve it on freebsd-questions@ several times..I think it involves making a minor change to pwd_mkdb and rebuilding. Please search the archives for that mailing list. Kris pgpCROcLmQGLn.pgp Description: PGP signature
Re: Long Format Directory Listing 15x Slower in FreeBSD 5.x
On Mon, Sep 12, 2005 at 08:51:44PM -0400, Kris Kennaway wrote: On Tue, Sep 13, 2005 at 11:55:55AM +1200, Nikolai Schupbach wrote: Hello, We have been trying to migrate to FreeBSD 5.4 from an older 4.x release for one of our busy mail servers. But we have encountered problems with directory listings on 5.4. Our /var/mail directory contains approximately 8,000 files doing a long directory listing (ls -l) takes approximately 5 min and during this time the CPU is running near 100%, on a FSBD 4 box the same directory contents takes less than 20 seconds to list. Even on a directory with 200 files, each file with a different owner, it will still take at least 4-5 seconds to list. The problems only seems to occur when the directory contains files from many different users. (as in /var/mail). If I chown all the files in the /var/mail directory to a single user the directory listing is near instant. It appears it has something to do with 'ls' looking up the id's in the password database, because if I instruct ls to display numeric IDs (ls -ln), rather than converting to user and group names, the directory with 8,000 files, with 8,000 different owners will list instantly. The reason this concerns me so much is we are also having a problem with our Washington IMAP / POP3 server on FBSD 5 using a lot of CPU while operating on small and even 0 byte mailboxes, when there are approx five or more concurrent POP3 or IMAP sessions. And I can't help thinking that the two problems are related. Does anyone have any ideas? Has anyone else noticed this problem? If I can't resolve it I'm most likely going to revert to using 4.11, but I'd really like to know what is going on. (and yes I'm using UFS_DIRHASH) Sounds like the real problem is that you have 8000 users on your machine and lookups are taking a long time. There has been discussion of this problem and how to solve it on freebsd-questions@ several times..I think it involves making a minor change to pwd_mkdb and rebuilding. Please search the archives for that mailing list. If you are using NIS and have any compat options in /etc/nsswitch.conf your performance will really suck in situations like this. IIRC, the compat code is worse than O(n^2) if you look up each user and the non-compat code is close to O(n). I'd really like to stop generating nsswitch.conf entries that use compat in 7.0. -- Brooks -- Any statement of the form X is the one, true Y is FALSE. PGP fingerprint 655D 519C 26A7 82E7 2529 9BF0 5D8E 8BE9 F238 1AD4 pgpPy7HoRC9yn.pgp Description: PGP signature
Re: Long Format Directory Listing 15x Slower in FreeBSD 5.x
On Mon, Sep 12, 2005 at 06:33:53PM -0700, Brooks Davis wrote: On Mon, Sep 12, 2005 at 08:51:44PM -0400, Kris Kennaway wrote: On Tue, Sep 13, 2005 at 11:55:55AM +1200, Nikolai Schupbach wrote: Hello, We have been trying to migrate to FreeBSD 5.4 from an older 4.x release for one of our busy mail servers. But we have encountered problems with directory listings on 5.4. Our /var/mail directory contains approximately 8,000 files doing a long directory listing (ls -l) takes approximately 5 min and during this time the CPU is running near 100%, on a FSBD 4 box the same directory contents takes less than 20 seconds to list. Even on a directory with 200 files, each file with a different owner, it will still take at least 4-5 seconds to list. The problems only seems to occur when the directory contains files from many different users. (as in /var/mail). If I chown all the files in the /var/mail directory to a single user the directory listing is near instant. It appears it has something to do with 'ls' looking up the id's in the password database, because if I instruct ls to display numeric IDs (ls -ln), rather than converting to user and group names, the directory with 8,000 files, with 8,000 different owners will list instantly. The reason this concerns me so much is we are also having a problem with our Washington IMAP / POP3 server on FBSD 5 using a lot of CPU while operating on small and even 0 byte mailboxes, when there are approx five or more concurrent POP3 or IMAP sessions. And I can't help thinking that the two problems are related. Does anyone have any ideas? Has anyone else noticed this problem? If I can't resolve it I'm most likely going to revert to using 4.11, but I'd really like to know what is going on. (and yes I'm using UFS_DIRHASH) Sounds like the real problem is that you have 8000 users on your machine and lookups are taking a long time. There has been discussion of this problem and how to solve it on freebsd-questions@ several times..I think it involves making a minor change to pwd_mkdb and rebuilding. Please search the archives for that mailing list. If you are using NIS and have any compat options in /etc/nsswitch.conf your performance will really suck in situations like this. IIRC, the compat code is worse than O(n^2) if you look up each user and the non-compat code is close to O(n). I'd really like to stop generating nsswitch.conf entries that use compat in 7.0. I'm pretty sure there's a problem with the database hash parameters used by pwd_mkdb on systems with large numbers of users..this is what I was alluding to above. Kris pgpP53wjAWGyt.pgp Description: PGP signature
Re: Long Format Directory Listing 15x Slower in FreeBSD 5.x
On Mon, Sep 12, 2005 at 09:37:27PM -0400, Kris Kennaway wrote: On Mon, Sep 12, 2005 at 06:33:53PM -0700, Brooks Davis wrote: On Mon, Sep 12, 2005 at 08:51:44PM -0400, Kris Kennaway wrote: On Tue, Sep 13, 2005 at 11:55:55AM +1200, Nikolai Schupbach wrote: Hello, We have been trying to migrate to FreeBSD 5.4 from an older 4.x release for one of our busy mail servers. But we have encountered problems with directory listings on 5.4. Our /var/mail directory contains approximately 8,000 files doing a long directory listing (ls -l) takes approximately 5 min and during this time the CPU is running near 100%, on a FSBD 4 box the same directory contents takes less than 20 seconds to list. Even on a directory with 200 files, each file with a different owner, it will still take at least 4-5 seconds to list. The problems only seems to occur when the directory contains files from many different users. (as in /var/mail). If I chown all the files in the /var/mail directory to a single user the directory listing is near instant. It appears it has something to do with 'ls' looking up the id's in the password database, because if I instruct ls to display numeric IDs (ls -ln), rather than converting to user and group names, the directory with 8,000 files, with 8,000 different owners will list instantly. The reason this concerns me so much is we are also having a problem with our Washington IMAP / POP3 server on FBSD 5 using a lot of CPU while operating on small and even 0 byte mailboxes, when there are approx five or more concurrent POP3 or IMAP sessions. And I can't help thinking that the two problems are related. Does anyone have any ideas? Has anyone else noticed this problem? If I can't resolve it I'm most likely going to revert to using 4.11, but I'd really like to know what is going on. (and yes I'm using UFS_DIRHASH) Sounds like the real problem is that you have 8000 users on your machine and lookups are taking a long time. There has been discussion of this problem and how to solve it on freebsd-questions@ several times..I think it involves making a minor change to pwd_mkdb and rebuilding. Please search the archives for that mailing list. If you are using NIS and have any compat options in /etc/nsswitch.conf your performance will really suck in situations like this. IIRC, the compat code is worse than O(n^2) if you look up each user and the non-compat code is close to O(n). I'd really like to stop generating nsswitch.conf entries that use compat in 7.0. I'm pretty sure there's a problem with the database hash parameters used by pwd_mkdb on systems with large numbers of users..this is what I was alluding to above. That wouldn't suprised me. For that matter it could be both problems. -- Brooks -- Any statement of the form X is the one, true Y is FALSE. PGP fingerprint 655D 519C 26A7 82E7 2529 9BF0 5D8E 8BE9 F238 1AD4 pgpJ1aPp9fcJg.pgp Description: PGP signature
Re: Long Format Directory Listing 15x Slower in FreeBSD 5.x
Brooks Davis wrote: On Mon, Sep 12, 2005 at 09:37:27PM -0400, Kris Kennaway wrote: On Mon, Sep 12, 2005 at 06:33:53PM -0700, Brooks Davis wrote: On Mon, Sep 12, 2005 at 08:51:44PM -0400, Kris Kennaway wrote: On Tue, Sep 13, 2005 at 11:55:55AM +1200, Nikolai Schupbach wrote: Hello, We have been trying to migrate to FreeBSD 5.4 from an older 4.x release for one of our busy mail servers. But we have encountered problems with directory listings on 5.4. Our /var/mail directory contains approximately 8,000 files doing a long directory listing (ls -l) takes approximately 5 min and during this time the CPU is running near 100%, on a FSBD 4 box the same directory contents takes less than 20 seconds to list. Even on a directory with 200 files, each file with a different owner, it will still take at least 4-5 seconds to list. The problems only seems to occur when the directory contains files from many different users. (as in /var/mail). If I chown all the files in the /var/mail directory to a single user the directory listing is near instant. It appears it has something to do with 'ls' looking up the id's in the password database, because if I instruct ls to display numeric IDs (ls -ln), rather than converting to user and group names, the directory with 8,000 files, with 8,000 different owners will list instantly. The reason this concerns me so much is we are also having a problem with our Washington IMAP / POP3 server on FBSD 5 using a lot of CPU while operating on small and even 0 byte mailboxes, when there are approx five or more concurrent POP3 or IMAP sessions. And I can't help thinking that the two problems are related. Does anyone have any ideas? Has anyone else noticed this problem? If I can't resolve it I'm most likely going to revert to using 4.11, but I'd really like to know what is going on. (and yes I'm using UFS_DIRHASH) Sounds like the real problem is that you have 8000 users on your machine and lookups are taking a long time. There has been discussion of this problem and how to solve it on freebsd-questions@ several times..I think it involves making a minor change to pwd_mkdb and rebuilding. Please search the archives for that mailing list. If you are using NIS and have any compat options in /etc/nsswitch.conf your performance will really suck in situations like this. IIRC, the compat code is worse than O(n^2) if you look up each user and the non-compat code is close to O(n). I'd really like to stop generating nsswitch.conf entries that use compat in 7.0. I'm pretty sure there's a problem with the database hash parameters used by pwd_mkdb on systems with large numbers of users..this is what I was alluding to above. That wouldn't suprised me. For that matter it could be both problems. -- Brook Looks like exact problem is described here: http://www.freebsd.org/cgi/query-pr.cgi?pr=75855 I'm not using NIS, so I will try the change /etc/nsswitch.conf group and passwd lines from 'compat' to 'files' and report back. -- Nikolai. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
