Re: [vfs] buf_daemon() slows down write() severely on low-speed CPU
On Tue, Mar 13, 2012 at 01:54:38PM +0100, Svatopluk Kraus wrote:
On Mon, Mar 12, 2012 at 7:19 PM, Konstantin Belousov
kostik...@gmail.com wrote:
On Mon, Mar 12, 2012 at 04:00:58PM +0100, Svatopluk Kraus wrote:
Hi,
I have solved a following problem. If a big file (according to
'hidirtybuffers') is being written, the write speed is very poor.
It's observed on system with elan 486 and 32MB RAM (i.e., low speed
CPU and not too much memory) running FreeBSD-9.
Analysis: A file is being written. All or almost all dirty buffers
belong to the file. The file vnode is almost all time locked by
writing process. The buf_daemon() can not flush any dirty buffer as a
chance to acquire the file vnode lock is very low. A number of dirty
buffers grows up very slow and with each new dirty buffer slower,
because buf_daemon() eats more and more CPU time by looping on dirty
buffers queue (with very low or no effect).
This slowing down effect is started by buf_daemon() itself, when
'numdirtybuffers' reaches 'lodirtybuffers' threshold and buf_daemon()
is waked up by own timeout. The timeout fires at 'hz' period, but
starts to fire at 'hz/10' immediately as buf_daemon() fails to reach
'lodirtybuffers' threshold. When 'numdirtybuffers' (now slowly)
reaches ((lodirtybuffers + hidirtybuffers) / 2) threshold, the
buf_daemon() can be waked up within bdwrite() too and it's much worse.
Finally and with very slow speed, the 'hidirtybuffers' or
'dirtybufthresh' is reached, the dirty buffers are flushed, and
everything starts from beginning...
Note that for some time, bufdaemon work is distributed among bufdaemon
thread itself and any thread that fails to allocate a buffer, esp.
a thread that owns vnode lock and covers long queue of dirty buffers.
However, the problem starts when numdirtybuffers reaches
lodirtybuffers count and ends around hidirtybuffers count. There are
still plenty of free buffers in system.
On the system, a buffer size is 512 bytes and the default
thresholds are following:
vfs.hidirtybuffers = 134
vfs.lodirtybuffers = 67
vfs.dirtybufthresh = 120
For example, a 2MB file is copied into flash disk in about 3
minutes and 15 second. If dirtybufthresh is set to 40, the copy time
is about 20 seconds.
My solution is a mix of three things:
1. Suppresion of buf_daemon() wakeup by setting bd_request to 1 in
the main buf_daemon() loop.
I cannot understand this. Please provide a patch that shows what do
you mean there.
curthread-td_pflags |= TDP_NORUNNINGBUF | TDP_BUFNEED;
mtx_lock(bdlock);
for (;;) {
- bd_request = 0;
+ bd_request = 1;
mtx_unlock(bdlock);
Is this a complete patch ? The change just causes lost wakeups for bufdaemon,
nothing more.
I read description of bd_request variable. However, bd_request should
serve as an indicator that buf_daemon() is in sleep. I.e., the
following paradigma should be used:
mtx_lock(bdlock);
bd_request = 0;/* now, it's only time when wakeup() will be meaningful */
sleep(bd_request, ..., hz/10);
bd_request = 1; /* in case of timeout, we must set it (bd_wakeup()
already set it) */
mtx_unlock(bdlock);
My patch follows the paradigma. What happens without the patch in
described problem: buf_daemon() fails in its job and goes to sleep
with hz/10 period. It supposes that next early wakeup will do nothing
too. bd_request is untouched but buf_daemon() doesn't know if its last
wakeup was made by bd_wakeup() or by timeout. So, bd_request could be
0 and buf_daemon() can be waked up before hz/10 just by bd_wakeup().
Moreover, setting bd_request to 0 when buf_daemon() is not in sleep
can cause time consuming and useless wakeup() calls without effect.
pgpgYYUkYWWIP.pgp
Description: PGP signature
Updating fscd
Hi, fscd by Tom Rhodes (see http://people.freebsd.org/~trhodes/fsc/) is a *very* nice tool for administrators, and I want to use it with our computers. But there were some usability flaws and he unfortunately didn't have time to continue working on it. These are my modifications, based on the last version he worked on: * there is a configuration file with one service per line, which should be monitored when starting up. * there is no database anymore, and no pidfiles being saved. You just need the service name (as processed by rc(8)), the rest will be fetched from rc(8)/service(8). * the whole startup of a service is now outsourced to service(8). We don't check pids anymore, we only ask service for the service's status and pids and then monitor these, the same after restarting. * several internal changes. The code from Tom Rhodes was nice, but when I worked on it it was easier for me to think of my own code structure instead of thinking into his ones. Anyway, most of his structure and code is still in it. * everything works and compiles in NetBSD. I'd like to see the service being named something more generic (bscd? ;), but I'm not the one to decide upon the code of others. There were some minor incompatibilities, though nothing you couldn't fix in five minutes, I'd prefer having a single codebase instead of two diverging ones. To use it, you: * power up fscd with `fscd` (nothing more) * enable a service with `fscadm enable $SERVICE` * stop monitoring it with `fscadm disable $SERVICE` * show status with `fscadm status` * close fscd with `fscadm shutdown` (or kill[all]) You can also supply the option `-s` to fscd and fscadm to specify the socket to be used instead of the standard one (/var/run/fscd.sock). You can as well supply the option `-c` to fscd to specify the configuration file to be used instead of the standard one (/etc/fscd.conf). You can download it here: https://vcs.in-berlin.de/schrank21_fscd/ (Files-Version-Tarball) NOTE: The manpages are *out of date*, they will be updated soon. Any comments, questions, further change requests, etc.? If you think the changes are sane, I'll also update the manpages. Regards, Julian PS: Please include me in the replies, I'm not subscribed to this list. signature.asc Description: PGP signature
Re: [vfs] buf_daemon() slows down write() severely on low-speed CPU
2012/3/15 Konstantin Belousov kostik...@gmail.com:
On Tue, Mar 13, 2012 at 01:54:38PM +0100, Svatopluk Kraus wrote:
On Mon, Mar 12, 2012 at 7:19 PM, Konstantin Belousov
kostik...@gmail.com wrote:
On Mon, Mar 12, 2012 at 04:00:58PM +0100, Svatopluk Kraus wrote:
Hi,
I have solved a following problem. If a big file (according to
'hidirtybuffers') is being written, the write speed is very poor.
It's observed on system with elan 486 and 32MB RAM (i.e., low speed
CPU and not too much memory) running FreeBSD-9.
Analysis: A file is being written. All or almost all dirty buffers
belong to the file. The file vnode is almost all time locked by
writing process. The buf_daemon() can not flush any dirty buffer as a
chance to acquire the file vnode lock is very low. A number of dirty
buffers grows up very slow and with each new dirty buffer slower,
because buf_daemon() eats more and more CPU time by looping on dirty
buffers queue (with very low or no effect).
This slowing down effect is started by buf_daemon() itself, when
'numdirtybuffers' reaches 'lodirtybuffers' threshold and buf_daemon()
is waked up by own timeout. The timeout fires at 'hz' period, but
starts to fire at 'hz/10' immediately as buf_daemon() fails to reach
'lodirtybuffers' threshold. When 'numdirtybuffers' (now slowly)
reaches ((lodirtybuffers + hidirtybuffers) / 2) threshold, the
buf_daemon() can be waked up within bdwrite() too and it's much worse.
Finally and with very slow speed, the 'hidirtybuffers' or
'dirtybufthresh' is reached, the dirty buffers are flushed, and
everything starts from beginning...
Note that for some time, bufdaemon work is distributed among bufdaemon
thread itself and any thread that fails to allocate a buffer, esp.
a thread that owns vnode lock and covers long queue of dirty buffers.
However, the problem starts when numdirtybuffers reaches
lodirtybuffers count and ends around hidirtybuffers count. There are
still plenty of free buffers in system.
On the system, a buffer size is 512 bytes and the default
thresholds are following:
vfs.hidirtybuffers = 134
vfs.lodirtybuffers = 67
vfs.dirtybufthresh = 120
For example, a 2MB file is copied into flash disk in about 3
minutes and 15 second. If dirtybufthresh is set to 40, the copy time
is about 20 seconds.
My solution is a mix of three things:
1. Suppresion of buf_daemon() wakeup by setting bd_request to 1 in
the main buf_daemon() loop.
I cannot understand this. Please provide a patch that shows what do
you mean there.
curthread-td_pflags |= TDP_NORUNNINGBUF | TDP_BUFNEED;
mtx_lock(bdlock);
for (;;) {
- bd_request = 0;
+ bd_request = 1;
mtx_unlock(bdlock);
Is this a complete patch ? The change just causes lost wakeups for bufdaemon,
nothing more.
Yes, it's a complete patch. And exactly, it causes lost wakeups which are:
1. !! UNREASONABLE !!, because bufdaemon is not sleeping,
2. not wanted, because it looks that it's correct behaviour for the
sleep with hz/10 period. However, if the sleep with hz/10 period is
expected to be waked up by bd_wakeup(), then bd_request should be set
to 0 just before sleep() call, and then bufdaemon behaviour will be
clear.
All stuff around bd_request and bufdaemon sleep is under bd_lock, so
if bd_request is 0 and bufdaemon is not sleeping, then all wakeups are
unreasonable! The patch is about that mainly.
I read description of bd_request variable. However, bd_request should
serve as an indicator that buf_daemon() is in sleep. I.e., the
following paradigma should be used:
mtx_lock(bdlock);
bd_request = 0; /* now, it's only time when wakeup() will be meaningful */
sleep(bd_request, ..., hz/10);
bd_request = 1; /* in case of timeout, we must set it (bd_wakeup()
already set it) */
mtx_unlock(bdlock);
My patch follows the paradigma. What happens without the patch in
described problem: buf_daemon() fails in its job and goes to sleep
with hz/10 period. It supposes that next early wakeup will do nothing
too. bd_request is untouched but buf_daemon() doesn't know if its last
wakeup was made by bd_wakeup() or by timeout. So, bd_request could be
0 and buf_daemon() can be waked up before hz/10 just by bd_wakeup().
Moreover, setting bd_request to 0 when buf_daemon() is not in sleep
can cause time consuming and useless wakeup() calls without effect.
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
FTSENT: name and path on `/' versus name and path on `*/'
Found a curious incongruent behavior in fts(3), wondering if there is some reason for this, or if it's just a bug. If you include the path `/' the FTSENT at depth 0 that is returned for the path has both fts_path = / and fts_name = /, compared to other entries, like /var which has fts_path = / and fts_path = / and fts_name = var, or /var/, which has fts_path = /var/ and fts_name = . Given the behavior of other paths used in fts(3), my expectation here is that FTSENT for path / on depth 0 would have fts_path = / and fts_name = . Haven't delved down into the code enough to figure out where this is happening, but from a cursory read through libc/gen/fts.c there doesn't seem to be any explicit special casing of the path /. Can anyone shed light on why this behavior is desirable, or if it's just a bug I'm happy to file a PR and delve further into fts.c ... -- regards, matt ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: compiling ports with SSP (was: [PATCH] Add -lssp_nonshared to GCC's LIB_SPEC unconditionally)=
Hi Bryan On Sun, Feb 26, 2012 at 09:41:07PM -0600, Bryan Drewery wrote: Thanks for this patch [1]! I've been building my ports tree with -fstack-protector on FreeBSD 6, 7 and 8. Once I upgraded to 8, I started running into the issue [2] this patch is fixing. I have a situation where non-ports applications are compiling statically, which ran into this. Specifically, the application is linking in security/openssl statically, which of course was compiled with -fstack-protector. Adding the /usr/lib/libc.ld fixed it without needing to hack at the failing non-port application. Would be nice if this, and PR 138228 were finally committed. Bryan Drewery [1] http://lists.freebsd.org/pipermail/freebsd-hackers/2011-June/035538.html [2] http://gcc.gnu.org/ml/gcc-help/2006-05/msg00092.html Wow, the perspective provided by those two posts makes me dizzy. This has been a very long standing project. The base system is now compiled with SSP, but doing so for ports still requires some manual hacking unfortenately. I've proposed a patch to compile ports with SSP a few years ago, but some ports with special building strategy suffered the problem described in [2]. Then I learned the possibilities of ld scripts and provided the patch in [1] last year. I think we have all the bits necessary to be able to compile ports with SSP painlessly. First the patch in [1] has to be committed in the base system. I think this can be done in CURRENT without any problem, I run it myself on my own servers without problem. Unfortunately it will probably never appear in RELENG_9 because it may be deemed too dangerous to make such a change in a stable branch. It would be nice to hear what kib@ and kan@ think about this. Next, the patch to bsd.port.mk in this PR [3] has to be applied to be able to compile ports with SSP using a single knob. (Other patches along this one can be thrown away, they were required hacks back when the libc ld script didn't exist.) Then portmgr@ will naturally want to make a full port build with this knob turned on to check, but last time I was told they had very few resource and that this couldn't be scheduled in the next couple of week, IIRC. I admit the situation is partly my fault, because I did the fun technical work but I didn't keep up with the lobbying part :). I asked once or twice, without success, and then went to other subjects. I would be really glad if we could proceed with this. FreeBSD-9.0 has just been release, this is probably a good time to step forward. [3] http://www.freebsd.org/cgi/query-pr.cgi?pr=ports/138228 Cheers, -- Jeremie Le Hen Men are born free and equal. Later on, they're on their own. Jean Yanne ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Capture states of all processes at the same time
I am unable to enter DDB. I used the command from this link: http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-online-ddb.html where it says: -- The second scenario is to drop to the debugger once the system has booted. There are two simple ways to accomplish this. If you would like to break to the debugger from the command prompt, simply type the command: # sysctl debug.kdb.enter=1 -- But when I type that the computer hangs! What do I do? My primary aim is to get the process states of all processes running at a particular time, save them and be able to retrieve them when needed so I can run the processes from those states. Also how can I know in which CPU (I am using an intel i5 multicore system) core each process is running? On 15 March 2012 01:57, Julian Elischer jul...@freebsd.org wrote: On 3/14/12 12:02 PM, Artem Belevich wrote: On Wed, Mar 14, 2012 at 11:25 AM, Maninya Mmani...@gmail.com wrote: Then typed this to force a panic: sysctl debug.kdb.panic=1 The computer just hung after this, and after waiting for a while I pressed the reboot button. It said no core dumps found while rebooting. First, make sure you have swap space configured. If minidump is not enabled (check sysctl debug.minidump) you will need to make sure you have more swap space than physical memory. Then make sure that dump device is set up correctly. See dumpdev in rc.conf(5) If that didn't work, you may be running into the issue in PR kern/155421: http://www.freebsd.org/cgi/**query-pr.cgi?pr=kern%2F155421**cat=http://www.freebsd.org/cgi/query-pr.cgi?pr=kern%2F155421cat= Alas, I don't know what to do about that. or just do ps from ddb and then continue. you can set things up in 9 (and maybe 8, I don't know) to capture the ddb output.. --Artem __**_ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/**mailman/listinfo/freebsd-**hackershttp://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscribe@** freebsd.org freebsd-hackers-unsubscr...@freebsd.org -- Maninya ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
