Re: [vfs] buf_daemon() slows down write() severely on low-speed CPU
On Wed, Mar 21, 2012 at 5:55 AM, Adrian Chadd adr...@freebsd.org wrote:
Hi,
I'm interested in this, primarily because I'm tinkering with file
storage stuff on my little (most wifi targetted) embedded MIPS
platforms.
So what's the story here? How can I reproduce your issue and do some
of my own profiling/investigation?
Adrian
Hi,
your interest has made me to do more solid/comparable investigation on
my embedded ELAN486 platform. With more test results, I made full
tracing of related VFS, filesystem, and disk function calls. It took
some time to understand what about the issue really is.
My test case:
Single file copy (no O_FSYNC). It means that no other filesystem
operation is served. The file size must be big enough according to
hidirtybuffers value. Other processes on machine, where the test was
run, almost were inactive. The real copy time was profiled. In all
tests, a machine was booted, a file was copied, file was removed, the
machine was rebooted. Thus, the the file was copied into same disk
layout.
The motivation is that my embedded machines don't do any writing to a
disk mostly. Only during software update, a single process is writing
to a disk (file by file). It doesn't need to be a problem at all, but
an update must be successful even under full cpu load. So, the writing
should be tuned up greatly to not affect other processes too much and
to finish in finite time.
On my embedded ELAN486 machines, a flash memory is used as a disk. It
means that a reading is very fast, but a writing is slow. Further, a
flash memory is divided into sectors and only complete sector can be
erased at once. A sector erasure is very time expensive action.
When I tried to tune up VFS by various parameters changing, I found
out that real copy time depends on two things. Both of them are a
subject of bufdaemon. Namely, its feature to try to work harder, if
its buffers flushing mission is failing. It's not suprise that the
best copy times were achived when bufdaemon was excluded from buffers
flushing at all (by VFS parameters setting).
This bufdaemon feature brings along (with respect to the real copy time):
1. bufdaemon runtime itself,
2. very frequent filesystem buffers flushing.
What really happens in the test case on my machine:
A copy program uses a buffer for coping. The default buffer size is
128 KiB in my case. The simplified sys_write() implementation for
DTYPE_VNODE and VREG type is following:
sys_write()
{
dofilewrite()
{
bwillwrite()
fo_write() = vn_write()
{
bwillwrite()
vn_lock()
VOP_WRITE()
VOP_UNLOCK()
}
}
}
So, all 128 KiB is written under VNODE lock. When I take back the
machine defaults:
hidirtybuffers: 134
lodirtybuffers: 67
bufdirtythresh: 120
buffer size (filesystem block size): 512 bytes
and do some simple calculations:
134 * 512 = 68608 - high water bytes count
120 * 512 = 61440
67 * 512 = 34304 - low water byte count
then it's obvious that bufdaemon has something to do during each
sys_write(). However, almost all dirty buffers belong to new file
VNODE and the VNODE is locked. What remains are filesystem buffers
only. I.e., superblock buffer and free block bitmap buffers. So,
bufdaemon iterates over all dirty buffers queue, what takes a
SIGNIFICANT time on my machine, and does not find any buffer to be
able to flush almost all time. If bufdaemon flushes one or two
buffers, kern_yield() is called, and new iteration is started until no
buffer is flushed. So, very often TWO full iteration over dirty
buffers queue is done to flush only one or two filesystem buffers and
to failed to reach lodirtybuffers threshold. A bufdaemon runtime is
growing up. Moreover, the frequent filesystem buffers flushing brings
along higher cpu load (geom down thread, geom up thread, disk thread
scheduling) and a disk blocks writing re-ordering. The correct disk
blocks writing order is important for the flash disk. Further, while
the file data buffers are aged but not flushed, filesystem buffers are
written repeatedly but flushed.
Of course, I use a sector cache in the flash disk, but I can't cache
too many sectors because of total memory size. So, filesystem disk
blocks often are written and that evokes more disk sector flushes. A
sector flush really takes long time, so real copy time grows up beyond
control. Last but not least, the flash memory are going to be aged
uselessly.
Well, this is my old story. Just to be honest, I quite forgot that my
kernel was compiled with FULL_PREEMPTION option. The things are very
much worse in this case. However, the option just makes the issue
worse, the issue doesn't disapper without it.
In this old story, I played a game with and focused to bufdirtythresh
value. However, bufdirtythresh is changing the way, how and by who
buffers are flushed, too much. I recorded disk sector flush count and
total disk_strategy() calls count with BIO_WRITE command (and total
bytes count to write). I used a file with size 2235517 bytes. When I
was caching
Re: [vfs] buf_daemon() slows down write() severely on low-speed CPU
2012/3/21 Konstantin Belousov kostik...@gmail.com:
On Thu, Mar 15, 2012 at 08:00:41PM +0100, Svatopluk Kraus wrote:
2012/3/15 Konstantin Belousov kostik...@gmail.com:
On Tue, Mar 13, 2012 at 01:54:38PM +0100, Svatopluk Kraus wrote:
On Mon, Mar 12, 2012 at 7:19 PM, Konstantin Belousov
kostik...@gmail.com wrote:
On Mon, Mar 12, 2012 at 04:00:58PM +0100, Svatopluk Kraus wrote:
Hi,
I have solved a following problem. If a big file (according to
'hidirtybuffers') is being written, the write speed is very poor.
It's observed on system with elan 486 and 32MB RAM (i.e., low speed
CPU and not too much memory) running FreeBSD-9.
Analysis: A file is being written. All or almost all dirty buffers
belong to the file. The file vnode is almost all time locked by
writing process. The buf_daemon() can not flush any dirty buffer as a
chance to acquire the file vnode lock is very low. A number of dirty
buffers grows up very slow and with each new dirty buffer slower,
because buf_daemon() eats more and more CPU time by looping on dirty
buffers queue (with very low or no effect).
This slowing down effect is started by buf_daemon() itself, when
'numdirtybuffers' reaches 'lodirtybuffers' threshold and buf_daemon()
is waked up by own timeout. The timeout fires at 'hz' period, but
starts to fire at 'hz/10' immediately as buf_daemon() fails to reach
'lodirtybuffers' threshold. When 'numdirtybuffers' (now slowly)
reaches ((lodirtybuffers + hidirtybuffers) / 2) threshold, the
buf_daemon() can be waked up within bdwrite() too and it's much worse.
Finally and with very slow speed, the 'hidirtybuffers' or
'dirtybufthresh' is reached, the dirty buffers are flushed, and
everything starts from beginning...
Note that for some time, bufdaemon work is distributed among bufdaemon
thread itself and any thread that fails to allocate a buffer, esp.
a thread that owns vnode lock and covers long queue of dirty buffers.
However, the problem starts when numdirtybuffers reaches
lodirtybuffers count and ends around hidirtybuffers count. There are
still plenty of free buffers in system.
On the system, a buffer size is 512 bytes and the default
thresholds are following:
vfs.hidirtybuffers = 134
vfs.lodirtybuffers = 67
vfs.dirtybufthresh = 120
For example, a 2MB file is copied into flash disk in about 3
minutes and 15 second. If dirtybufthresh is set to 40, the copy time
is about 20 seconds.
My solution is a mix of three things:
1. Suppresion of buf_daemon() wakeup by setting bd_request to 1 in
the main buf_daemon() loop.
I cannot understand this. Please provide a patch that shows what do
you mean there.
curthread-td_pflags |= TDP_NORUNNINGBUF | TDP_BUFNEED;
mtx_lock(bdlock);
for (;;) {
- bd_request = 0;
+ bd_request = 1;
mtx_unlock(bdlock);
Is this a complete patch ? The change just causes lost wakeups for
bufdaemon,
nothing more.
Yes, it's a complete patch. And exactly, it causes lost wakeups which are:
1. !! UNREASONABLE !!, because bufdaemon is not sleeping,
2. not wanted, because it looks that it's correct behaviour for the
sleep with hz/10 period. However, if the sleep with hz/10 period is
expected to be waked up by bd_wakeup(), then bd_request should be set
to 0 just before sleep() call, and then bufdaemon behaviour will be
clear.
No, your description is wrong.
If bufdaemon is unable to flush enough buffers and numdirtybuffers still
greater then lodirtybuffers, then bufdaemon enters qsleep state
without resetting bd_request, with timeouts of one tens of second.
Your patch will cause all wakeups for this case to be lost. This is
exactly the situation when we want bufdaemon to run harder to avoid
possible deadlocks, not to slow down.
OK. Let's focus to bufdaemon implementation. Now, qsleep state is
entered with random bd_request value. If someone calls bd_wakeup()
during bufdaemon iteration over dirty buffers queues, then bd_request
is set to 1. Otherwise, bd_request remains 0. I.e., sometimes qsleep
state only can be timeouted, sometimes it can be waked up by
bd_wakeup(). So, this random behaviour is what is wanted?
All stuff around bd_request and bufdaemon sleep is under bd_lock, so
if bd_request is 0 and bufdaemon is not sleeping, then all wakeups are
unreasonable! The patch is about that mainly.
Wakeups itself are very cheap for the running process. Mostly, it comes
down to locking sleepq and waking all threads that are present in the
sleepq blocked queue. If there is no threads in queue, nothing is done.
Are you serious? Is spin mutex really cheap? Many calls are cheap, but
they are not any matter where.
Svata
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
Re: [vfs] buf_daemon() slows down write() severely on low-speed CPU
On Thu, Mar 15, 2012 at 08:00:41PM +0100, Svatopluk Kraus wrote:
2012/3/15 Konstantin Belousov kostik...@gmail.com:
On Tue, Mar 13, 2012 at 01:54:38PM +0100, Svatopluk Kraus wrote:
On Mon, Mar 12, 2012 at 7:19 PM, Konstantin Belousov
kostik...@gmail.com wrote:
On Mon, Mar 12, 2012 at 04:00:58PM +0100, Svatopluk Kraus wrote:
Hi,
I have solved a following problem. If a big file (according to
'hidirtybuffers') is being written, the write speed is very poor.
It's observed on system with elan 486 and 32MB RAM (i.e., low speed
CPU and not too much memory) running FreeBSD-9.
Analysis: A file is being written. All or almost all dirty buffers
belong to the file. The file vnode is almost all time locked by
writing process. The buf_daemon() can not flush any dirty buffer as a
chance to acquire the file vnode lock is very low. A number of dirty
buffers grows up very slow and with each new dirty buffer slower,
because buf_daemon() eats more and more CPU time by looping on dirty
buffers queue (with very low or no effect).
This slowing down effect is started by buf_daemon() itself, when
'numdirtybuffers' reaches 'lodirtybuffers' threshold and buf_daemon()
is waked up by own timeout. The timeout fires at 'hz' period, but
starts to fire at 'hz/10' immediately as buf_daemon() fails to reach
'lodirtybuffers' threshold. When 'numdirtybuffers' (now slowly)
reaches ((lodirtybuffers + hidirtybuffers) / 2) threshold, the
buf_daemon() can be waked up within bdwrite() too and it's much worse.
Finally and with very slow speed, the 'hidirtybuffers' or
'dirtybufthresh' is reached, the dirty buffers are flushed, and
everything starts from beginning...
Note that for some time, bufdaemon work is distributed among bufdaemon
thread itself and any thread that fails to allocate a buffer, esp.
a thread that owns vnode lock and covers long queue of dirty buffers.
However, the problem starts when numdirtybuffers reaches
lodirtybuffers count and ends around hidirtybuffers count. There are
still plenty of free buffers in system.
On the system, a buffer size is 512 bytes and the default
thresholds are following:
vfs.hidirtybuffers = 134
vfs.lodirtybuffers = 67
vfs.dirtybufthresh = 120
For example, a 2MB file is copied into flash disk in about 3
minutes and 15 second. If dirtybufthresh is set to 40, the copy time
is about 20 seconds.
My solution is a mix of three things:
1. Suppresion of buf_daemon() wakeup by setting bd_request to 1 in
the main buf_daemon() loop.
I cannot understand this. Please provide a patch that shows what do
you mean there.
curthread-td_pflags |= TDP_NORUNNINGBUF | TDP_BUFNEED;
mtx_lock(bdlock);
for (;;) {
- bd_request = 0;
+ bd_request = 1;
mtx_unlock(bdlock);
Is this a complete patch ? The change just causes lost wakeups for
bufdaemon,
nothing more.
Yes, it's a complete patch. And exactly, it causes lost wakeups which are:
1. !! UNREASONABLE !!, because bufdaemon is not sleeping,
2. not wanted, because it looks that it's correct behaviour for the
sleep with hz/10 period. However, if the sleep with hz/10 period is
expected to be waked up by bd_wakeup(), then bd_request should be set
to 0 just before sleep() call, and then bufdaemon behaviour will be
clear.
No, your description is wrong.
If bufdaemon is unable to flush enough buffers and numdirtybuffers still
greater then lodirtybuffers, then bufdaemon enters qsleep state
without resetting bd_request, with timeouts of one tens of second.
Your patch will cause all wakeups for this case to be lost. This is
exactly the situation when we want bufdaemon to run harder to avoid
possible deadlocks, not to slow down.
All stuff around bd_request and bufdaemon sleep is under bd_lock, so
if bd_request is 0 and bufdaemon is not sleeping, then all wakeups are
unreasonable! The patch is about that mainly.
Wakeups itself are very cheap for the running process. Mostly, it comes
down to locking sleepq and waking all threads that are present in the
sleepq blocked queue. If there is no threads in queue, nothing is done.
I read description of bd_request variable. However, bd_request should
serve as an indicator that buf_daemon() is in sleep. I.e., the
following paradigma should be used:
mtx_lock(bdlock);
bd_request = 0; /* now, it's only time when wakeup() will be meaningful
*/
sleep(bd_request, ..., hz/10);
bd_request = 1; /* in case of timeout, we must set it (bd_wakeup()
already set it) */
mtx_unlock(bdlock);
My patch follows the paradigma. What happens without the patch in
described problem: buf_daemon() fails in its job and goes to sleep
with hz/10 period. It supposes that next early wakeup will do nothing
too. bd_request is untouched but buf_daemon() doesn't know if
Re: [vfs] buf_daemon() slows down write() severely on low-speed CPU
Hi, I'm interested in this, primarily because I'm tinkering with file storage stuff on my little (most wifi targetted) embedded MIPS platforms. So what's the story here? How can I reproduce your issue and do some of my own profiling/investigation? Adrian ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: [vfs] buf_daemon() slows down write() severely on low-speed CPU
On Tue, Mar 13, 2012 at 01:54:38PM +0100, Svatopluk Kraus wrote:
On Mon, Mar 12, 2012 at 7:19 PM, Konstantin Belousov
kostik...@gmail.com wrote:
On Mon, Mar 12, 2012 at 04:00:58PM +0100, Svatopluk Kraus wrote:
Hi,
I have solved a following problem. If a big file (according to
'hidirtybuffers') is being written, the write speed is very poor.
It's observed on system with elan 486 and 32MB RAM (i.e., low speed
CPU and not too much memory) running FreeBSD-9.
Analysis: A file is being written. All or almost all dirty buffers
belong to the file. The file vnode is almost all time locked by
writing process. The buf_daemon() can not flush any dirty buffer as a
chance to acquire the file vnode lock is very low. A number of dirty
buffers grows up very slow and with each new dirty buffer slower,
because buf_daemon() eats more and more CPU time by looping on dirty
buffers queue (with very low or no effect).
This slowing down effect is started by buf_daemon() itself, when
'numdirtybuffers' reaches 'lodirtybuffers' threshold and buf_daemon()
is waked up by own timeout. The timeout fires at 'hz' period, but
starts to fire at 'hz/10' immediately as buf_daemon() fails to reach
'lodirtybuffers' threshold. When 'numdirtybuffers' (now slowly)
reaches ((lodirtybuffers + hidirtybuffers) / 2) threshold, the
buf_daemon() can be waked up within bdwrite() too and it's much worse.
Finally and with very slow speed, the 'hidirtybuffers' or
'dirtybufthresh' is reached, the dirty buffers are flushed, and
everything starts from beginning...
Note that for some time, bufdaemon work is distributed among bufdaemon
thread itself and any thread that fails to allocate a buffer, esp.
a thread that owns vnode lock and covers long queue of dirty buffers.
However, the problem starts when numdirtybuffers reaches
lodirtybuffers count and ends around hidirtybuffers count. There are
still plenty of free buffers in system.
On the system, a buffer size is 512 bytes and the default
thresholds are following:
vfs.hidirtybuffers = 134
vfs.lodirtybuffers = 67
vfs.dirtybufthresh = 120
For example, a 2MB file is copied into flash disk in about 3
minutes and 15 second. If dirtybufthresh is set to 40, the copy time
is about 20 seconds.
My solution is a mix of three things:
1. Suppresion of buf_daemon() wakeup by setting bd_request to 1 in
the main buf_daemon() loop.
I cannot understand this. Please provide a patch that shows what do
you mean there.
curthread-td_pflags |= TDP_NORUNNINGBUF | TDP_BUFNEED;
mtx_lock(bdlock);
for (;;) {
- bd_request = 0;
+ bd_request = 1;
mtx_unlock(bdlock);
Is this a complete patch ? The change just causes lost wakeups for bufdaemon,
nothing more.
I read description of bd_request variable. However, bd_request should
serve as an indicator that buf_daemon() is in sleep. I.e., the
following paradigma should be used:
mtx_lock(bdlock);
bd_request = 0;/* now, it's only time when wakeup() will be meaningful */
sleep(bd_request, ..., hz/10);
bd_request = 1; /* in case of timeout, we must set it (bd_wakeup()
already set it) */
mtx_unlock(bdlock);
My patch follows the paradigma. What happens without the patch in
described problem: buf_daemon() fails in its job and goes to sleep
with hz/10 period. It supposes that next early wakeup will do nothing
too. bd_request is untouched but buf_daemon() doesn't know if its last
wakeup was made by bd_wakeup() or by timeout. So, bd_request could be
0 and buf_daemon() can be waked up before hz/10 just by bd_wakeup().
Moreover, setting bd_request to 0 when buf_daemon() is not in sleep
can cause time consuming and useless wakeup() calls without effect.
pgpgYYUkYWWIP.pgp
Description: PGP signature
Re: [vfs] buf_daemon() slows down write() severely on low-speed CPU
2012/3/15 Konstantin Belousov kostik...@gmail.com:
On Tue, Mar 13, 2012 at 01:54:38PM +0100, Svatopluk Kraus wrote:
On Mon, Mar 12, 2012 at 7:19 PM, Konstantin Belousov
kostik...@gmail.com wrote:
On Mon, Mar 12, 2012 at 04:00:58PM +0100, Svatopluk Kraus wrote:
Hi,
I have solved a following problem. If a big file (according to
'hidirtybuffers') is being written, the write speed is very poor.
It's observed on system with elan 486 and 32MB RAM (i.e., low speed
CPU and not too much memory) running FreeBSD-9.
Analysis: A file is being written. All or almost all dirty buffers
belong to the file. The file vnode is almost all time locked by
writing process. The buf_daemon() can not flush any dirty buffer as a
chance to acquire the file vnode lock is very low. A number of dirty
buffers grows up very slow and with each new dirty buffer slower,
because buf_daemon() eats more and more CPU time by looping on dirty
buffers queue (with very low or no effect).
This slowing down effect is started by buf_daemon() itself, when
'numdirtybuffers' reaches 'lodirtybuffers' threshold and buf_daemon()
is waked up by own timeout. The timeout fires at 'hz' period, but
starts to fire at 'hz/10' immediately as buf_daemon() fails to reach
'lodirtybuffers' threshold. When 'numdirtybuffers' (now slowly)
reaches ((lodirtybuffers + hidirtybuffers) / 2) threshold, the
buf_daemon() can be waked up within bdwrite() too and it's much worse.
Finally and with very slow speed, the 'hidirtybuffers' or
'dirtybufthresh' is reached, the dirty buffers are flushed, and
everything starts from beginning...
Note that for some time, bufdaemon work is distributed among bufdaemon
thread itself and any thread that fails to allocate a buffer, esp.
a thread that owns vnode lock and covers long queue of dirty buffers.
However, the problem starts when numdirtybuffers reaches
lodirtybuffers count and ends around hidirtybuffers count. There are
still plenty of free buffers in system.
On the system, a buffer size is 512 bytes and the default
thresholds are following:
vfs.hidirtybuffers = 134
vfs.lodirtybuffers = 67
vfs.dirtybufthresh = 120
For example, a 2MB file is copied into flash disk in about 3
minutes and 15 second. If dirtybufthresh is set to 40, the copy time
is about 20 seconds.
My solution is a mix of three things:
1. Suppresion of buf_daemon() wakeup by setting bd_request to 1 in
the main buf_daemon() loop.
I cannot understand this. Please provide a patch that shows what do
you mean there.
curthread-td_pflags |= TDP_NORUNNINGBUF | TDP_BUFNEED;
mtx_lock(bdlock);
for (;;) {
- bd_request = 0;
+ bd_request = 1;
mtx_unlock(bdlock);
Is this a complete patch ? The change just causes lost wakeups for bufdaemon,
nothing more.
Yes, it's a complete patch. And exactly, it causes lost wakeups which are:
1. !! UNREASONABLE !!, because bufdaemon is not sleeping,
2. not wanted, because it looks that it's correct behaviour for the
sleep with hz/10 period. However, if the sleep with hz/10 period is
expected to be waked up by bd_wakeup(), then bd_request should be set
to 0 just before sleep() call, and then bufdaemon behaviour will be
clear.
All stuff around bd_request and bufdaemon sleep is under bd_lock, so
if bd_request is 0 and bufdaemon is not sleeping, then all wakeups are
unreasonable! The patch is about that mainly.
I read description of bd_request variable. However, bd_request should
serve as an indicator that buf_daemon() is in sleep. I.e., the
following paradigma should be used:
mtx_lock(bdlock);
bd_request = 0; /* now, it's only time when wakeup() will be meaningful */
sleep(bd_request, ..., hz/10);
bd_request = 1; /* in case of timeout, we must set it (bd_wakeup()
already set it) */
mtx_unlock(bdlock);
My patch follows the paradigma. What happens without the patch in
described problem: buf_daemon() fails in its job and goes to sleep
with hz/10 period. It supposes that next early wakeup will do nothing
too. bd_request is untouched but buf_daemon() doesn't know if its last
wakeup was made by bd_wakeup() or by timeout. So, bd_request could be
0 and buf_daemon() can be waked up before hz/10 just by bd_wakeup().
Moreover, setting bd_request to 0 when buf_daemon() is not in sleep
can cause time consuming and useless wakeup() calls without effect.
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: [vfs] buf_daemon() slows down write() severely on low-speed CPU
On Mon, Mar 12, 2012 at 7:19 PM, Konstantin Belousov
kostik...@gmail.com wrote:
On Mon, Mar 12, 2012 at 04:00:58PM +0100, Svatopluk Kraus wrote:
Hi,
I have solved a following problem. If a big file (according to
'hidirtybuffers') is being written, the write speed is very poor.
It's observed on system with elan 486 and 32MB RAM (i.e., low speed
CPU and not too much memory) running FreeBSD-9.
Analysis: A file is being written. All or almost all dirty buffers
belong to the file. The file vnode is almost all time locked by
writing process. The buf_daemon() can not flush any dirty buffer as a
chance to acquire the file vnode lock is very low. A number of dirty
buffers grows up very slow and with each new dirty buffer slower,
because buf_daemon() eats more and more CPU time by looping on dirty
buffers queue (with very low or no effect).
This slowing down effect is started by buf_daemon() itself, when
'numdirtybuffers' reaches 'lodirtybuffers' threshold and buf_daemon()
is waked up by own timeout. The timeout fires at 'hz' period, but
starts to fire at 'hz/10' immediately as buf_daemon() fails to reach
'lodirtybuffers' threshold. When 'numdirtybuffers' (now slowly)
reaches ((lodirtybuffers + hidirtybuffers) / 2) threshold, the
buf_daemon() can be waked up within bdwrite() too and it's much worse.
Finally and with very slow speed, the 'hidirtybuffers' or
'dirtybufthresh' is reached, the dirty buffers are flushed, and
everything starts from beginning...
Note that for some time, bufdaemon work is distributed among bufdaemon
thread itself and any thread that fails to allocate a buffer, esp.
a thread that owns vnode lock and covers long queue of dirty buffers.
However, the problem starts when numdirtybuffers reaches
lodirtybuffers count and ends around hidirtybuffers count. There are
still plenty of free buffers in system.
On the system, a buffer size is 512 bytes and the default
thresholds are following:
vfs.hidirtybuffers = 134
vfs.lodirtybuffers = 67
vfs.dirtybufthresh = 120
For example, a 2MB file is copied into flash disk in about 3
minutes and 15 second. If dirtybufthresh is set to 40, the copy time
is about 20 seconds.
My solution is a mix of three things:
1. Suppresion of buf_daemon() wakeup by setting bd_request to 1 in
the main buf_daemon() loop.
I cannot understand this. Please provide a patch that shows what do
you mean there.
curthread-td_pflags |= TDP_NORUNNINGBUF | TDP_BUFNEED;
mtx_lock(bdlock);
for (;;) {
- bd_request = 0;
+ bd_request = 1;
mtx_unlock(bdlock);
I read description of bd_request variable. However, bd_request should
serve as an indicator that buf_daemon() is in sleep. I.e., the
following paradigma should be used:
mtx_lock(bdlock);
bd_request = 0;/* now, it's only time when wakeup() will be meaningful */
sleep(bd_request, ..., hz/10);
bd_request = 1; /* in case of timeout, we must set it (bd_wakeup()
already set it) */
mtx_unlock(bdlock);
My patch follows the paradigma. What happens without the patch in
described problem: buf_daemon() fails in its job and goes to sleep
with hz/10 period. It supposes that next early wakeup will do nothing
too. bd_request is untouched but buf_daemon() doesn't know if its last
wakeup was made by bd_wakeup() or by timeout. So, bd_request could be
0 and buf_daemon() can be waked up before hz/10 just by bd_wakeup().
Moreover, setting bd_request to 0 when buf_daemon() is not in sleep
can cause time consuming and useless wakeup() calls without effect.
2. Increment of buf_daemon() fast timeout from hz/10 to hz/4.
3. Tuning dirtybufthresh to (((lodirtybuffers + hidirtybuffers) /
2) - 15) magic.
Even hz / 10 is awfully long time on modern hardware.
The dirtybufthresh is already the sysctl that you can change.
Yes, I noted low-speed CPU. Don't forget that even if buf_daemon()
sleeps for hz/4 period (and this is expected to be rare case),
dirtybufthresh still works and helps. And I don't push the changes
(except bd_request one (a little)). I'm just sharing my experience.
The 32MB is indeed around the lowest amount of memory where recent
FreeBSD can make an illusion of being useful. I am not sure how much
should the system be tuned by default for such configuration.
Even recent FreeBSD on this configuration is useful pretty much. Of
course, file operations are not main concern ... IMHO, it's always
good to know how the system works (and its parts) in various
configurations.
The mention copy time is about 30 seconds now.
The described problem is just for information to anyone who can be
interested in. Comments are welcome. However, the bd_request thing is
more general.
bd_request (despite its description) should be 0 only when
buf_daemon() is in sleep(). Otherwise, wakeup() on bd_request channel
is useless. Therefore, setting bd_request to 1 in the main
buf_daemon() loop is
Re: [vfs] buf_daemon() slows down write() severely on low-speed CPU
On 12 March 2012 11:19, Konstantin Belousov kostik...@gmail.com wrote: The 32MB is indeed around the lowest amount of memory where recent FreeBSD can make an illusion of being useful. I am not sure how much should the system be tuned by default for such configuration. Some -new- embedded wifi hardware is shipping with 16MB of RAM. Just saying, Adrian ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
[vfs] buf_daemon() slows down write() severely on low-speed CPU
Hi, I have solved a following problem. If a big file (according to 'hidirtybuffers') is being written, the write speed is very poor. It's observed on system with elan 486 and 32MB RAM (i.e., low speed CPU and not too much memory) running FreeBSD-9. Analysis: A file is being written. All or almost all dirty buffers belong to the file. The file vnode is almost all time locked by writing process. The buf_daemon() can not flush any dirty buffer as a chance to acquire the file vnode lock is very low. A number of dirty buffers grows up very slow and with each new dirty buffer slower, because buf_daemon() eats more and more CPU time by looping on dirty buffers queue (with very low or no effect). This slowing down effect is started by buf_daemon() itself, when 'numdirtybuffers' reaches 'lodirtybuffers' threshold and buf_daemon() is waked up by own timeout. The timeout fires at 'hz' period, but starts to fire at 'hz/10' immediately as buf_daemon() fails to reach 'lodirtybuffers' threshold. When 'numdirtybuffers' (now slowly) reaches ((lodirtybuffers + hidirtybuffers) / 2) threshold, the buf_daemon() can be waked up within bdwrite() too and it's much worse. Finally and with very slow speed, the 'hidirtybuffers' or 'dirtybufthresh' is reached, the dirty buffers are flushed, and everything starts from beginning... On the system, a buffer size is 512 bytes and the default thresholds are following: vfs.hidirtybuffers = 134 vfs.lodirtybuffers = 67 vfs.dirtybufthresh = 120 For example, a 2MB file is copied into flash disk in about 3 minutes and 15 second. If dirtybufthresh is set to 40, the copy time is about 20 seconds. My solution is a mix of three things: 1. Suppresion of buf_daemon() wakeup by setting bd_request to 1 in the main buf_daemon() loop. 2. Increment of buf_daemon() fast timeout from hz/10 to hz/4. 3. Tuning dirtybufthresh to (((lodirtybuffers + hidirtybuffers) / 2) - 15) magic. The mention copy time is about 30 seconds now. The described problem is just for information to anyone who can be interested in. Comments are welcome. However, the bd_request thing is more general. bd_request (despite its description) should be 0 only when buf_daemon() is in sleep(). Otherwise, wakeup() on bd_request channel is useless. Therefore, setting bd_request to 1 in the main buf_daemon() loop is correct and better as it saves time spent by wakeup() on not existing channel. Svata ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: [vfs] buf_daemon() slows down write() severely on low-speed CPU
On Mon, Mar 12, 2012 at 04:00:58PM +0100, Svatopluk Kraus wrote: Hi, I have solved a following problem. If a big file (according to 'hidirtybuffers') is being written, the write speed is very poor. It's observed on system with elan 486 and 32MB RAM (i.e., low speed CPU and not too much memory) running FreeBSD-9. Analysis: A file is being written. All or almost all dirty buffers belong to the file. The file vnode is almost all time locked by writing process. The buf_daemon() can not flush any dirty buffer as a chance to acquire the file vnode lock is very low. A number of dirty buffers grows up very slow and with each new dirty buffer slower, because buf_daemon() eats more and more CPU time by looping on dirty buffers queue (with very low or no effect). This slowing down effect is started by buf_daemon() itself, when 'numdirtybuffers' reaches 'lodirtybuffers' threshold and buf_daemon() is waked up by own timeout. The timeout fires at 'hz' period, but starts to fire at 'hz/10' immediately as buf_daemon() fails to reach 'lodirtybuffers' threshold. When 'numdirtybuffers' (now slowly) reaches ((lodirtybuffers + hidirtybuffers) / 2) threshold, the buf_daemon() can be waked up within bdwrite() too and it's much worse. Finally and with very slow speed, the 'hidirtybuffers' or 'dirtybufthresh' is reached, the dirty buffers are flushed, and everything starts from beginning... Note that for some time, bufdaemon work is distributed among bufdaemon thread itself and any thread that fails to allocate a buffer, esp. a thread that owns vnode lock and covers long queue of dirty buffers. On the system, a buffer size is 512 bytes and the default thresholds are following: vfs.hidirtybuffers = 134 vfs.lodirtybuffers = 67 vfs.dirtybufthresh = 120 For example, a 2MB file is copied into flash disk in about 3 minutes and 15 second. If dirtybufthresh is set to 40, the copy time is about 20 seconds. My solution is a mix of three things: 1. Suppresion of buf_daemon() wakeup by setting bd_request to 1 in the main buf_daemon() loop. I cannot understand this. Please provide a patch that shows what do you mean there. 2. Increment of buf_daemon() fast timeout from hz/10 to hz/4. 3. Tuning dirtybufthresh to (((lodirtybuffers + hidirtybuffers) / 2) - 15) magic. Even hz / 10 is awfully long time on modern hardware. The dirtybufthresh is already the sysctl that you can change. The 32MB is indeed around the lowest amount of memory where recent FreeBSD can make an illusion of being useful. I am not sure how much should the system be tuned by default for such configuration. The mention copy time is about 30 seconds now. The described problem is just for information to anyone who can be interested in. Comments are welcome. However, the bd_request thing is more general. bd_request (despite its description) should be 0 only when buf_daemon() is in sleep(). Otherwise, wakeup() on bd_request channel is useless. Therefore, setting bd_request to 1 in the main buf_daemon() loop is correct and better as it saves time spent by wakeup() on not existing channel. Svata ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org pgpzucBWBKxIH.pgp Description: PGP signature
