Re: immense delayed write to file system (ZFS and UFS2), performance issues

2010-01-19 Thread O. Hartmann

On 01/19/10 10:09, krad wrote:

2010/1/18 Morgan Wesstr�m


O. Hartmann wrote:

I realise a strange behaviour of several FreeBSD 8.0-STABLE/amd64 boxes.
All boxes have the most recent STABLE. One box is a UP system, two
others SMP boxes, one with a Q6600 4-core, another XEON with 2x 4-cores
(Dell Poweredge III).

Symptome: All boxes have ZFS and UFS2 filesystems. Since two weeks or
so, sometimes the I/O performance drops massively when doing 'svn
update', 'make world' or even 'make kernel'. It doesn't matter what
memory and how many cpu the box has, it get stuck for several seconds
and freezing. On the UP box, this is sometimes for 10 - 20 seconds.
A very interesting phenomenon is the massively delayed file writing on
ZFS filesystems I realise. Editing a file in 'vi' running on one XTerm
and having in another Xterminal my shell for compiling this file, it
takes sometimes up to 20 seconds to get the file updated after it has
been written. It's like having an old, slow NFS connection with long
cache delays.
These massively delayed file transactions are not necessarely under
heavy load, sometimes they occur in a relaxed situation. They seem to
occur much more often on the UP box than on the SMP boxes, but this
strange phenomenon also occur on the Dell Poweredge II, which has 16GB
RAM and summa summarum 16 cores. This phenomenon does occur on ZFS- and
UFS2 filesystems as well. It is hardly reproducable.

Is there any known issue?

Ragrds,
Oliver



The disks involved don't happen to be Western Digital Green Power disks,
do they? The Intelli-Park function in these disks are wrecking havoc
with I/O in Linux-land at least, causing massive stalls and iowait
through the roof during the 25-30 seconds it takes for the heads to
unload after parking. I have two of these disks sitting on my desk now
collecting dust...
/Morgan
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "
freebsd-questions-unsubscr...@freebsd.org"




ZFS is copy on write, therefore to optimize the write performance it delays
writes for a long as possible, upto a set maximum time. It will then flush
to the disks. How long this time is depends on how much free ram you have
available. Assuming processes are eating up all your ram I would imagine you
are hitting the max limit. I'm not sure exactly what its set to on bsd but I
know the default on opensolaris is 30s. I think this explains your delayed
writes.

Not sure what will cause the lock ups though.
___
freebsd-sta...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"


This could end in a bad situation, where one process writes a files, say 
with some arbitrary stuff and another successing process is intended to 
read this file. even if the processes are run serial, those 'delays' 
could break the chain! The delay situation in a development environment 
is harsh, but in other circumstances it could develop very bad.


I see this strange behaviour now for several weeks, something essential 
has changed in the code, I guess.
On UP boxes the situation is worse sometimes, on SMp boxes with lots of 
RAM ( 8 and 16 GB and 4 or 8 CPU cores) it is still bad. I have a server 
that acts as a 'rsync' backup system gathering data from satellite 
servers from time to time. Since this problem of slowness occured, this 
4-core 8 gig RAM box crawls for minutes. Even when X11 is disabled 
working on console is 'bumpy': terminal out slows down, mouse pointer 
jumps etc.As I wrote, the same on a 8 core/16 gig box, but not that harsh.


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"


Re: immense delayed write to file system (ZFS and UFS2), performance issues

2010-01-19 Thread O. Hartmann

On 01/18/10 21:34, � wrote:

O. Hartmann wrote:

I realise a strange behaviour of several FreeBSD 8.0-STABLE/amd64 boxes.
All boxes have the most recent STABLE. One box is a UP system, two
others SMP boxes, one with a Q6600 4-core, another XEON with 2x 4-cores
(Dell Poweredge III).

Symptome: All boxes have ZFS and UFS2 filesystems. Since two weeks or
so, sometimes the I/O performance drops massively when doing 'svn
update', 'make world' or even 'make kernel'. It doesn't matter what
memory and how many cpu the box has, it get stuck for several seconds
and freezing. On the UP box, this is sometimes for 10 - 20 seconds.
A very interesting phenomenon is the massively delayed file writing on
ZFS filesystems I realise. Editing a file in 'vi' running on one XTerm
and having in another Xterminal my shell for compiling this file, it
takes sometimes up to 20 seconds to get the file updated after it has
been written. It's like having an old, slow NFS connection with long
cache delays.
These massively delayed file transactions are not necessarely under
heavy load, sometimes they occur in a relaxed situation. They seem to
occur much more often on the UP box than on the SMP boxes, but this
strange phenomenon also occur on the Dell Poweredge II, which has 16GB
RAM and summa summarum 16 cores. This phenomenon does occur on ZFS- and
UFS2 filesystems as well. It is hardly reproducable.

Is there any known issue?

Ragrds,
Oliver



The disks involved don't happen to be Western Digital Green Power disks,
do they? The Intelli-Park function in these disks are wrecking havoc
with I/O in Linux-land at least, causing massive stalls and iowait
through the roof during the 25-30 seconds it takes for the heads to
unload after parking. I have two of these disks sitting on my desk now
collecting dust...
/Morgan
___
freebsd-sta...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"



The disks in question are indeed WD on one box, but they are all Caviar 
Black and they performed well months ago with the very same hardware and 
an earlier FreeBSD 8 version.
The other boxes in questions do have a set of mixed type, Seagate, WD, 
Samsung (mostly Samsung F1 types). We do not use 'Green' drives, due to 
every box acts as a server and we found green-disks, even from WD, too slow.


Oliver

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"


Re: immense delayed write to file system (ZFS and UFS2), performance issues

2010-01-19 Thread Morgan Wesström
Emil Mikulic wrote:
> (off-list)
> 
> On Tue, Jan 19, 2010 at 04:32:34PM +0100, Morgan Wesstr?m wrote:
>> Emil Mikulic wrote:
>>> On Tue, Jan 19, 2010 at 09:16:41AM +0100, Gerrit K?hn wrote:
 Thanks for bringing up this topic here. I have drives showing up close to
 80 load cycle counts here. Guess it's time for that fix... :-|
>>> Device Model: WDC WD10EACS-00ZJB0
>>> Firmware Version: 01.01B01
>>> Serial Number:WD-WCAS
>>> [...]
>>>   9 Power_On_Hours   17046
>>> 193 Load_Cycle_Count 1045512
>>>
>>> The above drive is in a raidz of three.
>>> The other two drives from that batch have already failed.  :(
>> Did you RMA the failing drives? Did WD comment the Load_Cycle_Count?
> 
> No.  But apparently they're still under warranty, so maybe I'll give
> that a shot and let you know in ~3 weeks or so.  :)
> 
Thanks. I'd be really interested to know whether they'll make any
comments regarding that value or if they simply ignore it.
/Morgan
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"


Re: immense delayed write to file system (ZFS and UFS2), performance issues

2010-01-19 Thread krad
2010/1/18 Morgan Wesström 

> O. Hartmann wrote:
> > I realise a strange behaviour of several FreeBSD 8.0-STABLE/amd64 boxes.
> > All boxes have the most recent STABLE. One box is a UP system, two
> > others SMP boxes, one with a Q6600 4-core, another XEON with 2x 4-cores
> > (Dell Poweredge III).
> >
> > Symptome: All boxes have ZFS and UFS2 filesystems. Since two weeks or
> > so, sometimes the I/O performance drops massively when doing 'svn
> > update', 'make world' or even 'make kernel'. It doesn't matter what
> > memory and how many cpu the box has, it get stuck for several seconds
> > and freezing. On the UP box, this is sometimes for 10 - 20 seconds.
> > A very interesting phenomenon is the massively delayed file writing on
> > ZFS filesystems I realise. Editing a file in 'vi' running on one XTerm
> > and having in another Xterminal my shell for compiling this file, it
> > takes sometimes up to 20 seconds to get the file updated after it has
> > been written. It's like having an old, slow NFS connection with long
> > cache delays.
> > These massively delayed file transactions are not necessarely under
> > heavy load, sometimes they occur in a relaxed situation. They seem to
> > occur much more often on the UP box than on the SMP boxes, but this
> > strange phenomenon also occur on the Dell Poweredge II, which has 16GB
> > RAM and summa summarum 16 cores. This phenomenon does occur on ZFS- and
> > UFS2 filesystems as well. It is hardly reproducable.
> >
> > Is there any known issue?
> >
> > Ragrds,
> > Oliver
>
>
> The disks involved don't happen to be Western Digital Green Power disks,
> do they? The Intelli-Park function in these disks are wrecking havoc
> with I/O in Linux-land at least, causing massive stalls and iowait
> through the roof during the 25-30 seconds it takes for the heads to
> unload after parking. I have two of these disks sitting on my desk now
> collecting dust...
> /Morgan
> ___
> freebsd-questions@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "
> freebsd-questions-unsubscr...@freebsd.org"
>


ZFS is copy on write, therefore to optimize the write performance it delays
writes for a long as possible, upto a set maximum time. It will then flush
to the disks. How long this time is depends on how much free ram you have
available. Assuming processes are eating up all your ram I would imagine you
are hitting the max limit. I'm not sure exactly what its set to on bsd but I
know the default on opensolaris is 30s. I think this explains your delayed
writes.

Not sure what will cause the lock ups though.
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"


Re: immense delayed write to file system (ZFS and UFS2), performance issues

2010-01-18 Thread Daniel O'Connor
On Tue, 19 Jan 2010, Morgan Wesström wrote:
> The disks involved don't happen to be Western Digital Green Power
> disks, do they? The Intelli-Park function in these disks are wrecking
> havoc with I/O in Linux-land at least, causing massive stalls and
> iowait through the roof during the 25-30 seconds it takes for the
> heads to unload after parking. I have two of these disks sitting on
> my desk now collecting dust...

There's this..
http://www.silentpcreview.com/Terabyte_Drive_Fix

and you can get the tool at..
http://home.arcor.de/ghostadmin/wdidle3_1_00.zip

I am planning to try this out tonight..

-- 
Daniel O'Connor software and network engineer
for Genesis Software - http://www.gsoft.com.au
"The nice thing about standards is that there
are so many of them to choose from."
  -- Andrew Tanenbaum
GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C


signature.asc
Description: This is a digitally signed message part.


Re: immense delayed write to file system (ZFS and UFS2), performance issues

2010-01-18 Thread Morgan Wesström
O. Hartmann wrote:
> I realise a strange behaviour of several FreeBSD 8.0-STABLE/amd64 boxes.
> All boxes have the most recent STABLE. One box is a UP system, two
> others SMP boxes, one with a Q6600 4-core, another XEON with 2x 4-cores
> (Dell Poweredge III).
> 
> Symptome: All boxes have ZFS and UFS2 filesystems. Since two weeks or
> so, sometimes the I/O performance drops massively when doing 'svn
> update', 'make world' or even 'make kernel'. It doesn't matter what
> memory and how many cpu the box has, it get stuck for several seconds
> and freezing. On the UP box, this is sometimes for 10 - 20 seconds.
> A very interesting phenomenon is the massively delayed file writing on
> ZFS filesystems I realise. Editing a file in 'vi' running on one XTerm
> and having in another Xterminal my shell for compiling this file, it
> takes sometimes up to 20 seconds to get the file updated after it has
> been written. It's like having an old, slow NFS connection with long
> cache delays.
> These massively delayed file transactions are not necessarely under
> heavy load, sometimes they occur in a relaxed situation. They seem to
> occur much more often on the UP box than on the SMP boxes, but this
> strange phenomenon also occur on the Dell Poweredge II, which has 16GB
> RAM and summa summarum 16 cores. This phenomenon does occur on ZFS- and
> UFS2 filesystems as well. It is hardly reproducable.
> 
> Is there any known issue?
> 
> Ragrds,
> Oliver


The disks involved don't happen to be Western Digital Green Power disks,
do they? The Intelli-Park function in these disks are wrecking havoc
with I/O in Linux-land at least, causing massive stalls and iowait
through the roof during the 25-30 seconds it takes for the heads to
unload after parking. I have two of these disks sitting on my desk now
collecting dust...
/Morgan
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"


immense delayed write to file system (ZFS and UFS2), performance issues

2010-01-18 Thread O. Hartmann
I realise a strange behaviour of several FreeBSD 8.0-STABLE/amd64 boxes. 
All boxes have the most recent STABLE. One box is a UP system, two 
others SMP boxes, one with a Q6600 4-core, another XEON with 2x 4-cores 
(Dell Poweredge III).


Symptome: All boxes have ZFS and UFS2 filesystems. Since two weeks or 
so, sometimes the I/O performance drops massively when doing 'svn 
update', 'make world' or even 'make kernel'. It doesn't matter what 
memory and how many cpu the box has, it get stuck for several seconds 
and freezing. On the UP box, this is sometimes for 10 - 20 seconds.
A very interesting phenomenon is the massively delayed file writing on 
ZFS filesystems I realise. Editing a file in 'vi' running on one XTerm 
and having in another Xterminal my shell for compiling this file, it 
takes sometimes up to 20 seconds to get the file updated after it has 
been written. It's like having an old, slow NFS connection with long 
cache delays.
These massively delayed file transactions are not necessarely under 
heavy load, sometimes they occur in a relaxed situation. They seem to 
occur much more often on the UP box than on the SMP boxes, but this 
strange phenomenon also occur on the Dell Poweredge II, which has 16GB 
RAM and summa summarum 16 cores. This phenomenon does occur on ZFS- and 
UFS2 filesystems as well. It is hardly reproducable.


Is there any known issue?

Ragrds,
Oliver
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"