Re: [zfs-discuss] What causes slow performance under load?

2009-05-13 Thread Gary Mills
On Mon, Apr 27, 2009 at 04:47:27PM -0500, Gary Mills wrote: > On Sat, Apr 18, 2009 at 04:27:55PM -0500, Gary Mills wrote: > > We have an IMAP server with ZFS for mailbox storage that has recently > > become extremely slow on most weekday mornings and afternoons. When > > one of these incidents hap

Re: [zfs-discuss] What causes slow performance under load?

2009-05-13 Thread Joerg Schilling
Rince wrote: > On Tue, Apr 21, 2009 at 3:20 PM, Joerg Schilling > wrote: > > The license combination used by cdrtools was verified by several lawywers > > including Sun Legal and Eben Moglen and no lawyer did find a problem. > > [citation needed] What is the reason for restarting your FUD campa

Re: [zfs-discuss] What causes slow performance under load?

2009-05-12 Thread Rince
On Tue, Apr 21, 2009 at 3:20 PM, Joerg Schilling wrote: > The license combination used by cdrtools was verified by several lawywers > including Sun Legal and Eben Moglen and no lawyer did find a problem. [citation needed] https://lists.ubuntu.com/archives/ubuntu-news-team/2009-February/000413.ht

Re: [zfs-discuss] What causes slow performance under load?

2009-04-27 Thread Gary Mills
On Sat, Apr 18, 2009 at 04:27:55PM -0500, Gary Mills wrote: > We have an IMAP server with ZFS for mailbox storage that has recently > become extremely slow on most weekday mornings and afternoons. When > one of these incidents happens, the number of processes increases, the > load average increase

Re: [zfs-discuss] What causes slow performance under load?

2009-04-22 Thread Gary Mills
On Tue, Apr 21, 2009 at 04:09:03PM -0400, Oscar del Rio wrote: > There's a similar thread on hied-emailad...@listserv.nd.edu > that might help or at least can get you in touch with other University > admins in a similar situation. > > https://listserv.nd.edu/cgi-bin/wa?A1=ind0904&L=HIED-EMAILADMI

Re: [zfs-discuss] What causes slow performance under load?

2009-04-22 Thread David Dyer-Bennet
On Tue, April 21, 2009 14:20, Joerg Schilling wrote: > Miles Nordin wrote: >> First, there is plain-GPLv2, Linux-modified-GPLv2 with the ``or any >> later version'' clause deleted and the suspect ``interpretation'' of >> kernel modules, and plain-GPLv3: there are three GPL licenses to >> worry a

Re: [zfs-discuss] What causes slow performance under load?

2009-04-21 Thread Gary Mills
On Tue, Apr 21, 2009 at 09:34:57AM -0500, Patrick Skerrett wrote: > I'm fighting with an identical problem here & am very interested in this > thread. > > Solaris 10 127112-11 boxes running ZFS on a fiberchannel raid5 device > (hardware raid). You are about a year behind in kernel patches. The

Re: [zfs-discuss] What causes slow performance under load?

2009-04-21 Thread Miles Nordin
> "js" == Joerg Schilling writes: js> So what is your point? It was nothing to do with combinations of licenses within cdrkit, nor within cdrtools. It was that your changing your project's license to one incompatible with the GPL led to the forking of a project, so it would be better fo

Re: [zfs-discuss] What causes slow performance under load?

2009-04-21 Thread Joerg Schilling
Miles Nordin wrote: ... > chooses the license, _and_ can change the license later (which Linux > cannot, which sucks for mostly everyone). Please stop spreading FUD. > Especially since you brought us through this exact same thing before > the last time someone brought up dual-licensing. Please

Re: [zfs-discuss] What causes slow performance under load?

2009-04-21 Thread Miles Nordin
> "js" == Joerg Schilling writes: js> Do you really like Sun to be forced to verify that the kind of js> such a patch is below the interlectual creation level to be js> able to claim a copyright? the common and IMHO correct practice, and the practice Sun actually uses, is to assu

Re: [zfs-discuss] What causes slow performance under load?

2009-04-21 Thread Patrick Skerrett
I'm fighting with an identical problem here & am very interested in this thread. Solaris 10 127112-11 boxes running ZFS on a fiberchannel raid5 device (hardware raid). Randomly one lun on a machine will stop writing for about 10-15 minutes (during a busy time of day), and then all of a sudde

Re: [zfs-discuss] What causes slow performance under load?

2009-04-21 Thread Joerg Schilling
Bob Friesenhahn wrote: > To be more clear, standard GPL provides the option for the user to use > any later version. The Linux kernel uses a modified verison of GPLv2 Such an option is illegal in Europe anyway - you cannot agree with a contract that you don't know. > Due to this, ZFS would n

Re: [zfs-discuss] What causes slow performance under load?

2009-04-21 Thread Joerg Schilling
Bob Friesenhahn wrote: > On Mon, 20 Apr 2009, Tim wrote: > > > > Oracle: It should be an interesting ride to say the least. I guess we'll > > see just how much they love linux... either zfs et. all will become GPL, or > > we'll see their true colors. I'm secretly hoping for the latter (as long

Re: [zfs-discuss] What causes slow performance under load?

2009-04-20 Thread Gary Mills
On Sat, Apr 18, 2009 at 04:27:55PM -0500, Gary Mills wrote: > We have an IMAP server with ZFS for mailbox storage that has recently > become extremely slow on most weekday mornings and afternoons. When > one of these incidents happens, the number of processes increases, the > load average increase

Re: [zfs-discuss] What causes slow performance under load?

2009-04-20 Thread Bob Friesenhahn
On Mon, 20 Apr 2009, Eric D. Mudama wrote: GPL v2 may not be the most recent version, but a lot of people prefer GPLv2 to GPLv3, in the same way that some people might prefer Solaris 8 to Solaris 10, or Linux 2.4 kernels to the 2.6 series. To be more clear, standard GPL provides the option for

Re: [zfs-discuss] What causes slow performance under load?

2009-04-20 Thread Ray Van Dolson
On Mon, Apr 20, 2009 at 01:04:59PM -0700, Eric D. Mudama wrote: > On Mon, Apr 20 at 14:19, Bob Friesenhahn wrote: > > On Mon, 20 Apr 2009, Tim wrote: > >> > >> Oracle: It should be an interesting ride to say the least. I guess we'll > >> see just how much they love linux... either zfs et. all will

Re: [zfs-discuss] What causes slow performance under load?

2009-04-20 Thread Eric D. Mudama
On Mon, Apr 20 at 14:19, Bob Friesenhahn wrote: On Mon, 20 Apr 2009, Tim wrote: Oracle: It should be an interesting ride to say the least. I guess we'll see just how much they love linux... either zfs et. all will become GPL, or we'll see their true colors. I'm secretly hoping for the latter

Re: [zfs-discuss] What causes slow performance under load?

2009-04-20 Thread Bob Friesenhahn
On Mon, 20 Apr 2009, Tim wrote: Oracle: It should be an interesting ride to say the least. I guess we'll see just how much they love linux... either zfs et. all will become GPL, or we'll see their true colors. I'm secretly hoping for the latter (as long as they keep it open sourced). I don't

Re: [zfs-discuss] What causes slow performance under load?

2009-04-20 Thread Bob Friesenhahn
On Mon, 20 Apr 2009, Richard Elling wrote: I describe it as "free of the shackles of the corporate jail, I can now recognize and act upon any opportunity I find interesting." With Sun being bought by Oracle, I have a feeling there will be plenty of opportunity... Is this a forward-looking stat

Re: [zfs-discuss] What causes slow performance under load?

2009-04-20 Thread Tim
On Mon, Apr 20, 2009 at 11:35 AM, Richard Elling wrote: > Tim wrote: > >> >> >> On Sun, Apr 19, 2009 at 6:47 PM, Richard Elling >> > richard.ell...@gmail.com>> wrote: >> >>I see no evidence of an I/O or file system bottleneck here. While the >>service times are a little higher than I exp

Re: [zfs-discuss] What causes slow performance under load?

2009-04-20 Thread Richard Elling
Tim wrote: On Sun, Apr 19, 2009 at 6:47 PM, Richard Elling mailto:richard.ell...@gmail.com>> wrote: I see no evidence of an I/O or file system bottleneck here. While the service times are a little higher than I expect, I don't get worried until the %busy is high and actv is

Re: [zfs-discuss] What causes slow performance under load?

2009-04-19 Thread Tim
On Sun, Apr 19, 2009 at 6:47 PM, Richard Elling wrote: > I see no evidence of an I/O or file system bottleneck here. While the > service times are a little higher than I expect, I don't get worried until > the %busy is high and actv is high and asvc_t is high(er). I think your > problem is elsew

Re: [zfs-discuss] What causes slow performance under load?

2009-04-19 Thread Richard Elling
iostat measurements comment below... Gary Mills wrote: On Sat, Apr 18, 2009 at 11:45:54PM -0500, Mike Gerdts wrote: [perf-discuss cc'd] On Sat, Apr 18, 2009 at 4:27 PM, Gary Mills wrote: Many other layers are involved in this server. We use scsi_vhci for redundant I/O paths and Sun'

Re: [zfs-discuss] What causes slow performance under load?

2009-04-19 Thread Marion Hakanson
mi...@cc.umanitoba.ca said: > What would I look for with mpstat? Look for a CPU (thread) that might be 100% utilized; Also look to see if that CPU (or CPU's) has a larger number in the "ithr" column than all other CPU's. The idea here is that you aren't getting much out of the T2000 if only one

Re: [zfs-discuss] What causes slow performance under load?

2009-04-19 Thread Mike Gerdts
On Sun, Apr 19, 2009 at 10:58 AM, Gary Mills wrote: > On Sat, Apr 18, 2009 at 11:45:54PM -0500, Mike Gerdts wrote: >> Also, you may want to consider doing backups from the NetApp rather >> than from the Solaris box. > > I've certainly recommended finding a different way to perform backups. > >> As

Re: [zfs-discuss] What causes slow performance under load?

2009-04-19 Thread Gary Mills
On Sat, Apr 18, 2009 at 11:45:54PM -0500, Mike Gerdts wrote: > [perf-discuss cc'd] > > On Sat, Apr 18, 2009 at 4:27 PM, Gary Mills wrote: > > Many other layers are involved in this server.  We use scsi_vhci for > > redundant I/O paths and Sun's Iscsi initiator to connect to the > > storage on our

Re: [zfs-discuss] What causes slow performance under load?

2009-04-19 Thread Gary Mills
On Sat, Apr 18, 2009 at 09:41:39PM -0500, Tim wrote: > >On Sat, Apr 18, 2009 at 9:01 PM, Gary Mills <[1]mi...@cc.umanitoba.ca> >wrote: > > On Sat, Apr 18, 2009 at 06:53:30PM -0400, Ellis, Mike wrote: > > In case the writes are a problem: When zfs sends a sync-command > t

Re: [zfs-discuss] What causes slow performance under load?

2009-04-18 Thread Mike Gerdts
[perf-discuss cc'd] On Sat, Apr 18, 2009 at 4:27 PM, Gary Mills wrote: > Many other layers are involved in this server.  We use scsi_vhci for > redundant I/O paths and Sun's Iscsi initiator to connect to the > storage on our Netapp filer.  The kernel plays a part as well.  How > do we determine w

Re: [zfs-discuss] What causes slow performance under load?

2009-04-18 Thread Gary Mills
On Sat, Apr 18, 2009 at 06:06:49PM -0700, Richard Elling wrote: > [CC'ed to perf-discuss] > > Gary Mills wrote: > >We have an IMAP server with ZFS for mailbox storage that has recently > >become extremely slow on most weekday mornings and afternoons. When > >one of these incidents happens, the nu

Re: [zfs-discuss] What causes slow performance under load?

2009-04-18 Thread Tim
On Sat, Apr 18, 2009 at 9:01 PM, Gary Mills wrote: > On Sat, Apr 18, 2009 at 06:53:30PM -0400, Ellis, Mike wrote: > > In case the writes are a problem: When zfs sends a sync-command to > > the iscsi luns, does the netapp just ack it, or does it wait till it > > fully destages? Might make sense to

Re: [zfs-discuss] What causes slow performance under load?

2009-04-18 Thread Gary Mills
On Sat, Apr 18, 2009 at 09:58:19PM -0400, Ellis, Mike wrote: > I've found that (depending on the backup software) the backup agents > tend to run a single thread per filesystem. While that can backup > several filesystems concurrently, the single filesystem backup is > single-threaded... Yes, they

Re: [zfs-discuss] What causes slow performance under load?

2009-04-18 Thread Gary Mills
On Sat, Apr 18, 2009 at 06:53:30PM -0400, Ellis, Mike wrote: > In case the writes are a problem: When zfs sends a sync-command to > the iscsi luns, does the netapp just ack it, or does it wait till it > fully destages? Might make sense to disable write/sync in > /etc/system to be sure. So far I ha

Re: [zfs-discuss] What causes slow performance under load?

2009-04-18 Thread Gary Mills
On Sat, Apr 18, 2009 at 05:25:17PM -0500, Bob Friesenhahn wrote: > On Sat, 18 Apr 2009, Gary Mills wrote: > > >How do we determine which layer is responsible for the slow > >performance? > > If the ARC size is diminishing under heavy load then there must be > excessive pressure for memory from

Re: [zfs-discuss] What causes slow performance under load?

2009-04-18 Thread Richard Elling
[CC'ed to perf-discuss] Gary Mills wrote: We have an IMAP server with ZFS for mailbox storage that has recently become extremely slow on most weekday mornings and afternoons. When one of these incidents happens, the number of processes increases, the load average increases, but ZFS I/O bandwidt

[zfs-discuss] What causes slow performance under load?

2009-04-18 Thread Gary Mills
We have an IMAP server with ZFS for mailbox storage that has recently become extremely slow on most weekday mornings and afternoons. When one of these incidents happens, the number of processes increases, the load average increases, but ZFS I/O bandwidth decreases. Users notice very slow response