On Sat, Apr 18, 2009 at 06:06:49PM -0700, Richard Elling wrote: > [CC'ed to perf-discuss] > > Gary Mills wrote: > >We have an IMAP server with ZFS for mailbox storage that has recently > >become extremely slow on most weekday mornings and afternoons. When > >one of these incidents happens, the number of processes increases, the > >load average increases, but ZFS I/O bandwidth decreases. Users notice > >very slow response to IMAP requests. On the server, even `ps' becomes > >slow. > > If memory is being stolen from the ARC, then the consumer must be outside > of ZFS. I think this is a case for a traditional performance assessment.
It was being stolen from the ARC, but once we added memory, that was no longer the case. ZFS is still one of the suspects. > >The server is a T2000 running Solaris 10. It's a Cyrus murder back- > >end, essentially only an IMAP server. We did recently upgrade the > >front-end, from a 4-CPU SPARC box to a 16-core Intel box with more > >memory, also running Solaris 10. The front-end runs sendmail and > >proxies IMAP and POP connections to the back-end, and also forwards > >SMTP for local deliveries to the back-end, using LMTP. > > > >Cyrus runs thousands of `imapd' processes, with many `pop3d', and > >`lmtpd' processes as well. This should be an ideal workload for a > >Niagara box. All of these memory-map several moderate-sized > >databases, both Berkeley DB and skiplist types, and occasionally > >update those databases. Our EMC Networker client also often runs > >during the day, doing backups. All of the IMAP mailboxes reside on > >six ZFS filesystems, using a single 2-TB pool. It's only 51% occupied > >at the moment. > > > >Many other layers are involved in this server. We use scsi_vhci for > >redundant I/O paths and Sun's Iscsi initiator to connect to the > >storage on our Netapp filer. The kernel plays a part as well. How > >do we determine which layer is responsible for the slow performance? > > prstat is your friend. Find out who is consuming the resources and work > from there. What resources are visible with prstat, other than CPU and memory? Even at the busiest times, all of the processes only add up to about 6% of the CPU. The load average does rise alarmingly. Nothing is using large amounts of memory, although with thousands of processes, it would add up. > I've found that it often makes sense to create processor sets and segregate > dissimilar apps into different processor sets. mpstat can then clearly show > how each processor set consumes its processors. IMAP workloads can > be very tricky, because of the sort of I/O generated and because IMAP > allows searching to be done on the server, rather than the client (eg POP) What would I look for with mpstat? -- -Gary Mills- -Unix Support- -U of M Academic Computing and Networking- _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
