Re: [OpenIndiana-discuss] VERY slow server performance
Looking through /var/adm/messages, there is the following with respect to interrupts: Sep 7 21:03:33 mediaserver intrd[689]: [ID 702911 daemon.notice] Optimizing interrupt assignments Sep 7 21:03:33 mediaserver intrd[689]: [ID 702911 daemon.notice] Interrupt assignments optimized This was several days past the last system re-boot. Playing more with the system, if I run system monitor, file transfer speeds up dramatically as well as mouse operation almost returns to normal. It's as if interrupts are turned off for the mouse and the Ethernet port - if I run system monitor which starts to poll the Ethernet port as well as access the mouse to update the screen, operation almost gets back to normal. Any suggestions on where to go from here...greatly appreciated. Stuart -Original Message- From: Richard Elling [mailto:richard.ell...@richardelling.com] Sent: September-06-12 7:54 PM To: Discussion list for OpenIndiana Subject: Re: [OpenIndiana-discuss] VERY slow server performance On Sep 6, 2012, at 8:08 AM, Roel_D openindi...@out-side.nl wrote: Reading this it reminds me of the old days where IRQ's were important to systems. Those days my serial mouse could interfere with my modem. But I thought those days were way back.. Interrupt conflicts are syslogged at boot (and other times that the device tree is shaken). Check /var/adm/messages for clues. -- richard -Original Message- From: Ben Taylor [mailto:bentaylor.sol...@gmail.com] Sent: donderdag 6 september 2012 16:56 To: Discussion list for OpenIndiana Subject: Re: [OpenIndiana-discuss] VERY slow server performance On Thu, Sep 6, 2012 at 12:48 AM, Stuart Shirley antarc...@sympatico.ca wrote: Hi - Look for some assistance in improving disk access performance on my OpenIndiana install. Wanting to upgrade my server based on OpenSolaris and ZFS to 3TB drives, I upgraded the OS to OpenIndiana 151a. Performance is terrible over the network as compared to the previous OpenSolaris installation. Hardware is identical with the exception of a second SATA controller (a second identical Supermicro AOC-SAT2-MV8) and additional hard drives. Perhaps related, the USB mouse is virtually none-operational. It will only update the location of the mouse for about 0.5s out of every 10s. This has not been an issue, as I usually connect remotely. But still an indication that all is not well. Transfers on the machine, from drive to drive appears about the same as when running under OpenIndiana - but across the networks it's very slow. Moving a large file from my windows 7 machine across the GbE lan - transfers at ~11.5MB/s as reported by windows copy. Using zpool iostat, the write bandwidth is reported as high as 20M. Now if I run zpool iostat continuously (display every 3 seconds), copy performance increases - moving into the 16 to 17 MB/s range as reported by windows. Copying from one storage pool to another, zpool iostat will report write bandwidths of 26M. My pool configuration is detailed below. Other's slow performance had been pegged to flow control enabled on the Ethernet port - this was disabled. It did make a difference, but not that dramatic. Any suggestions on how to improve performance and how to fix the mouse would be greatly appreciated! Flow control properties: LINK PROPERTYPERM VALUE DEFAULTPOSSIBLE e1000g0 flowctrlrw no bi no,tx,rx,bi Zpool configurations: pool: rpool state: ONLINE scan: resilvered 4.63G in 0h7m with 0 errors on Wed Jun 6 22:12:16 2012 config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 mirror-0ONLINE 0 0 0 c3t0d0s0 ONLINE 0 0 0 c3t1d0s0 ONLINE 0 0 0 errors: No known data errors pool: tank_12T state: ONLINE scan: none requested config: NAMESTATE READ WRITE CKSUM tank_12TONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 c3t7d0 ONLINE 0 0 0 c3t3d0 ONLINE 0 0 0 c3t5d0 ONLINE 0 0 0 c3t6d0 ONLINE 0 0 0 c5d0ONLINE 0 0 0 c8d0ONLINE 0 0 0 errors: No known data errors pool: tank_m state: ONLINE scan: none requested config: NAMESTATE READ WRITE CKSUM tank_m ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 c3t2d0 ONLINE 0 0 0 c7t6d0 ONLINE 0 0 0 c6d0ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 c3t4d0 ONLINE 0 0 0 c7t7d0 ONLINE 0 0 0 c4d0ONLINE 0 0 0 errors: No known data errors The mouse interrupt
Re: [OpenIndiana-discuss] VERY slow server performance
On Sep 6, 2012, at 8:08 AM, Roel_D openindi...@out-side.nl wrote: Reading this it reminds me of the old days where IRQ's were important to systems. Those days my serial mouse could interfere with my modem. But I thought those days were way back.. Interrupt conflicts are syslogged at boot (and other times that the device tree is shaken). Check /var/adm/messages for clues. -- richard -Original Message- From: Ben Taylor [mailto:bentaylor.sol...@gmail.com] Sent: donderdag 6 september 2012 16:56 To: Discussion list for OpenIndiana Subject: Re: [OpenIndiana-discuss] VERY slow server performance On Thu, Sep 6, 2012 at 12:48 AM, Stuart Shirley antarc...@sympatico.ca wrote: Hi - Look for some assistance in improving disk access performance on my OpenIndiana install. Wanting to upgrade my server based on OpenSolaris and ZFS to 3TB drives, I upgraded the OS to OpenIndiana 151a. Performance is terrible over the network as compared to the previous OpenSolaris installation. Hardware is identical with the exception of a second SATA controller (a second identical Supermicro AOC-SAT2-MV8) and additional hard drives. Perhaps related, the USB mouse is virtually none-operational. It will only update the location of the mouse for about 0.5s out of every 10s. This has not been an issue, as I usually connect remotely. But still an indication that all is not well. Transfers on the machine, from drive to drive appears about the same as when running under OpenIndiana - but across the networks it's very slow. Moving a large file from my windows 7 machine across the GbE lan - transfers at ~11.5MB/s as reported by windows copy. Using zpool iostat, the write bandwidth is reported as high as 20M. Now if I run zpool iostat continuously (display every 3 seconds), copy performance increases - moving into the 16 to 17 MB/s range as reported by windows. Copying from one storage pool to another, zpool iostat will report write bandwidths of 26M. My pool configuration is detailed below. Other's slow performance had been pegged to flow control enabled on the Ethernet port - this was disabled. It did make a difference, but not that dramatic. Any suggestions on how to improve performance and how to fix the mouse would be greatly appreciated! Flow control properties: LINK PROPERTYPERM VALUE DEFAULTPOSSIBLE e1000g0 flowctrlrw no bi no,tx,rx,bi Zpool configurations: pool: rpool state: ONLINE scan: resilvered 4.63G in 0h7m with 0 errors on Wed Jun 6 22:12:16 2012 config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 mirror-0ONLINE 0 0 0 c3t0d0s0 ONLINE 0 0 0 c3t1d0s0 ONLINE 0 0 0 errors: No known data errors pool: tank_12T state: ONLINE scan: none requested config: NAMESTATE READ WRITE CKSUM tank_12TONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 c3t7d0 ONLINE 0 0 0 c3t3d0 ONLINE 0 0 0 c3t5d0 ONLINE 0 0 0 c3t6d0 ONLINE 0 0 0 c5d0ONLINE 0 0 0 c8d0ONLINE 0 0 0 errors: No known data errors pool: tank_m state: ONLINE scan: none requested config: NAMESTATE READ WRITE CKSUM tank_m ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 c3t2d0 ONLINE 0 0 0 c7t6d0 ONLINE 0 0 0 c6d0ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 c3t4d0 ONLINE 0 0 0 c7t7d0 ONLINE 0 0 0 c4d0ONLINE 0 0 0 errors: No known data errors The mouse interrupt issue is, in my experience, is not a new one. I've been seeing this on solaris laptops for as long as I can remember. The interesting issue for you is this only happened with the addition of another sata controller. This leads me to believe that there's some rather confused logic in the interrupt handler, such that the mouse interrupts basically gets preempted by the interrupts between the sata controller as data is passed from one controller to the other. Ben ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss -- illumos Day ZFS Day, Oct 1-2, 2012 San Fransisco www.zfsday.com richard.ell...@richardelling.com +1-760-896-4422
Re: [OpenIndiana-discuss] VERY slow server performance
On Thu, Sep 6, 2012 at 12:48 AM, Stuart Shirley antarc...@sympatico.ca wrote: Hi - Look for some assistance in improving disk access performance on my OpenIndiana install. Wanting to upgrade my server based on OpenSolaris and ZFS to 3TB drives, I upgraded the OS to OpenIndiana 151a. Performance is terrible over the network as compared to the previous OpenSolaris installation. Hardware is identical with the exception of a second SATA controller (a second identical Supermicro AOC-SAT2-MV8) and additional hard drives. Perhaps related, the USB mouse is virtually none-operational. It will only update the location of the mouse for about 0.5s out of every 10s. This has not been an issue, as I usually connect remotely. But still an indication that all is not well. Transfers on the machine, from drive to drive appears about the same as when running under OpenIndiana - but across the networks it's very slow. Moving a large file from my windows 7 machine across the GbE lan - transfers at ~11.5MB/s as reported by windows copy. Using zpool iostat, the write bandwidth is reported as high as 20M. Now if I run zpool iostat continuously (display every 3 seconds), copy performance increases - moving into the 16 to 17 MB/s range as reported by windows. Copying from one storage pool to another, zpool iostat will report write bandwidths of 26M. My pool configuration is detailed below. Other's slow performance had been pegged to flow control enabled on the Ethernet port - this was disabled. It did make a difference, but not that dramatic. Any suggestions on how to improve performance and how to fix the mouse would be greatly appreciated! Flow control properties: LINK PROPERTYPERM VALUE DEFAULTPOSSIBLE e1000g0 flowctrlrw no bi no,tx,rx,bi Zpool configurations: pool: rpool state: ONLINE scan: resilvered 4.63G in 0h7m with 0 errors on Wed Jun 6 22:12:16 2012 config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 mirror-0ONLINE 0 0 0 c3t0d0s0 ONLINE 0 0 0 c3t1d0s0 ONLINE 0 0 0 errors: No known data errors pool: tank_12T state: ONLINE scan: none requested config: NAMESTATE READ WRITE CKSUM tank_12TONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 c3t7d0 ONLINE 0 0 0 c3t3d0 ONLINE 0 0 0 c3t5d0 ONLINE 0 0 0 c3t6d0 ONLINE 0 0 0 c5d0ONLINE 0 0 0 c8d0ONLINE 0 0 0 errors: No known data errors pool: tank_m state: ONLINE scan: none requested config: NAMESTATE READ WRITE CKSUM tank_m ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 c3t2d0 ONLINE 0 0 0 c7t6d0 ONLINE 0 0 0 c6d0ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 c3t4d0 ONLINE 0 0 0 c7t7d0 ONLINE 0 0 0 c4d0ONLINE 0 0 0 errors: No known data errors The mouse interrupt issue is, in my experience, is not a new one. I've been seeing this on solaris laptops for as long as I can remember. The interesting issue for you is this only happened with the addition of another sata controller. This leads me to believe that there's some rather confused logic in the interrupt handler, such that the mouse interrupts basically gets preempted by the interrupts between the sata controller as data is passed from one controller to the other. Ben ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
Re: [OpenIndiana-discuss] VERY slow server performance
Reading this it reminds me of the old days where IRQ's were important to systems. Those days my serial mouse could interfere with my modem. But I thought those days were way back.. -Original Message- From: Ben Taylor [mailto:bentaylor.sol...@gmail.com] Sent: donderdag 6 september 2012 16:56 To: Discussion list for OpenIndiana Subject: Re: [OpenIndiana-discuss] VERY slow server performance On Thu, Sep 6, 2012 at 12:48 AM, Stuart Shirley antarc...@sympatico.ca wrote: Hi - Look for some assistance in improving disk access performance on my OpenIndiana install. Wanting to upgrade my server based on OpenSolaris and ZFS to 3TB drives, I upgraded the OS to OpenIndiana 151a. Performance is terrible over the network as compared to the previous OpenSolaris installation. Hardware is identical with the exception of a second SATA controller (a second identical Supermicro AOC-SAT2-MV8) and additional hard drives. Perhaps related, the USB mouse is virtually none-operational. It will only update the location of the mouse for about 0.5s out of every 10s. This has not been an issue, as I usually connect remotely. But still an indication that all is not well. Transfers on the machine, from drive to drive appears about the same as when running under OpenIndiana - but across the networks it's very slow. Moving a large file from my windows 7 machine across the GbE lan - transfers at ~11.5MB/s as reported by windows copy. Using zpool iostat, the write bandwidth is reported as high as 20M. Now if I run zpool iostat continuously (display every 3 seconds), copy performance increases - moving into the 16 to 17 MB/s range as reported by windows. Copying from one storage pool to another, zpool iostat will report write bandwidths of 26M. My pool configuration is detailed below. Other's slow performance had been pegged to flow control enabled on the Ethernet port - this was disabled. It did make a difference, but not that dramatic. Any suggestions on how to improve performance and how to fix the mouse would be greatly appreciated! Flow control properties: LINK PROPERTYPERM VALUE DEFAULTPOSSIBLE e1000g0 flowctrlrw no bi no,tx,rx,bi Zpool configurations: pool: rpool state: ONLINE scan: resilvered 4.63G in 0h7m with 0 errors on Wed Jun 6 22:12:16 2012 config: NAME STATE READ WRITE CKSUM rpool ONLINE 0 0 0 mirror-0ONLINE 0 0 0 c3t0d0s0 ONLINE 0 0 0 c3t1d0s0 ONLINE 0 0 0 errors: No known data errors pool: tank_12T state: ONLINE scan: none requested config: NAMESTATE READ WRITE CKSUM tank_12TONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 c3t7d0 ONLINE 0 0 0 c3t3d0 ONLINE 0 0 0 c3t5d0 ONLINE 0 0 0 c3t6d0 ONLINE 0 0 0 c5d0ONLINE 0 0 0 c8d0ONLINE 0 0 0 errors: No known data errors pool: tank_m state: ONLINE scan: none requested config: NAMESTATE READ WRITE CKSUM tank_m ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 c3t2d0 ONLINE 0 0 0 c7t6d0 ONLINE 0 0 0 c6d0ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 c3t4d0 ONLINE 0 0 0 c7t7d0 ONLINE 0 0 0 c4d0ONLINE 0 0 0 errors: No known data errors The mouse interrupt issue is, in my experience, is not a new one. I've been seeing this on solaris laptops for as long as I can remember. The interesting issue for you is this only happened with the addition of another sata controller. This leads me to believe that there's some rather confused logic in the interrupt handler, such that the mouse interrupts basically gets preempted by the interrupts between the sata controller as data is passed from one controller to the other. Ben ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss ___ OpenIndiana-discuss mailing list OpenIndiana-discuss@openindiana.org http://openindiana.org/mailman/listinfo/openindiana-discuss
