HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)
Hi, I've merged the NCQ branch to HEAD. NCQ is supported on ahcisata(4), siisata(4), and mvsata(4) Gen IIe at this moment. The code was quite extensively tested on that harware on amd64. Other archs and drivers compile, but I had no way to test them. Particularily, I had no chance to really test any real IDE disks, neither any non-PCI controller attachments. If you are in position to confirm them working, I'd appreciate it. Please report any problems, I'll fix any potential fallout ASAP. Performance wise, I've tested so far only sequential I/O via fio and dd with cache enabled. I observed very mild (>2%, 72.7->74.8 MB/s) performance increase for spinning rust HDD, but quite big increase for SSD (350->450 MB/s). With disabled disk cache, or random I/O, NCQ will probably have more effect. Jaromir
Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)
Hello, On Sat, 7 Oct 2017 18:34:04 +0200 Jaromír Doleček wrote: > I've merged the NCQ branch to HEAD. Nice, thanks! > The code was quite extensively tested on that harware on amd64. Other archs > and drivers compile, but I had no way to test them. Particularily, I had no > chance to really test any real IDE disks, neither any non-PCI controller > attachments. If you are in position to confirm them working, I'd appreciate > it. I tried siisata on sparc64 ( in a 64bit/66MHz slot ) and ahcisata on a Cubietruck, both seem to work as intended. > Please report any problems, I'll fix any potential fallout ASAP. > > Performance wise, I've tested so far only sequential I/O via fio and dd > with cache enabled. I observed very mild (>2%, 72.7->74.8 MB/s) performance > increase for spinning rust HDD, but quite big increase for SSD (350->450 > MB/s). With disabled disk cache, or random I/O, NCQ will probably have more > effect. I tried sequential reads ( dd if=/dev/rwd0c ... ) and throughput took a significant hit. I used to get about 120MB/s with the siisata, now it fluctuates between 80 and 90MB/s, ahcisata dropped from about 80MB/s to 70MB/s. Both spinning rust of varying vintage. I should probably do a bonnie run on either one before & after to see if there's any change in random access. have fun Michael
Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)
I've fixed the compilation for ALL kernels. 2017-10-10 17:34 GMT+02:00 Michael : > I tried sequential reads ( dd if=/dev/rwd0c ... ) and throughput took a > significant hit. I used to get about 120MB/s with the siisata, now it > fluctuates between 80 and 90MB/s, ahcisata dropped from about 80MB/s to > 70MB/s. Both spinning rust of varying vintage. > I should probably do a bonnie run on either one before & after to see > if there's any change in random access. I've seen this on one of my disks, too. It seems it's much slower in NCQ mode. I think the firmware might not utilise the disk cache properly when in NCQ mode. You can try switching it off via sysctl, hw.wdX.use_ncq. You can also try to turn off use_ncq_prio if that makes any difference. We might need to introduce some heuristics for this. Jaromir
Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)
On Tue, Oct 10, 2017 at 11:11:54PM +0200, Jarom??r Dole??ek wrote: > I've fixed the compilation for ALL kernels. > > 2017-10-10 17:34 GMT+02:00 Michael : > > I tried sequential reads ( dd if=/dev/rwd0c ... ) and throughput took a > > significant hit. I used to get about 120MB/s with the siisata, now it > > fluctuates between 80 and 90MB/s, ahcisata dropped from about 80MB/s to > > 70MB/s. Both spinning rust of varying vintage. > > I should probably do a bonnie run on either one before & after to see > > if there's any change in random access. > > I've seen this on one of my disks, too. It seems it's much slower in NCQ > mode. I think the firmware might not utilise the disk cache properly when > in NCQ mode. It probably has to do with our small maximum transfer size. The disk is probably trying to be safer and *not* caching tagged writes as aggressively, but with only 32 commands in-flight (SCSI/SAS allow 256) and a maximum transfer size of 64K (our MAXPHYS), that's only 2MB of in-flight data; maybe not enough to keep up with the actual bandwidth of the media at its real latency. Most other operating systems will hand the drive 32 x 128K transfers, or even larger. If you want to attack the -- fairly minor at this point, I think -- problems of rebasing and merging the tls-maxphys branch, I bet it would help here. I have the interest, but not the time, I'm afraid. Real life interferes again. Thor
Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)
Hello, On Tue, 10 Oct 2017 23:11:54 +0200 Jaromír Doleček wrote: > I've seen this on one of my disks, too. It seems it's much slower in NCQ > mode. I think the firmware might not utilise the disk cache properly when > in NCQ mode. > > You can try switching it off via sysctl, hw.wdX.use_ncq. You can also try > to turn off use_ncq_prio if that makes any difference. /home/ml# sysctl -w hw.wd1.use_ncq=0 hw.wd1.use_ncq: 1 -> 0 /home/ml# dd if=/dev/rwd1c of=/dev/null bs=1m count=2048 2048+0 records in 2048+0 records out 2147483648 bytes transferred in 21.747 secs (98748500 bytes/sec) /home/ml# sysctl -w hw.wd1.use_ncq=1 hw.wd1.use_ncq: 0 -> 1 /home/ml# dd if=/dev/rwd1c of=/dev/null bs=1m count=2048 2048+0 records in 2048+0 records out 2147483648 bytes transferred in 26.125 secs (82200331 bytes/sec) /home/ml# sysctl -w hw.wd1.use_ncq_prio=0 hw.wd1.use_ncq_prio: 1 -> 0 /home/ml# dd if=/dev/rwd1c of=/dev/null bs=1m count=2048 2048+0 records in 2048+0 records out 2147483648 bytes transferred in 24.187 secs (88786689 bytes/sec) ... that's the siisata: siisata0 at pci0 dev 2 function 0: vendor 1095 product 3124 (rev. 0x02) siisata0: interrupting at ivec 700 siisata0: SiI3124, 3.0Gb/s siisata0: 64-bit 66MHz PCI ... with one of these: siisata0 port 0: device present, speed: 3.0Gb/s wd1(siisata0:0:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 5 (Ultra/100) (using DMA), NCQ (30 tags) w/PRIO Model: WDC WD5000AAKX-083CA1, Rev: 17.01H17, Serial #: WD-WCAYUCL72135 World Wide Name: 50014EE104595309 Device type: ATA, fixed Capacity 500 Gbytes, 976773168 sectors, 512 bytes/sector Cylinders: 16383, heads: 16, sec/track: 63 ... Serial ATA capabilities: 1.5Gb/s signaling 3.0Gb/s signaling 6.0Gb/s signaling Native Command Queuing PHY Event Counters have fun Michael
Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)
t...@panix.com (Thor Lancelot Simon) writes: >It probably has to do with our small maximum transfer size. The disk is >probably trying to be safer and *not* caching tagged writes as aggressively, >but with only 32 commands in-flight (SCSI/SAS allow 256) and a maximum >transfer size of 64K (our MAXPHYS), that's only 2MB of in-flight data; >maybe not enough to keep up with the actual bandwidth of the media at >its real latency. With a larger MAXPHYS there wouldn't be as many transfers in flight. The disk here in the Thinkpad behaves differently. With up to 5 concurrent transfers (320k buffer size for dd reading the raw partition) everything is fine. With 6 concurrent transfers the speed goes down again, with 30 transfers we are at half speed. -- -- Michael van Elst Internet: mlel...@serpens.de "A potential Snark may lurk in every tree."
Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)
macal...@netbsd.org (Michael) writes: >/home/ml# sysctl -w hw.wd1.use_ncq=0 >hw.wd1.use_ncq: 1 -> 0 >/home/ml# dd if=/dev/rwd1c of=/dev/null bs=1m count=2048 >2048+0 records in >2048+0 records out >2147483648 bytes transferred in 21.747 secs (98748500 bytes/sec) Please try to use different buffer sizes. Maybe there is a pattern too. -- -- Michael van Elst Internet: mlel...@serpens.de "A potential Snark may lurk in every tree."
Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)
On Wed, Oct 11, 2017 at 08:19:13PM +, Chavdar Ivanov wrote: > I see you are on a ThinkPad as well. I have mentioned it elsewhere, on my > ThinkPad T61p I am getting reliably since the NCQ update the following: > ... > wd0 at atabus0 drive 0 > wd0: > wd0: drive supports 16-sector PIO transfers, LBA48 addressing > wd0: 298 GB, 620181 cyl, 16 head, 63 sec, 512 bytes/sect x 625142448 sectors > piixide0:0:0: bad state 0 in wdc_ata_bio_intr > panic: wdc_ata_bio_intr: bad state > fatal breakpoint trap in supervisor mode > trap type 1 code 0 rip 0x8021c0c5 cs 0x8 rflags 0x246 cr2 0 ilevel > 0x8 rsp 0xe40040003c38 > curlwp 0xe4013bb27840 pid 0.2 lowest kstack 0xe40042c0 > rebooting... Looks like bad handling of the drives RESET state. N.B. the T61p supports AHCI mode too, any reason why this is not selected in the BIOS? AHCI mode so far didn't show problems. Greetings, -- Michael van Elst Internet: mlel...@serpens.de "A potential Snark may lurk in every tree."
Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)
I see you are on a ThinkPad as well. I have mentioned it elsewhere, on my ThinkPad T61p I am getting reliably since the NCQ update the following: ... wd0 at atabus0 drive 0 wd0: wd0: drive supports 16-sector PIO transfers, LBA48 addressing wd0: 298 GB, 620181 cyl, 16 head, 63 sec, 512 bytes/sect x 625142448 sectors piixide0:0:0: bad state 0 in wdc_ata_bio_intr panic: wdc_ata_bio_intr: bad state fatal breakpoint trap in supervisor mode trap type 1 code 0 rip 0x8021c0c5 cs 0x8 rflags 0x246 cr2 0 ilevel 0x8 rsp 0xe40040003c38 curlwp 0xe4013bb27840 pid 0.2 lowest kstack 0xe40042c0 rebooting... ... The disk is on the place of the CD with an appropriate tray, this happens with kernels from releng, not only with mine. There is another disk, an SSD in the internal bay, with a Windows 10 and some Linux installation, the default boot is NetBSD from the tray. Chavdar Ivanov On Wed, 11 Oct 2017 at 20:57 Michael van Elst wrote: > t...@panix.com (Thor Lancelot Simon) writes: > > >It probably has to do with our small maximum transfer size. The disk is > >probably trying to be safer and *not* caching tagged writes as > aggressively, > >but with only 32 commands in-flight (SCSI/SAS allow 256) and a maximum > >transfer size of 64K (our MAXPHYS), that's only 2MB of in-flight data; > >maybe not enough to keep up with the actual bandwidth of the media at > >its real latency. > > With a larger MAXPHYS there wouldn't be as many transfers in flight. > > The disk here in the Thinkpad behaves differently. With up to 5 concurrent > transfers (320k buffer size for dd reading the raw partition) everything > is fine. With 6 concurrent transfers the speed goes down again, with > 30 transfers we are at half speed. > > -- > -- > Michael van Elst > Internet: mlel...@serpens.de > "A potential Snark may lurk in every tree." >
Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)
It supports AHCI mode only on the regular internal disk, not on a disk connected via a tray to the CDROM. On Thu, 12 Oct 2017, 00:11 Michael van Elst, wrote: > On Wed, Oct 11, 2017 at 08:19:13PM +, Chavdar Ivanov wrote: > > I see you are on a ThinkPad as well. I have mentioned it elsewhere, on my > > ThinkPad T61p I am getting reliably since the NCQ update the following: > > ... > > wd0 at atabus0 drive 0 > > wd0: > > wd0: drive supports 16-sector PIO transfers, LBA48 addressing > > wd0: 298 GB, 620181 cyl, 16 head, 63 sec, 512 bytes/sect x 625142448 > sectors > > piixide0:0:0: bad state 0 in wdc_ata_bio_intr > > panic: wdc_ata_bio_intr: bad state > > fatal breakpoint trap in supervisor mode > > trap type 1 code 0 rip 0x8021c0c5 cs 0x8 rflags 0x246 cr2 0 > ilevel > > 0x8 rsp 0xe40040003c38 > > curlwp 0xe4013bb27840 pid 0.2 lowest kstack 0xe40042c0 > > rebooting... > > Looks like bad handling of the drives RESET state. > > N.B. the T61p supports AHCI mode too, any reason why this is not > selected in the BIOS? AHCI mode so far didn't show problems. > > > Greetings, > -- > Michael van Elst > Internet: mlel...@serpens.de > "A potential Snark may lurk in every tree." >
Re: HEADS-UP: SATA NCQ support merged (from jdolecek-ncq branch)
On Wed, Oct 11, 2017 at 07:56:04PM -, Michael van Elst wrote: > t...@panix.com (Thor Lancelot Simon) writes: > > >It probably has to do with our small maximum transfer size. The disk is > >probably trying to be safer and *not* caching tagged writes as aggressively, > >but with only 32 commands in-flight (SCSI/SAS allow 256) and a maximum > >transfer size of 64K (our MAXPHYS), that's only 2MB of in-flight data; > >maybe not enough to keep up with the actual bandwidth of the media at > >its real latency. > > With a larger MAXPHYS there wouldn't be as many transfers in flight. That depends on the workload and the clustering behavior of the filesystem. Remember, in the relevant sense, data the disk has already accepted is "in flight" from the point of view of absorbing latency, in the non-NCQ, write-cache-enabled case. The host can't tell it's "in flight", but it still is -- so the host will continue to write aggressively even if the filesystem's clustering behavior would otherwise throttle it -- because to the host, these writes appear to be "complete". If the drive is not aggressively caching writes in the same way when NCQ is in use, this won't happen, since less writes will appear, to the host, to be completed. With only 32 tags available, we may need a larger transfer per tag to allow the same amount of in-flight data (if the disk, with NCQ enabled, has stopped lying about whether the transfers are really complete). -- Thor Lancelot Simont...@panix.com "The two most common variations translate as follows: illegitimi non carborundum = the unlawful are not silicon carbide illegitimis non carborundum = the unlawful don't have silicon carbide."