Re: em performs worse than igb (latency wise) in 12?
On 4/6/2019 2:56 AM, Pete French wrote: Something odd going on there there - I am using 12-STABLE and I have igb just fine, and it attaches to the same hardware that 11 did: It does work in 12, throughput is great, just that the latency is higher than 11. igb0: flags=8843 metric 0 mtu 1500 options=e527bb ether 38:ea:a7:8d:c1:6c inet 208.72.56.19 netmask 0xfc00 broadcast 208.72.59.255 inet6 fe80::3aea:a7ff:fe8d:c16c%igb0 prefixlen 64 scopeid 0x1 inet6 2602:ffb8::208:72:56:9 prefixlen 64 media: Ethernet autoselect (1000baseT ) status: active nd6 options=21 Do you have a custom kernel, and if so did you see this note in UPDATING? Yes I do, but it includes all of GENERIC which includes em drivers, otherwise it wouldn't even work with the network card. my custom kernel: include GENERIC ident CUSTOM makeoptions WITH_EXTRA_TCP_STACKS=1 options TCPHPTS options SC_KERNEL_CONS_ATTR=(FG_GREEN|BG_BLACK) options IPSTEALTH options AHC_REG_PRETTY_PRINT # Print register bitfields in debug options AHD_REG_PRETTY_PRINT # Print register bitfields in debug device cryptodev device aesni I did try without RACK just in case that was the culprit. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: em performs worse than igb (latency wise) in 12?
On 05/04/2019 16:01, Kris von Mach wrote: I've upgraded from Stable 11 to Stable 12, and noticed that igb has been removed and is now part of em driver. However, the performance seems a lot worse. This is using HP 366FLR which is just HP's version of Intel i350 I believe. Something odd going on there there - I am using 12-STABLE and I have igb just fine, and it attaches to the same hardware that 11 did: ... [petefrench@dilbert ~]$ ssh turpentine Last login: Fri Apr 5 18:52:50 2019 from 2a02:b90:3002:411::6 FreeBSD 12.0-STABLE r343538 GENERIC Baby baby turpentine... [webadmin@turpentine ~]$ ifconfig igb0 igb0: flags=8843 metric 0 mtu 1500 options=e507bb ether ac:1f:6b:46:5e:32 inet 10.32.10.5 netmask 0x broadcast 10.32.255.255 inet6 fe80::ae1f:6bff:fe46:5e32%igb0 prefixlen 64 scopeid 0x1 inet6 2a02:1658:1:2:e550::5 prefixlen 64 media: Ethernet autoselect (1000baseT ) status: active nd6 options=21 Do you have a custom kernel, and if so did you see this note in UPDATING? 20170109: The igb(4), em(4) and lem(4) ethernet drivers are now implemented via IFLIB. If you have a custom kernel configuration that excludes em(4) but you use igb(4), you need to re-add em(4) to your custom configuration. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: NVME aborting outstanding i/o
On Fri, Apr 5, 2019 at 1:33 AM Patrick M. Hausen wrote: > Hi all, > > > Am 04.04.2019 um 17:11 schrieb Warner Losh : > > There's a request that was sent down to the drive. It took longer than > 30s to respond. One of them, at least, was a trim request. > > […] > > Thanks for the explanation. > > This further explains why I was seeing a lot more of those and the system > occasionally froze for a couple of seconds after I increased these: > > vfs.zfs.vdev.async_write_max_active: 10 > vfs.zfs.vdev.async_read_max_active: 3 > vfs.zfs.vdev.sync_write_max_active: 10 > vfs.zfs.vdev.sync_read_max_active: 10 > > as recommended by Allan Jude reasoning that NVME devices could work on > up to 64 requests in parallel. I have since reverted that change and I am > running with the defaults. > > If I understand correctly, this: > > > hw.nvme.per_cpu_io_queues=0 > > essentially limits the rate at which the system throws commands at the > devices. Correct? > Yes. It de-facto limits the number of commands the system can throw at a nvme drive. Some drives have trouble with multiple CPUs submitting things. Others just have trouble with the volume of commands sometimes. This limits both. > So it’s not a real fix and there’s nothing fundamentally wrong with the > per CPU queue or > interrupt implementation. I will look into new firmware for my Intel > devices and > try tweaking the vfs.zfs.vdev.trim_max_active and related parameters. > Correct. It's a hack-a-around. > Out of curiosity: what happens if I disable TRIM? My knowledge is rather > superficial > and I just filed that under „TRIM is absolutely essential lest performance > will > suffer severely and your devices die - plus bad karma, of course …“ ;-) > TRIMs help the drive optimize their garbage collection by giving it a larger pool of free blocks to work with. This has the effect of reducing write amplification. Write amp is the measure of the amount of extra work the drive has to do for every user write it processes. Ideally, you want this number to be 1.0. You'll never get to 1.0, but numbers less than 1.5 are common and most of the models drive makers use to rate the lifetime of their NAND assume a write amp of about 2. So, if you eliminate the TRIMs you eliminate this optimization and write amp will increase. This has two bad effects. First, wear and tear on the NAND. Second, it takes resources away from the user. In practice, however, the bad effects are quite limited if you don't have a write intensive workload. Your drive is rated for so many drive writes per day (or equivalently total data written over the life of the drive). This will be on the spec sheet somewhere. If you don't have a write intensive workload (which I'd say is any sustained write load greater than about 1/10th the datasheet write limit), then if you think TRIMs are causing issues, you should disable them. The effects of not trimming are likely to be in the noise on such systems, and the benefits of having things TRIMed will be less. At work, for a large video streaming company, we enable the TRIMs, even though we're on the edge of the rule of thumb since we're unsure how long the machines really need to be in the field and don't want to risk it. Except for the version of Samsung nvme drives (PM963, no longer made) we got a while ago... those we turn TRIM off on because UFS' machine-gunning down of TRIMs and nvd's blind pass-through of TRIMs took down the drive. UFS now combines TRIMs and we've moved to using nda since it also combines TRIMs and it won't be so bad if we tried again today. Drive makers optimize different things. Enterprise drives handle TRIMs a lot better than consumer drives. consumer drives are cheaper (in oh so many ways), so some care is needed. Intel makes a wide range of drives, from the super duper awesome (with prices to match) to the somewhat disappointing (but incredibly cheap and good enough for a lot of applicaitons). Not sure where on this scale your drives fall on this spectrum. tl;dr: Unless you are writing the snot out of those Intel drives, disabling TRIM entirely will likely help avoid pushing so many commands they timeout. Warner ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
em performs worse than igb (latency wise) in 12?
Hello, I've upgraded from Stable 11 to Stable 12, and noticed that igb has been removed and is now part of em driver. However, the performance seems a lot worse. This is using HP 366FLR which is just HP's version of Intel i350 I believe. I've tried incorporating the sysctl settings I used in 11 into 12: dev.igb.0.fc=0 dev.igb.0.iflib.rx_budget=65535 hw.igb.rx_process_limit="-1" hw.igb.num_queues="2" hw.igb.rxd="2048" # (default 1024) hw.igb.txd="2048" # (default 1024) hw.igb.max_interrupt_rate="32000" # (default 8000) hw.em.rx_process_limit="-1" hw.em.num_queues="2" hw.em.rxd="2048" # (default 1024) hw.em.txd="2048" # (default 1024) hw.em.max_interrupt_rate="32000" But that didn't seem to help. Switching back to 11, of course restored the performance. The performance hit I'm talking about seems to be with latency, not throughput. To give you an example, this server is serving a website that is monitored from different locations and all locations the latency has increased by about 30-50% using stable 12. Any ideas as to how to improve latency in 12? Can you compile igb driver in 12? Has something else other than the network driver changed in 12 that could cause an increase in latency? ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Rare NVME related freeze at boot (was: Re: NVME aborting outstanding i/o)
Hi! > Am 05.04.2019 um 16:36 schrieb Warner Losh : > What normally comes after the nvme6 line in boot? Often times it's the next > thing after the last message that's the issue, not the last thing. nvme7 ;-) And I had hangs at nvme1, nvme3, … as well. Patrick -- punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe i...@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Rare NVME related freeze at boot (was: Re: NVME aborting outstanding i/o)
On Fri, Apr 5, 2019 at 6:41 AM Patrick M. Hausen wrote: > Hi all, > > in addition to the aborted commands every dozen of system boots or so > (this order of magnitude) the kernel simply hangs during initialisation of > one of the NVME devices: > > https://cloud.hausen.com/s/TxPTDFJwMe6sJr2 > > The particular device affected is not constant. > > A power cycle fixes it, the system has not shown hangs/freezes during > multiuser operation, yet. > > > Any ideas? > What normally comes after the nvme6 line in boot? Often times it's the next thing after the last message that's the issue, not the last thing. Warner ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Rare NVME related freeze at boot (was: Re: NVME aborting outstanding i/o)
Hi all, in addition to the aborted commands every dozen of system boots or so (this order of magnitude) the kernel simply hangs during initialisation of one of the NVME devices: https://cloud.hausen.com/s/TxPTDFJwMe6sJr2 The particular device affected is not constant. A power cycle fixes it, the system has not shown hangs/freezes during multiuser operation, yet. Any ideas? Patrick -- punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe i...@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: NVME aborting outstanding i/o
Hi all, > Am 04.04.2019 um 17:11 schrieb Warner Losh : > There's a request that was sent down to the drive. It took longer than 30s to > respond. One of them, at least, was a trim request. > […] Thanks for the explanation. This further explains why I was seeing a lot more of those and the system occasionally froze for a couple of seconds after I increased these: vfs.zfs.vdev.async_write_max_active: 10 vfs.zfs.vdev.async_read_max_active: 3 vfs.zfs.vdev.sync_write_max_active: 10 vfs.zfs.vdev.sync_read_max_active: 10 as recommended by Allan Jude reasoning that NVME devices could work on up to 64 requests in parallel. I have since reverted that change and I am running with the defaults. If I understand correctly, this: > hw.nvme.per_cpu_io_queues=0 essentially limits the rate at which the system throws commands at the devices. Correct? So it’s not a real fix and there’s nothing fundamentally wrong with the per CPU queue or interrupt implementation. I will look into new firmware for my Intel devices and try tweaking the vfs.zfs.vdev.trim_max_active and related parameters. Out of curiosity: what happens if I disable TRIM? My knowledge is rather superficial and I just filed that under „TRIM is absolutely essential lest performance will suffer severely and your devices die - plus bad karma, of course …“ ;-) Kind regards, Patrick -- punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe i...@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
