Re: [OmniOS-discuss] OmniOS Bloody update for Feb 18
Hello Dan McDonald and List, do your remember my feature request? ,-[ ]- | | at Friday, 3. Okt. 2014 17:22 Dan McDonald | at mid:a762c059-f687-42de-a1f7-6cf2a7f58...@omniti.com wrote: | | On Oct 3, 2014, at 9:29 AM, Alexander Lesle | gro...@tierarzt-mueller.de wrote: | | Hello Dan McDonald and List, | | in my mind this was a great proposal what you done at Sep, 02. | That's the way to build a great OmniOS near to the users. | | Here my request for the next release: | It would be convenient if open-vm-tools were installed in the new | release. http://sourceforge.net/projects/open-vm-tools/files/ | | Actually, this work has been done. I dropped it on the ground: | | https://github.com/omniti-labs/omnios-build/pull/39 | | I will be merging this pull requests now in the master branch, | and seeing how OOod chews on it. | | | I know there's other work to be done for OmniOS-as-a-guest as well. | | Thanks, | DAn | `--- Do you find the time to integrate open-vm-tools in the next stable version? Thanks. On Februar, 19 2015, 03:36 Dan McDonald wrote in [1]: There will be only 1-3 more updates prior to the next OmniOS stable (and for this time, this stable is also Long-Term Support) release. If you notice anything weird about the bloody bits, please let me know ASAP. I've heard little/no complaints about the updates to pkg(5), so either you're very happy, or not using it. :) This update will only be reaching the repo. * omnios-build master branch, revision f3d6d48 * Git to 2.3.0. * UnZIP fixes. * OpenJDK7 up to update 76, build 31. * Microtasking libraries (/lib/libmtsk*) are back as a distinct package (system/library/mtsk) now that they are not part of the (now open-source) Math libraries. * illumos-omnios master branch, revision cbf73e4 (last illumos-gate merge 336069c) * Bugfixes in PF_PACKET, SMB/CIFS, header files, NFS, and man pages. -- Best Regards Alexander Februar, 20 2015 [1] mid:f70c482d-68e8-43a2-bf73-86ade4b63...@omniti.com ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] OmniOS Bloody update for Feb 18
I will second that request as my OmniOS experiments always start as VMware VMs. -Chip On Fri, Feb 20, 2015 at 3:17 AM, Alexander Lesle gro...@tierarzt-mueller.de wrote: Hello Dan McDonald and List, do your remember my feature request? ,-[ ]- | | at Friday, 3. Okt. 2014 17:22 Dan McDonald | at mid:a762c059-f687-42de-a1f7-6cf2a7f58...@omniti.com wrote: | | On Oct 3, 2014, at 9:29 AM, Alexander Lesle | gro...@tierarzt-mueller.de wrote: | | Hello Dan McDonald and List, | | in my mind this was a great proposal what you done at Sep, 02. | That's the way to build a great OmniOS near to the users. | | Here my request for the next release: | It would be convenient if open-vm-tools were installed in the new | release. http://sourceforge.net/projects/open-vm-tools/files/ | | Actually, this work has been done. I dropped it on the ground: | | https://github.com/omniti-labs/omnios-build/pull/39 | | I will be merging this pull requests now in the master branch, | and seeing how OOod chews on it. | | | I know there's other work to be done for OmniOS-as-a-guest as well. | | Thanks, | DAn | `--- Do you find the time to integrate open-vm-tools in the next stable version? Thanks. On Februar, 19 2015, 03:36 Dan McDonald wrote in [1]: There will be only 1-3 more updates prior to the next OmniOS stable (and for this time, this stable is also Long-Term Support) release. If you notice anything weird about the bloody bits, please let me know ASAP. I've heard little/no complaints about the updates to pkg(5), so either you're very happy, or not using it. :) This update will only be reaching the repo. * omnios-build master branch, revision f3d6d48 * Git to 2.3.0. * UnZIP fixes. * OpenJDK7 up to update 76, build 31. * Microtasking libraries (/lib/libmtsk*) are back as a distinct package (system/library/mtsk) now that they are not part of the (now open-source) Math libraries. * illumos-omnios master branch, revision cbf73e4 (last illumos-gate merge 336069c) * Bugfixes in PF_PACKET, SMB/CIFS, header files, NFS, and man pages. -- Best Regards Alexander Februar, 20 2015 [1] mid:f70c482d-68e8-43a2-bf73-86ade4b63...@omniti.com ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] UPCOMING: End of Support Life approaching for OmniOS r151010
never mind, it's getting filled later on.
should have had a look at the whole of it before starting to bark :/
sorry for the noise!
On 02/20/2015 11:14 PM, Dominik Hassler wrote:
Dan,
I had a look at HVM-807 you mentioned.
Shouldn't line 251 in 'net/vnic.c' be
for (i = 0; i MIN(iovcnt, FRAMEIO_NVECS_MAX); i++, iov++) {
instead of
for (i = 0; i MIN(iovcnt, FRAMEIO_NVECS_MAX - 1); i++, iov++) {
As otherwise the last element of the frameio will not be used at all if
iovcnt is greater than the frameio buffer size?
Kind regards
Dominik
On 02/20/2015 09:17 PM, Dan McDonald wrote:
On Feb 20, 2015, at 2:58 PM, Tobias Oetiker t...@oetiker.ch wrote:
Hi Dan,
I do not have a spare machine to test this, but if the dramatic
network io performance regression for kvm present in r151012 has
not been fixed in r151014 then we will be stranded on r151010.
And the upstreaming of VND is likely not to happen with this release. That
blocks us from properly catching up with illumos-kvm-cmd.
There are several commits (Starting with the March 19th one we can't bring
in w/o VND) here:
https://github.com/joyent/illumos-kvm-cmd/commits/master
Currently, we only backport the most recent fix about QEMU using
preadv/pwritev recklessly. IF (big if) there are other commits we can
backport, you could try it using the patches/ directory of omnios-build.
I'm thinking maybe HVM-807's may help you (provided that's indepdent of VND
support).
Have you seen anything with dtrace or lockstat that could give a clue to
what's holding things up? I don't recall seeing anything observed.
Judging from the echo on the ML, not many people seem to see this
problem, or care about it.
I'm sorry, but that does appear to be the case.
Next week I disappear for other-customer work, but starting in March, I'm
doing nothing but r151014.
Dan
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
0xF9ECC6A5.asc
Description: application/pgp-keys
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] UPCOMING: End of Support Life approaching for OmniOS r151010
Dan,
I had a look at HVM-807 you mentioned.
Shouldn't line 251 in 'net/vnic.c' be
for (i = 0; i MIN(iovcnt, FRAMEIO_NVECS_MAX); i++, iov++) {
instead of
for (i = 0; i MIN(iovcnt, FRAMEIO_NVECS_MAX - 1); i++, iov++) {
As otherwise the last element of the frameio will not be used at all if
iovcnt is greater than the frameio buffer size?
Kind regards
Dominik
On 02/20/2015 09:17 PM, Dan McDonald wrote:
On Feb 20, 2015, at 2:58 PM, Tobias Oetiker t...@oetiker.ch wrote:
Hi Dan,
I do not have a spare machine to test this, but if the dramatic
network io performance regression for kvm present in r151012 has
not been fixed in r151014 then we will be stranded on r151010.
And the upstreaming of VND is likely not to happen with this release. That
blocks us from properly catching up with illumos-kvm-cmd.
There are several commits (Starting with the March 19th one we can't bring in
w/o VND) here:
https://github.com/joyent/illumos-kvm-cmd/commits/master
Currently, we only backport the most recent fix about QEMU using
preadv/pwritev recklessly. IF (big if) there are other commits we can
backport, you could try it using the patches/ directory of omnios-build. I'm
thinking maybe HVM-807's may help you (provided that's indepdent of VND
support).
Have you seen anything with dtrace or lockstat that could give a clue to
what's holding things up? I don't recall seeing anything observed.
Judging from the echo on the ML, not many people seem to see this
problem, or care about it.
I'm sorry, but that does appear to be the case.
Next week I disappear for other-customer work, but starting in March, I'm
doing nothing but r151014.
Dan
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
0xF9ECC6A5.asc
Description: application/pgp-keys
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] UPCOMING: End of Support Life approaching for OmniOS r151010
Unicast... Found this: http://echelog.com/logs/browse/smartos/1408053600 With this to say: rmustacc So, I can give you a copy of the analysis. rmustacc Basically the number of iovecs that we can receive is up to the virtio ring buffer size. rmustacc Which is a lot more than the number of frame io vectors. rmustacc And while this shouldn't generally happen, Windows has been guilt of it for non-jumbo frames. rmustacc Why you would break up a 1500 byte packet into 32 iovectors is beyond me. nahamu and when it does that QEMU crashes on the assert with the old code, correct? Dan ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] OmniOS Bloody update for Feb 18
Hello Dan McDonald and List, thanks for your offer, but I cant do that because I am not a omnios expert enough and my english is not very well. On Februar, 20 2015, 16:27 Dan McDonald wrote in [1]: Are you two volunteering for testing? If so, I can see what I can do about pushing the package (without it being in an incorporation) into the bloody repo server -- assuming I can build it properly. Dan Sent from my iPhone (typos, autocorrect, and all) On Feb 20, 2015, at 9:20 AM, Schweiss, Chip c...@innovates.com wrote: I will second that request as my OmniOS experiments always start as VMware VMs. -Chip On Fri, Feb 20, 2015 at 3:17 AM, Alexander Lesle gro...@tierarzt-mueller.de wrote: Hello Dan McDonald and List, do your remember my feature request? ,-[ ]- | | at Friday, 3. Okt. 2014 17:22 Dan McDonald | at mid:a762c059-f687-42de-a1f7-6cf2a7f58...@omniti.com wrote: | | On Oct 3, 2014, at 9:29 AM, Alexander Lesle | gro...@tierarzt-mueller.de wrote: | | Hello Dan McDonald and List, | | in my mind this was a great proposal what you done at Sep, 02. | That's the way to build a great OmniOS near to the users. | | Here my request for the next release: | It would be convenient if open-vm-tools were installed in the new | release. http://sourceforge.net/projects/open-vm-tools/files/ | | Actually, this work has been done. I dropped it on the ground: | | https://github.com/omniti-labs/omnios-build/pull/39 | | I will be merging this pull requests now in the master branch, | and seeing how OOod chews on it. | | | I know there's other work to be done for OmniOS-as-a-guest as well. | | Thanks, | DAn | `--- Do you find the time to integrate open-vm-tools in the next stable version? Thanks. On Februar, 19 2015, 03:36 Dan McDonald wrote in [1]: There will be only 1-3 more updates prior to the next OmniOS stable (and for this time, this stable is also Long-Term Support) release. If you notice anything weird about the bloody bits, please let me know ASAP. I've heard little/no complaints about the updates to pkg(5), so either you're very happy, or not using it. :) This update will only be reaching the repo. * omnios-build master branch, revision f3d6d48 * Git to 2.3.0. * UnZIP fixes. * OpenJDK7 up to update 76, build 31. * Microtasking libraries (/lib/libmtsk*) are back as a distinct package (system/library/mtsk) now that they are not part of the (now open-source) Math libraries. * illumos-omnios master branch, revision cbf73e4 (last illumos-gate merge 336069c) * Bugfixes in PF_PACKET, SMB/CIFS, header files, NFS, and man pages. -- Best Regards Alexander Februar, 20 2015 [1] mid:f70c482d-68e8-43a2-bf73-86ade4b63...@omniti.com ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -- Best Regards Alexander Februar, 20 2015 [1] mid:a19863f3-e121-445e-bf4b-da5c97ef3...@omniti.com ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] OmniOS Bloody update for Feb 18
On Feb 20, 2015, at 3:43 PM, Alexander Lesle gro...@tierarzt-mueller.de wrote: Hello Dan McDonald and List, thanks for your offer, but I cant do that because I am not a omnios expert enough and my english is not very well. If you are running the bloody version, all you need is this: pkg install open-vm-tools That's it! If you're waiting for something supported, you'll need to wait for r151014 to ship. Dan ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
[OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy
Hello all, Each of the things in the subject line are: 1: Horrendously broken 2: Have an extremely poor short-term outlook 3: Will take a huge investment of time by intelligent, dedicated, and insightful people to fix. It's common knowledge that the ixgbe driver in omnios/illumos/opensolaris is broke as hell. The point of this message is not to complain. The point is to pass on a configuration that is working for me, albeit in a screwed-up degraded fashion. I have four ESXi 5.5u2 host servers with 1 Intel PCI-e quad-gigabit NIC installed in each. Three of the gigabit ports on each client are dedicated to carry iSCSI traffic between each host and a single storage server. The storage server is based on a Supermicro X10SLM-F mainboard, which has three PCI-e slots. Two of the slots are used for storage controllers, and a single slot is used for an Intel X520 dual-port fiber 10G NIC. Previously, I had a single storage controller and two quad-gig NICs installed in the storage server, and was able to get close to line-rate on multipath iSCSI with three host clients. But when I added the fourth, I upgraded to 10G. After installation and configuration, I observed all kinds of bad behavior in the network traffic between the hosts and the server. All of this bad behavior is traced to the ixgbe driver on the storage server. Without going into the full troubleshooting process, here are my takeaways: 1: The only tuning factor that appears to have significant effect on the driver is MTU size. This applies to both the MTU of the ixgbe NIC as well as the MTU of the 1-gig NICs used in the hosts. 2: I have seen best performance with the MTU on the ixgbe set to 1000 bytes (yes, 1k). The MTU on the ESXi interfaces is set to 3000 bytes. 3: Setting 9000 byte MTUs on both sides results in about 150MB/s write speeds on a a linux vmware guest running a 10GB dd operation. But read speeds are at 5MB/s. 4: Testing of dd operations on the storage server itself shows that the filesystem is capable of performing 500MB/s+ reads and writes. 5: After setting the MTUs listed in point 2, I am able to get 270-300MB/s writes on the guest OS, and ~200MB/s reads. Not perfect, but I'll take it. 6: No /etc/system or other kernel tunings are in use. 7: Delayed ACK, Nagle, and L2 flow control tests had no effect. 8: pkg -u was performed before all tests, so I should be using the latest kernel code, etc. 9: When capturing traffic on omnios, I used the CSW distribution of tcpdump. It's worth noting that unlike EVERY ... OTHER ... IMPLEMENTATION ... of tcpdump I've ever used (BSD flavors, OSX, various linux distros, various embedded distros), libpcap doesn't appear to get individual frame reports from the omnios kernel, and so aggregates multi-frame TCP segments into a single record. This has the appearance of 20-60kB frames being transmitted by omnios when reading a packet capture with Wireshark. I cannot tell you how irritating this is when troubleshooting network issues. 10: At the wire level, the speed problems are clearly due to pauses in response time by omnios. At 9000 byte frame sizes, I see a good number of duplicate ACKs and fast retransmits during read operations (when omnios is transmitting). But below about a 4100-byte MTU on omnios (which seems to correlate to 4096-byte iSCSI block transfers), the transmission errors fade away and we only see the transmission pause problem. I'm in the process of aggregating the 10G ports and performing some IO testing with the vmware IO performance tool. That should show the performance of the 10G NIC when both physical ports are in use, and hopefully get me some more granularity on the MTU settings. If anyone has a list of kernel tuning parameters to test, I'm happy to try them out and report back. I've found a variety of suggestions online, but between Illumos, solaris, openindiana, Nexenta, opensolaris, etc, the supported variables are, um, inconsistent. -Warren V ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy
You should PLEASE share your original note with the illumos developer's list. Also, please keep in mind that it is VERY possible you're seeing crosscall effects where the interrupt-servicing CPU core is not on the same PCIe bus as where you're card is plugged in. I never got a chance to perform these tests fully when I *had* ixgbe HW handy, but I observed bizarre improvements, or the disappearance of bizarre effects, if I: - disabled the HT-inspired CPUs (psradm -f HT CPUs) - Disabled one of the two CPUs (again, using psradm). You may wish to try messing around with what OS-reported CPUs are on your Romley (Xeon E5) system. I will also note that it's high time for illumos to pull in the ixgbe updates from upstream. Intel is NOT being very helpful here, partially because of fear-of-Oracle, and partially because there aren't enough illumos customers to make a dent in their HW sales. In the past, illumos developers have found the time to yank in the newest driver updates from upstream. That hasn't happened recently. For the record, OmniTI might be able to contribute here IF AND ONLY IF a sufficiently paying customer motivates us. I suspect the same answer (sufficiently paying customer) applies to engineers from any other illumos shop as well. Dan ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy
On Feb 20, 2015, at 9:10 PM, Dan McDonald dan...@omniti.com wrote: I never got a chance to perform these tests fully when I *had* ixgbe HW handy, but I observed bizarre improvements, or the disappearance of bizarre effects, if I: - disabled the HT-inspired CPUs (psradm -f HT CPUs) - Disabled one of the two CPUs (again, using psradm). You may wish to try messing around with what OS-reported CPUs are on your Romley (Xeon E5) system. To see the layouts, the psrinfo(1M) command is your friend, especially psrinfo -vp. Dan ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy
After installation and configuration, I observed all kinds of bad behavior in the network traffic between the hosts and the server. All of this bad behavior is traced to the ixgbe driver on the storage server. Without going into the full troubleshooting process, here are my takeaways: [...] For what it's worth, we managed to achieve much better line rates on copper 10G ixgbe hardware of various descriptions between OmniOS and CentOS 7 (I don't think we ever tested OmniOS to OmniOS). I don't believe OmniOS could do TCP at full line rate but I think we managed 700+ Mbytes/sec on both transmit and receive and we got basically disk-limited speeds with iSCSI (across multiple disks on multi-disk mirrored pools, OmniOS iSCSI initiator, Linux iSCSI targets). I don't believe we did any specific kernel tuning (and in fact some of our attempts to fiddle ixgbe driver parameters blew up in our face). We did tune iSCSI connection parameters to increase various buffer sizes so that ZFS could do even large single operations in single iSCSI transactions. (More details available if people are interested.) 10: At the wire level, the speed problems are clearly due to pauses in response time by omnios. At 9000 byte frame sizes, I see a good number of duplicate ACKs and fast retransmits during read operations (when omnios is transmitting). But below about a 4100-byte MTU on omnios (which seems to correlate to 4096-byte iSCSI block transfers), the transmission errors fade away and we only see the transmission pause problem. This is what really attracted my attention. In our OmniOS setup, our specific Intel hardware had ixgbe driver issues that could cause activity stalls during once-a-second link heartbeat checks. This obviously had an effect at the TCP and iSCSI layers. My initial message to illumos-developer sparked a potentially interesting discussion: http://www.listbox.com/member/archive/182179/2014/10/sort/time_rev/page/16/entry/6:405/20141003125035:6357079A-4B1D-11E4-A39C-D534381BA44D/ If you think this is a possibility in your setup, I've put the DTrace script I used to hunt for this up on the web: http://www.cs.toronto.edu/~cks/src/omnios-ixgbe/ixgbe_delay.d This isn't the only potential source of driver stalls by any means, it's just the one I found. You may also want to look at lockstat in general, as information it reported is what led us to look specifically at the ixgbe code here. (If you suspect kernel/driver issues, lockstat combined with kernel source is a really excellent resource.) - cks ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] The ixgbe driver, Lindsay Lohan, and the Greek economy
Hello All, Thank you for your replies. I tried a few things, and found the following: 1: Disabling hyperthreading support in the BIOS drops performance overall by a factor of 4. 2: Disabling VT support also seems to have some effect, although it appears to be minor. But this has the amusing side effect of fixing the hangs I've been experiencing with fast reboot. Probably by disabling kvm. 3: The performance tests are a bit tricky to quantify because of caching effects. In fact, I'm not entirely sure what is happening here. It's just best to describe what I'm seeing: The commands I'm using to test are dd if=/dev/zero of=./test.dd bs=2M count=5000 dd of=/dev/null if=./test.dd bs=2M count=5000 The host vm is running Centos 6.6, and has the latest vmtools installed. There is a host cache on an SSD local to the host that is also in place. Disabling the host cache didn't immediately have an effect as far as I could see. The host MTU set to 3000 on all iSCSI interfaces for all tests. Test 1: Right after reboot, with an ixgbe MTU of 9000, the write test yields an average speed over three tests of 137MB/s. The read test yields an average over three tests of 5MB/s. Test 2: After setting ifconfig ixgbe0 mtu 3000, the write tests yield 140MB/s, and the read tests yield 53MB/s. It's important to note here that if I cut the read test short at only 2-3GB, I get results upwards of 350MB/s, which I assume is local cache-related distortion. Test 3: MTU of 1500. Read tests are up to 156 MB/s. Write tests yield about 142MB/s. Test 4: MTU of 1000: Read test at 182MB/s. Test 5: MTU of 900: Read test at 130 MB/s. Test 6: MTU of 1000: Read test at 160MB/s. Write tests are now consistently at about 300MB/s. Test 7: MTU of 1200: Read test at 124MB/s. Test 8: MTU of 1000: Read test at 161MB/s. Write at 261MB/s. A few final notes: L1ARC grabs about 10GB of RAM during the tests, so there's definitely some read caching going on. The write operations are easier to observe with iostat, and I'm seeing io rates that closely correlate with the network write speeds. Chris, thanks for your specific details. I'd appreciate it if you could tell me which copper NIC you tried, as well as to pass on the iSCSI tuning parameters. I've ordered an Intel EXPX9502AFXSR, which uses the 82598 chip instead of the 82599 in the X520. If I get similar results with my fiber transcievers, I'll see if I can get a hold of copper ones. But I should mention that I did indeed look at PHY/MAC error rates, and they are nil. -Warren V On Fri, Feb 20, 2015 at 7:25 PM, Chris Siebenmann c...@cs.toronto.edu wrote: After installation and configuration, I observed all kinds of bad behavior in the network traffic between the hosts and the server. All of this bad behavior is traced to the ixgbe driver on the storage server. Without going into the full troubleshooting process, here are my takeaways: [...] For what it's worth, we managed to achieve much better line rates on copper 10G ixgbe hardware of various descriptions between OmniOS and CentOS 7 (I don't think we ever tested OmniOS to OmniOS). I don't believe OmniOS could do TCP at full line rate but I think we managed 700+ Mbytes/sec on both transmit and receive and we got basically disk-limited speeds with iSCSI (across multiple disks on multi-disk mirrored pools, OmniOS iSCSI initiator, Linux iSCSI targets). I don't believe we did any specific kernel tuning (and in fact some of our attempts to fiddle ixgbe driver parameters blew up in our face). We did tune iSCSI connection parameters to increase various buffer sizes so that ZFS could do even large single operations in single iSCSI transactions. (More details available if people are interested.) 10: At the wire level, the speed problems are clearly due to pauses in response time by omnios. At 9000 byte frame sizes, I see a good number of duplicate ACKs and fast retransmits during read operations (when omnios is transmitting). But below about a 4100-byte MTU on omnios (which seems to correlate to 4096-byte iSCSI block transfers), the transmission errors fade away and we only see the transmission pause problem. This is what really attracted my attention. In our OmniOS setup, our specific Intel hardware had ixgbe driver issues that could cause activity stalls during once-a-second link heartbeat checks. This obviously had an effect at the TCP and iSCSI layers. My initial message to illumos-developer sparked a potentially interesting discussion: http://www.listbox.com/member/archive/182179/2014/10/sort/time_rev/page/16/entry/6:405/20141003125035:6357079A-4B1D-11E4-A39C-D534381BA44D/ If you think this is a possibility in your setup, I've put the DTrace script I used to hunt for this up on the web: http://www.cs.toronto.edu/~cks/src/omnios-ixgbe/ixgbe_delay.d This isn't the only potential source of driver stalls by any means, it's just the one I found. You may also want to look at lockstat in
Re: [OmniOS-discuss] OmniOS Bloody update for Feb 18
On Fri, Feb 20, 2015 at 9:27 AM, Dan McDonald dan...@omniti.com wrote: Are you two volunteering for testing? If so, I can see what I can do about pushing the package (without it being in an incorporation) into the bloody repo server -- assuming I can build it properly. Absolutely. I have several OmniOS VMs already. At least 1/2 of them need VMware tools. -Chip Dan Sent from my iPhone (typos, autocorrect, and all) On Feb 20, 2015, at 9:20 AM, Schweiss, Chip c...@innovates.com wrote: I will second that request as my OmniOS experiments always start as VMware VMs. -Chip On Fri, Feb 20, 2015 at 3:17 AM, Alexander Lesle gro...@tierarzt-mueller.de wrote: Hello Dan McDonald and List, do your remember my feature request? ,-[ ]- | | at Friday, 3. Okt. 2014 17:22 Dan McDonald | at mid:a762c059-f687-42de-a1f7-6cf2a7f58...@omniti.com wrote: | | On Oct 3, 2014, at 9:29 AM, Alexander Lesle | gro...@tierarzt-mueller.de wrote: | | Hello Dan McDonald and List, | | in my mind this was a great proposal what you done at Sep, 02. | That's the way to build a great OmniOS near to the users. | | Here my request for the next release: | It would be convenient if open-vm-tools were installed in the new | release. http://sourceforge.net/projects/open-vm-tools/files/ | | Actually, this work has been done. I dropped it on the ground: | | https://github.com/omniti-labs/omnios-build/pull/39 | | I will be merging this pull requests now in the master branch, | and seeing how OOod chews on it. | | | I know there's other work to be done for OmniOS-as-a-guest as well. | | Thanks, | DAn | `--- Do you find the time to integrate open-vm-tools in the next stable version? Thanks. On Februar, 19 2015, 03:36 Dan McDonald wrote in [1]: There will be only 1-3 more updates prior to the next OmniOS stable (and for this time, this stable is also Long-Term Support) release. If you notice anything weird about the bloody bits, please let me know ASAP. I've heard little/no complaints about the updates to pkg(5), so either you're very happy, or not using it. :) This update will only be reaching the repo. * omnios-build master branch, revision f3d6d48 * Git to 2.3.0. * UnZIP fixes. * OpenJDK7 up to update 76, build 31. * Microtasking libraries (/lib/libmtsk*) are back as a distinct package (system/library/mtsk) now that they are not part of the (now open-source) Math libraries. * illumos-omnios master branch, revision cbf73e4 (last illumos-gate merge 336069c) * Bugfixes in PF_PACKET, SMB/CIFS, header files, NFS, and man pages. -- Best Regards Alexander Februar, 20 2015 [1] mid:f70c482d-68e8-43a2-bf73-86ade4b63...@omniti.com ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] OmniOS Bloody update for Feb 18
Are you two volunteering for testing? If so, I can see what I can do about pushing the package (without it being in an incorporation) into the bloody repo server -- assuming I can build it properly. Dan Sent from my iPhone (typos, autocorrect, and all) On Feb 20, 2015, at 9:20 AM, Schweiss, Chip c...@innovates.com wrote: I will second that request as my OmniOS experiments always start as VMware VMs. -Chip On Fri, Feb 20, 2015 at 3:17 AM, Alexander Lesle gro...@tierarzt-mueller.de wrote: Hello Dan McDonald and List, do your remember my feature request? ,-[ ]- | | at Friday, 3. Okt. 2014 17:22 Dan McDonald | at mid:a762c059-f687-42de-a1f7-6cf2a7f58...@omniti.com wrote: | | On Oct 3, 2014, at 9:29 AM, Alexander Lesle | gro...@tierarzt-mueller.de wrote: | | Hello Dan McDonald and List, | | in my mind this was a great proposal what you done at Sep, 02. | That's the way to build a great OmniOS near to the users. | | Here my request for the next release: | It would be convenient if open-vm-tools were installed in the new | release. http://sourceforge.net/projects/open-vm-tools/files/ | | Actually, this work has been done. I dropped it on the ground: | | https://github.com/omniti-labs/omnios-build/pull/39 | | I will be merging this pull requests now in the master branch, | and seeing how OOod chews on it. | | | I know there's other work to be done for OmniOS-as-a-guest as well. | | Thanks, | DAn | `--- Do you find the time to integrate open-vm-tools in the next stable version? Thanks. On Februar, 19 2015, 03:36 Dan McDonald wrote in [1]: There will be only 1-3 more updates prior to the next OmniOS stable (and for this time, this stable is also Long-Term Support) release. If you notice anything weird about the bloody bits, please let me know ASAP. I've heard little/no complaints about the updates to pkg(5), so either you're very happy, or not using it. :) This update will only be reaching the repo. * omnios-build master branch, revision f3d6d48 * Git to 2.3.0. * UnZIP fixes. * OpenJDK7 up to update 76, build 31. * Microtasking libraries (/lib/libmtsk*) are back as a distinct package (system/library/mtsk) now that they are not part of the (now open-source) Math libraries. * illumos-omnios master branch, revision cbf73e4 (last illumos-gate merge 336069c) * Bugfixes in PF_PACKET, SMB/CIFS, header files, NFS, and man pages. -- Best Regards Alexander Februar, 20 2015 [1] mid:f70c482d-68e8-43a2-bf73-86ade4b63...@omniti.com ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
[OmniOS-discuss] UPCOMING: End of Support Life approaching for OmniOS r151010
Sometime during Q2CY2015 (no earlier than April 1st), OmniOS r151014 will be released. At that time, support for r151010 (previous stable) will be discontinued. This is in accordance with the OmniOS release cycle: http://omnios.omniti.com/wiki.php/ReleaseCycle If you can, please upgrade to r151012. We *should* be able to allow a jump from r151010 to r151014, as we must support a jump from r151012 to 014 AND from 006 to 014 (because r151014 will also be the next Long-Term Support release). As I mentioned the last time, beadm(1M) is your friend. Creating backup BEs in case of mistakes is easy. Thank you, Dan McDonald -- OmniOS Engineering ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] OmniOS Bloody update for Feb 18
On Feb 20, 2015, at 4:17 AM, Alexander Lesle gro...@tierarzt-mueller.de wrote: | Actually, this work has been done. I dropped it on the ground: | | https://github.com/omniti-labs/omnios-build/pull/39 | | I will be merging this pull requests now in the master branch, | and seeing how OOod chews on it. | Ahhh... turns out, this is *IN* bloody already. If you're running bloody, you can install open-vm-tools. Dan ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] UPCOMING: End of Support Life approaching for OmniOS r151010
You know what would be nice? an omnios.omniti.com/releases web page that has the release name, date and EOL [r151006 : LTS] [birth date] [end-of-life date] [r151010 : stable] [birth date] [end-of-life 2015-04-01] [r151013 : bloody] [birth date] [not supported] [r151014 : LTS] [birth we're expecting] [end-of-life date] We could make that nice an pretty and I think it would be very useful. On Fri, Feb 20, 2015 at 11:16 AM, Dan McDonald dan...@omniti.com wrote: Sometime during Q2CY2015 (no earlier than April 1st), OmniOS r151014 will be released. At that time, support for r151010 (previous stable) will be discontinued. This is in accordance with the OmniOS release cycle: http://omnios.omniti.com/wiki.php/ReleaseCycle If you can, please upgrade to r151012. We *should* be able to allow a jump from r151010 to r151014, as we must support a jump from r151012 to 014 AND from 006 to 014 (because r151014 will also be the next Long-Term Support release). As I mentioned the last time, beadm(1M) is your friend. Creating backup BEs in case of mistakes is easy. Thank you, Dan McDonald -- OmniOS Engineering ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss -- Theo Schlossnagle http://omniti.com/is/theo-schlossnagle ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] UPCOMING: End of Support Life approaching for OmniOS r151010
On Feb 20, 2015, at 2:58 PM, Tobias Oetiker t...@oetiker.ch wrote: Hi Dan, I do not have a spare machine to test this, but if the dramatic network io performance regression for kvm present in r151012 has not been fixed in r151014 then we will be stranded on r151010. And the upstreaming of VND is likely not to happen with this release. That blocks us from properly catching up with illumos-kvm-cmd. There are several commits (Starting with the March 19th one we can't bring in w/o VND) here: https://github.com/joyent/illumos-kvm-cmd/commits/master Currently, we only backport the most recent fix about QEMU using preadv/pwritev recklessly. IF (big if) there are other commits we can backport, you could try it using the patches/ directory of omnios-build. I'm thinking maybe HVM-807's may help you (provided that's indepdent of VND support). Have you seen anything with dtrace or lockstat that could give a clue to what's holding things up? I don't recall seeing anything observed. Judging from the echo on the ML, not many people seem to see this problem, or care about it. I'm sorry, but that does appear to be the case. Next week I disappear for other-customer work, but starting in March, I'm doing nothing but r151014. Dan ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] UPCOMING: End of Support Life approaching for OmniOS r151010
Hi Dan, Today Dan McDonald wrote: Sometime during Q2CY2015 (no earlier than April 1st), OmniOS r151014 will be released. At that time, support for r151010 (previous stable) will be discontinued. This is in accordance with the OmniOS release cycle: http://omnios.omniti.com/wiki.php/ReleaseCycle If you can, please upgrade to r151012. We *should* be able to allow a jump from r151010 to r151014, as we must support a jump from r151012 to 014 AND from 006 to 014 (because r151014 will also be the next Long-Term Support release). As I mentioned the last time, beadm(1M) is your friend. Creating backup BEs in case of mistakes is easy. I do not have a spare machine to test this, but if the dramatic network io performance regression for kvm present in r151012 has not been fixed in r151014 then we will be stranded on r151010. Judging from the echo on the ML, not many people seem to see this problem, or care about it. cheers tobi -- Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland www.oetiker.ch t...@oetiker.ch +41 62 775 9902 ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
