On Tue, 26 Mar 2013, Sašo Kiselkov wrote:
Well, seeing as his throughput to a 2x6-drive raidz2 seems totally expected, we are seeing a significant inconsistency here: 1) local ZFS throughput: 750 MB/s (OK) 2) COMSTAR over FC to ZFS-backed LUN: 200 MB/s (low?) 3) tmpfs throughput: <low> Please note that tmpfs and COMSTAR have *nothing* in common here (besides running on the same machine). Before going on to analyze performance pathologies, let's make at least sure the effect we're seeing here is real. To the simple question of "what kind of FC?" I've yet to hear an answer. As I said before, 200MB/s in writes on 4GbFC is something I've seen in reality before - in fact, I can reproduce it right now, at will:
I have no doubt. Writes using COMSTAR over FC to ZFS-backed LUN are likely be turned into synchronous writes whereas local writes are not. This becomes an apples/oranges comparison.
Even with 4Gbit FC, the channel would not be saturated if the writes are synchronous writes. The system hosting COMSTAR would need to ack the synchronous writes almost instantaneously and this is unlikely without very specialized zil hardware (or zil disabled).
It's entirely possible tmpfs has problems on 4-socket, 64-core Xeon E5 systems, but resolving them need not help FC/COMSTAR throughput. First we need to establish that what we're looking at here has a common source.
Yes. I would still be worried about unusually slow tmpfs performance. Semaphores are required to keep tmpfs consistent or else only one core could be allowed to be a writer. An issue with semaphores (or scheduling) would impact many facets of system performance.
Use of Richard Elling's zilstat.ksh (http://www.richardelling.com/Home/scripts-and-programs-1/zilstat) would be very useful in order to see what synchronous writes are occuring.
Bob -- Bob Friesenhahn [email protected], http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ ------------------------------------------- illumos-discuss Archives: https://www.listbox.com/member/archive/182180/=now RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175430-2e6923be Modify Your Subscription: https://www.listbox.com/member/?member_id=21175430&id_secret=21175430-6a77cda4 Powered by Listbox: http://www.listbox.com
