Re: [OmniOS-discuss] ZFS crash/reboot loop
On 7/13/15 12:02 PM, Dan McDonald wrote: > >> On Jul 13, 2015, at 11:56 AM, Derek Yarnell wrote: >> >> I don't need to hot patch (cold patch would be fine) so any update that >> I can apply and reboot would be fine. We have a second OmniOS r14 copy >> running that we are happy to patch in any way possible to get it mounted rw. > > IF (and only if) it's the bug I mentioned that's the problem. > > I want ZFS experts to take a look as well. It's on the ZFS list now, so > we'll see what happens. If you're REALLY feeling brave, I can build a > replacement ZFS module with 6033 in place for you to try, but I can't promise > it'll work. Hi Dan, I would be happy to try to test a build with 6033 on it to see if that is my issue. We have secured all the critical data and only have scratch data left. So at this point I would happy to take a chance to see if this will fix the issue. Thanks, derek -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] ZFS crash/reboot loop
>> ff0d4071ca98::print arc_buf_t b_hdr |::print arc_buf_hdr_t b_size > b_size = 0 >> > > Ouch. There's your zero. > > I'm going to forward this very note to the illumos ZFS list. I see ONE > possible bugfix post-r151014 that might help: > > commit 31c46cf23cd1cf4d66390a983dc5072d7d299ba2 > Author: Alek Pinchuk > Date: Tue Jun 30 09:44:11 2015 -0700 > > 6033 arc_adjust() should search MFU lists for oldest buffer when > adjusting MFU size > Reviewed by: Saso Kiselkov > Reviewed by: Xin Li > Reviewed by: Prakash Surya > Approved by: Matthew Ahrens > > It's a small bug, and I shudder to say this, even hot-patchable on a running > system if you're desperate. :) > Hi Dan, I don't need to hot patch (cold patch would be fine) so any update that I can apply and reboot would be fine. We have a second OmniOS r14 copy running that we are happy to patch in any way possible to get it mounted rw. Thanks, derek -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] ZFS crash/reboot loop
Hi Dan, Sorry I have not dealt with dumpadm/savecore that much but it looks like this is what you want. https://obj.umiacs.umd.edu/derek_support/vmdump.0 Thanks, derek On 7/13/15 12:55 AM, Dan McDonald wrote: > >> On Jul 12, 2015, at 9:18 PM, Richard Elling >> wrote: >> >> Dan, if you're listening, Matt would be the best person to weigh-in on this. > > Yes he would be, Richard.. > > The panic in the arc_get_data_buf() paths is similar to older problems we'd > seen in r151006. > > Derek, do you have a kernel coredump from these? I know you've been > panic-and-reboot-and-panic-ing, but if you can get savecore(1M) to do its > thing, having that dump would be useful. > > Thanks, > Dan > > -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] ZFS crash/reboot loop
On 7/12/15 3:21 PM, Günther Alka wrote: > First action: > If you can mount the pool read-only, update your backup We are securing all the non-scratch data currently before messing with the pool any more. We had backups as recent as the night before but it is still going to be faster to pull the current data from the readonly pool than from backups. > Then > I would expect that a single bad disk is the reason of the problem on a > write command. I would first check the system and fault log or > smartvalues for hints about a bad disk. If there is a suspicious disk, > remove that and retry a regular import. We have pulled all disks individually yesterday to test this exact theory. We have hit the mpt_sas disk failure panics before so we had already tried this. > If there is no hint > Next what I would try is a pool export. Then create a script that > imports the pool followed by a scrub cancel. (Hope that the cancel is > faster than the crash). Then check logs during some pool activity. If I have not imported the pool RW can I export the pool? I thought we have tried this but I will have to confer. > If this does not help, I would remove all data disks and bootup. > Then hot-plug disk by disk and check if its detected properly and check > logs. Your pool remains offline until enough disks come back. > Adding disk by disk and checking logs should help to find a bad disk > that initiates a crash This is interesting and we will try this once we secure the data. > Next option is, try a pool import where always one or next disk is > missing. Until there is no write, missing disks are not a problem with > ZFS (you may need to clear errors). Wouldn't this be the same as above hot-plugging disk by disk? > Last option: > use another server where you try to import (mainboard, power, hba or > backplane problem) remove all disks and do a nondestructive or smart > test on another machine Sadly we do not have a spare chassis with 40 slots around to test this. I am so far unconvinced that this is a hardware problem though. We will most likely boot up into linux live CD to run smartctl and see if it has any information on the disks. -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] ZFS crash/reboot loop
> The on-going scrub automatically restarts, apparently even in read-only > mode. You should 'zpool scrub -s poolname' ASAP after boot (if you can) > to stop the ongoing scrub. We have tried to stop the scrub but it seems you can not cancel a scrub when the pool is mounted readonly. -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
[OmniOS-discuss] ZFS crash/reboot loop
Hi, We just have had a catastrophic event on one of our OmniOS r14 file servers. In what seems to have been triggered by the weekly scrub of its one large zfs pool (~100T) it panics. This made it basically reboot continually and we have installed a second copy of OmniOS r14 in the mean time. We are able to mount the pool readonly and are currently securing the data as soon as possible. The underlying zpool (zvol00) was a Nexenta that was imported on June 18th and had gotten through more than one weekly scrub since we had upgraded. We had not upgraded the pool from 28 yet and we are not using dedup. I have the the vmdump[0] file available if that is useful from when we tried running `zpool import zvol00`. The hardware should be fully compatible Supermicro X8DTH-i/6/iF/6F, Hitachi/HGST SAS drives, STEC ZeusRAM slogs, STEC l2arc drives with LSI 2008 controllers. [0] - https://obj.umiacs.umd.edu/derek_support/vmdump.0 TIME UUID SUNW-MSG-ID Jul 11 2015 21:23:40.015738000 f120cfb4-7e83-c576-ca89-e522daf796e7 SUNOS-8000-KL TIME CLASS ENA Jul 11 21:23:39.9681 ireport.os.sunos.panic.dump_available 0x Jul 11 21:23:24.1537 ireport.os.sunos.panic.dump_pending_on_device 0x nvlist version: 0 version = 0x0 class = list.suspect uuid = f120cfb4-7e83-c576-ca89-e522daf796e7 code = SUNOS-8000-KL diag-time = 1436664219 981033 de = fmd:///module/software-diagnosis fault-list-sz = 0x1 fault-list = (array of embedded nvlists) (start fault-list[0]) nvlist version: 0 version = 0x0 class = defect.sunos.kernel.panic certainty = 0x64 asru = sw:///:path=/var/crash/unknown/.f120cfb4-7e83-c576-ca89-e522daf796e7 resource = sw:///:path=/var/crash/unknown/.f120cfb4-7e83-c576-ca89-e522daf796e7 savecore-succcess = 1 dump-dir = /var/crash/unknown dump-files = vmdump.0 os-instance-uuid = f120cfb4-7e83-c576-ca89-e522daf796e7 panicstr = assertion failed: c < (1ULL << 24) >> 9 (0x7f < 0x8000), file: ../../common/fs/zfs/zio.c, line: 221 panicstack = fba8b13d () | zfs:zio_buf_alloc+49 () | zfs:arc_get_data_buf+12b () | zfs:arc_buf_alloc+d2 () | zfs:arc_read+15b () | zfs:dsl_scan_prefetch+da () | zfs:dsl_scan_recurse+13c () | zfs:dsl_scan_visitbp+fd () | zfs:dsl_scan_visitdnode+b4 () | zfs:dsl_scan_recurse+435 () | zfs:dsl_scan_visitbp+fd () | zfs:dsl_scan_recurse+1b9 () | zfs:dsl_scan_visitbp+fd () | zfs:dsl_scan_recurse+1b9 () | zfs:dsl_scan_visitbp+fd () | zfs:dsl_scan_visitdnode+b4 () | zfs:dsl_scan_recurse+496 () | zfs:dsl_scan_visitbp+fd () | zfs:dsl_scan_visit_rootbp+5d () | zfs:dsl_scan_visit+273 () | zfs:dsl_scan_sync+247 () | zfs:spa_sync+2b3 () | zfs:txg_sync_thread+227 () | unix:thread_start+8 () | crashtime = 1436664019 panic-time = Sat Jul 11 21:20:19 2015 EDT (end fault-list[0]) fault-status = 0x1 severity = Major __ttl = 0x1 __tod = 0x55a1c19c 0xf02490 ### In readonly mode root@cbcbomni02:/root# zdb -e -bcsvL zvol00 assertion failed for thread 0xfd7fff162a40, thread-id 1: !claimed || !(zh->zh_flags & ZIL_CLAIM_LR_SEQ_VALID) || (max_blk_seq == claim_blk_seq && max_lr_seq == claim_lr_seq), file ../../../uts/common/fs/zfs/zil.c, line 367 Abort (core dumped) ### After mounting in readonly mode pool: zvol00 state: ONLINE status: The pool is formatted using a legacy on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on software that does not support feature flags. scan: scrub in progress since Sat Jul 11 11:00:02 2015 2.24G scanned out of 69.5T at 1/s, (scan is slow, no estimated time) 0 repaired, 0.00% done config: NAME STATE READ WRITE CKSUM zvol00 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 c4t5000CCA01A91D6F9d0 ONLINE 0 0 0 c4t5000CCA01A9D4B6Dd0 ONLINE 0 0 0 c4t5000CCA01A9ED1A9d0 ONLINE 0 0 0 c4t5000CCA01A9ED311d0 ONLINE 0 0 0 c4t5000CCA01A9ED345d0 ONLINE 0 0 0 c4t5000CCA01A9EDD21d0 ONLINE 0 0 0 raidz2-1 ONLINE 0 0 0 c4t5000CCA01A9EE1D1d0 ONLINE 0 0 0 c4t5000CCA01A9EE3D9d0 ONLINE 0 0 0 c4t5000CCA01A9F6659d0 ONLINE 0 0 0 c4t5000CCA01A9F69C5d0 ONLINE 0 0 0 c4t5000CCA01A9F81A9d0
Re: [OmniOS-discuss] NFSv4 client soft lockups
On 9/12/14 11:05 AM, Schweiss, Chip wrote: > Is you RHEL 6.5 client a virtual machine? If so this message is a red > herring to your problem. > > See the VMware KB article: > http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1009996 > > I run all NFSv4 from CentOS 6.5 to OmniOS, but do not use kerberos. It's > been rock solid. Hi Chip, No we are running on physical hardware. It also looks like the parameter they talk about is something different[1] post RHEL 6.1. We are trying to drop to just sec=krb5 to see if that provides any relief and may increase the watchdog timer a bit to see if this is just a race condition. [1] - https://access.redhat.com/solutions/57811? Thanks, derek -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
[OmniOS-discuss] NFSv4 client soft lockups
Hi, I am wondering if anyone else has experience running NFSv4 from OmniOS to RHEL (6.5) clients. We are running it with sec=krb5p and are getting these errors (soft lockups) at the end of this mail across all the nodes causing them to be unresponsive after. The OmniOS server doesn't report anything wrong in the logs. I have submitted a ticket to Red Hat about this already but since it happens on all my RHEL clients at once and references the same OmniOS server (172.19.0.98) then it may have something to do either Omni NFS server or the interoperability of the RHEL NFSv4 client and the Omni NFSv4 server. Thanks, derek Sep 11 23:00:54 nauseated kernel: BUG: soft lockup - CPU#5 stuck for 67s! [172.19.0.98-man:12273] Sep 11 23:00:54 nauseated kernel: Modules linked in: aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 aes_generic cbc cts nfs lockd fscache nfs_acl autofs4 rpcsec_gss_krb5 auth_rpcgss sunrpc ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 power_meter microcode iTCO_wdt iTCO_vendor_support dcdbas serio_raw lpc_ich mfd_core sg i7core_edac edac_core bnx2 ext4 jbd2 mbcache sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix mptsas mptscsih mptbase scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib] Sep 11 23:00:54 nauseated kernel: CPU 5 Sep 11 23:00:54 nauseated kernel: Modules linked in: aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 aes_generic cbc cts nfs lockd fscache nfs_acl autofs4 rpcsec_gss_krb5 auth_rpcgss sunrpc ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 power_meter microcode iTCO_wdt iTCO_vendor_support dcdbas serio_raw lpc_ich mfd_core sg i7core_edac edac_core bnx2 ext4 jbd2 mbcache sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix mptsas mptscsih mptbase scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib] Sep 11 23:00:54 nauseated kernel: Sep 11 23:00:54 nauseated kernel: Pid: 12273, comm: 172.19.0.98-man Not tainted 2.6.32-431.29.2.el6.x86_64 #1 Dell Inc. PowerEdge R610/0F0XJ6 Sep 11 23:00:54 nauseated kernel: RIP: 0010:[] [] nfs4_run_state_manager+0x38d/0x620 [nfs] Sep 11 23:00:54 nauseated kernel: RSP: 0018:88042b249e90 EFLAGS: 0206 Sep 11 23:00:54 nauseated kernel: RAX: 88042b249fd8 RBX: 88042b249ee0 RCX: 0034 Sep 11 23:00:54 nauseated kernel: RDX: 115b RSI: 880429e7ed90 RDI: a0247568 Sep 11 23:00:54 nauseated kernel: RBP: 8100bb8e R08: f000 R09: fbb8191d81fb3e00 Sep 11 23:00:54 nauseated kernel: R10: 0001 R11: R12: 88042b249ee0 Sep 11 23:00:54 nauseated kernel: R13: 8100bb8e R14: ff10 R15: a0247568 Sep 11 23:00:54 nauseated kernel: FS: () GS:88002824() knlGS: Sep 11 23:00:54 nauseated kernel: CS: 0010 DS: 0018 ES: 0018 CR0: 8005003b Sep 11 23:00:54 nauseated kernel: CR2: 006e19b8 CR3: 00042b026000 CR4: 07e0 Sep 11 23:00:54 nauseated kernel: DR0: DR1: DR2: Sep 11 23:00:54 nauseated kernel: DR3: DR6: 0ff0 DR7: 0400 Sep 11 23:00:54 nauseated kernel: Process 172.19.0.98-man (pid: 12273, threadinfo 88042b248000, task 8804299b9540) Sep 11 23:00:54 nauseated kernel: Stack: Sep 11 23:00:54 nauseated kernel: 0286 03fb 880429e7ecf9 880429e7ed00 Sep 11 23:00:54 nauseated kernel: 88042b249ee0 88042a5e1898 88042b249ef8 a02f5300 Sep 11 23:00:54 nauseated kernel: 880429e7ec00 88042cc89500 88042b249f40 8109abf6 Sep 11 23:00:54 nauseated kernel: Call Trace: Sep 11 23:00:54 nauseated kernel: [] ? nfs4_run_state_manager+0x0/0x620 [nfs] Sep 11 23:00:54 nauseated kernel: [] ? kthread+0x96/0xa0 Sep 11 23:00:54 nauseated kernel: [] ? child_rip+0xa/0x20 Sep 11 23:00:54 nauseated kernel: [] ? kthread+0x0/0xa0 Sep 11 23:00:54 nauseated kernel: [] ? child_rip+0x0/0x20 Sep 11 23:00:54 nauseated kernel: Code: 89 df e8 f7 8b 00 00 e9 e3 fc ff ff 66 90 48 89 df e8 d8 e4 ff ff 48 83 bb f8 00 00 00 00 0f 84 f7 fc ff ff f0 41 0f ba 2c 24 00 <19> c0 85 c0 0f 84 df fc ff ff e9 e1 fc ff ff 0f 1f 40 00 41 81 Sep 11 23:00:54 nauseated kernel: Call Trace: Sep 11 23:00:54 nauseated kernel: [] ? nfs4_run_state_manager+0x0/0x620 [nfs] Sep 11 23:00:54 nauseated kernel: [] ? kthread+0x96/0xa0 Sep 11 23:00:54 nauseated kernel: [] ? child_rip+0xa/0x20 Sep 11 23:00:54 nauseated kernel: [] ? kthread+0x0/0xa0 Sep 11 23:00:54 nauseated kernel: [] ? child_rip+0x0/0x20 -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies _
Re: [OmniOS-discuss] flock from RHEL7 client to OmniOS
On 6/28/14, 10:29 PM, Paul B. Henson wrote:
> On Sat, Jun 28, 2014 at 10:40:06AM -0400, Derek Yarnell wrote:
>> Is anyone else seeing a problem with flock over NFS on more recent
>> clients including RHEL7 (I am guessing recent fedora releases too)?
>
> Hmm, historically flock was not NFS compatible, you were supposed to use
> lockf or fcntl. I don't have an RHEL7 box at the moment, but the man
> page for flock(2) on an RHEL6 box says:
>
>flock() does not lock files over NFS. Use fcntl(2) instead: that does
>work over NFS, given a sufficiently recent version of Linux and a
>server which supports locking.
>
Well my problem is that createrepo uses it for whatever reason
open("/fs/UMyumrepos/rhel7/stable/repodata/locktest",
O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
flock(3, LOCK_EX) = -1 ENOLCK (No locks available)
write(2, "Could not create exclusive lock "..., 155Could not create
exclusive lock in /fs/UMyumrepos/rhel7/stable/repodata and sqlite
database generation enabled. Is this path on nfs? Is your lockd
running?) = 155
write(2, "\n", 1
) = 1
close(3)= 0
The code is really calling flock it seems. I am guessing that there is
some good reasons this is working on some NFS implementations and not
others, interesting that it works on Nexenta 3.x. I guess I will start
up a support ticket for this with Red Hat.
/usr/lib/python2.7/site-packages/createrepo/__init__.py:
if self.conf.database:
# do flock test on temp_final, temp_output
# if it fails raise MDError
for direc in [temp_final, temp_output]:
f = open(direc + '/locktest', 'w')
try:
fcntl.flock(f.fileno(), fcntl.LOCK_EX)
except (OSError, IOError), e:
raise MDError, _("Could not create exclusive lock in
%s and sqlite database generation enabled. Is this path on nfs? Is your
lockd running?") % direc
else:
os.unlink(direc + '/locktest')
--
Derek T. Yarnell
University of Maryland
Institute for Advanced Computer Studies
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss
[OmniOS-discuss] flock from RHEL7 client to OmniOS
Is anyone else seeing a problem with flock over NFS on more recent clients including RHEL7 (I am guessing recent fedora releases too)? fcntl works fine but flock has an error. Our Nexenta 3.x and NetApp servers work fine it seems but we also see the same issue with our Dell FluidFS (though we are at least a full revision behind on it). I have tested r151008 and r151010 both exhibit the same behavior for OmniOS. [root@walrus UMyumrepos]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.0 (Maipo) [root@walrus UMyumrepos]# df . Filesystem 1K-blocks Used Available Use% Mounted on umomni00:/volumes/zvol00/staff/UMyumrepos 8279117824 53210112 8225907712 1% /fs/UMyumrepos [root@walrus UMyumrepos]# flock ./test /bin/true flock: ./test: No locks available [root@rock UMyumrepos]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 6.5 (Santiago) [root@rock UMyumrepos]# df . Filesystem1K-blocks Used Available Use% Mounted on umomni00:/volumes/zvol00/staff/UMyumrepos 8279117824 53210112 8225907712 1% /fs/UMyumrepos [root@rock UMyumrepos]# flock ./test /bin/true [root@rock UMyumrepos]# echo $? 0 -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] where to look for VirtIO drivers? Keen to test OmniOS on OpenStack KVM
On 5/19/14, 10:06 PM, Dan McDonald wrote: > > On May 19, 2014, at 9:59 PM, Piers Dawson-Damer wrote: > >> Is anyone aware of Illumos/OmniOS VirtIO disk & nic drivers? >> I’m keen to test OmniOS on OpenStack KVM > > There are none in OmniOS or upstream in illumos-gate, but I believe Nexenta's > distro has them, and if someone with time & patience could upstream them to > illumos, we'd all benefit. Start by looking here: > > > https://github.com/Nexenta/illumos-nexenta/tree/master/usr/src/uts/common/io/ > Hi, This I don't think is totally true, VirtIO NIC driver support isn't in Illumos-gate yet but disk drivers have been in for a bit. https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/io/vioblk/vioblk.c I have been running r151005 which had them included for awhile now with no issues to report. Though they seem to not have been included in any recent stable OmniOS release (up to r151008, haven't upgraded to 10) so I don't know why that is. I haven't needed the network performance as much as the realtek support seems good enough for this small scale ZFS NFS fileserver we run using vioblk drivers. Thanks, derek -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Pliant/Sandisk SSD ZIL
On 2/18/14, 10:54 PM, Marion Hakanson wrote: >> I actually will test some spare DC S3700 drives as the slog >> devices that we have for a Ceph cluster in this in the next few days and >> will report back on this thread. > > Cool, I look forward to seeing what you find out. For a 44MB/1981 file tar file. With DC S3700 drive in slog (ATA-INTEL SSDSC2BA10-0265-93.16GB) # time tar -xf littletarfile.tar real0m8.337s user0m0.056s sys 0m0.659s With Pliant LB206S drive in slog (Pliant-LB206S-D323-186.31GB) # time tar -xf littletarfile.tar real8m43.268s user0m0.109s sys 0m1.493s With no slog # time tar -xf littletarfile.tar real9m17.243s user0m0.127s sys 0m1.409s -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Pliant/Sandisk SSD ZIL
On 2/18/14, 9:41 PM, Marion Hakanson wrote: > No better than with the ZIL running on the pool's HDD's? Really? Yes it would seem I was getting just as bad svc_t from these Pliant SSDs then from the pools spinners. > Out of curiosity, what HBA is being used for these drives (and slog) > in this R720xd, H710 or H310? Something else? This is a R720xd shipped with a pristine LSI card (mine even came with IT firmware). You will need to talk to your rep but they can do it. My hope here was to get a fully supported OmniOS box from Dell which the biggest problem was before making me take a H710/H800 card. While we have been successful running ZFS with a R510 w/ H700s drives all in R0 and setting zfs_nocacheflush it just isn't right. We have also never gone more than 9 data drives with the R0 configuration (also this has been mostly on Nexenta not Omni). > > I'm also looking at the R720xd here, though I was thinking of using the > Intel DC S3700 SSD for a log device (in one of the internal flex-bay's), > with the pool drives being of the 2.5" form-factor. The R720xd flex bays are no longer internal but in the back of the machine[1]. I actually will test some spare DC S3700 drives as the slog devices that we have for a Ceph cluster in this in the next few days and will report back on this thread. One also thing to note is that Dell "Enterprise" drives have this problem of not spinning up[2] which made for a very fun to get installed. I had to install on a non-dell disk that was equal or smaller than my OS disks then modify the /kernel/drv/sd.conf file, then mirror the rpool reboot and then detach the non-dell drive and add the other drive back to the mirror. [1] - http://www.anchor.com.au/blog/2012/10/shiny-new-hardware-dell-r720xd/ [2] - https://www.illumos.org/issues/2091 Thanks, derek -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
Re: [OmniOS-discuss] Pliant/Sandisk SSD ZIL
On 2/17/14, 7:31 PM, Richard Elling wrote: > On Feb 17, 2014, at 2:48 PM, Derek Yarnell wrote: > >> Hi, >> >> So we bought a new Dell R720xd with 2 Dell SLC SSDs which were shipped >> as a Pliant-LB206S-D323-186.31GB via format. > > Pliant SSDs (note: Pliant was purchased by Sandisk in 2011) are optimized for > lots of concurrent I/O operations. This is not the kind of workload generated > by > the ZIL, which is more contiguous, single thread-like. I realize that they may not be as good as Stec/HGST ZeusRAM drives for the slog. I still can't wrap my head around that it is as bad as having no discrete slog. > > Never ever use zpool iostat to measure application performance. zpool iostat > measures workload to the vdevs, showing back-end operations to disk. As such > there is no correlation to client-side operations of any sort, especially > writes and > metadata updates. You'll need to go up the stack and see what it is doing. > For NFS > I highly recommend nfssvrtop. To see the response time of the Pliant, use > "iostat -x" > or, as I prefer, "iostat -zxCn" Yes I realize that iostat will show me this information and the svc_t for the Pliant ssd(s) is anywhere from 2-7ms. But zpool iostat will show you your ZIL writes accurately no? I realize that it will then coalesce these into its transactions and write it out at the 5sec interval. > > Note: response time (measured by iostat -x as a variant of "svc_t") is the > critical > measurement for NFS workloads. Bandwidth is almost meaningless in analyzing > NFS. Yes and I have done this too. The average RTT on untaring is 43.000 ms. I guess we will just be getting another set of ZeusRAM drives. Thanks, derek -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
[OmniOS-discuss] Pliant/Sandisk SSD ZIL
Hi, So we bought a new Dell R720xd with 2 Dell SLC SSDs which were shipped as a Pliant-LB206S-D323-186.31GB via format. The bulk of the storage is just a 6+2 RaidZ2 with 3TB disks. When doing a simple untar (40MB/2000 files) over NFS (synchronous IO) having the mirrored slog of Pliant SSDs seems to offer horrendous performance. On the order of 18-20 OPs per second (via zpool iostat -v 1). We have a number of other ZFS builds with ZeusRAM slogs and they work very well and while the Pliant may not be as good a choice it shouldn't be as bad as this. # zfs get sync,dedup,logbias,compression,checksum zpool0 NAMEPROPERTY VALUE SOURCE zpool0 sync standard default zpool0 dedupoffdefault zpool0 logbias latencydefault zpool0 compression offdefault zpool0 checksum on default Adding a ramdisk slog device makes everything perform as expected. Setting sync=disabled of course makes things all better. # zpool status zpool0 pool: zpool0 state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zpool0 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 c1t5000C50057EC4B73d0 ONLINE 0 0 0 c1t5000C50057EC6EABd0 ONLINE 0 0 0 c1t5000C50057EC7CB7d0 ONLINE 0 0 0 c1t5000C50057EC8F23d0 ONLINE 0 0 0 c1t5000C50057EC998Bd0 ONLINE 0 0 0 c1t5000C50057ECB1BBd0 ONLINE 0 0 0 c1t5000C50057ECFEE7d0 ONLINE 0 0 0 c1t5000C50057ED03F7d0 ONLINE 0 0 0 logs mirror-1 ONLINE 0 0 0 c3t5001E820026DA2CAd0 ONLINE 0 0 0 c3t5001E820026DA45Ad0 ONLINE 0 0 0 errors: No known data errors -- Derek T. Yarnell University of Maryland Institute for Advanced Computer Studies ___ OmniOS-discuss mailing list OmniOS-discuss@lists.omniti.com http://lists.omniti.com/mailman/listinfo/omnios-discuss
