[OmniOS-discuss] ZFS crash/reboot loop

[Derek Yarnell]

Hi,

We just have had a catastrophic event on one of our OmniOS r14 file
servers.  In what seems to have been triggered by the weekly scrub of
its one large zfs pool (~100T) it panics.  This made it basically reboot
continually and we have installed a second copy of OmniOS r14 in the
mean time.  We are able to mount the pool readonly and are currently
securing the data as soon as possible.

The underlying zpool (zvol00) was a Nexenta that was imported on June
18th and had gotten through more than one weekly scrub since we had
upgraded.  We had not upgraded the pool from 28 yet and we are not using
dedup.  I have the the vmdump[0] file available if that is useful from
when we tried running `zpool import zvol00`.

The hardware should be fully compatible Supermicro X8DTH-i/6/iF/6F,
Hitachi/HGST SAS drives, STEC ZeusRAM slogs, STEC l2arc drives with LSI
2008 controllers.

[0] - https://obj.umiacs.umd.edu/derek_support/vmdump.0


TIME   UUID
SUNW-MSG-ID
Jul 11 2015 21:23:40.015738000 f120cfb4-7e83-c576-ca89-e522daf796e7
SUNOS-8000-KL

  TIME CLASS ENA
  Jul 11 21:23:39.9681 ireport.os.sunos.panic.dump_available
0x
  Jul 11 21:23:24.1537 ireport.os.sunos.panic.dump_pending_on_device
0x

nvlist version: 0
version = 0x0
class = list.suspect
uuid = f120cfb4-7e83-c576-ca89-e522daf796e7
code = SUNOS-8000-KL
diag-time = 1436664219 981033
de = fmd:///module/software-diagnosis
fault-list-sz = 0x1
fault-list = (array of embedded nvlists)
(start fault-list[0])
nvlist version: 0
version = 0x0
class = defect.sunos.kernel.panic
certainty = 0x64
asru =
sw:///:path=/var/crash/unknown/.f120cfb4-7e83-c576-ca89-e522daf796e7
resource =
sw:///:path=/var/crash/unknown/.f120cfb4-7e83-c576-ca89-e522daf796e7
savecore-succcess = 1
dump-dir = /var/crash/unknown
dump-files = vmdump.0
os-instance-uuid = f120cfb4-7e83-c576-ca89-e522daf796e7
panicstr = assertion failed: c < (1ULL << 24) >> 9
(0x7f < 0x8000), file: ../../common/fs/zfs/zio.c, line: 221
panicstack = fba8b13d () | zfs:zio_buf_alloc+49
() | zfs:arc_get_data_buf+12b () | zfs:arc_buf_alloc+d2 () |
zfs:arc_read+15b () | zfs:dsl_scan_prefetch+da () |
zfs:dsl_scan_recurse+13c () | zfs:dsl_scan_visitbp+fd () |
zfs:dsl_scan_visitdnode+b4 () | zfs:dsl_scan_recurse+435 () |
zfs:dsl_scan_visitbp+fd () | zfs:dsl_scan_recurse+1b9 () |
zfs:dsl_scan_visitbp+fd () | zfs:dsl_scan_recurse+1b9 () |
zfs:dsl_scan_visitbp+fd () | zfs:dsl_scan_visitdnode+b4 () |
zfs:dsl_scan_recurse+496 () | zfs:dsl_scan_visitbp+fd () |
zfs:dsl_scan_visit_rootbp+5d () | zfs:dsl_scan_visit+273 () |
zfs:dsl_scan_sync+247 () | zfs:spa_sync+2b3 () | zfs:txg_sync_thread+227
() | unix:thread_start+8 () |
crashtime = 1436664019
panic-time = Sat Jul 11 21:20:19 2015 EDT
(end fault-list[0])

fault-status = 0x1
severity = Major
__ttl = 0x1
__tod = 0x55a1c19c 0xf02490

### In readonly mode
root@cbcbomni02:/root# zdb -e -bcsvL zvol00
assertion failed for thread 0xfd7fff162a40, thread-id 1: !claimed ||
!(zh->zh_flags & ZIL_CLAIM_LR_SEQ_VALID) || (max_blk_seq ==
claim_blk_seq && max_lr_seq == claim_lr_seq), file
../../../uts/common/fs/zfs/zil.c, line 367
Abort (core dumped)

### After mounting in readonly mode
  pool: zvol00
 state: ONLINE
status: The pool is formatted using a legacy on-disk format.  The pool can
still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
pool will no longer be accessible on software that does not
support feature
flags.
  scan: scrub in progress since Sat Jul 11 11:00:02 2015
2.24G scanned out of 69.5T at 1/s, (scan is slow, no estimated time)
0 repaired, 0.00% done
config:

NAME   STATE READ WRITE CKSUM
zvol00 ONLINE   0 0 0
  raidz2-0 ONLINE   0 0 0
c4t5000CCA01A91D6F9d0  ONLINE   0 0 0
c4t5000CCA01A9D4B6Dd0  ONLINE   0 0 0
c4t5000CCA01A9ED1A9d0  ONLINE   0 0 0
c4t5000CCA01A9ED311d0  ONLINE   0 0 0
c4t5000CCA01A9ED345d0  ONLINE   0 0 0
c4t5000CCA01A9EDD21d0  ONLINE   0 0 0
  raidz2-1 ONLINE   0 0 0
c4t5000CCA01A9EE1D1d0  ONLINE   0 0 0
c4t5000CCA01A9EE3D9d0  ONLINE   0 0 0
c4t5000CCA01A9F6659d0  ONLINE   0 0 0
c4t5000CCA01A9F69C5d0  ONLINE   0 0 0
c4t5000CCA01A9F81A9d0

Re: [OmniOS-discuss] big zfs storage?

[Bryan Horstmann-Allen]

FWIW I have not had panics on disk failure. The pool will hang, and often not 
recover until a reboot -- depending on the disk failure mode. 

I blame sd for this rather than ZFS or FMA, but at the end of the day it is 
still suboptimal behavior.
-- 
bdha

> On Jul 11, 2015, at 16:11, Linda Kateley  wrote:
> 
> This really saddens me. To me my favorite part of solaris was FMA. But happy 
> about the 3008.
> 
> thanks 
> 
> linda
> 
>> On 7/10/15 12:31 PM, Schweiss, Chip wrote:
>> Unfortunately for the past couple years panics on disk failure has been the 
>> norm.   All my production systems are HA with RSF-1, so at least things come 
>> back online relatively quick.  There are quite a few open tickets in the 
>> Illumos bug tracker related to mpt_sas related panics.   
>> 
>> Most of the work to fix these problems has been committed in the past year, 
>> though problems still exist.  For example, my systems are dual path SAS, 
>> however, mpt_sas will panic if you pull a cable instead of dropping a path 
>> to the disks.  Dan McDonald is actively working to resolve this.   He is 
>> also pushing a bug fix in genunix from Nexenta that appears to fix a lot of 
>> the panic problems.   I'll know for sure in a few months after I see a disk 
>> or two drop if it truly fixes things.  Hans Rosenfeld at Nexenta is 
>> responsible for most of the updates to mpt_sas including support for 3008 
>> (12G SAS).
>> 
>> I haven't run any 12G SAS yet, but plan to on my next build in a couple 
>> months.   This will be about 300TB using an 84 disk JBOD.  All the code from 
>> Nexenta to support the 3008 appears to be in Illumos now, and they fully 
>> support it so I suspect it's pretty stable now.  From what I understand 
>> there may be some 12G performance fixes coming sometime.   
>> 
>> The fault manager is nice when the system doesn't panic.  When it panics, 
>> the fault manger never gets a chance to take action.  It is still the 
>> consensus that is is better to run pools without hot spares because there 
>> are situations the fault manager will do bad things.   I witnessed this 
>> myself when building a system and the fault manger replaced 5 disks in a 
>> raidz2 vdev inside 1 minute, trashing the pool.   I haven't completely yield 
>> to the "best practice".  I now run one hot spare per pool.  I figure with 
>> raidz2, the odds of the fault manager causing   something 
>> catastrophic is much less possible. 
>> 
>> -Chip
>> 
>> 
>> 
>>> On Fri, Jul 10, 2015 at 11:37 AM, Linda Kateley  
>>> wrote:
>>> I have to build and maintain my own system. I usually help others build(i 
>>> teach zfs and freenas classes/consulting). I really love fault management 
>>> in solaris and miss it. Just thought since it's my system and I get to 
>>> choose I would use omni. I have 20+ years using solaris and only 2 on 
>>> freebsd. 
>>> 
>>> I like freebsd for how well tuned for zfs oob. I miss the network, v12n and 
>>> resource controls in solaris. 
>>> 
>>> Concerned about panics on disk failure. Is that common?
>>> linda
>>> 
>>> 
 On 7/9/15 9:30 PM, Schweiss, Chip wrote:
 Linda,
 
 I have 3.5 PB running under OmniOS.  All my systems have LSI 2108 HBAs 
 which is considered the best choice for HBAs.   
 
 Illumos leaves a bit to be desired with handling faults from disks or SAS 
 problems, but things under OmniOS have been improving, much thanks to Dan 
 McDonald and OmniTI.   We have a paid support on all of our production 
 systems with OmniTI.  Their response and dedication has been very good.  
 Other than the occasional panic and restart from a disk failure, OmniOS 
 has been solid.   ZFS of course never has lost a single bit of 
 information. 
 
 I'd be curious why you're looking to move, have there been specific 
 problems under BSD or ZoL?  I've been slowly evaluating FreeBSD ZFS, but 
 of course the skeletons in the closet never seem to come out until you do 
 something big.
 
 -Chip
 
> On Thu, Jul 9, 2015 at 4:21 PM, Linda Kateley  
> wrote:
> Hey is there anyone out there running big zfs on omni?
> 
> I have been doing mostly zol and freebsd for the last year but have to 
> build a 300+TB box and i want to come back home to roots(solaris). 
> Feeling kind of hesitant :) Also, if you had to do over, is there 
> anything you would do different.
> 
> Also, what is the go to HBA these days? Seems like i saw stable code for 
> lsi 3008?
> 
> TIA
> 
> linda
> 
> 
> ___
> OmniOS-discuss mailing list
> OmniOS-discuss@lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>>> 
>>>  -- 
>>> Linda Kateley
>>> Kateley Company
>>> Skype ID-kateleyco
>>> http://kateleyco.com
> 
> ___
> OmniOS-discuss mailing list
> OmniOS-discuss@lists.omniti.com

Re: [OmniOS-discuss] big zfs storage?

[Linda Kateley]

This really saddens me. To me my favorite part of solaris was FMA. But 
happy about the 3008.


thanks

linda

On 7/10/15 12:31 PM, Schweiss, Chip wrote:
Unfortunately for the past couple years panics on disk failure has 
been the norm.   All my production systems are HA with RSF-1, so at 
least things come back online relatively quick.  There are quite a few 
open tickets in the Illumos bug tracker related to mpt_sas related 
panics.


Most of the work to fix these problems has been committed in the past 
year, though problems still exist.  For example, my systems are dual 
path SAS, however, mpt_sas will panic if you pull a cable instead of 
dropping a path to the disks.  Dan McDonald is actively working to 
resolve this.   He is also pushing a bug fix in genunix from Nexenta 
that appears to fix a lot of the panic problems. I'll know for sure in 
a few months after I see a disk or two drop if it truly fixes things.  
Hans Rosenfeld at Nexenta is responsible for most of the updates to 
mpt_sas including support for 3008 (12G SAS).


I haven't run any 12G SAS yet, but plan to on my next build in a 
couple months.   This will be about 300TB using an 84 disk JBOD.  All 
the code from Nexenta to support the 3008 appears to be in Illumos 
now, and they fully support it so I suspect it's pretty stable now.  
From what I understand there may be some 12G performance fixes coming 
sometime.


The fault manager is nice when the system doesn't panic. When it 
panics, the fault manger never gets a chance to take action.  It is 
still the consensus that is is better to run pools without hot spares 
because there are situations the fault manager will do bad things.   I 
witnessed this myself when building a system and the fault manger 
replaced 5 disks in a raidz2 vdev inside 1 minute, trashing the pool. 
  I haven't completely yield to the "best practice".  I now run one 
hot spare per pool.  I figure with raidz2, the odds of the fault 
manager causing something catastrophic is much less possible.


-Chip



On Fri, Jul 10, 2015 at 11:37 AM, Linda Kateley > wrote:


I have to build and maintain my own system. I usually help others
build(i teach zfs and freenas classes/consulting). I really love
fault management in solaris and miss it. Just thought since it's
my system and I get to choose I would use omni. I have 20+ years
using solaris and only 2 on freebsd.

I like freebsd for how well tuned for zfs oob. I miss the network,
v12n and resource controls in solaris.

Concerned about panics on disk failure. Is that common?

linda


On 7/9/15 9:30 PM, Schweiss, Chip wrote:

Linda,

I have 3.5 PB running under OmniOS.  All my systems have LSI 2108
HBAs which is considered the best choice for HBAs.

Illumos leaves a bit to be desired with handling faults from
disks or SAS problems, but things under OmniOS have been
improving, much thanks to Dan McDonald and OmniTI.   We have a
paid support on all of our production systems with OmniTI.  Their
response and dedication has been very good.  Other than the
occasional panic and restart from a disk failure, OmniOS has been
solid.   ZFS of course never has lost a single bit of information.

I'd be curious why you're looking to move, have there been
specific problems under BSD or ZoL?  I've been slowly evaluating
FreeBSD ZFS, but of course the skeletons in the closet never seem
to come out until you do something big.

-Chip

On Thu, Jul 9, 2015 at 4:21 PM, Linda Kateley
mailto:lkate...@kateley.com>> wrote:

Hey is there anyone out there running big zfs on omni?

I have been doing mostly zol and freebsd for the last year
but have to build a 300+TB box and i want to come back home
to roots(solaris). Feeling kind of hesitant :) Also, if you
had to do over, is there anything you would do different.

Also, what is the go to HBA these days? Seems like i saw
stable code for lsi 3008?

TIA

linda


___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com

http://lists.omniti.com/mailman/listinfo/omnios-discuss




-- 
Linda Kateley

Kateley Company
Skype ID-kateleyco
http://kateleyco.com




___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss