On 3/2/10 4:45 PM, Ethan Quach wrote: > Kristin, > > If you've got zones that were created by being cloned from > one another via 'zonadm clone', you're probably hitting > bug 10990, which wasn't fixed until 125.
It turns out that this is what happened. Luckily the zone that had been cloned was a test zone so it could be removed which should keep this from happening again next time. > > The workaround for systems running pre-125 unfortunately > isn't very generic, and has to be dealt with on a case by > case basis. > > Can you verify if the above is the case for you? > > If so, there are a couple of workarounds. The simpler one, > if this is possible for you, is to detach your zones before > update, and reattach when you've booted the new BE. > This doesn't leave you with fallback copies of the zones > of you need to boot your older BE however. This is what we did and it worked for all the zones bug one. This one zone ended up in a strange state that wouldn't allow us to boot, halt, detach it or much of anything > > Another workaround is to rename all of your zone root > dataset snapshots to all be something unique across the > snapshot names, before doing the image-update. We were able to detach the zones but the one zone mentioned above still got messed up and we had to use "zoneadm -z zonename mark -F configured", zoneadm list to verify the state of the zone, mount the dataset on the zones root mountpoint, and then do the zoneadm -z zonename attach -u. My apologies to Kristin for how long it took to get through this but all the zones are now attached, booted and upgraded. and they appear to be working for the most part at this point. -evan > > > -ethan > > > On 03/01/10 12:16, Kristin Amundsen-Cubanski wrote: >> Hi all, >> >> I had this on pkg-discuss but it seems the issue is not with pkg >> itself. I am trying to update our OpenSolaris 2009.06 server and it is >> getting stuck in a loop. >> >> kristin at waldorf:~# pkg image-update -v >> Creating Plan / Before evaluation: >> UNEVALUATED: >> (bunch of packages) >> >> After evaluation: >> (bunch of packages) >> Actuators: >> restart_fmri: svc:/system/manifest-import: >> default >> restart_fmri: svc:/application/desktop-cache/input-method-cache:default >> restart_fmri: >> svc:/application/desktop-cache/pixbuf-loaders-installer:default >> restart_fmri: svc:/application/desktop-cache/gconf-cache:default >> restart_fmri: svc:/application/desktop-cache/icon-cache:default >> None >> DOWNLOAD PKGS FILES XFER (MB) >> Completed 66/66 3927/3927 119.95/119.95 >> >> PHASE ACTIONS >> Removal Phase 167/167 >> Install Phase 614/614 >> Update Phase 6458/6458 >> PHASE ITEMS >> Reading Existing Index 8/8 >> Indexing Packages 66/66 >> Optimizing Index... >> PHASE ITEMS >> Indexing Packages 637/637 >> >> This is where it hangs. Truss of the process shows: >> >> ioctl(4, ZFS_IOC_OBJSET_STATS, 0x08044CD0) Err#12 ENOMEM >> ioctl(4, ZFS_IOC_OBJSET_STATS, 0x08044CD0) = 0 >> ioctl(4, ZFS_IOC_OBJSET_STATS, 0x08043060) = 0 >> ioctl(4, ZFS_IOC_OBJSET_STATS, 0x08043060) Err#12 ENOMEM >> ioctl(4, ZFS_IOC_OBJSET_STATS, 0x08043060) = 0 >> ioctl(4, ZFS_IOC_SNAPSHOT_LIST_NEXT, 0x08043480) = 0 >> ioctl(4, ZFS_IOC_OBJSET_STATS, 0x08041FE0) = 0 >> ioctl(4, ZFS_IOC_OBJSET_STATS, 0x08044CD0) Err#12 ENOMEM >> ioctl(4, ZFS_IOC_OBJSET_STATS, 0x08044CD0) = 0 >> ioctl(4, ZFS_IOC_OBJSET_STATS, 0x08043060) = 0 >> ioctl(4, ZFS_IOC_OBJSET_STATS, 0x08043060) Err#12 ENOMEM >> ioctl(4, ZFS_IOC_OBJSET_STATS, 0x08043060) = 0 >> ioctl(4, ZFS_IOC_SNAPSHOT_LIST_NEXT, 0x08043480) = 0 >> ioctl(4, ZFS_IOC_OBJSET_STATS, 0x08041FE0) = 0 >> >> just looping forever. >> >> I killed the process. Then beadm also hung when I was changing the >> boot environment back to the current one but it did change the boot >> environment. I killed that as well. After, using beadm to unmount the >> new image worked fine. >> >> I deleted the new boot env and tried the update again with the exact >> same hang in the same place. >> >> Here is zfs list: >> >> kristin at waldorf:~$ zfs list >> NAME >> USED AVAIL REFER MOUNTPOINT >> local0 84.1G 184G 20K /local0 >> local0/sites 10.4G 184G 19K /local0/sites >> local0/sites/database 10.4G 184G 10.4G /var/mysql/5.0/data >> local0/zones 73.6G 184G 25K /zones >> local0/zones/db01 28.4G 184G 22K /zones/db01 >> local0/zones/db01/ROOT 28.4G 184G 19K legacy >> local0/zones/db01/ROOT/zbe 1.50M 184G 612M legacy >> local0/zones/db01/ROOT/zbe-1 28.4G 184G 28.4G legacy >> local0/zones/db01/ROOT/zbe-2 72.5K 184G 28.4G legacy >> local0/zones/db02 2.44G 184G 22K /zones/db02 >> local0/zones/db02/ROOT 2.44G 184G 19K legacy >> local0/zones/db02/ROOT/zbe 2.44G 184G 2.11G legacy >> local0/zones/db02/ROOT/zbe-1 75.5K 184G 2.10G legacy >> local0/zones/web01 36.3G 184G 22K /zones/web01 >> local0/zones/web01/ROOT 36.3G 184G 19K legacy >> local0/zones/web01/ROOT/zbe 11.3G 184G 11.3G legacy >> local0/zones/web01/ROOT/zbe-1 25.0G 184G 24.9G legacy >> local0/zones/web01/ROOT/zbe-2 87.5K 184G 24.3G legacy >> local0/zones/web01_old 3.10G 184G 22K /zones/web01_old >> local0/zones/web01_old/ROOT 3.10G 184G 19K legacy >> local0/zones/web01_old/ROOT/zbe 3.10G 184G 14.4G legacy >> local0/zones/web01_old/ROOT/zbe-1 86.5K 184G 14.4G legacy >> local0/zones/web02 3.39G 184G 22K /zones/web02 >> local0/zones/web02/ROOT 3.39G 184G 19K legacy >> local0/zones/web02/ROOT/zbe 3.39G 184G 3.39G legacy >> local0/zones/web02/ROOT/zbe-1 86.5K 184G 3.38G legacy >> rpool 74.6G 59.3G 76K /rpool >> rpool/ROOT 13.2G 59.3G 18K legacy >> rpool/ROOT/opensolaris 26.5M 59.3G 3.37G / >> rpool/ROOT/opensolaris-1 40.6M 59.3G 3.79G / >> rpool/ROOT/opensolaris-2 12.7G 59.3G 10.2G / >> rpool/ROOT/opensolaris-3 529M 59.3G 10.3G / >> rpool/dump 16.0G 59.3G 16.0G - >> rpool/export 29.3G 59.3G 19K /export >> rpool/export/home 29.3G 59.3G 937K /export/home >> rpool/export/home/Admin 16.6M 59.3G 16.6M /export/home/Admin >> rpool/export/home/kristin 29.3G 59.3G 29.3G /export/home/kristin >> rpool/export/home/tcubansk 466K 59.3G 466K /export/home/tcubansk >> rpool/swap 16.0G 75.3G 16K - >> >> pstack output: >> >> kristin at waldorf:~# pstack `pgrep pkg` >> 27967: /usr/bin/python2.4 /usr/bin/pkg image-update -v >> fed819d5 ioctl (d8aa5e0, 5a14, 8043480, 1020) + 15 >> fdf5e7f0 zfs_iter_snapshots (d8aa5e0, fdf5fde8, 8045d00, 0) + 7c >> fdf60112 zfs_promote (cd81730, fe0283a0, 0, 0) + 166 >> fe01612c be_promote_ds_callback (cd81730, 0, 0, 0) + d0 >> fe015dce be_promote_zone_ds (caa2b70, c996038, 8047268, fe0145cd) + 2da >> fe014609 _be_activate (caa2b70, fe029008, 804728c, fe014188) + 3d9 >> fe0141d0 be_activate (c5f8528) + 68 >> fe051be5 beActivate (0, c783b4c, 0, feeceb7c) + 69 >> fef03125 call_function (804738c, 1, 23fe96c1, 82d5aac) + 3f5 >> fef00221 PyEval_EvalFrame (8142a64, 8295660, 828be84, 0) + 2b11 >> fef01d23 PyEval_EvalCodeEx (8295660, 828be84, 0, c4b975c, 1, c4b9760) >> + 903 >> fef032f4 fast_function (dcb3764, 804754c, 1, 1, 0, 0) + 164 >> fef02dff call_function (804754c, 1, 6f, 0) + cf >> fef00221 PyEval_EvalFrame (c4b95e4, 807def4, 828be84, 0) + 2b11 >> fef01d23 PyEval_EvalCodeEx (8295720, 828be84, 0, 84789fc, 1, 8478a00) >> + 903 >> fef032f4 fast_function (829656c, 804770c, 1, 1, 0, feebadd4) + 164 >> fef02dff call_function (804770c, 0, 200, 0) + cf >> fef00221 PyEval_EvalFrame (847888c, 827b6a0, 8279acc, 0) + 2b11 >> fef03288 fast_function (84dbb1c, 804784c, 1, 1, 0, feebadd4) + f8 >> fef02dff call_function (804784c, 0, 43b, 0) + cf >> fef00221 PyEval_EvalFrame (85061bc, 833bb20, 8079824, 0) + 2b11 >> fef03288 fast_function (84dd56c, 804798c, 2, 2, 0, feebadd4) + f8 >> fef02dff call_function (804798c, 2, de333679, 0) + cf >> fef00221 PyEval_EvalFrame (8124e64, 8346920, 8079824, 0) + 2b11 >> fef03288 fast_function (84ddaac, 8047acc, 0, 0, 0, 0) + f8 >> fef02dff call_function (8047acc, 0, 322, 0) + cf >> fef00221 PyEval_EvalFrame (80af5fc, 8346960, 8079824, 8079824) + 2b11 >> fef01d23 PyEval_EvalCodeEx (8346960, 8079824, 8079824, 0, 0, 0) + 903 >> feefd66e PyEval_EvalCode (8346960, 8079824, 8079824, 0) + 22 >> fef212a1 run_node (8061338, 8047df3, 8079824, 8079824, 8047c3c, 1) + 39 >> fef20425 PyRun_SimpleFileExFlags (fee037e0, 8047df3, 1, 8047c3c) + 14d >> fef26ebb Py_Main (4, 8047d1c, 8047d30, feffb7b4) + 86b >> 080509bd _start (4, 8047de0, 8047df3, 8047e00, 8047e0d, 0) + 7d >> >> My biggest question is if there is a way for me to fix or work around >> this right now. Our main server is panicking almost every day on an IP >> null pointer dereference that has been fixed. >> >> Thanks, >> >> -Kristin >> >> -- >> Kristin Amundsen-Cubanski >> CIO & Board of Directors >> The Mommies Network >> http://www.themommiesnetwork.org/ >> >> >> _______________________________________________ >> caiman-discuss mailing list >> caiman-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/caiman-discuss >> > > > _______________________________________________ > caiman-discuss mailing list > caiman-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/caiman-discuss
