Re: [OpenIndiana-discuss] VMware (OpenIndiana-discuss Digest, Vol 37, Issue 15) (OpenIndiana-discuss Digest, Vol 37, Issue 20)

[Ong Yu-Phing]

Hmm, sounds like some issue with the iscsi adapter on vmware?

This article isn't directly applicable, but the configuration discussion 
might give some pointers on what you can check?


http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007829

On 15/08/2013 03:42, James Relph wrote:

the same servers as iSCSI targets has no iSCSI errors at the same time as 
VMware is freaking out

Is VMware using iSCSI as well or NFS?


Tried it with both (iSCSI originally), and oddly it's basically the exact same 
issue (frequent disconnects) between NFS and iSCSI.  You would be convinced 
it's network related, but nothing shows up obviously wrong in the switch logs 
and obviously the OpenIndiana iSCSI initiators (two of which are guest OSs on 
the VMware cluster!) aren't affected at all.  You get a bizarre situation where 
VMware is complaining about iSCSI going up and down, yet the VMs themselves 
don't register any problems whatsoever.

Thanks,

James.






___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] VMware (OpenIndiana-discuss Digest, Vol 37, Issue 15)

[Ong Yu-Phing]

so far we've been discussing network.  How about the disk subsystem 
side?  I've had a situation where a rebuild (RAID10 equivalent with 3x 
RAID1 vdevs, had to replace a faulty disk), together with an overnight 
snapshot and replication to another server, was "enough" to cause iscsi 
timeouts.


On 13/08/2013 21:18, Doug Hughes wrote:

We have lacp working between force10, hp, and cisco switches in all possible 
combinations with no difficulties. We do monitor and alert on excessive errors 
and drops for interfaces, but lacp isnt a culprit. If anything, it's an 
underlying interface when we find them. Also, it beats the heck out of spanning 
tree and is 2 orders of magnitude simpler than ospf, and 1 order simpler and 
more portable than ecmp. I am quite surprised by your observations.

Sent from my android device.

-Original Message-
From: "Edward Ned Harvey (openindiana)" 
To: Discussion list for OpenIndiana 
Sent: Tue, 13 Aug 2013 7:22 AM
Subject: Re: [OpenIndiana-discuss] VMware


From: James Relph [mailto:ja...@themacplace.co.uk]
Sent: Monday, August 12, 2013 4:47 PM

No, we're not getting any ping loss, that's the thing.  The network looks
entirely faultless.  We've run pings for 24 hours with no ping loss.

Yeah, I swore you said you had ping loss before - but if not - I don't think ping alone 
is sufficient.  You have to find the error counters on the LACP interfaces.  Everybody 
everywhere seems to blindly assume LACP works reliably, but to me, simply saying the term 
"LACP" is a red flag.  It's extremely temperamental, and the resultant behavior 
is exactly as you've described.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss




Disclaimer: use of our emails are governed by terms at http://360-jambo.com/emd

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Recommendations for fast storage (OpenIndiana-discuss Digest, Vol 33, Issue 20)

[Ong Yu-Phing]

A heads up that 10-12TB means you'd need 11.5-13TB useable, assuming 
you'd need to keep used storage < 90% of total storage useable (or is 
that old news now?).


So, using Saso's RAID5 config of Intel DC3700s in 3xdisk raidz1, that 
means you'd need 21x Intel DC3700's at 800GB (21x800/3*2*.9=10.008) to 
get 10TB, or 27x to get 12.9TB useable, excluding root/cache etc.  Which 
means 50+K for SSDs, leaving you only 10K for the server platform, which 
might not be enough to get 0.5TB of RAM etc (unless you can get a bulk 
discount on the Intel DC3700s!).


Working set of ~50% is quite large; when you say data analysis I'd 
assume some sort of OLTP or real-time BI situation, but do you know the 
nature of your processing, i.e. is it latency dependent or bandwidth 
dependent?  Reason I ask, is because I think 10GB delivers better 
overall B/W, but 4GB infiniband delivers better latency.


10 years ago I've worked with 30+TB data sets which were preloaded into 
an Oracle database, with data structures highly optimized for the types 
of reads which the applications required (2-3 day window for complex 
analysis of monthly data).  No SSDs and fancy stuff in those days.  But 
if your data is live/realtime and constantly streaming in, then the work 
profile can be dramatically different.


On 15/04/2013 07:17, Sa?o Kiselkov wrote:

On 04/14/2013 05:15 PM, Wim van den Berge wrote:

Hello,

We have been running OpenIndiana (and its various predecessors) as storage
servers in production for the last couple of years. Over that time the
majority of our storage infrastructure has been moved to Open Indiana to the
point where we currently serve (iSCSI, NFS and CIFS) about 1.2PB from 10+
servers in three datacenters . All of these systems are pretty much the
same, large pool of disks, SSD for root, ZIL and L2ARC, 64-128GB RAM,
multiple 10Gb uplinks. All of these work like a charm.

However the next system is  going to be a little different. It needs to be
the absolute fastest iSCSI target we can create/afford. We'll need about
10-12TB of capacity and the working set will be 5-6TB and IO over time is
90% reads and 10% writes using 32K blocks but this is a data analysis
scenario so all the writes are upfront. Contrary to previous installs, money
is a secondary (but not unimportant) issue for this one. I'd like to stick
with a SuperMicro platform and we've been thinking of trying the new Intel
S3700 800GB SSD's which seem to run about $2K. Ideally I'd like to keep
system cost below $60K.

This is new ground for us. Before this one, the game has always been
primarily about capacity/data integrity and anything we designed based on
ZFS/Open Solaris has always more than delivered in the performance arena.
This time we're looking to fill up the dedicated 10Gbe connections to each
of the four to eight processing nodes as much as possible. The processing
nodes have been designed that they will consume whatever storage bandwidth
they can get.

Any ideas/thoughts/recommendations/caveats would be much appreciated.

Hi Wim,

Interesting project. You should definitely look at all-SSD pools here.
With the 800GB DC S3700 running in 3-drive raidz1's you're looking at
approximately $34k CAPEX (for the 10TB capacity point) just for the
SSDs. That leaves you ~$25k you can spend on the rest of the box, which
is *a lot*. Be sure to put lots of RAM (512GB+) into the box.

Also consider ditching 10GE and go straight to IB. A dual-port QDR card
can be had nowadays for about $1k (SuperMicro even makes motherboards
with QDR-IB on-board) and a 36-port Mellanox QDR switch can be had for
about $8k (this integrates the IB subnet manager, so this is all you
need to set up an IB network):
http://www.colfaxdirect.com/store/pc/viewPrd.asp?idcategory=7&idproduct=158

Cheers,
--
Saso






___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] using LSI 2308-IT onboard supermicro X9DRD-7LN4F-JBOD, openindiana not loading drivers? (OpenIndiana-discuss Digest, Vol 29, Issue 43) (OpenIndiana-discuss Digest, Vol 32, Is

[Ong Yu-Phing]

In my case, it's a pure storage server, went through ~5 days of 
intensive fio testing, then another month of zfs replication testing, 
and it has been stable (using SATA disks) except for a HW LED indicating 
temperature problems, which is supposed to be a supermicro firmware issue.


So I haven't seen this situation as yours, sorry.

On 01/03/2013 08:14, Heinrich van Riel wrote:

After close to 1TB of data added and 20+ virtual machine installs over
iSCSI, it is still running and performs better than expected at this point.
(the server was running windows storage server for a few months before,
stable but, disappointed performance wise so I know it is not hardware
related).
I am going to assume it was related to the vbox install failures and move
on, do not have the time to dig into it.
Thanks,


On Thu, Feb 28, 2013 at 2:41 PM, Heinrich van Riel <
heinrich.vanr...@gmail.com> wrote:


It had the same behavior with 151a5 and 151a7. It is not really doing
anything. Come to think of it the problem only started after attempting to
installing Virtualbox. I only ran the server for 1 or 2 hours before trying
to install it, locks up when loading kernel modules. I tried a few
different releases.
I re-installed it last night and did not attempt to install vbox. It is
configured as an iSCSI target and have been hit by VMware quite a bit since
last night (installing test machines) and it is not have the problem so
far. The strange thing is that I did go back and uninstall vbox each time
after the failed attempt (for a5 & a7) and it locked up every few hours.
iSCSI was not configured, was plain installs with failed attempts of vbox.
Currently it appears to be all good. Load will be increase quite a bit in
the next few days.
I will respond with the results. I am also still waiting for the sas disks
for the rpool, so a reinstall will happen at that point.



On Wed, Feb 27, 2013 at 9:33 PM, Bob Friesenhahn <
bfrie...@simple.dallas.tx.us> wrote:


On Wed, 27 Feb 2013, Heinrich van Riel wrote:

  I am using the same board with 20 disks. I seem to have some stability

issues, all are SAS disks except for the
2x rpool disks (SATA) that I have connected on the the backplane in the
back of the chassis since according to the supermicro documentation
SATA/SAS should not be mixed in the same backlpane. The board is in
the 6047R-E1R36L storage server from Supermicro.
The system would lockup after a few hours and also as soon as it tries to
load the kernel modules for Virtualbox during install.


What version of OpenIndiana are you using (uname -v)?

Is the system doing anything significant (other than starting VirtualBox)
when it locks up?

Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/**
users/bfriesen/ 
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/


__**_
OpenIndiana-discuss mailing list
OpenIndiana-discuss@**openindiana.org
http://openindiana.org/**mailman/listinfo/openindiana-**discuss








___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Zfs import fails (OpenIndiana-discuss Digest, Vol 31, Issue 28)

[Ong Yu-Phing]

I had a similar situation where after export, my disk paths were wrongly 
picked up by ZFS, even though they were recognised by the system.


===
The zpool import fails with this sort of message:
pool1   UNAVAIL  insufficient replicas
  mirror-0  UNAVAIL  corrupted data
c3t0d0  ONLINE
c3t1d0  ONLINE

When I check the disk labels on the disks:

zdb -l /dev/dsk/c3t0d0s0

LABEL 0

version: 28
name: 'pool1'
state: 1
txg: 7173951
pool_guid: 8370873525947507187
hostid: 13162267
hostname: 'openindiana'
top_guid: 15064987019855796782
guid: 4751459059166773513
vdev_children: 3
vdev_tree:
type: 'mirror'
id: 0
guid: 15064987019855796782
metaslab_array: 30
metaslab_shift: 34
ashift: 9
asize: 1998985625600
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 4751459059166773513
path: '/dev/dsk/c2t0d0s0'
devid: 'id1,sd@n600605b002e26410183d783b0e56515a/a'
phys_path: '/pci@0,0/pci8086,3410@9/pci1014,3c7@0/sd@0,0:a'
whole_disk: 1
DTL: 4119
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 7277976899319815787
path: '/dev/dsk/c2t1d0s0'
devid: 'id1,sd@n600605b002e264101839b39f123c1210/a'
phys_path: '/pci@0,0/pci8086,3410@9/pci1014,3c7@0/sd@1,0:a'
whole_disk: 1
DTL: 4118
create_txg: 4

So I noticed that even though the disk is c3t0d0, the path shows in the 
disk label as c2t0d0s0 (and similarly for the "child"/partner mirrored 
disk, c3t1d0, yet path shows as c2t1d0), is this what is causing the 
really strange behaviour, when even though the devices are considered 
ONLINE, the pool itself is UAVAIL?

===

Maybe you can check if you are hitting the same situation? 
Unfortunately, I never was able to fix this (caused by a live upgrade of 
OI from 148 to 151a3); instead, I booted from a live USB, then did a ZFS 
send to transfer the filesystems to another server.


On 07/02/2013 21:47, Ram Chander wrote:

"format" detects all 108 disks . I suspect it could be zfs issue. I ran out
of options now.

root@host1:~# format
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c1t0d0 
   /pci@0,0/pci8086,340b@4/pci1028,1f17@0/sd@0,0
1. c5t1d0 
   /ethdrv/sd@1,0
2. c5t1d1 
   /ethdrv/sd@1,1
3. c5t1d2 
   /ethdrv/sd@1,2
4. c5t1d3 
   /ethdrv/sd@1,3
5. c5t1d4 
   /ethdrv/sd@1,4
  etc

root@host1~# devfsadm -Cv
root@host1:~#



On Thu, Feb 7, 2013 at 6:44 PM, Sa?o Kiselkov wrote:


You have an issue with conectivity to your drives on the Coraid HBA
card. I suggest querying your HBA via its management tools to make sure
you can discover all the drives on your network. Chances are, they're
not all visible, which is why your pool is having trouble.

--
Saso

On 02/07/2013 01:49 PM, Ram Chander wrote:

The drives are in Coraid  connected to the server via Coraid 10G HBA

card.

Its exported and imported on the same host but after OS upgrade ( format

OS

disk and install again ). Before OS upgrade, zpool export was issued and
when tried to import it faults as below. The pool is not functional in

any

system now.  Tried -d /dev/dsk option but no luck.


root@storage1:~# zpool import
pool: pool1
  id: 10136140719439709374
   state: FAULTED
  status: The pool was last accessed by another system.
  action: The pool cannot be imported due to damaged devices or data.
 The pool may be active on another system, but can be imported

using

 the '-f' flag.
see: http://illumos.org/msg/ZFS-8000-EY
  config:

 pool1  FAULTED  corrupted data
   raidz1-0   ONLINE
 c5t1d0   UNAVAIL  corrupted data
 c5t1d1   UNAVAIL  corrupted data
 c5t1d2   UNAVAIL  corrupted data
 c5t1d3   UNAVAIL  corrupted data
 c5t1d4   UNAVAIL  corrupted data
   raidz1-1   ONLINE
 c5t1d5   UNAVAIL  corrupted data
 c5t1d6   UNAVAIL  corrupted data
 c5t1d7   UNAVAIL  corrupted data
 c5t1d8   UNAVAIL  corrupted data
 c5t1d9   UNAVAIL  corrupted data
   raidz1-2   ONLINE
 c5t1d10  UNAVAIL  corrupted data
 c5t1d11  UNAVAIL  corrupted data
 c5t1d12  UNAVAIL  corrupted data
 c5t1d13  UNAVAIL  corrupted data
 c5t1d14  UNAVAIL  corrupted data
   raidz1-3   ONLINE
 c5t1d15  UNAVAIL  corrupted data
 c5t1d16  UNAVAIL  corrupted data
 c5t1d17  UNAVAIL  corrupted data
 c5t1d18  UNAVAIL  corrupted data
 

Re: [OpenIndiana-discuss] using LSI 2308-IT onboard supermicro X9DRD-7LN4F-JBOD, openindiana not loading drivers? (OpenIndiana-discuss Digest, Vol 29, Issue 43)

[Ong Yu-Phing]

my bad and my typo, that's a 1 (one), not l (ell).

You've got sharp eyes, James!

On 02/01/2013 18:37, James C. McPherson wrote:

On 2/01/13 06:16 PM, Ong Yu-Phing wrote:

great, that worked, albeit slightly differently, so FYI to contribute
to the list and knowledge:



Once I had the hint about /etc/driver_alises, I man'd to find out
more, then checked using prtconf, noticed the card was referenced as
pci15d9,69l (rather than pciex1000,86), so used update_drv -a -i
"pci15d9,69l" mpt_sas, and the controller+disks were found.


If your driver alias includes non-hexadecimal characters
(apart from "pci" or "pciex" or "usb" or "class" etc) then
there is something very wrong.

Are you *sure* you needed the 'l' on the end of pci15d9,69  ?


James C. McPherson
--
Solaris kernel software engineer, system admin and troubleshooter
  http://www.jmcpdotcom.com/blog
Find me on LinkedIn @ http://www.linkedin.com/in/jamescmcpherson




___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] using LSI 2308-IT onboard supermicro X9DRD-7LN4F-JBOD, openindiana not loading drivers? (OpenIndiana-discuss Digest, Vol 29, Issue 43)

[Ong Yu-Phing]

great, that worked, albeit slightly differently, so FYI to contribute to 
the list and knowledge:


Once I had the hint about /etc/driver_alises, I man'd to find out more, 
then checked using prtconf, noticed the card was referenced as 
pci15d9,69l (rather than pciex1000,86), so used update_drv -a -i 
"pci15d9,69l" mpt_sas, and the controller+disks were found.


Thanks!

On 29/12/2012 07:53, Joshua M. Clulow wrote:

On 28 December 2012 03:12, Ong Yu-Phing  wrote:

I have a supermicro X9DRD-7LN4F-JBOD motherboard, which comes preconfigured
with an LSI 2308 onboard in IT mode.  The disks attached to the onboard LSI
2308 controller can be seen in linux from both dmesg and fdisk output, but
not in OI 148 or 151.
compatible:  pciex1000,86.15d9.69l.5
vendor-id:   1000
device-id:   86

Anybody else using this board or equivalent, that can help shed some light?

I have a machine with this board here, running SmartOS.  It's possible
that OI does not contain a mapping between the hardware you have and
the "mpt_sas" driver.  You can check thus:

[root@ra ~]# grep pciex1000,86 /etc/driver_aliases
mpt_sas "pciex1000,86"





___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

[OpenIndiana-discuss] using LSI 2308-IT onboard supermicro X9DRD-7LN4F-JBOD, openindiana not loading drivers?

[Ong Yu-Phing]

I have a supermicro X9DRD-7LN4F-JBOD motherboard, which comes 
preconfigured with an LSI 2308 onboard in IT mode.  The disks attached 
to the onboard LSI 2308 controller can be seen in linux from both dmesg 
and fdisk output, but not in OI 148 or 151.


FYI the MPT firmware in the LSI controller is 13.00.57.00-IT.

When I boot up in e.g. OI 148, scanpci output finds "lsi logic / symbios 
logic sas2208 pci-express fusion-mpt sas-2", but modinfo | grep -i mpt 
doesn't show any drivers loaded.  I tried to manually load them (e.g. 
modload -p drv/kernel/amd64/mpt) but format still doesn't show the disks 
(it does show the 2x Intel 520 SSDs connected to the SATA interface, 
though).


When I startup the Device Driver Manager in OI 151, it shows as Storage 
: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS2 : UNK 
(i.e. no device driver loaded).  Right click to show more info:


node name:   pci15d9,69l
Vendor:  LSI Logic / Symbios Logic
Device:  SAS2308 PCI-Express FUsion-MPT SAS-2
Sub-Vendor:  Super Micro Computer Inc
binding name:pci15d9,69l
devfs path:  /pci@0,0/pci8086,3c06@2,2/pci15d9,69l
compatible name: 
(pciex1000,86.15d9.69l.5)(pciex1000,86.15d9.69l)(pciex1000,86.5)(pciex1000,86)(pciexclass,010700)(pciexclass,0107)(pci1000,86.15d9.69l.5)(pci1000,86.15d9.69l)(pci15d9.69l)(pci1000,86.5)((pci1000,86)(pciclass,010700)(pciclass,0107)

driver name: unknown
acpi-namespace:  _SB_.PCI0.NPE5.LSAS
assigned-addresses:  81030010
reg: 3
compatible:  pciex1000,86.15d9.69l.5
model:   Serial Attached SCSI Controller
power-consumption:   1
devsel-speed:0
interrupts:  1
subsystem-vendor-id: 15d9
subsystem-id:69l
unit-address:0
class-code:  10700
revision-id: 5
vendor-id:   1000
device-id:   86


Anybody else using this board or equivalent, that can help shed some light?
___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] import of pool fails, due to changed device path? (OpenIndiana-discuss Digest, Vol 28, Issue 30)

[Ong Yu-Phing]

yes, I've already removed /etc/zfs/zpool.cache

without doing this, the upgraded system would hang (in fact, sometimes 
it would even reboot straight after the grub prompt).


I then tried:
1) Boot LiveUSB
2) zpool import -f pool
3) zpool export pool
4) Try to boot main OS installation and use the pool
When I boot up in the upgraded system, this is when the pool is shown as 
offline


Haven't tried zpool import -R yet, but will give it a try.

> What was your upgrade procedure? Did you "pkg upgrade" or reinstall?
> It might help to look around such files as /etc/path_to_inst,
> /etc/driver*, /etc/device*, /dev/*, /devices* and try to rename
> the storage controller device nodes so that they report what your
> pool wants. This is a kind of woodoo magic, so should be last resort :)

it was a pkg-img update, but even a new install (I had a couple of spare 
servers for a DR test, so luckily I could swap disks and servers) with 
151a3 failed as per above.  I just think there's some strange naming of 
devices, which a USB boot gets around somehow (both 148 and 151a3 live USB).


I'll leave the voodoo magic for later, past experience has had me 
destroy some installs (non-zfs) by doing funky mknod stuff, but I 
understand what you are trying to achieve.  This system is now in test 
mode, since I've had to zfs send the various filesystems off to another 
server for production use, so I don't really mind if it's trashed, but I 
would like to figure this out for future reference.


Thanks for your suggestions!

On 22/11/2012 21:45, Jim Klimov wrote:

On 2012-11-22 11:18, Ong Yu-Phing wrote:

I have an interesting situation where an upgrade of OI 151a3 is unable
to import my pool, yet booting into a live USB allows access to the pool
without problems.

The zpool import fails with this sort of message:
 pool1   UNAVAIL  insufficient replicas
   mirror-0  UNAVAIL  corrupted data
 c3t0d0  ONLINE
 c3t1d0  ONLINE


...

So I noticed that even though the disk is c3t0d0, the path shows in the
disk label as c2t0d0s0 (and similarly for the "child"/partner mirrored
disk, c3t1d0, yet path shows as c2t1d0), is this what is causing the
really strange behaviour, when even though the devices are considered
ONLINE, the pool itself is UAVAIL?  And if so, is there any way to fix
this?


To recap, if you boot into LiveUSB, you can import the pool and there
are no errors? The zpool status from above is reported by your live
system?
Does it work for you to:
1) Boot LiveUSB
2) zpool import (-f?) pool
3) zpool export pool
4) Try to boot main OS installation and use the pool?
(This is the procedure to "fix" similar looking problems with rpools)

Is the situation influenced by presence of removable storage (USB,
eSATA) at boot of the installed OS? (I guess it can "move" the
controller names around) For example, if you boot the installed
OI while the liveusb is still plugged in - is the situation the
same?

Remove /etc/zfs/zpool.cache on your OI installation and have it
search for your pool components during zpool import completely.
Alternately, import the pool with different or none cache file:
  zpool import -R / pool
This should work around the device/driver renamings if that's
the only problem.

What was your upgrade procedure? Did you "pkg upgrade" or reinstall?
It might help to look around such files as /etc/path_to_inst,
/etc/driver*, /etc/device*, /dev/*, /devices* and try to rename
the storage controller device nodes so that they report what your
pool wants. This is a kind of woodoo magic, so should be last resort :)

HTH,
//Jim Klimov







___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

[OpenIndiana-discuss] import of pool fails, due to changed device path?

[Ong Yu-Phing]

I have an interesting situation where an upgrade of OI 151a3 is unable 
to import my pool, yet booting into a live USB allows access to the pool 
without problems.


The zpool import fails with this sort of message:
pool1   UNAVAIL  insufficient replicas
  mirror-0  UNAVAIL  corrupted data
c3t0d0  ONLINE
c3t1d0  ONLINE

When I check the disk labels on the disks:

zdb -l /dev/dsk/c3t0d0s0

LABEL 0

version: 28
name: 'pool1'
state: 1
txg: 7173951
pool_guid: 8370873525947507187
hostid: 13162267
hostname: 'openindiana'
top_guid: 15064987019855796782
guid: 4751459059166773513
vdev_children: 3
vdev_tree:
type: 'mirror'
id: 0
guid: 15064987019855796782
metaslab_array: 30
metaslab_shift: 34
ashift: 9
asize: 1998985625600
is_log: 0
create_txg: 4
children[0]:
type: 'disk'
id: 0
guid: 4751459059166773513
path: '/dev/dsk/c2t0d0s0'
devid: 'id1,sd@n600605b002e26410183d783b0e56515a/a'
phys_path: '/pci@0,0/pci8086,3410@9/pci1014,3c7@0/sd@0,0:a'
whole_disk: 1
DTL: 4119
create_txg: 4
children[1]:
type: 'disk'
id: 1
guid: 7277976899319815787
path: '/dev/dsk/c2t1d0s0'
devid: 'id1,sd@n600605b002e264101839b39f123c1210/a'
phys_path: '/pci@0,0/pci8086,3410@9/pci1014,3c7@0/sd@1,0:a'
whole_disk: 1
DTL: 4118
create_txg: 4

So I noticed that even though the disk is c3t0d0, the path shows in the 
disk label as c2t0d0s0 (and similarly for the "child"/partner mirrored 
disk, c3t1d0, yet path shows as c2t1d0), is this what is causing the 
really strange behaviour, when even though the devices are considered 
ONLINE, the pool itself is UAVAIL?  And if so, is there any way to fix 
this?


Thanks.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Current ZFS Backup projects (OpenIndiana-discuss Digest, Vol 26, Issue 34) (OpenIndiana-discuss Digest, Vol 26, Issue 39)

[Ong Yu-Phing]

Yes, there are two slightly different goals in terms of backup, (1) to 
backup ZFS based CIFS servers (2) to backup non-ZFS based servers.  For 
(1) I've been trying to use tools like zetaback, warmer, etc.  For (2) 
I've been using good-ol' backuppc.


I'm quite glad this thread started, because lo and behold we now have 
zrep pointed out by Frank (written by Philip Brown, according to the 
website), which looks exactly like what I'm interested in for (1), and 
mdbackup from Julius, which looks like I could use for (2), although I 
might still stay with backuppc as my staff are quite familiar with this.


So thanks all for your valuable inputs!

Phing

On 13/09/2012 17:23, Jim Klimov wrote:

2012-09-12 6:05, Ong Yu-Phing ?:

Jim, I assume you are referring to this:
http://wiki.openindiana.org/oi/rsync+daemon+service+on+OpenIndiana, 
thanks!


Yes, I think that's it ;)


My concern is that typically rsync will take quite a while to traverse a
large set of files before sending only changed files; a classic example
is backing up say 1TB of maildir emails, it may take 4+ hours, and you
now have to deal with a situation where your midnight backup is really a
"somewhere between midnight and 4am" backup.  And of course, if you want
to take snapshots/backups of comstar volumes, rsync isn't quite the
right fit.

On the other hand, a zfs snapshot gives an almost-at-the-time backup
(give or take a few seconds), versus the aforementioned rsync. The zfs
snapshot can then be sent off-site, independent of the backup activity.


I got an impression that you needed to backup some "client" machines
with varied OSes, such as Windows or Linux desktops, onto a ZFS server.
In that case rsync should help, although you're right that it would
take long to scan the directory trees for changes.

With client FSes that support snapshots (ZFS, NTFS shadow copy) you
might have some luck making scripts that take the client's snapshot,
hold it (in case of ZFS, to avoid it being destroyed while you're
working), rsync the changes from the snapshot (so the 0am backup is
really the state at 0am) and release/destroy the snapshot on client.
In case of ZFS at least, you might have some optimization by using
"zfs diff" to determine changed files between two snapshots on the
client - but then you should not destroy rsynced snapshots right
away, but keep a backlog of one or two at least. And you should have
some locking to prevent several instances of the backup job crawling
the same client space and bringing IOPS to a halt.

Now, before you ask "why not zfs-send client snapshots directly?" -
there may be reasons, such as incompatible ZFS versions on client
and server, differing dataset layouts, flaky network preventing
transfer of large zfs-send streams (though that should have been
addressed with resumable zfs-send feature, if that was integrated).

HTH,
//Jim Klimov







___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Current ZFS Backup projects (OpenIndiana-discuss Digest, Vol 26, Issue 34)

[Ong Yu-Phing]

Jim, I assume you are referring to this: 
http://wiki.openindiana.org/oi/rsync+daemon+service+on+OpenIndiana, thanks!


My concern is that typically rsync will take quite a while to traverse a 
large set of files before sending only changed files; a classic example 
is backing up say 1TB of maildir emails, it may take 4+ hours, and you 
now have to deal with a situation where your midnight backup is really a 
"somewhere between midnight and 4am" backup.  And of course, if you want 
to take snapshots/backups of comstar volumes, rsync isn't quite the 
right fit.


On the other hand, a zfs snapshot gives an almost-at-the-time backup 
(give or take a few seconds), versus the aforementioned rsync.  The zfs 
snapshot can then be sent off-site, independent of the backup activity.


However, having used zetaback, warmer, as well as other zfs scripts, 
what I've noticed so far is that sometimes, the snapshot send/receive 
"hangs" (for want of a better word), then the other automated, scheduled 
jobs then queue and "hang" behind the first (since they tend to be 
incremental).


If anybody has encountered and found a solution or even a way to 
troubleshoot why that particular job "hangs", I'd appreciate any feedback!


Phing

On 12/09/2012 01:47, Jim Klimov wrote:

2012-09-11 18:56, Mark Creamer wrote:

A recent thread caused me to look for open source projects that leverage
ZFS to backup systems. I found a couple, such as OmniTI's
Zetaback,
but that one appears to be dead - at least the links don't work and 
the Git

page shows no recent activity. Nexenta's commercial product for Windows,
"Delorean", also appears to have been killed (unfortunately without 
first
being released to the community as far as I can tell). Wondering if 
anyone
knows of any other projects that use scripts or some other method to 
manage
system backups with zfs. I'm hoping to build on the ideas of someone 
more

knowledgeable to automate my snapshot and recovery efforts.



Regarding backup of other OSes to ZFS servers and leveraging ZFS
from there on, I can suggest rsync (or cwrsync in case of Windows
clients). This is easily automatable on clients (push to rsync
server) or on the server (pull from clients), provided that you
set up the security/logins appropriately.

The ZFS-enabled server can take care of snapshotting successful
rsync run results, deduplication if appropriately spec'ed, etc.

I posted an enhanced SMF script and config file snippets on OI
wiki to run the rsync server and take ZFS auto-snapshots after
successful completion of incoming rsync sessions (push mode);
it is even more trivial in pull mode initiated by the server -
it can "mkdir .../.zfs/snapshot/$SNAPNAME" given the proper
access via "zfs allow").

HTH,
//Jim Klimov






___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Very slow write performance on a 3 disk raidz1-0 zpool (OpenIndiana-discuss Digest, Vol 25, Issue 34)

[Ong Yu-Phing]

I think you just answered your own question; 2TB with about 1.5x dedup, 
and only 1GB of RAM isn't going to work.  ZFS will need to check the 
dedup tables everytime you write, and if the dedup tables don't fit* 
into 1/4 of RAM (or was it that ZFS only uses 3/4 of available RAM by 
default, then dedup only uses 1/4 of that ZFS RAM? I forget!), then 
there will be a lot of iops running as ZFS keeps having to read the 
dedup tables from disk instead.


I had a similar system, AMD64 3000 (I think, or 3200) with 2GB and when 
I bumped it up to 8GB write speed exponentially improved.


* There is some calculation you can go through to work out dedup 
requirements for RAM, but the rough rule of thumb is 2GB (I'd even say 
4GB or more, if it's a live/nearline system, versus a backup/archive 
system) of RAM for every 1TB; and you likely still need to tweak 
arc_meta_limit.


For a larger example, I have an archive server with 20TB RAIDZ3 (13x2TB 
disks), 1.62x dedup with 10.8T used and 12.9T free, and 48GB of RAM, 
where arc_meta_limit has been tweaked to use 32GB of RAM instead.  I get 
about 20-30mbps (with dedup off I would normally be getting 50-60mbps) 
over 1gbps ethernet.


On 24/08/2012 18:25, Julius Roberts wrote:

On 24 August 2012 15:03, Robbie Crash  wrote:

Are you using compression or dedup on the FS?

Yes, both.  We're getting about 1.5x dedup on the Backups pool, not
sure how to calculate compression



___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Office apps unable to write to ZFS over CIFS (OpenIndiana-discuss Digest, Vol 23, Issue 43)

[Ong Yu-Phing]

In my environment I have 3 OI_148's integrated into two different ADs, 
with the majority of users being Windows users.  They don't see any 
issues like this.  They are heavy users of cad, design tools like 
photoshop/sketchup etc, as well as MS office/libreoffice, ~12TB of data 
all up (and 30+snapshots per filesystem).


Are you sharing via NFS, or are the Windows users accessing via the ZFS 
sharesmb=thsfs?


In my situation, I don't have both NFS and sharesmb turned on 
simultaneously, nor do I have compression enabled, not sure if these 
make a difference.


My nbmand is off, though, as I'm not using both NFS and CIFS.  Maybe 
this is your problem, since it specifically deals with locking 
(http://docs.oracle.com/cd/E19082-01/820-2429/configurecrossprotocollockingtask/index.html), 
specifically "When the nbmand mount option is not set, the Solaris CIFS 
service will enforce mandatory share reservations and byte-range locking 
internally for all CIFS clients. However, without nbmand set, there is 
only limited coordination with NFS and local processes."?


On 29/06/2012 05:08, Martin Frost wrote:

I'm running oi_148 as a fileserver, exporting via NFS and the
kernel CIFS service for ZFS.

But Windows users (XP and probably all Windows versions) are unable to
write files from any MS Office applications into the shares from ZFS.
They always get: "Access denied.  Contact your administrator."  Same
result whether they're trying to overwrite a file or write a new file.

Any ideas what's causing that?  This is driving me crazy.  I've
seen the same problem under Linux with Samba, where disabling
locking seemed to help.

After that error, the users save the file to the local disk and copy
it over the CIFS connection into the ZFS system successfully.  So they
clearly have write access into ZFS from Windows, and the filesystem
has lots of free space, but Office can't write any files to ZFS.  I
assume this is some sort of locking problem.  I have nbmand=on, which
is what I've read it should be set to for CIFS sharing.

The directories and files they're trying to edit are owned by the
actual user (defined by matching passwd and smbpasswd entries on the
OI machine) and have 700 permissions and full_set ACLS:

   owner@:rwxpdDaARWcCos:fd-:allow
everyone@:rwxpdDaARWcCos:fd-:deny

Some relevant services:

online Feb_11   svc:/network/shares/group:smb
online Mar_01   svc:/network/smb/server:default
online Feb_27   svc:/network/nfs/status:default
online Feb_27   svc:/network/nfs/rquota:default
online Feb_27   svc:/network/nfs/nlockmgr:default
online May_08   svc:/network/nfs/mapid:default
online May_09   svc:/network/nfs/server:default
online May_09   svc:/network/nfs/cbd:default

Here are most properties of the filesystem:

NAME PROPERTY VALUE  SOURCE
gname/thefs  mounted  yes-
gname/thefs  reservation  none   default
gname/thefs  mountpoint   /gname/thefs   default
gname/thefs  sharenfs sec=sys,rw=gname   local
gname/thefs  checksum on default
gname/thefs  compression  gzip   inherited from gname
gname/thefs  atimeoffinherited from gname
gname/thefs  devices  on default
gname/thefs  exec on default
gname/thefs  readonly offdefault
gname/thefs  zonedoffdefault
gname/thefs  aclinherit   passthroughlocal
gname/thefs  canmount on default
gname/thefs  xattron default
gname/thefs  version  5  -
gname/thefs  utf8only off-
gname/thefs  normalizationnone   -
gname/thefs  casesensitivity  mixed  -
gname/thefs  vscanoffdefault
gname/thefs  nbmand   on local
gname/thefs  sharesmb name=thefs local
gname/thefs  refquota none   default
gname/thefs  refreservation   none   default
gname/thefs  logbias  latencydefault
gname/thefs  dedupoffdefault
gname/thefs  mlslabel none   default
gname/thefs  sync standard   default

Thanks for any suggestions!

Martin







___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Static routes (OpenIndiana-discuss Digest, Vol 23, Issue 42)

[Ong Yu-Phing]

Assuming that your subnet mask is 255.255.255.0, this means your two 
subnets are separate (192.168.0.0/24 and 192.168.10.0/24).


Then accessing 192.168.10.2 and 192.168.10.1 would be on a separate 
media (your crossover cable; although a modern switch, not hub, would 
also do, since they do point-to-point links, e.g. the HP 2510g's I use 
have 96Gbps capacity across 48ports.  If you are really paranoid about 
any data leakage, you can also VLAN them)


So all you need to do is an rsync using the IP rather than hostname (or 
create an entry in the hosts file for the private IP:

192.168.0.100 machineA
192.168.10.2 privateA
)

e.g. from machine A, rsync -av user@192.168.10.1:/whatever

On 28/06/2012 09:26, drsgrid wrote:

Im not a full time admin but im trying to use rsync across 10GE (or even
1GE -as a starting point) from one openindana box to another. Both machines
are also connected to a broader network to receive updates (eth0)  but im
trying to use a xover cable for eth1 so i can rsync across and not take up
a lot of bandwith on the broader network. From my reading think I need to
setup a static route on eth1.   All my ip addresses are static: -

machineA(eth0)=192.168.0.100, machineA(eth1)=192.168.10.2
machineB(eth0)=192.168.0.108, machineB(eth1)=192.168.10.1

how do i setup the machines so that i can do an rsync across the
192.168.10.x domain so as not to disturb bandwith on the 192.168.0.x domain?






___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] intermittent CIFS loss, spontaneous-reboot with OI148/151a and IBM Megaraid M5015?

[Ong Yu-Phing]

Hi Ken,

The OI148 servers do occasionally "lose" SMB/CIFS access.  But no 
spontaneous reboot (yet!).


Hi James,

/etc/syslog.conf is setup with the following:
*.err;kern.notice;auth.notice   @loghost
*.err;kern.debug;daemon.notice;mail.crit@loghost

and in the loghost, this is the output (slightly sanitised) from around 
when the server rebooted (just yesterday!):
Feb  1 12:06:01 san7.local svc.startd[10]: [ID 122153 daemon.warning] 
svc:/network/smb/server:default: Method or service exit timed out.  
Killing contract 1511.
Feb  1 12:06:29 san7.local svc.startd[10]: [ID 122153 daemon.warning] 
svc:/network/smb/server:default: Method or service exit timed out.  
Killing contract 1511.

Feb  1 12:08:10 san7.local svc.startd[10]: last message repeated 61 times
Feb  1 12:09:11 san7.local svc.startd[10]: last message repeated 61 times
Feb  1 12:09:28 san7.local svc.startd[10]: last message repeated 16 times

coreadm is set by default for only per-process core dumps, and similarly 
dumpadm shows defaults, I've enable savecore, so will see what i get for 
a core dump in future.


Thanks all.

regards, Yu-Phing

On 02/02/2012 00:50, ken mays wrote:

Ong,

Any issues with the oi_148 servers?

~ Ken Mays


*From:* James Carlson 
*To:* Discussion list for OpenIndiana 


*Cc:* Ong Yu-Phing 
*Sent:* Wednesday, February 1, 2012 9:10 AM
*Subject:* Re: [OpenIndiana-discuss] intermittent CIFS loss, 
spontaneous-reboot with OI148/151a and IBM Megaraid M5015?


On 02/01/12 03:29, Ong Yu-Phing wrote:
> We've a number of IBM 3630M3 servers, equipped with BBU M5014/5015s, 
running as CIFS server, with a mixture of OI148 and OI151a.  Nothing 
fancy (no dedup, no compression), just a pool of mirrored disks aka 
RAID10, with CIFS access authenticated via MS AD.

>
> Intermittently, CIFS/SMB will go down, sometimes this can be 
restored via restarting the smb service ("enable -r smb/server"), 
other times it necessitates a server reset ("svcs | grep smb" shows 
that smb/server has an * next to it).


If you do "svcs -xv", it should show references to log files for the
services that are in trouble.  For smb/server, I'd expect that to be
/var/svc/log/network-smb-server:default.log.  Examining that file would
be a good first step here.

Also, it's common for services to log via syslog.  /var/adm/messages
might be a good place to start there.

> And one of the servers (always the same, so far...) will 
intermittently reboot (more frequently than the SMB service going 
down).  Sometimes in the middle of the day, sometimes in the evening 
(once it was around 6pm).  This particular server will reboot and come 
back up without much delay, and the pool and zfs shares come back 
online fine.


Spontaneous reboot has to be either a kernel panic or a hardware
problem.  "dumpadm" should tell you where the kernel dumps are going --
the "savecore" directory; usually /var/crash.  Look for files there.

Running mdb on the files and using ::status and ::stack commands might
give a good enough signature that someone could identify the cause.

(I'm not a CIFS expert, but if you gather some basic log information
about the problem, I imagine one may be able to help.)

--
James Carlson42.703N 71.076W <mailto:carls...@workingcode.com>>


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org 
<mailto:OpenIndiana-discuss@openindiana.org>

http://openindiana.org/mailman/listinfo/openindiana-discuss



___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

[OpenIndiana-discuss] intermittent CIFS loss, spontaneous-reboot with OI148/151a and IBM Megaraid M5015?

[Ong Yu-Phing]

We've a number of IBM 3630M3 servers, equipped with BBU M5014/5015s, running as 
CIFS server, with a mixture of OI148 and OI151a.  Nothing fancy (no dedup, no 
compression), just a pool of mirrored disks aka RAID10, with CIFS access 
authenticated via MS AD.

Intermittently, CIFS/SMB will go down, sometimes this can be restored via 
restarting the smb service ("enable -r smb/server"), other times it 
necessitates a server reset ("svcs | grep smb" shows that smb/server has an * 
next to it).

And one of the servers (always the same, so far...) will intermittently reboot 
(more frequently than the SMB service going down).  Sometimes in the middle of 
the day, sometimes in the evening (once it was around 6pm).  This particular 
server will reboot and come back up without much delay, and the pool and zfs 
shares come back online fine.

I'm not sure if these events, the CIFS/SMB service going down, and the 
intermittent server reboot are related, and I'm not sure if its also related to 
the mr_sas/mpt bug (https://www.illumos.org/issues/618 and 
https://www.illumos.org/issues/1069), as the M501[45] are just LSI controllers, 
using SUNWmrsas.

Anybody have any suggestions about how to investigate this further, or if this 
is indeed the behaviour associated with the aforementioned mr_sas/mpt bugs?

Thanks... Yu-Phing


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss