Re: [OpenIndiana-discuss] zabbix-agent

[Edward Ned Harvey (openindiana)]

> From: Pawel Stefanski [mailto:pejo...@gmail.com]
> 
> here you have complete instruction
> https://www.zabbix.com/wiki/howto/install/solaris/opensolaris

I know.  I described that as Plan B.  See:


> > Plan A is to get it from a standard package
> > repository, and Plan B is to get the solaris binaries from zabbix.com.

If Plan A doesn't exist, that's ok.  I just thought zabbix-agent was in some 
standard package repository because of Adam Stevko's and Andrzej Szeszo's 
emails discussing it in the package repo.  Andrzej is apparently the maintainer 
of  http://wiki.openindiana.org/oi/Package+Repositories so...  That led to 
credible belief that I should expect to find the package in there...

___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

[OpenIndiana-discuss] zabbix-agent

[Edward Ned Harvey (openindiana)]

Hi.  I'm trying to figure out the best & easiest way to get zabbix-agent 
installed on openindiana.

The post here suggests that it should already be included in some standard 
package repository, but I don't see it in dev, sfe, or sfe-encumbered.
http://openindiana.org/pipermail/oi-dev/2012-January/001092.html

Sure I can build from source, or download solaris binaries from zabbix.com.  I 
will if I have to.  Plan A is to get it from a standard package repository, and 
Plan B is to get the solaris binaries from zabbix.com.

___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] ZFS allowed characters (valid characters)

[Edward Ned Harvey (openindiana)]

> From: Edward Ned Harvey (openindiana)
> [mailto:openindi...@nedharvey.com]
> 
> Is ZFS using Unicode or ASCII?  Or something else?
> 
> Are there disallowed characters?  '\0' or @ or '/' or anything else?  I know
> these characters generally would be *difficult* to use just because of
> limitations of your application environment (for example, bash will always
> parse the '/' as a directory delimiter, but at least in hfs+, that character 
> isn't
> actually forbidden by the filesystem.)  So I'm not asking which characters
> would be *practically* difficult to use, I'm asking which characters are 
> *valid*
> according to the filesystem itself.

Are the valid characters the same for filesystem names, and filenames?  (e.g. 
mountpoints & zvol's, versus directories and files)

___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

[OpenIndiana-discuss] ZFS allowed characters (valid characters)

[Edward Ned Harvey (openindiana)]

Is ZFS using Unicode or ASCII?  Or something else?

Are there disallowed characters?  '\0' or @ or '/' or anything else?  I know 
these characters generally would be *difficult* to use just because of 
limitations of your application environment (for example, bash will always 
parse the '/' as a directory delimiter, but at least in hfs+, that character 
isn't actually forbidden by the filesystem.)  So I'm not asking which 
characters would be *practically* difficult to use, I'm asking which characters 
are *valid* according to the filesystem itself.
___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Real performance hit raidz2 5 disks vs 6?

[Edward Ned Harvey (openindiana)]

> From: Hans J Albertsson [mailto:hans.j.alberts...@gmail.com]
> 
> I will use it for media files over cifs and/or dlna, almost exclusively.
> 
> Thus it'll be almost consistently write once/read many, and most files will
> be large.

In your case, performance will be almost irrelevant.  Because even with a 
single disk, you would be able to do over 1Gbit/sec, but with your raidz or 
raidz2, you'll be able to go much faster.  But I bet the network you're 
connected to is 1Gbit or slower...  And...  Even if you're streaming the full 
BluRay disc or something, you only need about 8Mbit.  

So your demands will be far lower than even the lowest performance setup you 
could imagine creating.

___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Is this dell xeon a decent buy

[Edward Ned Harvey (openindiana)]

> From: Harry Putnam [mailto:rea...@newsguy.com]
>  
> http://www.ebay.com/itm/Dell-Poweredge-6650-Enterprise-Server-2-x-
> Intel-Xeon-1-5-GHz-8-GB-
> /261423732486?pt=COMP_EN_Servers&hash=item3cde119706

I didn't actually follow the link, so this might be irrelevant:

IMHO, don't use the motherboard integrated broadcom ethernet.  Use an add-on 
intel nic.

___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

[OpenIndiana-discuss] test from yahoo

[Edward Ned Harvey via openindiana-discuss]

asdf

___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Testing from myself

[Edward Ned Harvey (openindiana)]

again

> -Original Message-
> From: Edward Ned Harvey (openindiana)
> [mailto:openindi...@nedharvey.com]
> Sent: Friday, May 23, 2014 8:51 AM
> To: 'openindiana-discuss@openindiana.org'
> Subject: [OpenIndiana-discuss] Testing from myself
> 
> hi
> ___
> openindiana-discuss mailing list
> openindiana-discuss@openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss

___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

[OpenIndiana-discuss] Testing from myself

[Edward Ned Harvey (openindiana)]

hi
___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Critical security issue notification

[Edward Ned Harvey (openindiana)]

> From: Udo Grabowski (IMK) [mailto:udo.grabow...@kit.edu]
> 
> Moral: Never run a changing system !

Heheh, I hope the irony is intentional.  ;-)  Like "Never get vaccines, because 
sometimes vaccines cause problems."   ;-)  It's true that sometimes updates 
cause problems, but there are *more* problems without.  The irony of suggesting 
that 0.9.8 is better than 1.1.0...  If anybody cares, could be easily 
dismantled by just reading the changelog of the openssl releases... 
http://git.openssl.org/gitweb/?p=openssl.git;a=blob_plain;f=CHANGES;hb=HEAD

The latest 0.9.8 is 4 years old.  Since then, I see many security 
vulnerabilities fixed... CVE-2010-3864, CVE-2010-4252, CVE-2010-4180, 
CVE-2011-0014,  etc.  

Point is, as soon as there's any security vulnerability discovered, it both 
gets *published* so the world knows about it, and it also gets patched.  If you 
don't keep up with patches, you're literally publishing your vulnerabilities to 
the world, for everyone to see, and then sitting back and neglecting to patch 
it up.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] USB boot Sometimes?

[Edward Ned Harvey (openindiana)]

> From: Udo Grabowski (IMK) [mailto:udo.grabow...@kit.edu]
> 

So - Thank you, Udo and Wim.

I tried several USB2 thumb drives and USB2 hard drives, and they all worked.  I 
only have one USB3 device, and it doesn't work, even though the machine itself 
has USB2 and therefore USB3 isn't being used.  So I think most likely the 
correct explanation is as Udo said.  Most likely the USB3 thumb drive itself 
doesn't do good USB2 backward compatibility.  So anybody out there reading this 
- 

Either get the fastest USB2 device you can find, or get an assortment of USB3 
devices so you can hopefully find one that's backward compatible USB2.  

The non-bootable USB3 devices I have which don't work:  Mushkin Atom  
MKNUFDAM32GB

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

[OpenIndiana-discuss] USB boot Sometimes?

[Edward Ned Harvey (openindiana)]

I installed oi server to an old slow 4G usb thumb drive as a test, just to see 
if it's possible.  It worked fine; it's just slow as hell.  So I bought a pair 
of 32GB usb3 fast devices, and installed oi to one of them...  But grub fails.  
It just boots up to a grub menu and stops there.

So I thought maybe I need to upgrade my bios...  Upgraded, no effect.  So I 
thought maybe there's a device size limit that I'm exceeding...  So I hooked up 
an external USB 80GB hard drive.  I installed to the USB 80G and booted just 
fine.

Something smaller works.  Something bigger works.  The only thing that 
*doesn't* work is the thing I care about.  Can anybody shed any light?
___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] USB boot Sometimes?

[Edward Ned Harvey (openindiana)]

> From: w...@vandenberge.us [mailto:w...@vandenberge.us]
> 
> My guess would be that this is due to the lack of USB3 support in
> OpenIndiana.
> Have you tried plugging the drive into a USB2 port, forcing the device into
> USB2
> mode, and seeing if it works (I know that's not what you really want but it
> would narrow down the issue)

Good guess.  I checked - although the usb device is usb3, the machine hardware 
is usb2.  So I *think* that rules out the usb3 incompatibility possibility.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] How to let find 'see' .zfs directories

[Edward Ned Harvey (openindiana)]

> From: Timothy Coalson [mailto:tsc...@mst.edu]
> 
> You could instead test for the existence of the .zfs directory in all
> folders, with some kind of "find . -type d -exec 'test -d {}/.zfs'"

This is what zhist does under the hood.
It's not the same as 'find' but useful in a lot of cases.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] NFS

[Edward Ned Harvey (openindiana)]

> From: Ryan John [mailto:john.r...@bsse.ethz.ch]
> 
> Being reliant on NFS myself, I decided to test this. I just updated one test
> machine from OI_a8 to OI_a9, and another from OI_a7 to OI_a9
> Both machines give the same results, IE: NFSv3 works okay.
> My clients are RetHat EL6.

I wonder what version of nfs ships with rhel/centos6?  If you don't post back, 
I'll look it up later today and post.  It seems also, I'll have to start 
questioning if there's simply something wrong with my *individual* system, 
rather than the distro.  In other words, verify checksum of installation media, 
repeat on another system...

Also, I wonder if there's a difference between your system and mine, resulting 
from the fact you did an upgrade, whereas I installed fresh.  

Are you using OI desktop, or OI server?   I'm using OI server 151a9...   And 
I'm using OI desktop 151a7.


> OpenIndiana (powered by illumos)SunOS 5.11oi_151a9November 2013
> root@ openindiana:~# zfs create dataPool/nfstest
> root@ openindiana :~# zfs set sharenfs=on dataPool/nfstest
> root@ openindiana:~# sharectl set -p server_versmax=3 nfs

Did you remember to restart nfs server after making that change?

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] NFS

[Edward Ned Harvey (openindiana)]

> From: Edward Ned Harvey (openindiana)

It *appears* that NFSv4 is fine in both 151a7 and 151a9.
It *appears* that NFSv3 is broken in 151a9.  Which was unfortunately, necessary 
to support ESXi client and Ubuntu 10.04 client.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] NFS

[Edward Ned Harvey (openindiana)]

> From: Jonathan Adams [mailto:t12nsloo...@gmail.com]
> 
> to test if it is a permissions problem, can you just set sharenfs=on? and
> then try to access from the other machines?

Thanks for the help everyone.  I decided to take it a step further than that:

On both the 151a7 (homer) and 151a9 (marge) machines:
sudo sharectl set -p client_versmax=4 nfs
sudo sharectl set -p server_versmax=4 nfs
sudo svcadm restart nfs/server

On 151a9 (marge)
sudo zfs create storage/NfsExport
sudo zfs set sharenfs=on storage/NfsExport

On 151a7 (homer)
sudo zfs create storagepool/NfsExport
sudo zfs set sharenfs=on storagepool/NfsExport

Now, from ubuntu 10.04, try mounting them both:
(as root)
mkdir /mnt/151a7
mkdir /mnt/151a9

mount -v -t nfs marge:/storage/NfsExport /mnt/151a9
mount.nfs: timeout set for Thu Jan 30 16:00:50 2014
mount.nfs: text-based options: 'addr=10.10.10.13'
mount.nfs: mount(2): Input/output error
mount.nfs: mount system call failed

mount -v -t nfs homer:/storagepool/NfsExport /mnt/151a7
mount.nfs: timeout set for Thu Jan 30 16:01:29 2014
mount.nfs: text-based options: 'addr=10.10.10.242'
homer:/storagepool/NfsExport on /mnt/151a7 type nfs (rw)

(Notice, it worked for 151a7, and failed for 151a9)

Now, I'll have the two OI machines mount each other - which should pretty well 
answer any questions about firewall and RPC ports, etc.

(151a7 machine mounting 151a9)  (Success)
sudo mount -F nfs marge:/storage/NfsExport /mnt
eharvey@homer:~$ df -h /mnt
FilesystemSize  Used Avail Use% Mounted on
marge:/storage/NfsExport
   11T   31K   11T   1% /mnt

(151a9 machine mounting 151a7)  (Success)
sudo mount -F nfs homer:/storagepool/NfsExport /mnt
eharvey@marge:~$ df -h /mnt
FilesystemSize  Used Avail Use% Mounted on
homer:/storagepool/NfsExport
  4.3T   44K  4.3T   1% /mnt

Now dismount them all, on all machines.

Reduce the versions to 3, (on both 151a7 and 151a9)
sudo sharectl set -p client_versmax=3 nfs
sudo sharectl set -p server_versmax=3 nfs
sudo svcadm restart nfs/server

And try again.

Attempt to mount again from ubuntu client.  Once again, 151a7 works and 151a9 
fails.

Attempt to mutually mount 151a7 to 151a9 and vice-versa...

151a7 cannot mount 151a9
sudo mount -F nfs marge:/storage/NfsExport /mnt
nfs mount: marge: : RPC: Rpcbind failure - RPC: Authentication error
nfs mount: retrying: /mnt
nfs mount: marge: : RPC: Rpcbind failure - RPC: Authentication error

151a9 mounts 151a7 just fine
sudo mount -F nfs homer:/storagepool/NfsExport /mnt
eharvey@marge:~$ df -h /mnt
FilesystemSize  Used Avail Use% Mounted on
homer:/storagepool/NfsExport
  4.3T   44K  4.3T   1% /mnt

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] NFS

[Edward Ned Harvey (openindiana)]

> From: Jonathan Adams [mailto:t12nsloo...@gmail.com]
> Sent: Thursday, January 30, 2014 4:06 AM
> 
> If a share was mounted on the client and you change the underlying NFS
> version on the server then you will need to get the client to unmount all
> shares from the server before they can see the version 3 shares ... is this
> the case in your instance?
> 
> Are your shares auto-mounted? if so it depends on which system you're
> using
> but it might be quicker to reboot ... :(

For now, I'm just trying to make it work.  Later I can make it automount or 
whatever, so at present, it goes like this:

(On server)
sudo zfs set sharenfs=rw=@10.10.10.14/32,root=@10.10.10.14/32 storage/ad1

(on ubuntu 10.04 client)
root@orion:~# mount -v -t nfs storage1:/storage/ad1 /mnt
mount.nfs: timeout set for Thu Jan 30 09:01:24 2014
mount.nfs: text-based options: 'addr=10.10.10.13'
mount.nfs: mount(2): Input/output error
mount.nfs: mount system call failed

Since that didn't work, I try with ESXi 5.5 client, I repeat, eliminating the @ 
symbols, eliminating the /32, and putting them back in there...  Set the 
versions as follows:
sudo sharectl set -p server_versmax=3 nfs
sudo svcadm refresh  svc:/network/nfs/server:default

Retry all the variations of set sharenfs and repeat trying to mount...  Still 
nothing works...

I wondered if maybe I have firewall enabled on the server.  So I used "nc" and 
"telnet" from the client to confirm the port is open.  (111 and 2049).  No 
problem.

The only thing that *does* work:  When I have the 151a7 box mount the 151a9 box 
using nfs v4, then it works.  But if I reduce the server and client both to v3, 
then even THEY fail to mount too.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

[OpenIndiana-discuss] NFS

[Edward Ned Harvey (openindiana)]

At home, I have oi_151a7 and ESXi 5.1.
I wrote down precisely how to share NFS, and mount from the ESXi machine.
sudo zfs set sharenfs=rw=@192.168.5.5/32,root=@192.168.5.5/32 
mypool/somefilesystem

I recall it was a pain to get the syntax correct, especially thanks to some 
innacuracy in the man page.  But I got it.

Now at work I have oi_151a9 and ESXi 5.5.  I also have oi_151a7 and some ubuntu 
12.04 servers and centos 5 & 6 servers.

On the oi_151a9 machine, I do the precise above sharenfs command.  Then the 
oi_151a7 machine can mount, but both centos, ubuntu, and ESXi clients all fail 
to mount.  So I think, "ah-hah!"  That sounds like a NFS v3 conflict with v4!

So then I do this:
sudo sharectl set -p client_versmax=3 nfs
sudo sharectl set -p server_versmax=3 nfs
sudo svcadm refresh  svc:/network/nfs/server:default

Now, *none* of the clients can mount.  So I put it back to 4, and the 
openindiana client can mount.

Is NFS v3 simply broken in the latest OI?

When I give the "-v" option to mount, I get nothing useful.
There is also nothing useful in the nfsd logs.

The only thing I have left to test...  I could try sharing NFS from the *old* 
oi server, and see if the new ESXi is able to mount it.  If ESXi is able to 
mount the old one, and not able to mount the new one, that would be a pretty 
solid indicator v3 is broken in 151_a9.
___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Expanding storage with JBOD

[Edward Ned Harvey (openindiana)]

> From: Dave Pooser [mailto:dave...@pooserville.com]
> Sent: Tuesday, January 7, 2014 12:22 PM
> 
> Sans Digital makes one-- their part number TR4M6GNC, selling at NewEgg for
> $134.99. I'd never use one myself, because they seem to be a recipe for
> weird flakiness and I/O hangs, and ZFS does not deal well with SATA kinda-
> sorta-maybe failure modes. But they exist.

Thanks for pointing that out.  Interestingly, the product I was looking at was 
the Sans Digital ES104T, which is just a chassis power supply, and four ESATA 
pass-thru cables going to the individual drives.

I know ZFS works fine with SATA in general, as long as there isn't some junky 
SATA chip or driver in the mix.  So the flakiness you described might be on a 
different product, or it might be introduced by the SATA multiplier in that 
system.  The aforementioned ES104T, I would have absolute faith in, because 
it's literally brainless.   ;-)   And hence comes with the disadvantage of 
requiring the 4 separate cables.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Expanding storage with JBOD

[Edward Ned Harvey (openindiana)]

> From: Saso Kiselkov [mailto:skiselkov...@gmail.com]
> 
> Nope, SATA does have port multipliers, though I agree that beyond a certain
> point it becomes a mess.

Now that you mention it, when I look around, everything that I find called a 
SATA port multiplier is some sort of add-on card that takes an upstream SATA 
port and gives you several downstream SATA ports.  I don't see the point.  If 
you have room for an add-on card, I would think, it would make more sense to 
just add another SATA adapter to the PCIe bus.

The main point is, a port multiplier might be useful if it's *outside* your 
system.  Supposing you have an external enclosure that holds 4 drives or 
something like that...  You might say, "I have SATA 6Gbit bus which is good 
enough for the 4 drives in that enclosure."  And I would tend to agree, if they 
let you run a single 6Gbit SATA cable to the enclosure and then internally they 
use a SATA port multiplier, that would be pretty nice.  But when I look around, 
I don't see any 4-drive enclosures that use a single upstream 6Gbit ESATA 
bus...  I see enclosures with 4 bays, and 4 ESATA ports.  One for each drive.  
By comparison, the same exact class of products certainly exist, with 4 drives 
(or even 16 or 24) on a single SAS (Mini-SAS, SFF-8088) cable.  Internally, the 
enclosures have SAS Expander to fanout to the drives.
___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Expanding storage with JBOD

[Edward Ned Harvey (openindiana)]

> From: Hans J Albertsson [mailto:hans.j.alberts...@branneriet.se]
> Sent: Tuesday, January 7, 2014 5:01 AM
> 
> I thought that might be what you were talking about, but I wanted to be
> sure.
> Shelf, fix icy box and PS, short pins by clips-and-wire and add som isolation
> for safe op?
> Can you suggest a cheap and commonly available sata pci card that can work
> in an X7BSi mb based computer?

This has been a long thread, that has confused me a bit.  Can you remind us 
what you're trying to do?

Since I'm currently doing this, and learned the hard way, I'd like to inform 
you:

Suppose you have a computer chassis, with enough internal SATA ports, but not 
enough internal space to hold drives.  So you want to add external drives.  So 
you dangle a SATA cable out the computer, and you want to power an external 
drive using an external ATX power supply.  (This is what I did).

Then you short the pins, as Saso suggested, in order to get the ATX power 
supply to turn on.  BUT you MUST connect the external ATX ground to the 
computer ground.  Simply sharing the common ground pin on your wall outlet 
isn't enough.  (I fried two of my hard drives.  That's how I learned the hard 
way.)  The easiest way is to simply take out one of the screws of the ATX power 
supply, put an eyelet tipped wire there (with the screw put back in obviously) 
and connect the other end of the wire to the computer chassis.  Just wire the 
two chassis' together.

If you look at an ESATA cable, it's exactly the same as the SATA cable, except 
that it includes an external ground connector.  If you buy something like a 
Sans Digital (I like the brand) ESATA external enclosure connected via ESATA, 
then the ESATA cable guarantees the external chassis and internal chassis will 
share the common ground.

If all you want to add is a couple (up to maybe 4) drives, then SATA works 
fine.  Beyond that point, SAS becomes the obvious better solution.  Because in 
SATA, you have to run a separate cable for every drive.  While in SAS, you get 
24 Gbit in a single cable.  (4 channels of shared 6Gbit)
___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Expanding storage with JBOD

[Edward Ned Harvey (openindiana)]

> From: Roman Naumenko [mailto:ro...@naumenko.ca]
> 
> I don't know if even 2TB will fill fast enough to justifying any "investment"
> into storage expansion.

I don't get that comment.


> Speaking about storage expansion, even HBA cards are dirt cheap, pricing on
> enclosures with integrated SAS expander is just nuts. Can't figure out how to
> add 8-12 disks externally to the storage server without paying for this 2x 
> cost
> of the server itself.

Depends how much you paid for the server itself.  ;-)  But that's kind of 
irrelevant.  Often, the storage *is* more expensive than the rest of the 
server.  Depends on how much storage you're adding.  There's a cost for the 
HBA, for the drive bay, and then a cost for the disks, multiplied by the number 
of disks.  So the disks add up quickly.

I'm a fan of Sans Digital products for the price.  And I'm a fan of using the 
disks that they recommend.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] ZFS keeps finding errors

[Edward Ned Harvey (openindiana)]

> -Original Message-
> From: Jim Klimov [mailto:jimkli...@cos.ru]
> Sent: Saturday, December 21, 2013 10:13 AM

Recycle.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] real-time syncing two directories across servers (unison ?)

[Edward Ned Harvey (openindiana)]

> From: Geoff Nordli [mailto:geo...@gnaa.net]
> 

Oh - since OP mentioned unison & bittorrent, I'll mention, I have a low opinion 
of unison, but siber goodsync is quite excellent.  Only problem is, it's not 
"real time."  It's something you run periodically or triggered, and in order 
for it to sync, it must scan the whole tree on both sides, and it doesn't 
understand permissions and ownership.  So it's probably not what you're looking 
for...

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] real-time syncing two directories across servers (unison ?)

[Edward Ned Harvey (openindiana)]

> From: Gregory Youngblood [mailto:greg...@youngblood.me]
> 
> Check out owncloud. The open source components might be useful.

I personally, and two other IT guys that I've spoken with from different 
companies, have been burned by placing any trust in owncloud.  In fact, I'm 
still subscribed to their mailing list, and at least once a day, new people 
write in, asking for help, because it either fails to sync files it should 
sync, or some data seems to be lost.  My advice:  Steer clear.

In Microsoft, what you're looking for is called DFS.  I don't know any good 
linux/unix based bidirectional tools such as requested, but if you start 
searching for alternatives to DFS (or if you simply pay MS and deploy DFS) then 
you have an answer, and hopefully can find some FOSS solution out there.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] 10GigE vs Infiniband vs SCSI Target ...

[Edward Ned Harvey (openindiana)]

> From: jason matthews [mailto:ja...@broken.net]
>  
> does that help?

Thank you, what I was looking for was:  I want to connect the vmware servers to 
the openindiana server using SAS hardware.  Beat the performance of Ether, and 
not as expensive (or as difficult) as Infiniband.  Let the openindiana server 
present a zvol (or whatever) as a scsi target on the SAS bus, so as far as 
vmware can tell, there's just a hard disk on the other end of this SAS cable.  
Vmware would have no idea it was actualy a ZFS volume or anything.

I think Saso answered it.  "there was a SAS target driver for some LSI 
1068-based chips in the old 2009-era OpenSolaris days, but
ultimately that didn't lead anywhere and it fell by the wayside."  and "nobody 
appears to be working on a SAS target driver ATM, as far as I can tell."

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] 10GigE vs Infiniband vs SCSI Target ...

[Edward Ned Harvey (openindiana)]

No responses
Anybody?


> -Original Message-
> From: Edward Ned Harvey (openindiana)
> [mailto:openindi...@nedharvey.com]
> Sent: Monday, November 18, 2013 7:35 AM
> To: 'openindiana-discuss@openindiana.org'
> Subject: [OpenIndiana-discuss] 10GigE vs Infiniband vs SCSI Target ...
> 
> ZFS is great to manage backend storage in a SAN environment.  So then
> you're likely to use 10GigE, or Infiniband as the transport...
> 
> I only recently discovered SAS SFF-8088.  Gives you 4x 6Gbit buses yielding 24
> Gbit with very low overhead, low cost.  A lot of performance for the buck.
> 
> I also recently discovered Linux has something called SCST, a driver of sorts,
> that turns some linux HBA into a scsi target.  Does openindiana have
> something similar?  It would sure beat the pants off 10GigE, and while
> Infiniband would still be faster, it would be very useful to do the scsi 
> target
> thing for smaller systems...
> ___
> OpenIndiana-discuss mailing list
> OpenIndiana-discuss@openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

[OpenIndiana-discuss] 10GigE vs Infiniband vs SCSI Target ...

[Edward Ned Harvey (openindiana)]

ZFS is great to manage backend storage in a SAN environment.  So then you're 
likely to use 10GigE, or Infiniband as the transport...

I only recently discovered SAS SFF-8088.  Gives you 4x 6Gbit buses yielding 24 
Gbit with very low overhead, low cost.  A lot of performance for the buck.

I also recently discovered Linux has something called SCST, a driver of sorts, 
that turns some linux HBA into a scsi target.  Does openindiana have something 
similar?  It would sure beat the pants off 10GigE, and while Infiniband would 
still be faster, it would be very useful to do the scsi target thing for 
smaller systems...
___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] [zfs] problem on my zpool

[Edward Ned Harvey (openindiana)]

> From: Clement BRIZARD [mailto:clem...@brizou.fr]
> 
>   NAME STATE READ WRITE CKSUM
>   nas  UNAVAIL 63 2 0  insufficient 
> replicas
> raidz1-0   DEGRADED 0 0 0
>   c8t50024E9004993E6Ed0p0  ONLINE   0 0 0
>   c8t50024E92062E7524d0ONLINE   0 0 0
>   c8t50024E900495BE84d0p0  ONLINE   0 0 0
>   c8t50014EE25A5EEC23d0p0  ONLINE   0 0 0
>   c8t50024E9003F03980d0p0  ONLINE   0 0 1  (repairing)
>   c8t50014EE2B0D3EFC8d0ONLINE   0 0 0
>   c8t50014EE6561DDB4Cd0p0  DEGRADED 0 0   211  too many
> errors  (repairing)
>   c8t50024E9003F03A09d0p0  ONLINE   0 018  (repairing)
> raidz1-1   UNAVAIL131 9 0  insufficient 
> replicas
>   c50t8d0  REMOVED  0 0 0  (repairing)
>   c2d0 ONLINE   0 0 0  (repairing)
>   c1d0 ONLINE   0 0 0  (repairing)
>   c50t11d0 ONLINE   0 0 0  (repairing)
>   c50t10d0 REMOVED  0 0 0

Something bad happened.  You've had more failures than the redundancy level 
protects - which means some data is lost.

Sometimes other people here chime in with zdb debugging information and other 
stuff, that helps you recover gracefully - but the first and most obvious 
question is backups.  Do you have backups you could restore from?

Also, it's not clear what the cause of failure is.  Could be two disks 
coincidentally going bad at the same time.  Could be controller failure.  Or 
CPU or memory error.  Could be you're running on unsupported hardware with some 
kind of driver or firmware bug.  Even if you can restore, will the problem soon 
come back?  Not clear based solely on what info you've provided...

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Good enterprise hardware

[Edward Ned Harvey (openindiana)]

> From: Mark Creamer [mailto:white...@gmail.com]
> Sent: Sunday, October 20, 2013 12:19 AM
> To: Discussion list for OpenIndiana
> Subject: Re: [OpenIndiana-discuss] Good enterprise hardware
> 
> For whatever it's worth, I've been buying SM servers for the last couple
> years. I had a motherboard failure and they had a replacement to me next
> morning. I didn't have to ship anything except the defective board once I
> replaced it.
> Mark

Dang, this is a recent conversation between me and the would-be sales person at 
Silicon Mechanics:

> He said:
3yr 24x7 4Hr sameday on-site warranty
We don't recommend this option, the 4-hour response is meaningless unless you 
have spare parts on hand.  We recommend Next Business Day with a spares kit.. 
The tech will not come on site unless you have replacement parts on hand.  If 
you are able to replace a mem stick, HDD or add on card.. you should buy just a 
spares kit and do it yourself. IF it's a mother board or back plane failure, 
most likely the box will come back to us.
 
Our warranty
3 years return to depot/advance parts replacement when available.

> I said:
Shoot.  Normally, "4 hr same day onsite warranty" means the vendor keeps parts 
available, and will dispatch a technician with parts, same day, for any and all 
types of failures - motherboard, cpu, power supply, whatever.  This is critical 
for business critical servers.  Never send back to a depot.  Business can't 
afford the risk of downtime stretching on for days or weeks.  Nor can they 
afford to simply buy two of everything.

> He said:
Thanks for the refresher in services. I do this for a living and deploy 
hardware to some of the major data centers in the u.s. and all over the world 
for some major corporations that have found value in doing business with us and 
use the support services that I offered you.  The reality is that hardware 
fails and we design that hardware to be as redundant as your application 
requires.. I do recall that the first solution I submitted had redundacy in 
mind..but you did not  see the value in that.

(I don't know why he said that, cuz it's not correct)

We and most of the white box integrators offer support through 3rd party 
services. It would be impossible for them to show up with parts for every 
system out there...take a look at the supermicro web offering and you might 
understand...  The cost of a drive or a stick of memory over lets say 4 years 
are minimal. You can buy lots of spare parts for the cost of the support level 
you want..  By the way, return to depot with advace replacement is a pretty 
good thing..

My suggestion to you is to go with one of the boutique vendors that are more in 
line with your needs such as hp ibm etc

> I didn't bother replying to the above, but the systems he's quoting me are 
> on-par (slightly cheaper, but not dramatically) with the oracle systems 
> including that level of service.  So the idea of buying lots of spare parts 
> instead of enterprise support ... might hold true if you're buying a lot 
> cheaper stuff, but in the present situation, I'm not seeing it.

If he wanted to focus on "advance replacement" as you mentioned above Mark, 
then maybe there would be a good solution here.  If "advance replacement" means 
they ship you replacement parts super super fast - like within 4 hrs, or even 
next day (including non-business days) then there might be an acceptable level 
of support here.  But I didn't even get to the comment about "advance 
replacement is a pretty good thing" before I had already made up my mind based 
on attitude and lack of support...

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

[OpenIndiana-discuss] Good enterprise hardware

[Edward Ned Harvey (openindiana)]

I'm planning to build a ZFS storage server in the datacenter.  Mission critical 
storage for virtualization, requiring hardware support, 24/7, 4hr, sameday 
onsite.

I thought I was going with silicon mechanics, and just learned, that their 24/7 
4hr sameday service is pointless - because they don't get parts to you.  You 
have to buy spare parts including memory and whatever else, but if something 
like a motherboard or CPU dies, you still have to ship it back to the depot.

My next thought is, obviously I can get some good hardware with good support 
from oracle.  But I'd like to know what other alternatives there are.

Recommendations?
___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Opinions on LSI RAID vs. ZFS

[Edward Ned Harvey (openindiana)]

> From: Joshua M. Clulow [mailto:j...@sysmgr.org]
> 
> This is emphatically false.  Though the pestilence of cfgadm(1M) and
> the idea that device replacement is somehow advantageously manual had
> persisted inside Sun's walls until its untimely demise, SunOS itself
> is certainly capable of hot-plugging devices.
> 
> If you have a SAS controller attached via the mpt_sas driver, for
> instance, you can absolutely expect hot plugged disks to go away and
> return automatically.  If not, then you are experiencing bugs in the
> system and should report them as such so that we can fix them.
> 
> SATA disks attached to SATA controllers supported through the sata
> framework are something of a special case.  There is a tuneable that
> prevents SATA hot plug from working, which is unfortunately enabled by
> default -- again due to history and the misguided manual configuration
> crowd.  You can enable it in /etc/system, though.  The variable in
> question is "sata_auto_online", visible (with comment) here:

The system I just tested on is a Dell Precision workstation, and it uses 
sata.so.1.  I mostly mention that, because others here have mentioned assuming 
the use of mpt_sas, which is different, and uses some variation of " -x 
remove_device" instead of "-c disconnect"

The following procedure worked for me:
export EXTERNALDISK=c3t4d0
export SATAPORT=`sudo cfgadm -al | grep $EXTERNALDISK | sed 's/:.*//'` 
; echo "" ; echo "EXTERNALDISK is: $EXTERNALDISK" ; echo "SATAPORT is: 
$SATAPORT" ; echo ""

# Result:
# EXTERNALDISK is: c3t4d0
# SATAPORT is: sata0/4

sudo cfgadm -y -c unconfigure $SATAPORT
sudo cfgadm -y -c disconnect $SATAPORT

Now it is safe to disconnect your drive, and connect a new drive in its 
place.

After new drive is connected, do this:
sudo cfgadm -y -c connect $SATAPORT
sudo cfgadm -y -c configure $SATAPORT
sudo devfsadm -Cv

As Josh describes, perhaps the unconfigure/disconnect/connect/configure can be 
skipped, by tuning sata_auto_online.  But as far as I'm concerned, the present 
result is good enough.  I don't care to bother testing sata_auto_online.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Opinions on LSI RAID vs. ZFS

[Edward Ned Harvey (openindiana)]

> From: Aneurin Price [mailto:aneurin.pr...@gmail.com]
> 
> Is this in IDE emulation mode per-chance? If your controller acts this
> way in AHCI mode, it's defective and should be replaced

Alright, I'll try again.  Gimme a couple days to respond, as this is what I do 
for offsite data rotation.

I would never intentionally use IDE mode.  I would always use AHCI mode when/if 
available.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Opinions on LSI RAID vs. ZFS

[Edward Ned Harvey (openindiana)]

> From: Edward Ned Harvey (openindiana)
> 
> If it were truly hot plug, then the OS should get a drive
> disconnected signal, and c1t1d1 should not exist anymore.  As is the case
> with a USB drive or firewire.

(or hotplug capable HBA)

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Opinions on LSI RAID vs. ZFS

[Edward Ned Harvey (openindiana)]

> From: Stefan Müller-Wilken [mailto:stefan.mueller-wil...@acando.de]
> Sent: Wednesday, October 09, 2013 3:34 AM
> 
> ... so just to make sure I got that right: for the T3-1 you'd suggest to dump
> the PCIe HBA and go with the on-board SAS controller - even for production
> use? And for the DL380, where this is no option, place each drive in a single
> drive logical volum? And then, in both cases, build RAIDZ pools on top of
> that? Anyone a clue why our local Oracle dealer's tech support didn't even
> mention this option?

I think there's really only one answer for you:  You have to understand the 
limitations of each option, and make up your own mind based on what you care 
about.

You should definitely find a solution that handles the raid in ZFS instead of 
in hardware.  Due to data integrity capabilities of ZFS.

If you are forced to use a single volume raid-1 or raid-0 in hardware, be aware 
that your disks are likely not portable to another system, unless the other 
system has the same type of HBA.  For this reason, if your HBA supports JBOD, 
then JBOD is preferred.

With or without HBA, consider the hotplug capabilities of your system, and the 
red blinking light capabilities.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Opinions on LSI RAID vs. ZFS

[Edward Ned Harvey (openindiana)]

> From: Christopher Chan [mailto:christopher.c...@bradbury.edu.hk]
> Sent: Tuesday, October 08, 2013 8:42 PM
> 
> Er...isn't hotswap capability PART of the specs whether the drives are
> SAS or SATA? I can do this on a cheap desktop motherboard but you cannot
> on a server board with getting a HBA?

Maybe you're using a different bios, or maybe you're hotswapping like for like 
drives that coincidentally work in your situation or something, but ... 

In BIOS, I have the option to enable/disable SATA port 0, 1,2,3.  If the port 
is enabled and nothing connected, it throws and error during POST.  If 
something is connected, it's identified as a 1TB or whatever drive, and 
presented to the OS as c1t1d1 or whatever.  Later if I disconnect that drive 
while the OS is running, the OS still thinks c1t1d1 exists, but if I try to 
access it, I'll get an IO error.  If it were truly hot plug, then the OS should 
get a drive disconnected signal, and c1t1d1 should not exist anymore.  As is 
the case with a USB drive or firewire.

And if I stick a different drive on that line, it *may* behave correctly, but I 
won't trust it.  The chances of correct behavior are improved if you're 
swapping the drive for another of the same model, but I seem to recall bad 
behavior when swapping a 1T drive for a 2T drive or 500G.

If it works for you, I say great.  I only feel comfortable with using something 
that truly supports hotplug, or rebooting.

PS.  I accidentally fried a mac laptop SATA hard drive once, when I thought the 
computer was off (it was actually asleep).  I was transplanting the hard drive 
from one laptop to another, due to a broken screen.  The moment I pulled the 
SATA hard drive out of the old laptop, while it was still powered on, that hard 
drive was fried and never again functional.  Wouldn't spin up anymore (or if it 
spun up, it was never recognized as a drive ever again.)

Back at that time, I looked it up, and found as you said, hot plug is 
supposedly incorporated into the SATA spec.  But it's poorly implemented, 
rarely tested, and not to be relied upon, unless you have another layer beneath 
it (such as truly hotplug capable HBA) which will make up for the deficiencies 
of the SATA hotplug.  Basically, if your product was *intended* to be used for 
hotplug, and it advertises itself as hotplug, then it's hotplug.  But if you're 
just blindly assuming all motherboard internal built-in SATA adapters support 
it ... you're taking your data into your own hands.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Opinions on LSI RAID vs. ZFS

[Edward Ned Harvey (openindiana)]

> From: Saso Kiselkov [mailto:skiselkov...@gmail.com]
> 
> > I find, with the motherboard built-in sas controller, you usually need to
> power off in order to swap a drive, and you need to "devfsadm -Cv" after
> coming up, in order to make the new drive available.
> 
> What kind of a messed up SAS controller would that be? I've yet to see
> this happen to me with any on-board SAS controller (in fact, all
> on-board SAS controllers that I've encountered so far were LSI chips,
> just placed on the motherboard instead on a plug-in board).

The on-board SAS controller isn't externally facing, and not expecting hotswap 
drives.  Drive detection is performed by BIOS during POST.  Dell optiplex, 
precision, and poweredge.  I have all 3 sitting in my basement right now, with 
SATA cables dangling out the chassis through holes I cut, in order to attach 
removable disks externally.  It all works but I have to power off the host 
while connecting/disconnecting drives.

The poweredge server also has an HBA (basic SAS) with hotswappable drives on 
the front.  These do not require any poweroff to swap the drives, as the name 
suggests.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Opinions on LSI RAID vs. ZFS

[Edward Ned Harvey (openindiana)]

> From: Peter Tribble [mailto:peter.trib...@gmail.com]
> 
> This is where I get a little confused. Why an extra HBA at all - what's
> wrong with using the SAS ports on the system board?

My usual reason for using an HBA is for hot plug, and red blinking lights on 
failed drives.  If you have degraded redundancy, the last thing you want is to 
accidentally remove the wrong drive.

I find, with the motherboard built-in sas controller, you usually need to power 
off in order to swap a drive, and you need to "devfsadm -Cv" after coming up, 
in order to make the new drive available.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Opinions on LSI RAID vs. ZFS

[Edward Ned Harvey (openindiana)]

> From: Stefan Müller-Wilken [mailto:stefan.mueller-wil...@acando.de]
>  
> Question now: what would you recommend? 8 LSI RAID0 LVs under ZFS, 8
> drives under LSI RAID-5, or switch to an Oracle certified controller with JBOD
> mode (which one??)? Does it make sense to go for Soft RAID anyway, with
> that configuration?

As you said, JBOD is best.  But if you're running on HBA's that don't support 
JBOD, you need to use a bunch of RAID-0 volumes with single disks in them.  

You said something about can't do failover, hotspare, or something, with the 
RAID-0 volumes.  I suggest you look at that again.  There is nothing preventing 
you from failing over to hotspares etc, just because you're using RAID-0 on 
your HBA.

If you can get the controller that supports JBOD, then great.  The main 
advantage of that is portability of those drives to other systems.  When an HBA 
requires making drives into RAID-0, the HBA necessarily stores metadata in 
something similar to a partition scheme, before presenting the remainder of the 
disk to the OS.  This has the side effect of making the drives unusable if you 
happen to require taking them out and attaching to a different system.  (In the 
event of an HBA failure for example.)  

The JBOD vs RAID-0 advantage is both simplicity of management, and better 
portability.  But if you have to live with RAID-0, it's completely possible.

JBOD or RAID-0 with ZFS...  Both ways are *far* better than hardware controlled 
RAID-5.
___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] RAM based devices as ZIL

[Edward Ned Harvey (openindiana)]

> From: Steve Gonczi [mailto:gon...@comcast.net]
> 
> For a fast (high ingest rate) system, 4G may not be enough.
> 
> If your Ram device space is not sufficient to hold all the in-flight Zill 
> blocks,
> it will fail over, and the Zil will just redirect to your main data pool.
> 
> This is hard to notice, unless you have an idea of how much
> data should be flowing to your pool, as you monitor it with
> zpool iostat. Then you may notice the extra data being written to your
> data pool.
> 
> The calculation of how much Zil space you need is not straight forward,
> because blocks in general are freed in a delayed manner.
> 
> In other words, it is possible that the some Zil blocks are no longer needed
> because the transactions they represent already committed, but the blocks
> have not made it back to "available" status because of the conservative
> nature of the freed block recycling algo.
> 
> Rule of thumb, 3 to 5 txg-s worth of ingest, depending on who you ask.
> 
> Dedup and compression makes Slog sizing harder, because the Zil is neither
> compressed nor deduped. I would say if you dedup and / or compress,
> all bets are off.

The part that's missing from the above is discussion of size of ZIL vs the 
speed of the pool.  (And the nature of your writes.)  

If your pool, for example, is made of a single disk (or a single mirror) then 
the maximum speed you'll ever be able to write is 1 Gbit/sec, or 1Gbyte in 8 
seconds.  Even if you have a pool of hundreds of disks, each SSD that you might 
consider using as ZIL has approximately the same limitation - approx 1 Gbyte in 
8 seconds.  Maybe it's faster, call it 1 GB in 5 seconds.  Conveniently the 
same as the TXG flush interval.  So even with a 4x factor worked in as Steve 
mentioned above, you're still going to use only 4GB on the ZIL device.  (But as 
he said, writing a lot of highly compressible sync writes might be a factor.)

But even those numbers are unrealistic to attain.  I forget the name of the 
parameter, but there's an evil tuning parameter that says, sync mode writes 
above a certain size will not go to the log device - they will immediately go 
into the next TXG and trigger an immediate flush.  This means, the only sync 
mode writes that *actually* hit the log device are small in size, and your 
aggregate throughput to the device can never allow you to write anywhere near 
4GB to the device.

Basically, this holds true unless your workload is specifically designed to 
violate it.  If you have a multithreaded infinite loop of really small random 
sync mode writes that are highly compressible...  Then you might hit the limit.

If your work load is serving NFS or iscsi ... which are among the most 
demanding sync write services you can provide ... the bottleneck is going to be 
your network.  Even with infiniband or 10GB ether, most of the actual workload 
is going to be larger blocks, and it will be extremely tough to get a 
sufficient level of small sync write blocks to add up to more than 4GB within 
the speed of the devices.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] trying to get 4K aligned root pool on ESX

[Edward Ned Harvey (openindiana)]

> From: Steve Goldthorpe [mailto:openindi...@waistcoat.org.uk]
> Sent: Sunday, August 18, 2013 12:23 PM
> 
> No matter what I try I can't seem to get a 4K aligned root pool using the
> OpenIndiana installer (oi151-a7 live image).
> I'm using ESXi using 4K aligned disks.  

You shouldn't put ESX volumes in rpool.  Create a separate pool - if necessary 
first partition your disk - and put the volumes into the second pool.

If you partition your disk, you should be aware that zfs only enables the disk 
hardware write-back cache for systems where zfs has control of the whole disk.  
So you might have to script something with "format" to enable the disk cache on 
those devices.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] VMware

[Edward Ned Harvey (openindiana)]

> From: James Relph [mailto:ja...@themacplace.co.uk]
> Sent: Monday, August 12, 2013 4:47 PM
> 
> No, we're not getting any ping loss, that's the thing.  The network looks
> entirely faultless.  We've run pings for 24 hours with no ping loss.

Yeah, I swore you said you had ping loss before - but if not - I don't think 
ping alone is sufficient.  You have to find the error counters on the LACP 
interfaces.  Everybody everywhere seems to blindly assume LACP works reliably, 
but to me, simply saying the term "LACP" is a red flag.  It's extremely 
temperamental, and the resultant behavior is exactly as you've described.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] VMware

[Edward Ned Harvey (openindiana)]

> -Original Message-
> From: James Relph [mailto:ja...@themacplace.co.uk]
> Sent: Sunday, August 11, 2013 10:59 AM
> 
> although would we lose pings
> with that (had pings running to test for a network issue and never had packet
> loss)?  It's a bit of a puzzler!

Hold on now ...


> From: James Relph [mailto:ja...@themacplace.co.uk]
> Sent: Sunday, August 11, 2013 12:59 PM
> 
>dedicated physical 10Gb network for iSCSI/NFS traffic, with 4x 10Gb
> links (in an LACP bond) per device.  Should be pretty solid really.

I think we found your smoking gun.  You're getting ping loss on a local 
network, and you're using 4x 10Gb LACP bonded network.  And for some reason you 
say "should be pretty solid."  What you've described is basically the 
definition of unstable, if you ask me.

Before anything else, know this:  In LACP, only one network interface can be 
used per data stream.  So if you have a server with LACP, then each client can 
go up to 10Gb, but if you have 4 clients simultaneously, they can each go up to 
10Gb.  You cannot push 40Gb to a single client.

Also, your hard disks are all 1Gbit.  So every 10 disks you have in the server 
add up to a single 10Gb network interface.  It is absolutely pointless to use 
LACP in this situation unless you have a huge honking server.  (Meaning >40 
disks).

In my experience, LACP is usually unstable, unless you buy a really expensive 
switch and QA test the hell out of your configuration before using it.  I hear 
lots of people say their LACP is stable and reliable where they are - but it's 
only because they have never tested it and haven't noticed the problems.  The 
problems are specifically as you've described.  Occasional packet loss, which 
people tend to think is ok, but in reality, the only acceptable level of packet 
loss is 0%.

Here's what you need to do:

Figure out how to observe & clear the error counters on all the network 
interfaces.  Login to the switch to measure them there ...  Login to the server 
to measure them there ...  Login to each client to measure them there.  Reset 
them all to 0.  And then start hammering the shit out of the whole system.  Get 
all the clients to drive the network hard, both transmit and receive.  If you 
see error counters increasing, you have a problem.

Based on what you've said so far, I guarantee you're going to see error 
counters increasing.  Unless you ignore my advice and don't do these tests ... 
because these tests are difficult to do, and like I said, several times I've 
seen sysadmins swear their system was reliable, only to be proven wrong when 
*actually* put to the test.

I also encounter a lot:  Mailing lists exactly like this one, I say something 
just like above, and other people come back and argue about it, *insisting* 
that it's ok to have occasional packet loss, on a LAN or WAN.  I swear to you, 
as an IT consultant, this provides a lot of my sustenance - I get called into 
places with either storage problems or internet problems, and if there is 
packet loss >0% that is ultimately the root cause of their problem.  Never seen 
an exception.

Because this argument invariably leads to argument, I won't respond to any of 
it.  I've simply grown tired of arguing about it with other people elsewhere.  
It's definitely a trending pattern.  The way I see it, I provide you free 
advice on a mailing list, if you don't take it, so be it.  I continue to get 
paid.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] VMware

[Edward Ned Harvey (openindiana)]

> From: James Relph [mailto:ja...@themacplace.co.uk]
> Sent: Saturday, August 10, 2013 6:12 AM
> 
> Is anybody using Oi as a data store for VMware using NFS or iSCSI?

I have done both.  What do you want to know?

I couldn't measure any performance difference nfs vs iscsi.  Theoretically, 
iscsi should be more reliable, by default setting the refreservation and 
supposedly guaranteeing there will always be disk space available for writes, 
but I haven't found that to be reality.  I have bumped into full disk problems 
with iscsi just as much as nfs, so it's important to simply monitor and manage 
intelligently.  And the comstar stuff seems to be kind of unreliable, not to 
mention confusing.  NFS seems to be considerably easier to manage.  So I would 
recommend NFS rather than iscsi.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Using OI and zfs from a windows machine

[Edward Ned Harvey (openindiana)]

> From: Robbie Crash [mailto:sardonic.smi...@gmail.com]
> 
> I always see this bandied about. Following the Oracle documentation on how
> to join OI to a domain for the built in CIFS serving has worked for me,
> flawlessly on 10 different OI installations.
> 
> Every time I hear about people with issues with it, they're always using
> Samba. What benefit does using an additional module have over the built in
> CIFS server?

I wasted ages, following oracle documentation, configuring sharesmb, getting 
crap for results.  Adding a little more info - One of the applications I 
support is Acronis TrueImage.  A ghost-like whole-system backup software you 
run inside windows.  When using sharesmb, I was able to access the server just 
fine to create backups, but that's only half the requirement.  You need to also 
be able to boot from the rescue media, which is a dos-like environment, and 
then *read* the backup.  For this, sharesmb was simply failing.  Not making the 
server visible on the network.  If you don't care about that, there's a good 
chance your sharesmb experience would be better than mine.


> Is it just that people want to use smb.conf instead of
> managing shares through zfs set sharesmb?

That wasn't a goal, but it turned out to be a positive side-effect.

I found, if I were to manage permissions via solaris ACL's, the implementation 
is completely different from anything else - so I would have to learn yet 
another platform specific idiosyncrasy.  Which I would be willing to do, but 
don't have any specific desire to do.  When I finally gave up and went for 
samba, I was happy that I didn't need to learn anything.

I seem to recall, there is a simple option to turn off ACL's and use posix 
permissions instead.  Which is significantly simpler, if you don't need ACL 
granularity.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Using OI and zfs from a windows machine

[Edward Ned Harvey (openindiana)]

> From: Harry Putnam [mailto:rea...@newsguy.com]
> 
> When I did run OI, I had recurring problems with various permissions
> type problems when accessing the zfs server from windows.

Technically supported.  I personally wouldn't trust ... and your complaint adds 
validation to my superstition.

I personally configure samba to not join any windows domain.  Just have local 
user accounts inside the samba box.  Manage all the perms with chown/chmod.  
Keep it simple and well within the beaten path.


> Working from windows across a gigabit network on a zfs server.  Didn't
> seem to work well... I'm talking about running adobe tools on files
> from windows when the files are on a zfs server across the network.
> 
> The problems I remember best were sloth and freezups on the windows
> machine.
> 
> So, all I really want to know right now is if that should be entirely
> possible .. and if there are users here who do that daily that can
> vouch for it.

I do it regularly.  The trick is configuring samba.  I found the built-in cifs 
server to be crap, and switched to the actual samba service.  The reason it's 
tricky to configure is cuz ... OI doesn't come with any sample config files, 
and try as I might, I simply can't get SWAT to run on OI.  So I created a linux 
machine, ran swat on the linux machine, and then copied over the smb.conf file 
and destroyed linux.

I'll happily provide my documentation, on how I configured my server, if 
desired.


> Or maybe gigabit just isn't a big enough pipe for heavy graphics work?

With a 1Gbit link, you should get performance very comparable to a single 
locally attached disk.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] running VirtualBox headless

[Edward Ned Harvey (openindiana)]

> From: Jan Owoc [mailto:jso...@gmail.com]
> 
> 1) in the case of "synchronous=off", I had an order of magnitude speed
> increase for writes (i.e. during the install :-) ). Are there general
> guidelines for what kinds of workloads are "safe" to have the ZIL
> disabled?

You mean "sync=disabled."   ;-)
It goes like this:  ZFS will aggregate writes into transactions, which will 
then be atomically flushed to disk.  So under no circumstance, does corrupt 
data get written to disk, nor in the wrong temporal order.  In the event of an 
ungraceful system interruption (kernel panic, power outage, etc) the on-disk 
data is always self-consistent, a snapshot of data that at some point was valid.

The risk is this:  In order to aggregate those transactions, there's a period 
of time (5sec) where data might be at risk because it exists solely in RAM that 
has yet to be flushed to disk.  As long as you can accept that risk, then you 
are safe to run with sync=disabled.  But it's not always easy to determine if 
you can safely accept that risk.  Read on:

Generally, if your system is a standalone system, that doesn't have any 
stateful clients watching it, then you're safe to disable sync.  But for 
example, if you have a bunch of NFS clients, and your ZFS server is a NFS 
server...  In the event your NFS server crashes and "rewinds" 5 sec, your NFS 
clients will all remember what they were doing at the time of the crash, and 
you'll have an inconsistent state between your server and clients.  You can 
remedy this situation by restarting all the NFS clients at the same time you 
restart the NFS server.  

The point is:  You need to first of all ask yourself if you can accept 5 sec of 
potential data loss in the event of a crash, and you need to secondly ask 
yourself, what services are being provided by the server, and if there is any 
stateful client that will notice or care, if the server were to suddenly crash 
and "rewind" as much as 5 sec.


> 2) I was able to get the machine to autostart using SMF, but it was a
> lot of work; should I expand that section of the wiki with my
> findings? I have no idea how to get it to "autoshutdown" when the
> machine is shutting down.

There is a section on the wiki discussing SMF.  You should look at vboxsvc and 
simplesmf.  Jim writes and maintains vboxsvc, and I write and maintain 
simplesmf.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] running VirtualBox headless

[Edward Ned Harvey (openindiana)]

> From: Jan Owoc [mailto:jso...@gmail.com]
> 
> I'm running a home NAS using OI 151a7 "server" (vs. "desktop"). I was
> thinking of running Ubuntu Server in a virtual machine on OI, ideally
> configured to startup/shutdown when OI starts/shuts down. I can
> connect a monitor to the machine, but it generally should run
> headless.
> 
> I found there is a very helpful entry on the wiki [1] describing most
> of the steps. Has anyone successfully installed VirtualBox on a
> headless OI and have any other tips before I dive in?
> 
> [1] http://wiki.openindiana.org/oi/7.2+VirtualBox

I do this quite a bit.  In fact, I'm a contributor to the wiki page you 
mentioned.  A long time ago, I tried doing this on "server" and found I 
couldn't get VirtualBox to install, or to run, or something like that.  So I 
switched to "desktop" and everything was smooth.  It was a long time ago and I 
don't recall details.  It could simply be that I hadn't yet figured out how to 
run headless, so I required X and gnome, and a desktop, and couldn't get that 
working.  Now that I think of it, I think that's the most likely truth.  
Hopefully it will go smoothly for you.  I'm also the author of simplesmf, which 
is mentioned on that wiki page.  I have some updated scripts that haven't been 
committed to the project yet, but I think they're an improvement, so if 
interested, let me know and I'll provide.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] ZFS read speed(iSCSI)

[Edward Ned Harvey (openindiana)]

> From: Jim Klimov [mailto:jimkli...@cos.ru]
> 
> > With 90 VM's on 8 servers, being served ZFS iscsi storage by 4x 1Gb
> > ethernet in LACP, you're really not going to care about any one VM being
> > able to go above 1Gbit.  Because it's going to be so busy all the time, 
> > that the
> > 4 LACP bonded ports will actually be saturated.  I think your machines are
> > going to be slow.  I normally plan for 1Gbit per VM, in order to be 
> > comparable
> > with a simple laptop.
> >
> > You're going to have a lot of random IO.  I'll strongly suggest you switch 
> > to
> > mirrors instead of raidz.
> 
> I'll leave your practical knowledge in higher regard than my theoretical
> hunches, but I believe typical PCs (including VDI desktops) don't do
> much disk IO after they've loaded the OS or a requested application.

Agreed, disk is mostly idle except when booting or launching apps.  Some apps 
write to disk, such as internet browsing caching stuff, and MS Office 
constantly hitting the PST or OST file, and Word/Excel autosave, etc.

But there are 90 of them.  So even "idle" time multiplied by 90 is no longer 
idle time.  And most likely, when they *do* get used, a whole bunch of them 
will get used at the same time.  (20 students all browsing the internet in 
between classes, or 20 students all doing homework between 5pm and 9pm, but 
they're all asleep from 4am to 6am, so all 90 instances are idle during that 
time...)


> And from what I read, if his 8 VM servers would contact the ZFS storage
> box with requests to many more targets, then on average all NICs will
> likely get their share of work, for one connection or another, even as
> part of LACP trunks (which may be easier to manage than VLAN-based
> MPxIO, with its separate benefits however). Right?..

Yup, with 8 VM servers, each having 11 VM guests, even if each server has a 
single 1Gb link to the 4x LACP storage server, I expect the 4x LACP links will 
see pretty heavy and well distributed usage.


> It might seem like a good idea
> to use dedup as well, 

Not if you care at all about performance, or usability.


> So here's my 2c, but they may be wrong ;)

:-)

I guess the one thing I can still think to add here is this:

If the 90 VM's all originated as clones of a single system, and the deviation 
*from* that original system remains minimal, then the ARC & L2ARC cache will do 
wonders.  Because when the first VM requests the boot blocks and the OS blocks, 
and all the blocks to launch Internet Explorer (or whatever app)...  Those 
things can get served from cache to satisfy the 89 subsequent requests to do 
the same actions.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] ZFS read speed(iSCSI)

[Edward Ned Harvey (openindiana)]

> From: Heinrich van Riel [mailto:heinrich.vanr...@gmail.com]
> 
> I will post my findings, but might take some time to fix the network in
> time and they will have to deal with 1Gbps for the storage. The request is
> to run ~90 VMs on 8 servers connected.

With 90 VM's on 8 servers, being served ZFS iscsi storage by 4x 1Gb ethernet in 
LACP, you're really not going to care about any one VM being able to go above 
1Gbit.  Because it's going to be so busy all the time, that the 4 LACP bonded 
ports will actually be saturated.  I think your machines are going to be slow.  
I normally plan for 1Gbit per VM, in order to be comparable with a simple 
laptop.

You're going to have a lot of random IO.  I'll strongly suggest you switch to 
mirrors instead of raidz.

And I'll strongly suggest adding a log device.  Your cache device might not 
give you benefit because 90 VM's might be too large to yield a substantial 
cache hit, but the log device almost certainly will benefit, as all your iscsi 
writes will be sync writes.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] ZFS read speed(iSCSI)

[Edward Ned Harvey (openindiana)]

> From: Christopher Chan [mailto:christopher.c...@bradbury.edu.hk]
> 
> I may be wrong but I don't think a single connection will be split over
> two interfaces in LACP to achieve higher throughput. If you have
> concurrent connections then perhaps you may see more throughput.

That is correct.  But there is a very good chance the iscsi initiator is 
intelligent enough to launch multiple threads.  However, the OP says Hyper-V 
system shows never more than 54% utilization, which tends to support this 
theory.  The MS system might be single-threaded, and therefore limited to 1Gbit 
on the 2x bonded LACP.


> zfs has to seek all over the place to find the snapshots which slows
> things down.

That is also correct, but only if the pool is heavily fragmented.  This will 
occur if you've had systems in production for a long time, creating, modifying, 
and deleting snapshots.  I may be wrong, but I assume the OP is deploying a new 
setup, which probably hasn't seen such dramatic use of snapshot fragmentation 
over time.


> From: Heinrich van Riel [mailto:heinrich.vanr...@gmail.com]
> 
> I have 2 x rz2 of 10x 3TB NL-SAS each in the pool.

Do you have any log devices?  I presume you have the default sync=enabled.


> When I copy to an iSCSI disk from a local disk, it copies at around 200MB/s
> and thats fine. 

I question this result.  Although 200MB/s (1.6Gbit/sec) is a reasonable rate 
(slightly slow) across 2Gbit network, you're writing to iscsi without any log 
device, and you should be hammered by sync operations.  I have a feeling you're 
seeing data written into memory cache, and not actually flushed to disk at this 
rate.

To really find out the actual throughput to disk, run "zpool iostat" on the 
server.  I would say "zpool iostat 30"


> When I copy from the iSCSI disk to the local disk I get no
> more that 80-90MB/s 

You have raidz3 on the server.  If you're doing sequential operations (read or 
write) then it should perform very well.  But if you're doing random 
operations, it should perform no better than a single disk.  (Probably half as 
good as a single disk.)  

If your system has been in production a while, getting fragmented, then you 
probably see a lot of random IO, as Christopher suggested.

If you're successfully using your LACP bonding with multiple threads, guess 
what, multiple threads implies random IO.  It's lose-lose.

How are you measuring your throughput?  If you want to see good throughput, 
make sure you're serially reading/writing sequential data.


> Even when I do a zfs send/recv is seems that reads are slower. I assume
> this is the expected behavior.

Now this I have to know more about.  Because you're obviously not doing zfs 
send/recv on Hyper-V.  Do you have another OI system connected over iscsi?

Also, the instantaneous performance of zfs send/recv varies quite a lot.  You 
have to watch it for several minutes, and see how high it peaks, and how low it 
bottoms out.


>* With this type of workload load would there be a noticeable
> improvement in by adding a cache disk?

Adding cache helps improve read performance.  Generally, the more cache, the 
better.  (But cache does leave a RAM footprint, so you want to have lots of RAM 
too).

Adding log helps improve sync writes.  If you're using iscsi, then basically 
all writes are sync writes.  It really doesn't matter how much log you add; the 
smallest size possible would be plenty.  (Even 4G.)  The only thing that 
matters is super fast IO, and obviously, nonvolatile.

If you want to temporarily measure the benefits of adding log, you can "zfs set 
sync=disabled" and see how fast it goes.  This will give you an upper bound.  
But for a VM environment over iscsi, it's not safe to leave it that way.


>* System has 2x 6core E5 2.4, 64GB mem; would compression help?

For most work loads, compression improves performance, but it depends on your 
work load.  Lots of uncompressible data is, of course, uncompressible.


>* Would it make more sense to create the pool with mirror sets?

A bunch of mirrors can do much faster random IO than a raidz.  But a raidz can 
do the same sustained serial speed for lower cost.  All of this is only 
relevant if your bottleneck is the actual storage, and not the network.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] shell script mutex locking or process signaling

[Edward Ned Harvey (openindiana)]

> From: Gary Mills [mailto:gary_mi...@fastmail.fm]
> 
> On Fri, May 31, 2013 at 01:52:01PM -0400, James Carlson wrote:
> >
> > Are your methods returning before the action they're supposed to take is
> > completely finished?  If so, then that sounds like the underlying
> > problem to me.  The methods are not supposed to exit until they're done
> > doing whatever it is they do.  If they do, then the design of that
> > service is incomplete.
> 
> That requirement is described briefly in the smf_method man page:
> 
>  The required behavior of a start method is to delay  exiting
>  until the service instance is ready to answer requests or is
>  otherwise functional.
> 
> If your method script starts a server process in the background and
> then immediately exits, it is broken.

Fortunately for me, I don't have that problem.  The start & stop scripts do 
indeed stay running until the service has reached a running or stopped state.  
The issue is with regard to concurrency - if a service takes a long time to 
start or stop, or if there's a problem with the start or stop, then the ability 
to launch "stop" and have it kill a "start" that's already in progress.  I 
swear, I've seen multiple instances of the script running concurrently before, 
but now I can't reproduce that.  At least, it can be run multiple times in 
parallel - manually.  If the admin issues "stop" while the "start" is still in 
progress (now that I've added this locking etc) then the start script dies 
gracefully, and the stop script does its thing.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] shell script mutex locking or process signaling

[Edward Ned Harvey (openindiana)]

> From: James Carlson [mailto:carls...@workingcode.com]
> 
> At least for me, when an SMF method takes time, I end up with that "*"
> marker in the svcs output, and it doesn't let me to do overlapping changes.

Huh.  Now I go explicitly check, and I see the same thing.  As long as the 
service is in transition state (with *) any subsequent calls to 
enable/disable/refresh the service seem to be completely ignored.  (I wonder 
what happens if you disable a service while it's in transition starting?  Does 
it queue the commands and immediately call the "stop" after the "start" 
finished?  Does it ignore your command and stay enabled?  Does it become 
disabled, even though it's started?)

Now I wonder how I ever saw multiple concurrent instances of the start method 
running.  Did I really do it manually, by running "/lib/svc/... start" and not 
remember that particular detail afterward?  Was it a behavior that changed from 
151a5 to 151a7?  (Why do I remember seeing the behavior?)  It seems unlikely 
that I would do that, and not remember it.  But I've been known to be a doofus 
from time to time.  So maybe.  Uggh, I hope not.

I swear, I saw multiple concurrent start methods running, frequently enough, 
that I kept making the mental note to implement locking.  Now I finally did it, 
and can't reproduce the problem.  How annoying.

For what it's worth, I just tested refreshing 50 times in rapid succession.  
SMF queues the requests and processes them serially.

So I have a new question (probably warrants a new thread).

Suppose you have a SMF service that could take a while to start, and suppose in 
the meantime, an admin is noticing it's going poorly.  And would like to 
disable the service.  (While it's in transition starting.)  Is there a simple 
SMF toggle property that will *allow* the service to be disabled (and call the 
stop method) while the start method is still running?

I've gone to the effort of writing all the locking BS now.  Only to discover I 
can't use it.   Pt.   

At present, if there's a problem bringing up some machines, and I want to 
terminate the virtualbox guest startup process, I guess I can call 
"/lib/svc/method/...  stop"  But sometimes that's hard to remember under 
pressure.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] shell script mutex locking or process signaling

[Edward Ned Harvey (openindiana)]

> From: Udo Grabowski (IMK) [mailto:udo.grabow...@kit.edu]
> 
> If you don't understand SMF (which is a bit clumsy, but indeed has
> all what you want), use this little generator, it will do the hard
> work for you:
> 
> 

That's a pretty cool generator.  But apparently, I understand SMF as well as it 
does.  Because it allows you to specify start & stop & refresh methods ...  But 
if those happen to take *time* to complete, and you start 
enabling/disabling/refreshing the service while the previous instances of 
enable/disable/refresh (start/stop/refresh) are still running, then multiple 
concurrent instances of those things get launched.  Which is how I got where I 
am now.

In any event, I went forward and implemented with "mkdir" locking.  It works, 
and usually will, except in the event of ungraceful things.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] shell script mutex locking or process signaling

[Edward Ned Harvey (openindiana)]

> From: Gary Mills [mailto:gary_mi...@fastmail.fm]
> 
> SMF is actually well documented, but you do have to jump around from
> man page to man page.  Start with `man smf'.  There are also lots of
> examples to follow, both of manifests and methods.  They are all text
> files.

Ok, so here's a quasi-recent example of a difficulty I've encountered trying to 
use SMF.  I have a service, which is configured for a single instance.  I then 
wanted to break it into individual instances, svc:foo and svc:bar.  I looked at 
examples, I did what I thought made sense, tried to import it, and SMF puked on 
the xml, saying something generic like "invalid configuration."  If SMF is 
actually well documented and good to work with, I need something to (a) guide 
me creating good xml, and (b) validate the xml, letting me know if something's 
wrong, and how to fix it.

Last I knew, there are a bunch of good html editors out there.  You start 
typing something, and based on context, the tool knows what's valid to use in 
the spot where you are working, so it will suggest and autocomplete tags, and 
properties inside of tags.  If you start typing  when you're not inside 
 or  they throw warning signs at you.  Last I knew, there isn't any 
such thing as a DTD aware XML editor.  So when I sit down and start typing XML, 
I have no idea what tags belong in the place where I'm typing.

In my example above, it turned out, I was putting the exec method before the 
dependency name, or something like that.  Order matters in XML, and I got it 
wrong just by trying to read and copy some example into my service manifest.  
To debug, I forget the exact process I followed, but I recall it being 
painfully iterative and manual.


> I'd recommend using the facilities of SMF, rather than trying to do it
> all outside of SMF.  These facilities are extensive and complete. 

You say SMF has capabilities that make this all go away.  But I read "man smf" 
and I don't see it there.  I don't know what to look for, and I'm not going to 
read the DTD from top to bottom, hoping to find something that fits the bill.

 
> Have you considered the contract facility?  It's used internally by
> SMF, but you can use it elsewhere as well.  The shell commands are
> ctrun(1), ctstat(1), ctwatch(1), and pkill(1).

There may be a solution there, but I'm not very familiar with solaris contract 
subsystem - it looks like you define the behavior of one process, and you use 
another process to monitor it.  If this is correct, it would make a very 
convoluted solution - A SMF service launches the "start" method, and while it's 
running, the same service launches the "stop" or "refresh" method ... Rather 
than executing the method directly, in each situation, utilizing contracts, the 
method would actually start a contract to monitor a sub-method for executing 
the "start," "stop," or "refresh."  And if the user (or system) is repeating 
calls to start/stop/refresh, each of these instances need to be made aware of 
each other, so the later method calls signal the earlier ones that they should 
terminate their contracts ...   *blah*

In any event, for the problem at hand in this thread, I used the easy solution:

Script starts.  Script uses mkdir $LOCKDIR which is /tmp/something
Script chugs along, and at select moments, checks for the existence of 
$BREAKLOCKDIR, which is /tmp/somethingelse
If a script starts and fails to get lock on LOCKDIR, then the script locks 
BREAKLOCKDIR and starts polling for the non-existence of LOCKDIR.
LOCKDIR is a signal that a script is already running.
BREAKLOCKDIR is a signal that a later process wants to steal lock.

If LOCKDIR becomes stale (for example, system power cycled while lock existed) 
any script that *has* lock guarantees to release it in less than 60 seconds.  
So if the BREAKLOCK script detects LOCK exists for more than 60 seconds, assume 
it's a stale lock and steal it forcibly.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] shell script mutex locking or process signaling

[Edward Ned Harvey (openindiana)]

> From: Laurent Blume [mailto:laurent...@elanor.org]
> 
> That's why I pointed out mine are in /tmp or /var/run - tmpfs, so it's
> guaranteed cleared on reboot, graceful or not :-)

The behavior of clearing out /tmp is a configurable feature, and the default 
varies by OS.  Some OSes clear it on every reboot, some clear it according to 
an aging schedule, some don't clear it.

I'm not sure what the default is for solaris / openindiana.  For the problem at 
hand (SMF service) obviously, it doesn't matter what the linux defaults are.   
;-)

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] shell script mutex locking or process signaling

[Edward Ned Harvey (openindiana)]

> From: Aneurin Price [mailto:aneurin.pr...@gmail.com]
> 
> I don't know about pure/POSIX shell, but at least bash and ksh support
> noclobber, which should do the trick. I've been using the following
> idiom for some time without problems:

I read somewhere (possibly obsolete, and also can't relocate) that noclobber on 
solaris doesn't always work right.  And in any event, I don't see the advantage 
of using noclobber on a file, versus using mkdir.  (Mkdir has the advantage of 
definitely being atomic, regardless of platform.)  Either way, you could still 
have stale locks left around (even with trap), after a "kill -KILL" or a 
kernel, storage, or power failure.

It would be *really* nice to have a locking mechanism that exists solely in 
ram, so it would go away and automatically release locks, in the event of a 
system ungraceful reboot.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] shell script mutex locking or process signaling

[Edward Ned Harvey (openindiana)]

> From: Gary Mills [mailto:gary_mi...@fastmail.fm]
> 
> On Thu, May 30, 2013 at 02:15:12PM +0000, Edward Ned Harvey
> (openindiana) wrote:
> >
> > Here's the problem I'm trying to solve: SMF service is configured
> > to launch things like VirtualBox during startup / shutdown.  This
> > startup process can take a long time (10, 20 minutes) so if there's
> > a problem of any kind for any reason, you might do things like
> > enable and disable or refresh the service ... but each time the
> > script gets launched, it's not aware of the previous one.
> 
> Are you saying that the method script itself might manipulate the
> service, or that the system admin might do it?  SMF has ways to
> prevent multiple instances of the method from running and to make
> enable and disable requests synchronous.  Can you do what you want
> within SMF, or does the method script had to do all the process
> manipulation?

Not familiar with that, but, if you can name some XML tags, or things to search 
for, I'd be willing to take a look.  I can say this: I'm biased to believe SMF 
is complex and confusing, mostly due to lack of understandable documentation 
surrounding the XML, and lack of any good XML tools to edit, validate, or 
otherwise produce good XML.   So I'm biased to expect it's probably better to 
use locking within the script.

Also, the desired behavior is NOT to prevent multiple scripts from running.  
The desired behavior is to let the later instances supersede the previous 
instances, letting the previous instances die gracefully (without putting SMF 
into mainetnance or anything like that.)  So a certain amount of shell script 
hackery is required, no matter what.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Moving ZFS pool between systems ?

[Edward Ned Harvey (openindiana)]

> From: Svavar Örn Eysteinsson [mailto:sva...@fiton.is]
> 
> So my question, is it as simple as "zpool export datapool" on the orginal
> machine
> and "zpool import" on the new one ?

Should be that simple.  Yes.  As long as you're not doing any proprietary 
hardware RAID on the disks, and you ensure the zpool capabilities on the new 
system are at least as good as the zpool capabilities on the old system.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

[OpenIndiana-discuss] shell script mutex locking or process signaling

[Edward Ned Harvey (openindiana)]

Here's the problem I'm trying to solve:  SMF service is configured to launch 
things like VirtualBox during startup / shutdown.  This startup process can 
take a long time (10, 20 minutes) so if there's a problem of any kind for any 
reason, you might do things like enable and disable or refresh the service ... 
but each time the script gets launched, it's not aware of the previous one.  
I'm really looking for a good way to allow multiple instances of some shell 
script to signal each other.  Let the earlier instances die and the later 
instances take over control.

This problem has two parts.  Atomicity of signaling operations (acquiring / 
releasing mutex, etc), and inter-process signaling.  (Let the later instance 
signal the earlier instance that it should die.)  It seems easy enough, as long 
as you have a good atomic operation for locking, the process that acquired lock 
can write its PID into a file, and later instances can "kill" that PID.

I'm looking around, and not finding any great answers.  So far, using mkdir, 
it's easy to see there exists a way to do mutex locking, and you could easily 
write your PID into the subdir that was just created; unfortunately, the 
problem is when a script gets killed, leaving the stale lock.  So I'm looking 
for something better than "mkdir" to use for locking.

I see there are a bunch of C constructs available ... mutex_init, etc.  Surely 
there must be a wrapper application around this kind of thing, right?

Thanks, everyone, for any suggestions you may offer.
___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] replacing an open solaris box

[Edward Ned Harvey (openindiana)]

> From: Kristoff Bonne [mailto:krist...@skypro.be]
>  
> Is there a list of applications that does work OK? As said, my
> requirements are not that special: thunderbird, firefox, virtual box.

thunderbird and firefox are included in the standard OS package repositories.  
So yes, those work.

Virtualbox definitely works too.  But I recommend reading the openindiana wiki 
page on virtualbox, because, although it will *work* simply, there are a lot of 
ways you can optimize it beyond the default.
http://wiki.openindiana.org/oi/7.+Virtualization

No, there isn't a "list" of software that works.  It's a fully functional OS, 
with thousands of packages in the standard distribution repositories, plus 
thousands of other products available from vendors who happen to support 
openindiana (or solaris) including VirtualBox.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] export pool at shutdown and import at boot up

[Edward Ned Harvey (openindiana)]

> From: Bill Sommerfeld [mailto:sommerf...@alum.mit.edu]
> 
> On 05/27/13 17:25, Edward Ned Harvey (openindiana) wrote:
> > I have some external storage which isn't super reliable.  If I don't
> > export it before shutdown, it will often cause the boot to fail, as
> > it doesn't mount properly.
> >
> > I would like to make that pool automatically export during shutdown,
> > or somehow flag it so the system will never try to import or mount it
> > during bootup.  ... is there a better way, like
> > setting some sort of pool property, or sticking a flag in some
> > defaults file, or tweaking the zfs cache file or something?
> 
> See the -c cachefile option to zpool import and the "cachefile" pool
> property; pools that aren't listed in the default system pool cache file
> aren't auto-imported at boot time.

Looks like this should do it:
zpool set cachefile=none externalpool

Or if I wanted to do it manually, 
zpool import -c none externalpool

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

[OpenIndiana-discuss] export pool at shutdown and import at boot up

[Edward Ned Harvey (openindiana)]

I have some external storage which isn't super reliable.  If I don't export it 
before shutdown, it will often cause the boot to fail, as it doesn't mount 
properly.

I would like to make that pool automatically export during shutdown, or somehow 
flag it so the system will never try to import or mount it during bootup.

My best idea is to write a smf wrapper around "zpool export" and "zpool 
import."  Is that the best way, or is there a better way, like setting some 
sort of pool property, or sticking a flag in some defaults file, or tweaking 
the zfs cache file or something?
___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] replacing an open solaris box

[Edward Ned Harvey (openindiana)]

> From: Nikola M. [mailto:minik...@gmail.com]
> 
> On 05/24/13 02:46 PM, Edward Ned Harvey (openindiana) wrote:
> >> From: Kristoff Bonne [mailto:krist...@skypro.be]
> >> Sent: Friday, May 24, 2013 5:27 AM
> >>
> >> What Operating System would you now advice for a personal workstation
> >> for "simple" work (telnet/ssh to other devices, perl, firefox,
> >> thunderbird, ...). Also important is the ability to run VirtualBox.
> > For personal workstation, openindiana desktop.
> And it is called illumos ;)

Ummm...  Illumos is the open source fork of the core of the opensolaris 
operating system.  But illumos itself is not an installable or usable OS.  It 
needs to be packaged into a distribution.  There are several distributions 
based on illumos - The one that most closely resembles opensolaris is called 
openindiana.  Openindiana has both a full GUI desktop version, and a server 
version.  (I always use the desktop version, even on servers, because I use 
VirtualBox, which has difficult to satisfy dependencies if installed on the 
"server" version.)  There are some other distros - most notably, SmartOS.  This 
one is specifically designed for specific functions in the server room, and if 
you don't know it, I suggest checking their webpage for more info.

For upgrading from opensolaris to openindiana, please see here:
http://wiki.openindiana.org/oi/OpenIndiana+Wiki+Home
Type in "opensolaris" into the search bar.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] replacing an open solaris box

[Edward Ned Harvey (openindiana)]

> From: Kristoff Bonne [mailto:krist...@skypro.be]
> Sent: Friday, May 24, 2013 5:27 AM
> 
> What Operating System would you now advice for a personal workstation
> for "simple" work (telnet/ssh to other devices, perl, firefox,
> thunderbird, ...). Also important is the ability to run VirtualBox.

For personal workstation, openindiana desktop.

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] 2.88Mb floppy image file.

[Edward Ned Harvey (openindiana)]

> From: Jonathan Adams [mailto:t12nsloo...@gmail.com]
> 
> Hi ... I was recently trying to create a 2.88Mb floppy file to try and BIOS
> upgrade a Dell computer that wouldn't boot anything graphical.
> 
> I know I'm probably going about this wrong, but I cannot seem to use
> fdformat, or mkfs -F pcfs on a file, in order to burn the file to a cdrom

I'm reasonably certain if you check dell's website again, you'll find a better 
download (especially if you have support; call them up and they'll provide you 
with an iso) but if you really want to boot from a floppy image burned to a 
CD...

mkisofs -J -r -o somefile.iso -b floppy.img

___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Problem with Dell iDRAC

[Edward Ned Harvey (openindiana)]

Oh.  I see now, Rich's solution about updated driver.  That sounds better.  ;-)


> -Original Message-
> From: Edward Ned Harvey (openindiana)
> Sent: Tuesday, April 23, 2013 8:34 AM
> To: openindiana-discuss@openindiana.org
> Subject: RE: [OpenIndiana-discuss] Problem with Dell iDRAC
> 
> > From: Kris Henriksson [mailto:kt...@cornell.edu]
> >
> > I've been having a long-standing issue with using the iDRAC on my
> > server with OpenIndiana, and I thought before giving up completely, I
> > could try asking the mailing list. The problem is that the iDRAC is
> > completely inaccessible while OpenIndiana is running.
> >
> > The system is a Dell PowerEdge T610, with Dell iDRAC 6 Express. If I
> > run Linux on it, the iDRAC can be accessed just fine, and before
> > OpenIndiana has started booting, I can access it, but with OI running
> > it is inaccessible. The DRAC shares a physical network port with the
> > OS, but has a separate MAC address and independent network traffic.
> 
> I've seen that before.
> 
> Got over it...  I forget precisely how ...  I think I had to go into BIOS, and
> "disable" the first NIC.  This makes it inaccessible to the OS, but not
> inaccessible to the iDRAC.  Then, obviously, connect two separate ethernet
> cables.  One for the management interface, and one for the OS.
> 
> Also, if you ping monitor the OS and the iDRAC...  It is normal to see the
> iDRAC disappear at certain moments during the boot process.  So don't
> assume it's failed the moment ping begins to fail.  Wait for the OS to come up
> completely, and perhaps a minute longer.
> 
> Oh yeah ...
> 
> This might be separate, but the built-in broadcom NIC was never stable in
> solaris 10 / opensolaris.  Symptom was a weird sort of black-screen lockup
> while still responding to ping, which occurred approx once a week.  We made
> this problem go away by buying an add-on Intel server NIC.  So it's distinctly
> possible, that the actual iDRAC solution is to "disable" both the broadcom
> ethernets in BIOS (use only by iDRAC) and only use the Intel NIC in the OS.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Problem with Dell iDRAC

[Edward Ned Harvey (openindiana)]

> From: Kris Henriksson [mailto:kt...@cornell.edu]
> 
> I've been having a long-standing issue with using the iDRAC on my
> server with OpenIndiana, and I thought before giving up completely, I
> could try asking the mailing list. The problem is that the iDRAC is
> completely inaccessible while OpenIndiana is running.
> 
> The system is a Dell PowerEdge T610, with Dell iDRAC 6 Express. If I
> run Linux on it, the iDRAC can be accessed just fine, and before
> OpenIndiana has started booting, I can access it, but with OI running
> it is inaccessible. The DRAC shares a physical network port with the
> OS, but has a separate MAC address and independent network traffic.

I've seen that before.

Got over it...  I forget precisely how ...  I think I had to go into BIOS, and 
"disable" the first NIC.  This makes it inaccessible to the OS, but not 
inaccessible to the iDRAC.  Then, obviously, connect two separate ethernet 
cables.  One for the management interface, and one for the OS.

Also, if you ping monitor the OS and the iDRAC...  It is normal to see the 
iDRAC disappear at certain moments during the boot process.  So don't assume 
it's failed the moment ping begins to fail.  Wait for the OS to come up 
completely, and perhaps a minute longer.

Oh yeah ...

This might be separate, but the built-in broadcom NIC was never stable in 
solaris 10 / opensolaris.  Symptom was a weird sort of black-screen lockup 
while still responding to ping, which occurred approx once a week.  We made 
this problem go away by buying an add-on Intel server NIC.  So it's distinctly 
possible, that the actual iDRAC solution is to "disable" both the broadcom 
ethernets in BIOS (use only by iDRAC) and only use the Intel NIC in the OS.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Pounding on well trod ground..

[Edward Ned Harvey (openindiana)]

> From: Harry Putnam [mailto:rea...@newsguy.com]
> 
> Sorry to go over what must have been already covered many times but I
> dropped out of OI participation for a good long while.
> 
> Hopefully someone will feel kindly disposed and post a brief outline
> of how to go from zero to running a vb vm of current OI on a win7
> 64bit host.

Download and install virtualbox.
Download the openindiana desktop iso.
In virtualbox, click Add, new machine, select solaris (I think) or perhaps 
solaris 64bit.  Boot from the iso.

Simple as that.

You'll have some self explanatory prompts, such as creating a virtual guest 
hard drive, and clicking on "Install" and choosing the language.

The only part that isn't obvious or self-expanatory:  
After the guest is installed, it's advisable to click "Install Guest Additions" 
in the host window.  This will virtually insert the guest additions CD rom into 
guest, which should automatically launch.


> Once I have a running install, is the syntax for the packages still
> something like REPONAME/dev?

sudo pkg search subversion
sudo pkg install subversion
(or whatever)


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] vdev reliability was: Recommendations for fast storage

[Edward Ned Harvey (openindiana)]

> From: Timothy Coalson [mailto:tsc...@mst.edu]
> 
> As for what I said about resilver speed, I had not accounted for the fact
> that data reads on a raid-z2 component device would be significantly
> shorter than for the same data on 2-way mirrors.  Depending on whether
> you
> are using enormous block sizes, or whether your data is allocated extremely
> linearly in the way scrub/resilver reads it, this could be the limiting
> factor on platter drives due to seek times, and make raid-z2 take much
> longer to resilver.  I fear I was thinking of raid-z2 in terms of raid6.

I'm not sure if you misunderstand something, or if I misunderstand what you're 
saying, but ...

Even if you are using enormous block sizes, it's actually just enormous *max* 
block sizes.  If you write a 1 byte file (very slowly such that no write 
accumulation can occur) then ZFS only writes a 1 byte file, into a block.  So 
the enormous block sizes only come into play when you're writing large amounts 
of data ...  And when you're writing large amounts of data, you're likely to 
simply span multiple sequential blocks anyway.  So all-in-all, the blocksize is 
rarely very important.  There are some situations where it matters, but ...  
All this is a tangent.

The real thing I'm addressing here, is, you said scrub/resilver progresses 
extremely linearly.  This is unfortunately, about as wrong as it can be.  In 
actuality, scrub / resilver proceed in approximately temporal order, which in 
the typical situation of a long-time server with frequent creation & 
destruction of snapshots, results in approximately random disk order.

Here's the evidence I observed:  I had a ZFS server running in production for 
about 2 years, and a disk failed.  I had measured previously, on this server, 
each disk sustains 1 Gbit/sec sequentially.  With 1T disks, linearly 
resilvering the entire disk including empty space, it should take about 2 hrs 
to resilver.  But ZFS doesn't resilver the whole disk; it only resilvers used 
space.  This would be great, if your pool is mostly empty, or if it was disk 
linearly ordered.  But it actually took 12 hours to resilver that disk.  I went 
to zfs-discuss and discussed.  Learned about the temporal ordering.  Got my 
explanation how resilvering just the used portions could take several times 
longer than resilvering the whole disk.




___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] vdev reliability was: Recommendations for fast storage

[Edward Ned Harvey (openindiana)]

> From: Jim Klimov [mailto:jimkli...@cos.ru]
> 
> Well, thanks to checksums we can know which variant of userdata
> is correct, and thanks to parities we can verify which bytes are
> wrong in a particular block. If there's relatively few such bytes,
> it is theoretically possible to brute-force match values into the
> "wrong" bytes and recalculate checksums. So if a "broken" range
> is on the order of 30-40 bytes (which someone said is typical
> for a CRC error and HDD returning uncertain data) you have a
> chance of recovering the block in a few days if lucky ;)
> 
> This is a very compute-intensive task; I proposed this idea half
> a year ago on the zfs list (I had unrecoverable errors on raidz2
> made of 4 data disks and 2 parity disks, meaning corruptions on
> 3 or more drives, but not necessarily whole-sector corruptions)
> and tried to take known byte values from different components at
> known "bad" byte offsets and put them into the puzzle. Complexity
> (size of recursive iteration) grows very quickly even if we only
> have about 5 values to match (unlike 256 in full recovery above),
> and we estimated that for a 4096 byte block it would take Earth's
> compute resources longer than the lifetime of the universe to do
> the full search and recovery. So such approach is really limited
> to just a few dozen broken bytes. But it is possible :)

I think you're misplacing a decimal, confusing bits for bytes, and mixing up 
exponents.  Cuz you're way off.

With merely 70 unknown *bits* that is, less than 10 bytes, you'll need a 
3-letter government agency devoting all its computational resources to the 
problem for a few years.

Furthermore, when you find a matching cksum, you haven't found the correct data 
yet.  You'll need to exhaustively search the entire space requiring 2^70 
operations, find all the matches (there will be a lot) and from those matches, 
choose the one you think is right.

Even with merely 70 unknown bits, and a 32-bit cksum (the default in zfs 
fletcher-4) you will have 2^38 (that is, 256 billion) results that produce the 
right cksum.  You'll have to rely on your knowledge of the jpg file or txt file 
or whatever, to choose which one of the 256 billion cksum-passing-results is 
*actually* the right result.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] vdev reliability was: Recommendations for fast storage

[Edward Ned Harvey (openindiana)]

> From: Sebastian Gabler [mailto:sequoiamo...@gmx.net]
> 
> AFAIK, a bit error in Parity or stripe data can be specifically
> dangerous when it is raised during resilvering, and there is only one
> layer of redundancy left. 

You're saying "error in parity," but that's because you're thinking of raidz, 
which I don't usually use.  You really mean "error in redundant copy," and the 
only risk, as you've identified, is the error in the *last* redundant copy.

The answer to this is:  You *do* scrub every week or two, don't you?You 
should.


> I do not think that zfs will have better resilience against rot of
> parity data than conventional RAID.

That's incorrect, because conventional raid cannot scrub proactively.

Sure, if you have a pool with only one level of redundancy, and the bit error 
creeped in between the most recent scrub and the present failure time, then 
that's a problem, and zfs cannot protect you against it.  This is, by 
definition, simultaneous failure of all redundant copies of the data.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Recommendations for fast storage

[Edward Ned Harvey (openindiana)]

> From: Jay Heyl [mailto:j...@frelled.us]
> 
> I now realize you're talking about 8 separate 2-disk
> mirrors organized into a pool. "mirror x1 y1 mirror x2 y2 mirror x3 y3..."

Yup.  That's normal, and the only way.


> I also realize that almost every discussion I've seen online concerning
> mirrors proposes organizing the drives in the way I was thinking about it

Hmmm...   What alternative are you thinking of?There is no alternative.


> This also starts to make a lot more sense. Confused the hell out of me the
> first three times I read it. I'm going to have to ponder this a bit more as
> my thinking has been heavily influenced by the more conventional mirror
> arrangement.

What are you talking about?
___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Recommendations for fast storage

[Edward Ned Harvey (openindiana)]

> From: Timothy Coalson [mailto:tsc...@mst.edu]
> 
> Did you also compare the probability of bit errors causing data loss
> without a complete pool failure?  2-way mirrors, when one device
> completely
> dies, have no redundancy on that data, and the copy that remains must be
> perfect or some data will be lost.  

I had to think about this comment for a little while to understand what you 
were saying, but I think I got it.  I'm going to rephrase your question:

If one device in a 2-way mirror becomes unavailable, then the remaining device 
has no redundancy.  So if a bit error is encountered on the (now non-redundant) 
device, then it's an uncorrectable error.  Question is, did I calculate that 
probability?

Answer is, I think so.  Modelling the probability of drive failure (either 
complete failure or data loss) is very complex and non-linear.  Also dependent 
on the specific model of drive in question, and the graphs are typically not 
available.  So what I did was to start with some MTBDL graphs that I assumed to 
be typical, and then assume every data-loss event meant complete drive failure. 
 Already I'm simplifying the model beyond reality, but the simplification 
focuses on worst case, and treats every bit error as complete drive failure.  
This is why I say "I think so," to answer your question.  

Then, I didn't want to embark on a mathematician's journey of derivatives and 
integrals over some non-linear failure rate graphs, so I linearized...  I 
forget now (it was like 4-6 years ago) but I would have likely seen that drives 
were unlikely to fail in the first 2 years, and about 50% likely to fail after 
3 years, and nearly certain to fail after 5 years, so I would have likely 
modeled that as a linearly increasing probability of failure rate up to 4 
years, where it's assumed 100% failure rate at 4 years.

Yes, this modeling introduces inaccuracy, but that inaccuracy is in the noise.  
Maybe in the first 2 years, I'm 25% off in my estimates to the positive, and 
after 4 years I'm 25% off in the negative, or something like that.  But when 
the results show 10^-17 probability for one configuration and 10^-19 
probability for a different configuration, then the 25% error is irrelevant.  
It's easy to see which configuration is more probable to fail, and it's also 
easy to see they're both well within acceptable limits for most purposes 
(especially if you have good backups.)


> Also, as for time to resilver, I'm guessing that depends largely on where
> bottlenecks are (it has to read effectively all of the remaining disks in
> the vdev either way, but can do so in parallel, so ideally it could be the
> same speed), 

No.  The big factor for resilver time is (a) the number of operations that need 
to be performed, and (b) the number of operations per second.

If you have one big vdev making up a pool, then the number of operations to be 
performed is equal to the number of objects in the pool.  The number of 
operations per second is limited by the worst case random seek time for any 
device in the pool.  If you have an all-SSD pool, then it's equal to a single 
disk performance.  If you have an all-HDD pool, then with increasing number of 
devices in your vdev, you approach 50% of the IOPS of a single device.

If your pool is broken down into a bunch of smaller vdev's, Let's say N mirrors 
that are all 2-way.  Then the number of operations to resilver the degraded 
mirror is 1/N of the total objects in the pool.  And the number of operations 
per second is equal to the performance of a single disk.  So the resilver time 
in the big vdev raidz is 2N times longer than the resilver time for the mirror.

As you mentioned, other activity in the pool can further reduce the number of 
operations per second.  If you have N mirrors, then the probability of the 
other activity affecting the degraded mirror is 1/N.  Whereas, with a single 
big vdev, you guessed it, all other activity is guaranteed to affect the 
resilvering vdev.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Recommendations for fast storage

[Edward Ned Harvey (openindiana)]

> From: Jay Heyl [mailto:j...@frelled.us]
> 
> Ah, that makes much more sense. Thanks for the clarification. Now that you
> put it that way I have to wonder how I ever came under the impression it
> was any other way.

I've gotten lost in the numerous mis-communications of this thread, but just to 
make sure there is no confusion:

If you have a mirror (or any vdev with redundancy, radizN) you issue a read, 
then normally only one side of the mirror gets read (not the redundant copies.) 
 If the cksum fails, then redundant copies are read successively, until a 
successful cksum is found (still, some redundant copies might not have been 
read.)

If you perform a scrub, then all copies of all information are read and 
validated.

The advantage of reading only one side of the mirror is performance.  If one 
device is busy satisfying one read request, then the other sides of the mirror 
are available to satisfy other read requests.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Recommendations for fast storage

[Edward Ned Harvey (openindiana)]

> From: Sašo Kiselkov [mailto:skiselkov...@gmail.com]
> 
> Raid-Z indeed does stripe data across all
> leaf vdevs (minus parity) and does so by splitting the logical block up
> into equally sized portions. 

Jay, there you have it.  You asked why use mirrors, and you said you would use 
raidz2 or raidz3 unless cpu overhead is too much.  I recommended using mirrors 
and avoiding raidzN, and here is the answer why.

If you have 16 disks arranged in 8x mirrors, versus 10 disks in raidz2 which 
stripes across 8 disks plus 2 parity disks, then the serial write of each 
configuration is about the same; that is, 8x the sustained write speed of a 
single device.  But if you have two or more parallel sequential read threads, 
then the sequential read speed of the mirrors will be 16x while the raidz2 is 
only 8x.  The mirror configuration can do 8x random write while the raidz2 is 
only 1x.  And the mirror can do 16x random read while the raidz2 is only 1x.

In the case you care about the least, they're equal.  In the case you care 
about most, the mirror configuration is 16x faster.

You also said the raidz2 will offer more protection against failure, because 
you can survive any two disk failures (but no more.)  I would argue this is 
incorrect (I've done the probability analysis before).  Mostly because the 
resilver time in the mirror configuration is 8x to 16x faster (there's 1/8 as 
much data to resilver, and IOPS is limited by a single disk, not the "worst" of 
several disks, which introduces another factor up to 2x, increasing the 8x as 
high as 16x), so the smaller resilver window means lower probability of 
"concurrent" failures on the critical vdev.  We're talking about 12 hours 
versus 1 week, actual result of my machines in production.  Also, while it's 
possible to fault the pool with only 2 failures in the mirror configuration, 
the probability is against that happening.  The first disk failure probability 
is 1/16 for each disk ... And then if you have a 2nd concurrent failure, 
there's a 14/15 probability that it occurs on a separately independent (safe) 
mirror.  The 3rd concurrent failure 12/14 chance of being safe.  The 4th 
concurrent failure 10/13 chance of being safe.  Etc.  The mirror configuration 
can probably withstand a higher number of failures, and also the resilver 
window for each failure is smaller.  When you look at the total probability of 
pool failure, they were both like 10^-17 or something like that.  In other 
words, we're splitting hairs but as long as we are, we might as well point out 
that they're both about the same.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Recommendations for fast storage

[Edward Ned Harvey (openindiana)]

> From: Mehmet Erol Sanliturk [mailto:m.e.sanlit...@gmail.com]
> 
> SSD units are very vulnerable to power cuts during work up to complete
> failure which they can not be used any more to complete loss of data .

If there are any junky drives out there that fail so dramatically, those are 
junky and the exception.  Just imagine how foolish the engineers would have to 
be, "Power loss?  I didn't think of that...  Complete drive failure in power 
loss is acceptable behavior."   Definitely an inaccurate generalization about 
SSD's.  There is nothing inherent about flash memory as compared to magnetic 
material, that would cause such a thing.

I repeat:  I'm not saying there's no such thing as a SSD that has such a 
problem.  I'm saying if there is, it's junk.  And you can safely assume any 
good drive doesn't have that problem.


> MLC ( Multi-Level Cell ) SSD units have a short life time if they are
> continuously written ( they are more suitable to write once ( in a limited
> number of writes sense ) - read many )  .

It's a fact that NAND has a finite number of write cycles, and it gets slower 
to write, the more times it's been re-written.  It is also a fact that when 
SSD's were first introduced to the commodity market about 11 years ago, that 
they failed quickly due to OSes (windows) continually writing the same sectors 
over and over.  But manufacturers have been long since aware of this problem, 
and solved it by overprovisioning and wear-leveling.

Similar to ZFS copy-on-write, which has the ability to logically address some 
blocks and secretly re-map them to different sectors behind the scenes...  
SSD's with wear-leveling secretly remap sectors during writes.


> SSD units may fail due to write wearing in an unexpected time , making them
> very unreliable for mission critical works .

Every page has a write counter, which is used to predict failure.  A very 
predictable, and very much *not* unexpected time.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Recommendations for fast storage

[Edward Ned Harvey (openindiana)]

> From: Jay Heyl [mailto:j...@frelled.us]
> 
> > So I'm just assuming you're going to build a pool out of SSD's, mirrored,
> > perhaps even 3-way mirrors.  No cache/log devices.  All the ram you can fit
> > into the system.
> 
> What would be the logic behind mirrored SSD arrays? With spinning platters
> the mirrors improve performance by allowing the fastest of the mirrors to
> respond to a particular command to be the one that defines throughput.

When you read from a mirror, ZFS doesn't read the same data from both sides of 
the mirror simultaneously and let them race, wasting bus & memory bandwidth to 
attempt gaining smaller latency.  If you have a single thread doing serial 
reads, I also have no cause to believe that zfs reads stripes from multiple 
sides of the mirror to accelerate - rather, it relies on the striping across 
multiple mirrors or vdev's.

But if you have multiple threads requesting independent random read operations 
that are on the same mirror, I have measured the results that you get very 
nearly n-times a single disk random read performance by using a n-way mirror 
and at least n or 2n independent random read threads.


> There is no
> latency due to head movement or waiting for the proper spot on the disc to
> rotate under the heads. 

Nothing, including ZFS, has such an in-depth knowledge of the inner drive 
geometry as to know how long is necessary for the rotational latency to come 
around.  Also, rotational latency is almost nothing compared to head seek.  For 
this reason, short-stroking makes a big difference, when you have a data usage 
pattern that can easily be confined to a small number of adjacent tracks.  I 
believe, if you use a HDD for log device, it's aware of itself and does 
short-stroking, but I don't actually know.  Also, this is really a completely 
separate subject.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Recommendations for fast storage

[Edward Ned Harvey (openindiana)]

> From: Sašo Kiselkov [mailto:skiselkov...@gmail.com]
> 
> If you are IOPS constrained, then yes, raid-zn will be slower, simply
> because any read needs to hit all data drives in the stripe. 

Saso, I would expect you to know the answer to this question, probably:
I have heard that raidz is more similar to raid-1e than raid-5.  Meaning, when 
you write data to raidz, it doesn't get striped across all devices in the raidz 
vdev...  Rather, two copies of the data get written to any of the available 
devices in the raidz.  Can you confirm?

If the behavior is to stripe across all the devices in the raidz, then the 
raidz iops really can't exceed that of a single device, because you have to 
wait for every device to respond before you have a complete block of data.  But 
if it's more like raid-1e and individual devices can read independently of each 
other, then at least theoretically, the raidz with n-devices in it could return 
iops performance on-par with n-times a single disk. 


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Recommendations for fast storage (OpenIndiana-discuss Digest, Vol 33, Issue 20)

[Edward Ned Harvey (openindiana)]

> From: Bob Friesenhahn [mailto:bfrie...@simple.dallas.tx.us]
> 
> It would be difficult to believe that 10Gbit Ethernet offers better
> bandwidth than 56Gbit Infiniband (the current offering).  The swiching
> model is quite similar.  The main reason why IB offers better latency
> is a better HBA hardware interface and a specialized stack.  5X is 5X.

Put another way, the reason infiniband is so much higher throughput and lower 
latency than ethernet is because the switching (at the physical layer) is 
completely different from ethernet, and messages are passed directly from 
user-level to user-level on remote system ram via RDMA, bypassing the OSI layer 
model and other kernel overhead.  I read a paper from vmware, where they 
implemented RDMA over ethernet and doubled the speed of vmotion (but still not 
as fast as infiniband, by like 4x.)

Beside the bypassing of OSI layers and kernel latency, IB latency is lower 
because Ethernet switches use store-and-forward buffering managed by the 
backplane in the switch, in which a sender sends a packet to a buffer on the 
switch, which then pushes it through the backplane, and finally to another 
buffer on the destination.  IB uses cross-bar, or cut-through switching, in 
which the sending host channel adapter signals the destination address to the 
switch, then waits for the channel to be opened.  Once the channel is opened, 
it stays open, and the switch in between is nothing but signal amplification 
(as well as additional virtual lanes for congestion management, and other 
functions).  The sender writes directly to RAM on the destination via RDMA, no 
buffering in between.  Bypassing the OSI layer model.  Hence much lower latency.

IB also has native link aggregation into data-striped lanes, hence the 1x, 2x, 
4x, 16x designations, and the 40Gbit specifications.  Something which is 
quasi-possible in ethernet via LACP, but not as good and not the same.  IB 
guarantees packets delivered in the right order, with native congestion control 
as compared to ethernet which may drop packets and TCP must detect and 
retransmit...  

Ethernet includes a lot of support for IP addressing, and variable link speeds 
(some 10Gbit, 10/100, 1G etc) and all of this asynchronous.  For these reasons, 
IB is not a suitable replacement for IP communications done on ethernet, with a 
lot of variable peer-to-peer and broadcast traffic.  IB is designed for 
networks where systems want to establish connections to other systems, and 
those connections remain mostly statically connected.  Primarily clustering & 
storage networks.  Not primarily TCP/IP.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Recommendations for fast storage

[Edward Ned Harvey (openindiana)]

> From: Wim van den Berge [mailto:w...@vandenberge.us]
> 
> multiple 10Gb uplinks
> 
> However the next system is  going to be a little different. It needs to be
> the absolute fastest iSCSI target we can create/afford. 

So I'm just assuming you're going to build a pool out of SSD's, mirrored, 
perhaps even 3-way mirrors.  No cache/log devices.  All the ram you can fit 
into the system.

You've been using 10G ether so far.  Expensive, not too bad.  I'm going to 
recommend looking into infiniband instead.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] ZfS migration scenario including zvols

[Edward Ned Harvey (openindiana)]

> From: Sebastian Gabler [mailto:sequoiamo...@gmx.net]
> Sent: Saturday, April 13, 2013 11:38 AM
> 
> - zfs send mainbranch@1 -R > /pool2/mainbranch.dmp for each nfs, iscsi,
> smb

It is advisable, if possible, to create a new zpool with your new tmp storage, 
and zfs send | zfs receive.  (Don't store a zfs send data stream in a file.)  
Because the "zfs receive" does all the cksumming, and will detect and notify 
you in the event of a data transmission problem.

But if all you have is a network store, or something else that you can't format 
with zfs for some reason...  So be it.  You do what you have to do.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] ZfS migration scenario including zvols

[Edward Ned Harvey (openindiana)]

> From: Sebastian Gabler [mailto:sequoiamo...@gmx.net]

Be careful, there are lots of ways to screw this up.  Fortunately, not many of 
them result in data loss or anything like that.  Just bad behavior.

Specifically, I thinking, you want to send including properties, to preserve 
the nfs & iscsi properties, etc.  But unfortunately, you don't want to send 
with properties, which also preserves the mountpoint.  So the receiving 
filesystem fails to mount, and then ... I don't know what.  Perhaps you have to 
export both pools and re-import, or manually tweak the mountpoints on the 
recipient filesystem...

Um ... Also, most likely since you have nested filesystems, each of them has 
been able to snapshot separately from each other.  The list of snapshots is 
probably not 100% identical on all filesystems, which means, if you try to do 
the recursive+recursive incremental send from the root, it will fail.  The 
*easiest* thing to do is simply take a recursive snapshot now, and send a 
recursive stream based on that, but you'll give up snapshot history.  The 
*best* thing would be to handle each filestem/zvol individually, so you can do 
the recursive incremental (without going recursive filesystems.)

I recommend looking at the list of zfs properties on your datasets, and think 
about which ones you want to preserve, and which ones you don't.  


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] 3737 days of uptime

[Edward Ned Harvey (openindiana)]

> From: Ben Taylor [mailto:bentaylor.sol...@gmail.com]
> 
> Patching is a bit of arcane art.  Some environments don't have
> test/acceptance/pre-prod with similar hardware and configurations, so
> minimizing impact is understandable, which means patching only what is
> necessary.

This thread has long since become pointless and fizzled, but just for the fun 
of it:

I recently started a new job, where updates had not been applied to any of the 
production servers in several years.  (By decree of former CIO).  We recently 
ran into an obstacle where some huge critical deliverable was not possible 
without applying the updates.  So we were forced, the whole IT team working 
overnight on the weekend, to apply several years' backlog of patches to all the 
critical servers worldwide.  Guess how many patch-related issues were 
discovered.  (Hint:  none.)

Patching is extremely safe.  But let's look at the flip side.  Suppose you 
encounter the rare situation where patching *does* cause a problem.  It's been 
known to happen; heck, it's been known to happen *by* *me*.  You have to ask 
yourself, which is the larger risk?  Applying the patches, or not applying the 
patches?  

First thing to point out:  Suppose you patch something and it goes wrong ...  
Generally speaking you can back out of the patch.  Suppose you don't apply the 
patch, and you get a virus or hacked, or some data corruption.  Generally 
speaking, that is not reversible.

For the approx twice in my life that I've seen OS patches cause problems, and 
then had to reverse out the patches...  I've seen dozens of times that somebody 
inadvertently sets a virus loose on the internal network, or a server's memory 
or storage became corrupted due to misbehaving processes or subsystem, or some 
server has some kind of instability and needs periodic rebooting, or becomes 
incompatible with the current release of some critical software or hardware, 
until you apply the patches.  

Patches are "bug fixes" and "security fixes" for known flaws in the software.  
You can't say "if it ain't broke, don't fix it."  It is broke, that's why they 
gave you the fix for it.  At best, you can say, "I've been ignoring it, and we 
haven't noticed any problems yet."


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] How to configure OI for proper daylight savings adjustments.

[Edward Ned Harvey (openindiana)]

> From: Robin Axelsson [mailto:gu99r...@student.chalmers.se]
> 
> Is there anyone who has got this
> working properly?

I confirm that it works correctly out of the box, for EST/EDT (New York time), 
with 

this line in /etc/default/init
TZ=US/Eastern

this line in /etc/rtc_config
zone_info=US/Eastern


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Added a page on the wiki about using Xvnc for multiple simultaneous graphical logins.

[Edward Ned Harvey (openindiana)]

> From: Hans J. Albertsson [mailto:hans.j.alberts...@branneriet.se]
> 
> http://wiki.openindiana.org/oi/4.7+Remote+Graphical+Login:+Using+Xvnc+
> and+gdm

That's interesting ... So correct me if I'm wrong, you VNC to your server on 
5900, and you get a login prompt as if you were sitting down in front of the 
physical console.  Every new connection to 5900 gets a new login prompt, so you 
don't have to consume more ports (5901, 5902, etc).

But if your VNC client crashes for some reason, what happens?  I'm guessing the 
VNC session dies and all the programs inside it die too.  There is no session 
manager; when you login to 5900 you can't reconnect to a previously existing 
session.  Right?


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Dhcp woes, (was Re: Anyone using OpenIndiana in production?)

[Edward Ned Harvey (openindiana)]

> From: Michael Stapleton [mailto:michael.staple...@techsologic.com]
> 
> The Dhcp files can be stored on NFS and used by multiple servers.

It defeats the purpose of redundant dhcp servers if you make them both 
dependent on non-redundant storage.  But that's only tangential.  The upshot of 
what you're saying is that the config files are in some directory, and they 
could be versioned and distributed just like I'm presently doing with svn.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Dhcp woes, (was Re: Anyone using OpenIndiana in production?)

[Edward Ned Harvey (openindiana)]

> From: Jim Klimov [mailto:jimkli...@cos.ru]
> 
> Well, at the time I documented this page, it worked (at oi_151a5
> timeframe, I believe):
> 
> http://wiki.openindiana.org/oi/Using+host-
> only+networking+to+get+from+build+zones+and+test+VMs+to+the+Intern
> et

Yikes.

Thanks for writing that up.  But ..

I have two servers, and some reservations, and the config files are stored in 
subversion.  So at present, when I create a reservation for a new machine, I 
just edit one file (duplicate & modify a line) and commit.  Svn post-commit 
hook then verifies integrity, and deploys to the mirror.  Very easy.

I can't believe there's this new dhcpmgr, dhcpconfig,  several config files, 
some binfiles that I presume I can't read or edit...  3+ packages that need to 
be installed...

I'm sure it's very powerful, and maybe can even do what I'm doing *even* 
better.  But that's very complicated and more than I want to invest.  It's no 
wonder I didn't get it working.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] Anyone using OpenIndiana in production?

[Edward Ned Harvey (openindiana)]

> From: Gerry Weaver [mailto:ger...@compvia.com]
> 
> I have been checking out OpenIndiana as a possible file server and KVM host.

I have several sites using OI for samba, dns, and VirtualBox.  As far as 
stability is concerned, yes it's stable.  But it's not amazingly mature (see 
below).

Others have said that some of the packages are sort of not greatly maintained, 
and lack configuration.  I needed to configure bind and samba from scratch by 
hand, which was a huge pain.  I actually built some linux VM's just to get 
their default config files and copy them over to OI and destroy the linux 
guests.  I would like to run dhcp from OI, rather than running it from a linux 
guest.  But I never got dhcp working on OI, so I still have the linux dhcp 
guests just for this purpose.

I recommend the latest 4.1.x version of VirtualBox, which was super awesome and 
stable.  Unfortunately I updated to 4.2.x, and it's still very buggy.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] 3737 days of uptime

[Edward Ned Harvey (openindiana)]

> From: dormitionsk...@hotmail.com [mailto:dormitionsk...@hotmail.com]
> Sent: Tuesday, March 19, 2013 11:42 PM
> 
> A Sun Solaris machine was shut down last week in Hungary, I think, after 3737
> days of uptime.  Below are links to the article and video.
>
> Warning:  It might bring a tear to your eye...

It would only bring a tear to my eye, because of how foolishly irresponsible 
that is.  3737 days of uptime means 10 years of never applying security patches 
and bugfixes.  Whenever people are proud of a really long uptime, it's a sign 
of a bad sysadmin.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] DLNA server

[Edward Ned Harvey (openindiana)]

> From: Hans J. Albertsson [mailto:hans.j.alberts...@branneriet.se]
> Sent: Saturday, March 09, 2013 2:28 PM
> 
> Of course, there's NO serviio package in any std repository.
> I thought you already had a serviio installed from the linux distro from
> serviio.org??

Sorry, my mistake.  I just pulled out my actual notes, and here's what I 
*actually* do:

adapted from http://wiki.serviio.org/doku.php?id=howto:solaris:install

sudo pkg set-publisher -p http://pkg.openindiana.org/sfe-encumbered
sudo pkg set-publisher -p http://pkg.openindiana.org/sfe

sudo pkg install ffmpeg

sudo mkdir /usr/local
cd /usr/local
sudo tar xzf serviio-1.1-linux.tar.gz
(Yes, I know that says linux.)

sudo chown -R eharvey:staff /usr/local/serviio-1.1

Get onto a GUI (vnc) as eharvey.
cd /usr/local/serviio-1.1 
./bin/serviio.sh & 
./bin/serviio-console.sh &

Icon appears in status tray. Right click it, open serviio console
Under "Network Settings" / "Bound IP address" enter your IP explicitly.

Under Library / Shared Folders, add something to share.

Create the script:  ~/bin/serviio.sh
#!/usr/bin/bash
cd /usr/local/serviio-1.1
./bin/serviio.sh & 
./bin/serviio-console.sh &

Assuming you already have vnc set to launch as a smf service, 
just add the serviio.sh script to System / Preferences / Startup 
Applications

Type "which ffmpeg" to get its location, and then edit JAVA_OPTS at the 
bottom of /usr/local/serviio-1.1/bin/serviio.sh and add 
-Dffmpeg.location=/usr/bin/ffmpeg

Restart serviio.
~/bin/serviio.sh


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] DLNA server

[Edward Ned Harvey (openindiana)]

> From: Michelle Knight [mailto:miche...@msknight.com]
> 
> It looks like I'm going to have to install something on the server to
> publish the video directories in DLNA, which I've got no experience of.

This is what I do:

sudo pkg set-publisher -p http://pkg.openindiana.org/sfe-encumbered
sudo pkg set-publisher -p http://pkg.openindiana.org/sfe
sudo pkg -y install serviio

As others have mentioned, configuring a profile specifically compatible with 
you device is crucial.  With my TV, so far I've only been able to play, pause, 
FFx1 FFx2.  Any other button (REW, or FFx4) will halt the TV, requiring a power 
cycle.  It's very annoying.

I haven't put in the effort to fix a profile for my TV.  As Hans said, you need 
to go to http://forum.serviio.org/ and basically just figure it out.  And 
hopefully share profiles so other people with the same device can benefit.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] IO Stalls

[Edward Ned Harvey (openindiana)]

> From: Grant Albitz [mailto:gr...@schultztechnology.com]
> 
> I have been chasing an issue with my openindiana host for some time. It is
> stable for a few weeks but then I find it rebooted with no kernel errors.

This sounds like a driver issue.  I've had similar problems on an R510 or R520, 
or something similar, I forget exactly which.  That system was unstable and 
rebooted or crashed about once a week.  The most effective thing we did was to 
add on an Intel NIC and stop using the built-in broadcom NIC, but even so, it 
still crashed about once every 3 weeks or a month.

I've had bad luck in general running solaris or openindiana on dell servers.  
(No problems on dell workstations.)  Other people have had good luck running on 
dell servers.

If possible, eliminate a PERC and use a plain old stupid SATA/SAS card.

Sorry I don't have anyhting more helpful to offer.   Good luck.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] multiple IP addresses, same NIC

[Edward Ned Harvey (openindiana)]

> From: Reginald Beardsley [mailto:pulask...@yahoo.com]
> Sent: Wednesday, March 06, 2013 3:34 PM
> 
> How about summarizing on the wiki?

I'm in favor, but in this case, I don't think there's anything to summarize ... 
 Here is the summary:

sudo dladm create-vnic -l e1000g0 vnic0
sudo ipadm create-addr -T static -a 192.168.2.100/24 vnic0/v4static
sudo route -p add 192.168.10.0/24 192.168.2.1

And voila.  New IP address and new MAC address on the same wire with my 
pre-existing LAN subnet, with a static route.  Actually ... I believe all these 
commands are already on the wiki.  I think I actually *got* these answers from 
the wiki, once I knew what to look for.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] multiple IP addresses, same NIC

[Edward Ned Harvey (openindiana)]

> From: Robbie Crash [mailto:sardonic.smi...@gmail.com]
> 
> This is something that should be handled at the router, not at the client
> in software.

It turns out, I reached a conclusion with the NAT possibility.  In pfsense, you 
can NAT traffic before it goes across an openvpn, but you can't NAT traffic 
before it goes across an ipsec vpn.  (Just a limitation of their software, 
until at least the next release, when they *might* add that feature.)  At 
present, in pfsense, I would need one firewall to establish the VPN connection, 
and another firewall to NAT from that subnet to my internal subnet.

Thanks to Jim's idea of VNIC, I have a solution in client-side software.  So 
this thread really doesn't need to continue...  But it was an interesting and 
fun exercise to talk about.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] multiple IP addresses, same NIC

[Edward Ned Harvey (openindiana)]

> From: Robbie Crash [mailto:sardonic.smi...@gmail.com]
> 
> > The problem is at the remote side.  If they have a huge internal corporate
> > network that happens to include 192.168.10.x/24 and 192.168.1.x/24 ...
> When
> > I VPN to them and my LAN is 192.168.1.x/24, I have a subnet that overlaps
> > with their pre-existing subnet.  They can't route traffic to me without
> > breaking one of their internal subnets.
> >
> 
> I get that, but in your original email you stated you don't need to access
> their 192.168.1.0 subnet, unless all their traffic routes over that subnet
> internally you shouldn't have an issue. Their side will see the request
> coming from your VPN point, and will send traffic there and your VPN server
> will send it to the proper client. 

No, there's something you seem to be missing.  I'm making up the details in 
this email, but the concept stands:  They have 192.168.1.x/24 in Buffalo.  
192.168.10.x/24 in Syracuse.  10.10.10.x/24 in Toronto.  172.16.14.x/24 in 
Vancouver...  and a hundred other sites.  They have all their routers 
configured to support this.  If somebody at any site sends traffic to 
192.168.1.x/24, their routers know the traffic is routed to Buffalo.  So if I 
get inside the network, using 192.168.1.x/24 in Boston, all those other sites 
can't talk to me, or can't talk to Buffalo.  I have to either use a subnet that 
doesn't conflict, or I have to NAT and virtually use a subnet that doesn't 
conflict.

If I actually use the new subnet, 192.168.2.x/24 which isn't used anywhere else 
in the company, then all traffic is routable to and from my network, which is 
good.  But if I virtuallly NAT my 192.168.1.x/24 network, making my traffic 
appear as 192.168.2.x/24 as far as the company's concerned ...  Then I have no 
way to access their 192.168.1.x/24 because my systems will think the 
destination is local and hence not use the router.  I am saying that I'm ok 
using this NAT solution to avoid the need to renumber my systems.  I'm only 
blocking the traffic from my local 192.168.1.x to the company's 192.168.1.x 
(and vice-versa) but I don't care about connecting to anything in the company's 
192.168.1.x range.

Make sense now?;-)


> What IP address are you receiving from
> the VPN server? 

Their VPN server doesn't assign an IP address.  This is not a mobile client VPN 
we're talking about, it's a site-to-site VPN.  Firewall to firewall.  Corporate 
home office.

And I'm the IT guy.  So I can do whatever I want and support whatever I want.  
The question is what do I want.  Well, I have about a dozen or two systems in 
my house, including a mobile vpn server, site-to-site vpn's with other 
companies, two windows active directory domains, a few dns zones, and a 
virtualization infrastructure.  While I *can* renumber, it'll cost me about a 
day's work.  So the NAT solution is attractive.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] multiple IP addresses, same NIC

[Edward Ned Harvey (openindiana)]

> From: Jim Klimov [mailto:jimkli...@cos.ru]
> 
> 1) Why not use VNICs to avoid the MAC-related problem altogether?

Ye...  That did it.  Problem solved, thank you.
I have never used VNIC before ... it took very little googling to figure out 
what you were talking about, figure out how to use it, and make it work.  
Again, thank you.   :-)


> 2) Can you run the VPN client on the client computer so that it "has"

Long story short, thanks for the idea, but due to technical problems, it would 
be a huge hassle, and not available in the near future.  In fact, I already 
have the VPN client, and I already use it.  But I can only access the LAN 
that's directly attached to the VPN server.  If I need to hop across another 
router to reach another subnet, that requires the "hairpinning" feature which 
we don't currently have.  It would be a hassle to make that work.

I am totally satisfied with the present solution however.  :-)  The vnic was 
the magic I was looking for.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] multiple IP addresses, same NIC

[Edward Ned Harvey (openindiana)]

> From: Robbie Crash [mailto:sardonic.smi...@gmail.com]
> 
> If you're not accessing clients on the remote 192.168.1.0 subnet, why are
> you adding the second network?
> 
> Why are you not handling this on the router instead of the client? Static
> routes on a client are bad mojo. It's the router's job to route, let it do
> that. All you should need to do is tell the router to route all traffic for
> 192.168.10.0/24 to use whatever the VPN interface is.

The problem is at the remote side.  If they have a huge internal corporate 
network that happens to include 192.168.10.x/24 and 192.168.1.x/24 ... When I 
VPN to them and my LAN is 192.168.1.x/24, I have a subnet that overlaps with 
their pre-existing subnet.  They can't route traffic to me without breaking one 
of their internal subnets.

The most elegant solution (aside from renumbering my network) would be NAT.  It 
would be nice to eliminate 192.168.2.x/24 from my house, and configure the 
firewall so when I send a packet to the VPN network, let my source IP be NAT'd 
to 192.168.2.x/24.  However, I have not yet had any luck configuring pfsense to 
NAT the traffic first and then route it, NAT'd across the VPN.

At present, I have two problems I'm trying to solve in parallel.  If I can 
either make OI behave as expected, then I can use the 
multiple-subnets-on-a-single-LAN solution and move forward.  Or if I can get 
the firewall to NAT as expected, then I can scrap the multiple-subnets idea and 
move forward.


> The issue here sounds like since the OI box already knows that it has a
> route to 192.168.10.10 over its default route, it doesn't need to use the
> secondary IP.

That's not quite correct.  Sure, if I didn't add the static route 192.168.10.x 
via 192.168.2.1, then OI would try to reach 192.168.10.x via the default 
gateway.  But that's irrelevant - By adding the 192.168.2.1 route, the system 
does in fact know it's supposed to reach 192.168.10.x via 192.168.2.1.  The 
evidence is when a packet leaves the NIC destined for 192.168.10.x, its 
destination MAC corresponds to 192.168.2.1.  But unfortunately, the source IP 
is wrong.


> If you can't configure the router, PCI NICs are $9 these days, and that'll
> work for sure.

I might do that.  The main obstacle is knowing I would have to wait for it to 
arrive, and it will require downtime on the VM host, to solve something that 
should be solvable in software.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss

Re: [OpenIndiana-discuss] multiple IP addresses, same NIC

[Edward Ned Harvey (openindiana)]

> From: Doug Hughes [mailto:d...@will.to]
> 
> 2) explicitly set the route for 192.168.10.x :
> route add 192.168.10.0/ 192.168.2.1

That's what I'm saying I have already done.  I set the default route to 
192.168.1.1, and I set a static route, 192.168.10.x/24 via 192.168.2.1.  The 
route is in effect, as evidenced:

For simplicity, let's say 192.168.1.1 has MAC 11:11:11:11:11:11 and let's say 
192.168.2.1 has mac 22:22:22:22:22:22.   

When I ping something on the internet, I see a packet go out my NIC, source IP 
192.168.1.100, destination MAC 11:11:11:11:11:11 and destination IP 8.8.8.8.  
It all works, I get a ping response.

When I ping 192.168.2.1 directly, I see a packet go out my NIC, source IP 
192.168.2.100, destination MAC 22:22:22:22:22:22 and destination IP 
192.168.2.1.  It all works, I get a ping response.

When I ping something on the other end of the VPN, I see a packet go out of my 
NIC, source IP 192.168.1.100, destination MAC 22:22:22:22:22:22 and destination 
IP 192.168.10.10 (or whatever.)  The firewall drops the packet, because duh, 
the source IP isn't in the same subnet as the firewall.

I am also exploring the NAT option, assuming I'm not going to be able to 
resolve the above problem.


___
OpenIndiana-discuss mailing list
OpenIndiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss