Re: [OmniOS-discuss] 4kn or 512e with ashift=12

[Fred Liu]

> -Original Message-
> From: OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] On
> Behalf Of Chris Siebenmann
> Sent: 星期二, 三月 22, 2016 3:27
> To: Richard Elling
> Cc: omnios-discuss@lists.omniti.com
> Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
> 
> > > Adding the ashift argument to zpool was discussed every few years
> > > and so far was always deemed not enterprisey enough for the Solaris
> > > heritage, so the setup to tweak sd driver reports and properly rely
> > > on that layer was pushed instead.
> >
> > The issue is that once a drive model lies, then the Solaris approach
> > is to encode the lie into a whitelist, so that the lie is always
> > handled properly. The whitelist is in the sd.conf file.
> >
> > By contrast, the ZFSonLinux approach requires that the sysadmin knows
> > there is a lie and manually corrects for every invocation. This is
> > unfortunately a very error-prone approach.
> 
>  This implicitly assumes that the only reason to set ashift=12 is if you are
> currently using one or more drives that require it. I strongly disagree with 
> this
> view. Since ZFS cannot currently replace a 512n drive with a 512e one, I feel

*In theory* this replacement should work well if the lie works *correctly*.
 In ZoL, for the "-o ashift" is supported in "zpool replace", the replacement 
should also work in mixed sector sizes.
 And in illumos the whitelist will do the same.
 What errors have you ever seen?

> that you really want to force-create all pools with ashift=12 even on 512n 
> drives
> unless you're absolutely confident that you will be able to obtain replacement
> 512n drives over the lifetime of your pool (or unless the impact of using
> ashift=12 is so drastic on your usage case that you will need to totally 
> rethink
> your pool if you become unable to get 512n replacement drives).

Same as my comments above.
> 
>  For many usage cases, somewhat more space usage and perhaps somewhat
> slower pools are vastly preferable to a loss of pool redundancy over time. I 
> feel
> that OmniOS should at least give you the option here (in a less crude way than
> simply telling it that absolutely all of your drives are 4k drives, partly 
> because
> such general lies are problematic in various situations).
> 

The whitelist (sd.conf) should fit into this consideration. But not sure how 
mixed sector sizes impact the performance.


Thanks.

Fred
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

[Fred Liu]

> -Original Message-
> From: OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] On
> Behalf Of Jim Klimov
> Sent: 星期二, 三月 22, 2016 2:11
> To: Hanno Hirschberger; omnios-discuss@lists.omniti.com
> Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
> 
> 21 марта 2016 г. 10:02:03 CET, Hanno Hirschberger
>  пишет:
> >On 21.03.2016 08:00, Fred Liu wrote:
> >> So that means illumos can handle 512n and 4kn automatically and
> >properly?
> >
> >Not necessarily as far as I know. Sometime drives are emulating 512
> >blocks and don't properly tell the OS about that and Illumos ZFS is
> >aligning the drives with ashift=9 which leads to enormous performance
> >issues. Also forcing the system to handle drives with a specific sector
> >
> >size with the sd.conf doesn't turn out to be reliable in some cases (at
> >
> >least on my workstations). Here's what I do to ensure ashift=12 values:
> >
> >Reboot the system with a Linux live disk of your choice and install ZoL
> >
> >in the live session. Then create the ZFS pool, export it and reboot the
> >
> >machine. OmniOS / Illumos can import the new pool without problems and
> >the ashift value is correctly set. There was a fixed zpool binary
> >(Solaris 11 binary) flying around the internet which can handle the "-o
> >
> >shift=12" parameter and works with OmniOS but unfortunately I can't
> >find it again right now. This would make the reboot into a live session
> >obsolete.
> >
> >Does anyone know if the "ashift" parameter will be implemented in the
> >OmniOS / Illumos zpool binary in the near future?
> >
> >Best regards
> >
> >Hanno
> >___
> >OmniOS-discuss mailing list
> >OmniOS-discuss@lists.omniti.com
> >http://lists.omniti.com/mailman/listinfo/omnios-discuss
> 
> Adding the ashift argument to zpool was discussed every few years and so far
> was always deemed not enterprisey enough for the Solaris heritage, so the
> setup to tweak sd driver reports and properly rely on that layer was pushed
> instead.
> 
> That said, the old tweaked binary came with a blog post detailing the source
> changes; you're welcome to try a d port and rti it (I'd say there is enough 
> user
> demand to back the non-enterprisey fix to be on par with other OpenZFS
> siblings). At worst, you can publish the modernized binary as the original
> blogger did ;)

Is the post/source still accessible?


Thanks.

Fred
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

[Fred Liu]

> -Original Message-
> From: OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] On
> Behalf Of Hanno Hirschberger
> Sent: 星期一, 三月 21, 2016 17:02
> To: omnios-discuss@lists.omniti.com
> Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
> 
> On 21.03.2016 08:00, Fred Liu wrote:
> > So that means illumos can handle 512n and 4kn automatically and properly?
> 
> Not necessarily as far as I know. Sometime drives are emulating 512 blocks and
> don't properly tell the OS about that and Illumos ZFS is aligning the drives 
> with
> ashift=9 which leads to enormous performance issues. Also forcing the system

That is the goal(ashift=9) that 512e wants to realize. Isn't it? Emulation is 
to guarantee
compatiblity not performace, right?

> to handle drives with a specific sector size with the sd.conf doesn't turn 
> out to
> be reliable in some cases (at least on my workstations). Here's what I do to

What errors have you ever seen?

> ensure ashift=12 values:
> 
> Reboot the system with a Linux live disk of your choice and install ZoL in 
> the live
> session. Then create the ZFS pool, export it and reboot the machine. OmniOS /
> Illumos can import the new pool without problems and the ashift value is

This operation is driver-dependant. I have the similar workaround. It works for 
avago HBAs.
But it doesn't apply in NVMe drivers. The zpool comprised of NVMe SSDs created 
under ZoL can't be imported under Illumos
in my tests.

> correctly set. There was a fixed zpool binary (Solaris 11 binary) flying 
> around the
> internet which can handle the "-o shift=12" parameter and works with OmniOS
> but unfortunately I can't find it again right now. This would make the reboot
> into a live session obsolete.
> 
> Does anyone know if the "ashift" parameter will be implemented in the OmniOS
> / Illumos zpool binary in the near future?

I think the whitelist(sd.conf) is the choice by Illumos. But we may try build 
it ourselves if the source is available.

Thanks.

Fred 
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

[Fred Liu]

> -Original Message-
> From: OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] On
> Behalf Of Richard Elling
> Sent: 星期二, 三月 22, 2016 3:21
> To: Bob Friesenhahn
> Cc: omnios-discuss@lists.omniti.com
> Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
> 
> 
> > On Mar 21, 2016, at 12:19 PM, Bob Friesenhahn
>  wrote:
> >
> > On Mon, 21 Mar 2016, Richard Elling wrote:
> >>>
> >>> Adding the ashift argument to zpool was discussed every few years and so
> far was always deemed not enterprisey enough for the Solaris heritage, so the
> setup to tweak sd driver reports and properly rely on that layer was pushed
> instead.
> >>
> >> The issue is that once a drive model lies, then the Solaris approach
> >> is to encode the lie into a whitelist, so that the lie is always
> >> handled properly. The whitelist is in the sd.conf file.
> >
> > Does this approach require that Illumos users only use drive hardware much
> older than the version of Illumos they happen running since outdated whitelist
> won't know about the new lies?
> 
> Forunately, lies are becoming less common. But this raises a good point: if 
> your
> drive doesn't lie, then you don't need to workaround.

That means 512n and 4kn are handled correctly by default, right?


Thanks.

Fred
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

[Fred Liu]

> -Original Message-
> From: OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] On
> Behalf Of Richard Elling
> Sent: 星期二, 三月 22, 2016 3:19
> To: Richard Jahnel
> Cc: omnios-discuss@lists.omniti.com
> Subject: Re: [OmniOS-discuss] 4kn or 512e with ashift=12
> 
> 
> > On Mar 21, 2016, at 12:00 PM, Richard Jahnel 
> wrote:
> >
> > Both approaches have their error points.
> >
> > FWIW I would very very much like to be able to force my new pools into
> ashift=12. It would make drive purchasing and replacement a lot more straight
> forward and future resistant.
> 
> The fundamental problem is that this is a vdev-settable, not a pool settable.
> Today, it is very common for pools to have multiple ashifts active. Recently,
> per-vdev ZAP objects have been proposed and that code is working its way
> through the review and integration process.

Not sure how multiple-ashifts impact the performace.
> 
> Meanwhile, use one or more of the dozens of methods documented for doing
> this.
> 
> FWIW, most people with HDDs want space efficiency, because HDDs lost the
> performance race to SSDs long ago. In general, forcing everything to 4k 
> reduces
> space efficiency, so it is unlikely to be a good default.

Agree.

Thanks.

Fred
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

[Chris Siebenmann]

> >  This implicitly assumes that the only reason to set ashift=12 is
> > if you are currently using one or more drives that require it. I
> > strongly disagree with this view. Since ZFS cannot currently replace
> > a 512n drive with a 512e one, I feel [...]
> 
> *In theory* this replacement should work well if the lie works *correctly*.
>  In ZoL, for the "-o ashift" is supported in "zpool replace", the
> replacement should also work in mixed sector sizes.
>  And in illumos the whitelist will do the same.
>  What errors have you ever seen?

 We have seen devices that changed between (claimed) 512n and
(claimed) 512e/4k *within the same model number*; the only thing that
distinguished the two was firmware version (which is not something that
you can match in sd.conf). This came as a complete surprise to us the
first time we needed to replace an old (512n) one of these with a new
(512e) one.

 The sd.conf whitelist also requires a reboot to activate if you need
to add a new entry, as far as I know.

(Nor do I know what happens if you have some 512n disks and some
512e disks, both correctly recognized and in different pools, and
now you need to replace a 512n disk with a spare 512e disk so you
change sd.conf to claim that all of the 512e disks are 512n. I'd
like to think that ZFS will carry on as normal, but I'm not sure.
This makes it somewhat dangerous to change sd.conf on a live system.)

> >  For many usage cases, somewhat more space usage and perhaps
> > somewhat slower pools are vastly preferable to a loss of pool
> > redundancy over time. I feel that OmniOS should at least give you
> > the option here (in a less crude way than simply telling it that
> > absolutely all of your drives are 4k drives, partly because such
> > general lies are problematic in various situations).
>
> The whitelist (sd.conf) should fit into this consideration. But not
> sure how mixed sector sizes impact the performance.

 Oh, 512e disks in a 512n pool will probably have not great performance.
ZFS does a lot of unaligned reads and writes, unlike other filesystems;
if you say your disks are 512n, it really believes you and behaves
accordingly.

- cks
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] supermicro 3U all-in one storage system

[Geoff Nordli]

I found out there is an internal cage you can get to add two more 
disks.  Part # MCP-220-82611-0N




On 16-03-21 04:14 PM, Kevin Swab wrote:

These things work with the ahci driver:

http://www.addonics.com/products/ad4mspx2-a.php

Performance is marginal for use as an l2arc or slog, but they're fine
for installing the OS.  Hot-swapping a failed device is tricky but
possible - I use nylon screws in case I drop one on the MB...

Kevin



On 03/21/2016 04:07 PM, Geoff Nordli wrote:

On 16-03-19 01:28 PM, Richard Elling wrote:

On Mar 18, 2016, at 4:12 PM, Geoff Nordli  wrote:

Hi.

I have had good luck with the SuperStorage 6037R-E1R16L chassis with
the LSI 2308 IT mode HBA.

We have several similar servers. The X10DRH is fine. For a non-HA
system, single expander backplane
is ok (BPN-SAS3-836EL1).

Any thoughts on how to put 4 SATA drives in that chassis?   2 are easy,
since they fit in the back of the server, but the other two would need
to go in the front.

Is the nvme driver stable enough yet to use as l2arc/slog?

thanks,

Geoff
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss


___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] 4kn or 512e with ashift=12

[Richard Elling]

> On Mar 22, 2016, at 7:41 AM, Chris Siebenmann  wrote:
> 
>>> This implicitly assumes that the only reason to set ashift=12 is
>>> if you are currently using one or more drives that require it. I
>>> strongly disagree with this view. Since ZFS cannot currently replace
>>> a 512n drive with a 512e one, I feel [...]
>> 
>> *In theory* this replacement should work well if the lie works *correctly*.
>> In ZoL, for the "-o ashift" is supported in "zpool replace", the
>> replacement should also work in mixed sector sizes.
>> And in illumos the whitelist will do the same.
>> What errors have you ever seen?
> 
> We have seen devices that changed between (claimed) 512n and
> (claimed) 512e/4k *within the same model number*; the only thing that
> distinguished the two was firmware version (which is not something that
> you can match in sd.conf). This came as a complete surprise to us the
> first time we needed to replace an old (512n) one of these with a new
> (512e) one.
> 
> The sd.conf whitelist also requires a reboot to activate if you need
> to add a new entry, as far as I know.
> 
> (Nor do I know what happens if you have some 512n disks and some
> 512e disks, both correctly recognized and in different pools, and
> now you need to replace a 512n disk with a spare 512e disk so you
> change sd.conf to claim that all of the 512e disks are 512n. I'd
> like to think that ZFS will carry on as normal, but I'm not sure.
> This makes it somewhat dangerous to change sd.conf on a live system.)

What is missing from
http://wiki.illumos.org/display/illumos/ZFS+and+Advanced+Format+disks 

is:

1. how to change the un_phy_blocksize for any or all uns
2. how to set a default setting for all drives in sd.conf by setting attributes 
to
the "" of ""  (see sd(7d))

I am aware of no new HDDs with 512n, so this problem will go away for HDDs.
However, there are many SSDs that work better with un_phy_blocksize = 8192
and some vendors set sd.conf or source appropriately.
 -- richard

> 
>>> For many usage cases, somewhat more space usage and perhaps
>>> somewhat slower pools are vastly preferable to a loss of pool
>>> redundancy over time. I feel that OmniOS should at least give you
>>> the option here (in a less crude way than simply telling it that
>>> absolutely all of your drives are 4k drives, partly because such
>>> general lies are problematic in various situations).
>> 
>> The whitelist (sd.conf) should fit into this consideration. But not
>> sure how mixed sector sizes impact the performance.
> 
> Oh, 512e disks in a 512n pool will probably have not great performance.
> ZFS does a lot of unaligned reads and writes, unlike other filesystems;
> if you say your disks are 512n, it really believes you and behaves
> accordingly.
> 
>   - cks

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss