Re: HAST, zfs and local mirroring

[Sami Halabi]

hi,
another suggestion would be using mirrored zfsOnRoot on each host (out of
the 2 local disks) and then use zrep to sync the pools.
use with combination of carp to active/standby cluster.

Sami
בתאריך 3 ביוני 2016 11:11 PM,‏ "Peter Jeremy"  כתב:

> On 2016-Jun-03 22:12:55 +0700, Eugene Grosbein  wrote:
> >> all your media content is valid.  I've also had bad experiences with
> >> gmirror volumes silently getting out of sync on a crash.
> >
> >
> >gmirror or (gmirror+gjournal) ones?
>
> Plain gmirror.
>
> --
> Peter Jeremy
>
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: HAST, zfs and local mirroring

[Peter Jeremy]

On 2016-Jun-03 22:12:55 +0700, Eugene Grosbein  wrote:
>> all your media content is valid.  I've also had bad experiences with
>> gmirror volumes silently getting out of sync on a crash.
>
>
>gmirror or (gmirror+gjournal) ones?

Plain gmirror.

-- 
Peter Jeremy


signature.asc
Description: PGP signature

Re: HAST, zfs and local mirroring

[Freddie Cash]

On Fri, Jun 3, 2016 at 1:40 AM, Eugene M. Zheganin 
wrote:

> Hi.
>
> On 02.06.16 19:50, Slawa Olhovchenkov wrote:
> > I am suggesting next setup:
> >
> > node0:
> >   own pool zroot0: mirror-0: local_disk0
> >remote-iscsi_disk1/1
> >   local_disk1: exported by iscsi as remote-iscsi_disk0/1 to node1
> >
> > node1:
> >   own pool zroot1: mirror-0: local_disk0
> >remote-iscsi_disk0/1
> >   local_disk1: exported by iscsi as remote-iscsi_disk1/1 to node0
> >
> >
> > No HAST.
> > Disks synced by ZFS over iSCSI.
> But this way I will get two independent zfs pools (or I still didn't get
> it), one half of each will be stored on another machine. And I need
> something different - a continuously replicated disk resource, that
> would be available on both machines in case either will crash. Cluster
> filesystem would be fine, but as far as I know there's no such thing on
> FreeBSD, so I accept it will be unavailable on slave while mounted as
> read-write on the master. HAST looks like a thing thats fits my
> requirement, only that I wank it to be redundant inside each host too,
> and all the documentation shows examples without local redundancy.
>

​Which is exactly the setup that I showed you.

Host1:  diskA  diskB

Host2:  diskC  diskD

You create two HAST resources:
  hast1:  using diskA and diskC
  hast2:  using diskB and diskD

Then create a ZFS pool using hast1 and hast2 as a mirror vdev.

That way, if you lose diskA, then diskC takes over, and ZFS never even
notices.

If you lose both diskA and diskC, you still have half the ZFS mirror vdev
running, so the pool is intact.

If you lose all of Host1, Host2 takes over, imports the pool, and carries
on.

HAST + CARP (to manage the fail-over and pool export/import) + ZFS gives
you everything (well, mostly) that you want.  There will be a smidgeon of
downtime during the host-to-host fail-over that may or may not be workable
in your setup.

The other option is to investigate the Ceph clustered filesystem.  I
believe there's been work ongoing this year to get it working on top of ZFS
on FreeBSD.

There's also GlusterFS, which has seen a bunch of work to get it working on
top of ZFS on FreeBSD.

No idea how well those last two work.


-- 
Freddie Cash
fjwc...@gmail.com
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: HAST, zfs and local mirroring

[Eugene Grosbein]

03.06.2016 2:48, Peter Jeremy пишет:

On 2016-Jun-02 12:12:35 +0500, "Eugene M. Zheganin"  wrote:

differs a lot ? And why should I prefere this overcomplicated scheme
over the geom_mirror, which seems rather simple when comparing. Seems
like I can point HAST to /dev/mirror/whatever device, right ?


Because using RAID of any sort under ZFS defeats a lot of the smarts
in ZFS.  In particular, you can no longer rely on scrub verifying that
all your media content is valid.  I've also had bad experiences with
gmirror volumes silently getting out of sync on a crash.



gmirror or (gmirror+gjournal) ones?


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: HAST, zfs and local mirroring

[Pete French]

> You message lead me to a though that I could use iSCSI to replicate the
> zfs pool from node1 to both iSCSI-provided disk on a node2 in a 4-way
> mirror, right ? Are there any obvious obstacles to this, that I don't
> see, considering the bandwith will be enough ?

I have spent a long time doing this kind of thing - always needing
a pair of machines, 2 discs in each and 4-way mirroring. over the last
decade weve used ggated/ggatec + gmirrors, iscsi + gmirror, iscsi + zfs and
finally hast + zfs.

Out of all that I have to say that hnast has been a godsend - its is by far
the best of all the solutions. so my recommentation would be to
hast the drives into two hast resources and zpool on top of that. yes, you
have two hast devices, but why is that an issue ? what we also do is
to add separate ether cards into the machines just dedicated to the hast
traffic between them, so that is not a bottlneck.

the reason hwy hast is so good compared to all the other network
block evel solutions we have tried is that it behaves ell in the case
when things fail. Theres nothing wrong with either ggatec or iscsi
in the normal case, but in the past I had a lot of issues with the mirroring
layer not understanding properly when the unerlying devices vanish
due to crashes or other unexpected events. As the purpose of all of this is,
preseumably, fault tolerance, then thats unfortunate. this may have chnaged,
I havent tried these technologies out in a while, but with hast you
dont ever see a device simply vanish rom underneath your zpool, and the
whole thing continues to work reliably and smoothly until the remote node comes
back up, whereupon it resyncs perfectly.

really, go with hast + zpool unless you have a good reason not to...

-pete.

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: HAST, zfs and local mirroring

[Slawa Olhovchenkov]

On Fri, Jun 03, 2016 at 02:05:07PM +0500, Eugene M. Zheganin wrote:

> Hi,
> 
> On 02.06.16 19:50, Slawa Olhovchenkov wrote:
> >
> > I am suggesting next setup:
> >
> > node0:
> >   own pool zroot0: mirror-0: local_disk0
> >  remote-iscsi_disk1/1
> >   local_disk1: exported by iscsi as remote-iscsi_disk0/1 to node1
> >
> > node1:
> >   own pool zroot1: mirror-0: local_disk0
> >  remote-iscsi_disk0/1
> >   local_disk1: exported by iscsi as remote-iscsi_disk1/1 to node0
> You message lead me to a though that I could use iSCSI to replicate the
> zfs pool from node1 to both iSCSI-provided disk on a node2 in a 4-way
> mirror, right ? Are there any obvious obstacles to this, that I don't
> see, considering the bandwith will be enough ?

No, just regular mirror.
And yes, two independed pool.
10G link prefered. 1G link acts like old HDD, about 70MB/s transfer.
Every transit switch degrade performance.

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: HAST, zfs and local mirroring

[Slawa Olhovchenkov]

On Fri, Jun 03, 2016 at 01:40:54PM +0500, Eugene M. Zheganin wrote:

> Hi.
> 
> On 02.06.16 19:50, Slawa Olhovchenkov wrote:
> > I am suggesting next setup:
> >
> > node0:
> >   own pool zroot0: mirror-0: local_disk0
> >  remote-iscsi_disk1/1
> >   local_disk1: exported by iscsi as remote-iscsi_disk0/1 to node1
> >
> > node1:
> >   own pool zroot1: mirror-0: local_disk0
> >  remote-iscsi_disk0/1
> >   local_disk1: exported by iscsi as remote-iscsi_disk1/1 to node0
> >
> >
> > No HAST.
> > Disks synced by ZFS over iSCSI.
> But this way I will get two independent zfs pools (or I still didn't get
> it), one half of each will be stored on another machine.

Yes.

> And I need
> something different - a continuously replicated disk resource, that
> would be available on both machines in case either will crash.

I think this is imposible.
All modern FS (cluster also) believe exclusive modifivation of on-disk
data. Cluster FS just acts as services (but still believe exclusive
access to HDD)


> Cluster
> filesystem would be fine, but as far as I know there's no such thing on
> FreeBSD, so I accept it will be unavailable on slave while mounted as
> read-write on the master. HAST looks like a thing thats fits my
> requirement, only that I wank it to be redundant inside each host too,
> and all the documentation shows examples without local redundancy.

You can't touch local HDD HASTed to remote.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: HAST, zfs and local mirroring

[Eugene M. Zheganin]

Hi,

On 02.06.16 19:50, Slawa Olhovchenkov wrote:
>
> I am suggesting next setup:
>
> node0:
>   own pool zroot0: mirror-0: local_disk0
>remote-iscsi_disk1/1
>   local_disk1: exported by iscsi as remote-iscsi_disk0/1 to node1
>
> node1:
>   own pool zroot1: mirror-0: local_disk0
>remote-iscsi_disk0/1
>   local_disk1: exported by iscsi as remote-iscsi_disk1/1 to node0
You message lead me to a though that I could use iSCSI to replicate the
zfs pool from node1 to both iSCSI-provided disk on a node2 in a 4-way
mirror, right ? Are there any obvious obstacles to this, that I don't
see, considering the bandwith will be enough ?

Thanks.
Eugene.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: HAST, zfs and local mirroring

[Eugene M. Zheganin]

Hi.

On 02.06.16 19:50, Slawa Olhovchenkov wrote:
> I am suggesting next setup:
>
> node0:
>   own pool zroot0: mirror-0: local_disk0
>remote-iscsi_disk1/1
>   local_disk1: exported by iscsi as remote-iscsi_disk0/1 to node1
>
> node1:
>   own pool zroot1: mirror-0: local_disk0
>remote-iscsi_disk0/1
>   local_disk1: exported by iscsi as remote-iscsi_disk1/1 to node0
>
>
> No HAST.
> Disks synced by ZFS over iSCSI.
But this way I will get two independent zfs pools (or I still didn't get
it), one half of each will be stored on another machine. And I need
something different - a continuously replicated disk resource, that
would be available on both machines in case either will crash. Cluster
filesystem would be fine, but as far as I know there's no such thing on
FreeBSD, so I accept it will be unavailable on slave while mounted as
read-write on the master. HAST looks like a thing thats fits my
requirement, only that I wank it to be redundant inside each host too,
and all the documentation shows examples without local redundancy.

Thanks.
Eugene.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: HAST, zfs and local mirroring

[Peter Jeremy]

On 2016-Jun-02 12:12:35 +0500, "Eugene M. Zheganin"  wrote:
>differs a lot ? And why should I prefere this overcomplicated scheme
>over the geom_mirror, which seems rather simple when comparing. Seems
>like I can point HAST to /dev/mirror/whatever device, right ?

Because using RAID of any sort under ZFS defeats a lot of the smarts
in ZFS.  In particular, you can no longer rely on scrub verifying that
all your media content is valid.  I've also had bad experiences with
gmirror volumes silently getting out of sync on a crash.

-- 
Peter Jeremy


signature.asc
Description: PGP signature

Re: HAST, zfs and local mirroring

[Slawa Olhovchenkov]

On Thu, Jun 02, 2016 at 12:12:35PM +0500, Eugene M. Zheganin wrote:

> Hi.
> 
> On 01.06.16 18:23, Slawa Olhovchenkov wrote:
> >
> > Only FS support changed data bypass FS layer is Files-11 ODS-2 level,
> > may be hardware support required.
> >
> > Can you use ZFS mirror with one vdev local and other vdev by iSCSI?
> > Every node using separate ZFS pool in this case.
> If you mean that I should distribute the HAST one-disk device via iSCSI
> and then use it as a half of zfs mirrored pool on each node, then I
> should ask how the pool will decide which half is more recent - local or
> iSCSI, after I will import it and the kernel will found that it's vdevs
> differs a lot ? And why should I prefere this overcomplicated scheme
> over the geom_mirror, which seems rather simple when comparing. Seems
> like I can point HAST to /dev/mirror/whatever device, right ?

I am suggesting next setup:

node0:
  own pool zroot0: mirror-0: local_disk0
 remote-iscsi_disk1/1
  local_disk1: exported by iscsi as remote-iscsi_disk0/1 to node1

node1:
  own pool zroot1: mirror-0: local_disk0
 remote-iscsi_disk0/1
  local_disk1: exported by iscsi as remote-iscsi_disk1/1 to node0


No HAST.
Disks synced by ZFS over iSCSI.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: HAST, zfs and local mirroring

[Eugene M. Zheganin]

Hi.

On 01.06.16 18:23, Slawa Olhovchenkov wrote:
>
> Only FS support changed data bypass FS layer is Files-11 ODS-2 level,
> may be hardware support required.
>
> Can you use ZFS mirror with one vdev local and other vdev by iSCSI?
> Every node using separate ZFS pool in this case.
If you mean that I should distribute the HAST one-disk device via iSCSI
and then use it as a half of zfs mirrored pool on each node, then I
should ask how the pool will decide which half is more recent - local or
iSCSI, after I will import it and the kernel will found that it's vdevs
differs a lot ? And why should I prefere this overcomplicated scheme
over the geom_mirror, which seems rather simple when comparing. Seems
like I can point HAST to /dev/mirror/whatever device, right ?

Thanks.
Eugene.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: HAST, zfs and local mirroring

[Slawa Olhovchenkov]

On Tue, May 31, 2016 at 11:18:32PM +0500, Eugene M. Zheganin wrote:

> Hi.
> 
> I wat to start using HAST, I have two nodes and a pair of disk on each 
> node. So I want to use HASt in an environment where each HAST resource 
> would be mirrored. What is the preferred approach if I want to use ZFS 
> on an end-device to avoid exsessive fscking, and, in the same time, I 
> want to have some redundancy on a block level ? I see two possibility: 
> HAST on a zvol of a mirrored pool, and a ZFS on a hast. But recently I 
> heard that nested zfs (like zfs on zvol) is clamed unsupported. 
> Futhermore, I have zfs on a geli on a zvol, and this solution proved 
> itself to be very affected to livelocking - when disk i/o on a such fs 
> is above some treshold, system is locking, and the only way out is to 
> reset it. Should I chose geom_mirror to provide a device for HAST and 
> the build ZFS on it ?

Only FS support changed data bypass FS layer is Files-11 ODS-2 level,
may be hardware support required.

Can you use ZFS mirror with one vdev local and other vdev by iSCSI?
Every node using separate ZFS pool in this case.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: HAST, zfs and local mirroring

[Eugene M. Zheganin]

Hi.

On 01.06.16 02:49, Freddie Cash wrote:
> On Tue, May 31, 2016 at 11:18 AM, Eugene M. Zheganin
> >wrote:
>
> I wat to start using HAST, I have two nodes and a pair of disk on
> each node. So I want to use HASt in an environment where each HAST
> resource would be mirrored. What is the preferred approach if I
> want to use ZFS on an end-device to avoid exsessive fscking, and,
> in the same time, I want to have some redundancy on a block level
> ? I see two possibility: HAST on a zvol of a mirrored pool, and a
> ZFS on a hast. But recently I heard that nested zfs (like zfs on
> zvol) is clamed unsupported. Futhermore, I have zfs on a geli on a
> zvol, and this solution proved itself to be very affected to
> livelocking - when disk i/o on a such fs is above some treshold,
> system is locking, and the only way out is to reset it. Should I
> chose geom_mirror to provide a device for HAST and the build ZFS
> on it ?
>
>
> ​The generally recommend way to do this is to create a HAST resource
> out of 1 disk from each system, and then build the ZFS pool using the
> HAST resources as the "disks".
>
> That way, your ZFS pool is made up of 2 HAST devices in a mirror vdev.
>
> And each of the two HAST devices uses one disk from each server (total
> of four disks).
> ​
>  
>
>
But I don't need two HAST devices, I need one, but redundant on each server.

Eugene.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: HAST, zfs and local mirroring

[Freddie Cash]

On Tue, May 31, 2016 at 11:18 AM, Eugene M. Zheganin 
wrote:

> I wat to start using HAST, I have two nodes and a pair of disk on each
> node. So I want to use HASt in an environment where each HAST resource
> would be mirrored. What is the preferred approach if I want to use ZFS on
> an end-device to avoid exsessive fscking, and, in the same time, I want to
> have some redundancy on a block level ? I see two possibility: HAST on a
> zvol of a mirrored pool, and a ZFS on a hast. But recently I heard that
> nested zfs (like zfs on zvol) is clamed unsupported. Futhermore, I have zfs
> on a geli on a zvol, and this solution proved itself to be very affected to
> livelocking - when disk i/o on a such fs is above some treshold, system is
> locking, and the only way out is to reset it. Should I chose geom_mirror to
> provide a device for HAST and the build ZFS on it ?
>

​The generally recommend way to do this is to create a HAST resource out of
1 disk from each system, and then build the ZFS pool using the HAST
resources as the "disks".

That way, your ZFS pool is made up of 2 HAST devices in a mirror vdev.

And each of the two HAST devices uses one disk from each server (total of
four disks).
​


-- 
Freddie Cash
fjwc...@gmail.com
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"