Re: [OpenIndiana-discuss] Antwort: Transferring everything in one zpool to be a, subtree in another larger zpool (Hans J. Albertsson)

2015-05-06 Thread Hans J Albertsson
Hear hear!

Hans J. Albertsson
From my Nexus 5
Den 5 maj 2015 22:36 skrev Timothy Coalson tsc...@mst.edu:

 If you transfer many snapshots, especially if some are small, there is a
 benefit from buffering: the receive end, after each snapshot, stops
 consuming data, possibly for several seconds.  Without a buffer, this will
 stall the sending side, so that nothing gets done during those pauses.  I
 have put a small buffer on the sending side and a larger buffer on the
 receiving side (as ssh is my current bottleneck, from encryption as you
 note, but it is fairly close to the network speed anyway), and it helps get
 things finished faster.

 Tim


 On Tue, May 5, 2015 at 8:32 AM, Doug Hughes d...@will.to wrote:

  you can also use ttcp or iperf just fine. I do that all the time and they
  are pretty much available anywhere. (ttcp is so simple it's just a single
  .c source file). The buffering isn't really the important part, using as
  close to raw tcp transport gets you the biggest benefit (vs
  encryption/decryption overhead and slowdown).
 
 
  Sent from my android device.
 
  -Original Message-
  From: Hans J. Albertsson hans.j.alberts...@gmail.com
  To: Discussion list for OpenIndiana openindiana-discuss@openindiana.org
 
  Sent: Tue, 05 May 2015 9:31
  Subject: Re: [OpenIndiana-discuss] Antwort: Transferring everything in
 one
  zpool to be a, subtree in another larger zpool (Hans J. Albertsson)
 
  mbuffer isn't included in FreeNAS.
 
  It IS available for FreeBSD, but I don't know if that version will
  behave responsibly in FreeNAS.
 
  FreeNAS devs say mbuffer won't be included in FreeNAS because we
  already do proper buffering via dd.
 
  Would dd work for buffering in Solaris/OI??
 
  On 2015-05-05 14:26, Sebastian Gabler wrote:
   Am 05.05.2015 um 14:00 schrieb
   openindiana-discuss-requ...@openindiana.org:
   As a side note; just piping to ssh is excruciatingly slow, using
 netcat,
   nc, speeds things up at least 4fold.
   The method that worked fine for me was using mbuffer. (I couldn't make
   netcat work, shame on me, but I also remember that there were several
   pros of using mbuffer. I think one of them was the ability to use
   digest to secure the transfer.)
  
   See:
  
 
 http://everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/
  
   That being said, it's exactly two years when I last did a migration
   with zfs send | zfs receive. I hope that still works.
  
   BR,
  
   Sebastian
  
   ___
   openindiana-discuss mailing list
   openindiana-discuss@openindiana.org
   http://openindiana.org/mailman/listinfo/openindiana-discuss
 
 
  ___
  openindiana-discuss mailing list
  openindiana-discuss@openindiana.org
  http://openindiana.org/mailman/listinfo/openindiana-discuss
  ___
  openindiana-discuss mailing list
  openindiana-discuss@openindiana.org
  http://openindiana.org/mailman/listinfo/openindiana-discuss
 
 ___
 openindiana-discuss mailing list
 openindiana-discuss@openindiana.org
 http://openindiana.org/mailman/listinfo/openindiana-discuss

___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss


Re: [OpenIndiana-discuss] Antwort: Transferring everything in one zpool to be a, subtree in another larger zpool (Hans J. Albertsson)

2015-05-05 Thread Sebastian Gabler

Am 05.05.2015 um 14:00 schrieb openindiana-discuss-requ...@openindiana.org:

As a side note; just piping to ssh is excruciatingly slow, using netcat,
nc, speeds things up at least 4fold.
The method that worked fine for me was using mbuffer. (I couldn't make 
netcat work, shame on me, but I also remember that there were several 
pros of using mbuffer. I think one of them was the ability to use digest 
to secure the transfer.)


See: 
http://everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/


That being said, it's exactly two years when I last did a migration with 
zfs send | zfs receive. I hope that still works.


BR,

Sebastian

___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss


Re: [OpenIndiana-discuss] Antwort: Transferring everything in one zpool to be a, subtree in another larger zpool (Hans J. Albertsson)

2015-05-05 Thread Hans J. Albertsson

mbuffer isn't included in FreeNAS.

It IS available for FreeBSD, but I don't know if that version will 
behave responsibly in FreeNAS.


FreeNAS devs say mbuffer won't be included in FreeNAS because we 
already do proper buffering via dd.


Would dd work for buffering in Solaris/OI??

On 2015-05-05 14:26, Sebastian Gabler wrote:
Am 05.05.2015 um 14:00 schrieb 
openindiana-discuss-requ...@openindiana.org:

As a side note; just piping to ssh is excruciatingly slow, using netcat,
nc, speeds things up at least 4fold.
The method that worked fine for me was using mbuffer. (I couldn't make 
netcat work, shame on me, but I also remember that there were several 
pros of using mbuffer. I think one of them was the ability to use 
digest to secure the transfer.)


See: 
http://everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/


That being said, it's exactly two years when I last did a migration 
with zfs send | zfs receive. I hope that still works.


BR,

Sebastian

___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss



___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss


Re: [OpenIndiana-discuss] Antwort: Transferring everything in one zpool to be a, subtree in another larger zpool (Hans J. Albertsson)

2015-05-05 Thread Doug Hughes
you can also use ttcp or iperf just fine. I do that all the time and they are 
pretty much available anywhere. (ttcp is so simple it's just a single .c source 
file). The buffering isn't really the important part, using as close to raw tcp 
transport gets you the biggest benefit (vs encryption/decryption overhead and 
slowdown).


Sent from my android device.

-Original Message-
From: Hans J. Albertsson hans.j.alberts...@gmail.com
To: Discussion list for OpenIndiana openindiana-discuss@openindiana.org
Sent: Tue, 05 May 2015 9:31
Subject: Re: [OpenIndiana-discuss] Antwort: Transferring everything in one 
zpool to be a, subtree in another larger zpool (Hans J. Albertsson)

mbuffer isn't included in FreeNAS.

It IS available for FreeBSD, but I don't know if that version will 
behave responsibly in FreeNAS.

FreeNAS devs say mbuffer won't be included in FreeNAS because we 
already do proper buffering via dd.

Would dd work for buffering in Solaris/OI??

On 2015-05-05 14:26, Sebastian Gabler wrote:
 Am 05.05.2015 um 14:00 schrieb 
 openindiana-discuss-requ...@openindiana.org:
 As a side note; just piping to ssh is excruciatingly slow, using netcat,
 nc, speeds things up at least 4fold.
 The method that worked fine for me was using mbuffer. (I couldn't make 
 netcat work, shame on me, but I also remember that there were several 
 pros of using mbuffer. I think one of them was the ability to use 
 digest to secure the transfer.)

 See: 
 http://everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/

 That being said, it's exactly two years when I last did a migration 
 with zfs send | zfs receive. I hope that still works.

 BR,

 Sebastian

 ___
 openindiana-discuss mailing list
 openindiana-discuss@openindiana.org
 http://openindiana.org/mailman/listinfo/openindiana-discuss


___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss
___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss


Re: [OpenIndiana-discuss] Antwort: Transferring everything in one zpool to be a subtree in another larger zpool

2015-05-05 Thread Hans J. Albertsson

In OI 151a7, the option -r is NOT available in zfs send.
See https://www.illumos.org/issues/2811

Has this been fixed in later OI versions???

On 2015-05-04 10:25, Thorsten Heit wrote:

2) Send the pool to your new machine:

root@oldmachine:# zfs send -R oldpool@snapshot_name | \
 ssh root@new_machine '/usr/sbin/zfs receive newpool/new_name'

-R creates a replication stream package that sends everything
recursively up to the named snapshot, including properties, snapshots,
descendent file systems and clones.

If you're not interested in older snapshots, use -rp instead of -R
whereas -r creates a recursive stream package and -p tells zfs send
to include properties.



___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss


Re: [OpenIndiana-discuss] Antwort: Transferring everything in one zpool to be a, subtree in another larger zpool (Hans J. Albertsson)

2015-05-05 Thread Hans J. Albertsson

ttcp?? ToolTalk CoPy?

On 2015-05-05 15:32, Doug Hughes wrote:

you can also use ttcp or iperf just fine. I do that all the time and they are 
pretty much available anywhere. (ttcp is so simple it's just a single .c source 
file). The buffering isn't really the important part, using as close to raw tcp 
transport gets you the biggest benefit (vs encryption/decryption overhead and 
slowdown).


Sent from my android device.




___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss


Re: [OpenIndiana-discuss] Antwort: Transferring everything in one zpool to be a subtree in another larger zpool

2015-05-05 Thread Hans J. Albertsson
I should have put this in the subject; the receiving zpool is on a 
FreeNas 9.3 box.


When I did as described by you, i e zfs send -R pool/name@snap | ...
the resulting dataset on Freenas didn't really work.
When trying to set up a share for the dataset on Freenas, I got 
unsupported share protocol:1 and sharing failed.


It seems this has to do with zfs properties and the fact that ZFS has 
hooks in the kernel in OI but not in FreeNas.


I'm guessing that just doing a zfs send pool/name@snap, i e not using 
-rp or -R, will make a working dataset on FreeNas which can then be shared.


I'll get back with a HowTo if I succeed or a warning if I don't.

As a side note; just piping to ssh is excruciatingly slow, using netcat, 
nc, speeds things up at least 4fold.


On the receiving end:

nc -w 600 -l 8023 | zfs recv pool/name

and on the sending end

zfs send pool/data@snap|nc -w 20 receive-address 8023



On 2015-05-04 10:25, Thorsten Heit wrote:

Hi,


I have this old machine running oi151a7 with a zpool containing a set of
filesystems used for media files of various types, and I want to

transfer

them all, in one go, safely but not unnecessarily slowly, to a new zpool

on

a different machine.

The old zpool is on a 2tb two-disk mirror on the oi151a7,  while the new
zpool is a freenas raidz2 of 12tb or so actual data space.

I want your advice on what is the correct, safest and/or most practical

way

to do it.

Both machines are on a GB network, and neither of them will be doing
anything else while transferring the data.

I'm guessing zfs send/recv over the network, using nc for buffering, but
I've never done this before, so please advice me.


Quite easy with zfs and a little time for the sending/receiving process:

1) Create a snapshot of the pool you intend to move:

root@oldmachine:# zfs snapshot -r oldpool@snapshot_name


2) Send the pool to your new machine:

root@oldmachine:# zfs send -R oldpool@snapshot_name | \
 ssh root@new_machine '/usr/sbin/zfs receive newpool/new_name'

-R creates a replication stream package that sends everything
recursively up to the named snapshot, including properties, snapshots,
descendent file systems and clones.

If you're not interested in older snapshots, use -rp instead of -R
whereas -r creates a recursive stream package and -p tells zfs send
to include properties.


3) Have a coffee or two and wait :-)


4) Optionally move the received zfs filesystems, volumes, ... one level up
in your new pool:

root@newmachine:# zfs rename newpool/new_name/filesystem1
newpool/filesystem1


I have done this several times in the past without problems. You could
optionally compress the stream with gzip and uncompress it when receiving:

root@oldmachine:# zfs send -R oldpool@snapshot_name | gzip | \
 ssh root@new_machine '/usr/bin/gunzip | /usr/sbin/zfs receive
newpool/new_name'


In my experience with some 100GB of data this took far longer in our LAN
than the same without compressing although the sending and receiving
machines should have been fast enough...


If you don't want to send the whole pool, replace oldpool above with
the file system / data set you're interested in; the commands stay the
same.


Best regards

Thorsten
___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss



___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss


Re: [OpenIndiana-discuss] Antwort: Transferring everything in one zpool to be a, subtree in another larger zpool (Hans J. Albertsson)

2015-05-05 Thread Timothy Coalson
If you transfer many snapshots, especially if some are small, there is a
benefit from buffering: the receive end, after each snapshot, stops
consuming data, possibly for several seconds.  Without a buffer, this will
stall the sending side, so that nothing gets done during those pauses.  I
have put a small buffer on the sending side and a larger buffer on the
receiving side (as ssh is my current bottleneck, from encryption as you
note, but it is fairly close to the network speed anyway), and it helps get
things finished faster.

Tim


On Tue, May 5, 2015 at 8:32 AM, Doug Hughes d...@will.to wrote:

 you can also use ttcp or iperf just fine. I do that all the time and they
 are pretty much available anywhere. (ttcp is so simple it's just a single
 .c source file). The buffering isn't really the important part, using as
 close to raw tcp transport gets you the biggest benefit (vs
 encryption/decryption overhead and slowdown).


 Sent from my android device.

 -Original Message-
 From: Hans J. Albertsson hans.j.alberts...@gmail.com
 To: Discussion list for OpenIndiana openindiana-discuss@openindiana.org
 Sent: Tue, 05 May 2015 9:31
 Subject: Re: [OpenIndiana-discuss] Antwort: Transferring everything in one
 zpool to be a, subtree in another larger zpool (Hans J. Albertsson)

 mbuffer isn't included in FreeNAS.

 It IS available for FreeBSD, but I don't know if that version will
 behave responsibly in FreeNAS.

 FreeNAS devs say mbuffer won't be included in FreeNAS because we
 already do proper buffering via dd.

 Would dd work for buffering in Solaris/OI??

 On 2015-05-05 14:26, Sebastian Gabler wrote:
  Am 05.05.2015 um 14:00 schrieb
  openindiana-discuss-requ...@openindiana.org:
  As a side note; just piping to ssh is excruciatingly slow, using netcat,
  nc, speeds things up at least 4fold.
  The method that worked fine for me was using mbuffer. (I couldn't make
  netcat work, shame on me, but I also remember that there were several
  pros of using mbuffer. I think one of them was the ability to use
  digest to secure the transfer.)
 
  See:
 
 http://everycity.co.uk/alasdair/2010/07/using-mbuffer-to-speed-up-slow-zfs-send-zfs-receive/
 
  That being said, it's exactly two years when I last did a migration
  with zfs send | zfs receive. I hope that still works.
 
  BR,
 
  Sebastian
 
  ___
  openindiana-discuss mailing list
  openindiana-discuss@openindiana.org
  http://openindiana.org/mailman/listinfo/openindiana-discuss


 ___
 openindiana-discuss mailing list
 openindiana-discuss@openindiana.org
 http://openindiana.org/mailman/listinfo/openindiana-discuss
 ___
 openindiana-discuss mailing list
 openindiana-discuss@openindiana.org
 http://openindiana.org/mailman/listinfo/openindiana-discuss

___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss


Re: [OpenIndiana-discuss] Antwort: Transferring everything in, one zpool to be a, subtree in another larger zpool (Hans J. Albertsson)

2015-05-05 Thread Sebastian Gabler
You could try to build it from source on your FreeNAS machine. If it 
builds, I guess it will work.


Here is the project page: http://www.maier-komor.de/mbuffer.html

BR

Sebastian

Am 05.05.2015 um 19:38 schrieb openindiana-discuss-requ...@openindiana.org:

Message: 3
Date: Tue, 05 May 2015 15:30:11 +0200
From: Hans J. Albertssonhans.j.alberts...@gmail.com
To: Discussion list for OpenIndiana
openindiana-discuss@openindiana.org
Subject: Re: [OpenIndiana-discuss] Antwort: Transferring everything in
one zpool to be a, subtree in another larger zpool (Hans J.
Albertsson)
Message-ID:5548c5e3.3000...@gmail.com
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

mbuffer isn't included in FreeNAS.

It IS available for FreeBSD, but I don't know if that version will
behave responsibly in FreeNAS.


___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss


Re: [OpenIndiana-discuss] Antwort: Transferring everything in one zpool to be a subtree in another larger zpool

2015-05-05 Thread Thorsten Heit
Hi,
 
 Thanks, that was exactly it.
 
 To add a few bits of info:
 
 You first must get a pair of private/public keys
 
 ssh-keyget -t rsa
 
 for the OI user to run the zfs send.

Um, yes, sorry, I assumed that you already did this ;-)


 I suppose you could use password login for root on freenas, but freenas 
 is set up to use rsa key controlled login for replication. So this way 

 probably has been tested a lot more.

I don't know whether receiving a data stream via SSH using a password will 
work at all; I'm normally using only public key authentification...


Regards

Thorsten
___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss


Re: [OpenIndiana-discuss] Antwort: Transferring everything in one zpool to be a subtree in another larger zpool

2015-05-05 Thread Thorsten Heit
Hi,

 I should have put this in the subject; the receiving zpool is on a 
 FreeNas 9.3 box.
 
 When I did as described by you, i e zfs send -R pool/name@snap | ...
 the resulting dataset on Freenas didn't really work.
 When trying to set up a share for the dataset on Freenas, I got 
 unsupported share protocol:1 and sharing failed.
 
 It seems this has to do with zfs properties and the fact that ZFS has 
 hooks in the kernel in OI but not in FreeNas.

A short look in the FreeBSD's zfs man page gives the impression that it 
only supports sharing via NFS (OI/Solaris also support CIFS), and you have 
to enable the NFS server part in your /etc/rc.conf; see for example 
https://forums.freebsd.org/threads/creating-a-zfs-network-share-over-nfs.34828/


 I'm guessing that just doing a zfs send pool/name@snap, i e not using 
 -rp or -R, will make a working dataset on FreeNas which can then be 
shared.

If you're only interested in the data sets, not in their properties, -r 
will suffice.


 As a side note; just piping to ssh is excruciatingly slow, using netcat, 

 nc, speeds things up at least 4fold.
 
 On the receiving end:
 
 nc -w 600 -l 8023 | zfs recv pool/name
 
 and on the sending end
 
 zfs send pool/data@snap|nc -w 20 receive-address 8023

Hm, I can't confirm that. Perhaps this has to do with the hardware you're 
using...?


Regards

Thorsten
___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss


Re: [OpenIndiana-discuss] Antwort: Transferring everything in one zpool to be a subtree in another larger zpool

2015-05-05 Thread Hans J. Albertsson

On 2015-05-05 11:56, Thorsten Heit wrote:

Hi,


I should have put this in the subject; the receiving zpool is on a
FreeNas 9.3 box.

When I did as described by you, i e zfs send -R pool/name@snap | ...
the resulting dataset on Freenas didn't really work.
When trying to set up a share for the dataset on Freenas, I got
unsupported share protocol:1 and sharing failed.

It seems this has to do with zfs properties and the fact that ZFS has
hooks in the kernel in OI but not in FreeNas.

A short look in the FreeBSD's zfs man page gives the impression that it
only supports sharing via NFS (OI/Solaris also support CIFS), and you have
to enable the NFS server part in your /etc/rc.conf; see for example
https://forums.freebsd.org/threads/creating-a-zfs-network-share-over-nfs.34828/


For FreeBSD you're right, but FreeNAS has done that already, and can 
handle exporting via CIFS and NFS from the webgui.





I'm guessing that just doing a zfs send pool/name@snap, i e not using
-rp or -R, will make a working dataset on FreeNas which can then be

shared.

If you're only interested in the data sets, not in their properties, -r
will suffice.
-r just gets me an error message invalid option r. I'm on OI 151a7, 
which might be relevant.

send w/o option seems to get the data across, ignoring any options and such.





As a side note; just piping to ssh is excruciatingly slow, using netcat,
nc, speeds things up at least 4fold.

On the receiving end:

nc -w 600 -l 8023 | zfs recv pool/name

and on the sending end

zfs send pool/data@snap|nc -w 20 receive-address 8023


Well, I don't know, I got this info from the web, and it supposedly 
applies to SmartOS and several other environments, and FreeNAS.


And to my setup.


Hm, I can't confirm that. Perhaps this has to do with the hardware you're
using...?


Regards

Thorsten
___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss



___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss


[OpenIndiana-discuss] Antwort: Transferring everything in one zpool to be a subtree in another larger zpool

2015-05-04 Thread Thorsten Heit
Hi,

 I have this old machine running oi151a7 with a zpool containing a set of
 filesystems used for media files of various types, and I want to 
transfer
 them all, in one go, safely but not unnecessarily slowly, to a new zpool 
on
 a different machine.
 
 The old zpool is on a 2tb two-disk mirror on the oi151a7,  while the new
 zpool is a freenas raidz2 of 12tb or so actual data space.
 
 I want your advice on what is the correct, safest and/or most practical 
way
 to do it.
 
 Both machines are on a GB network, and neither of them will be doing
 anything else while transferring the data.
 
 I'm guessing zfs send/recv over the network, using nc for buffering, but
 I've never done this before, so please advice me.


Quite easy with zfs and a little time for the sending/receiving process:

1) Create a snapshot of the pool you intend to move:

root@oldmachine:# zfs snapshot -r oldpool@snapshot_name


2) Send the pool to your new machine:

root@oldmachine:# zfs send -R oldpool@snapshot_name | \
ssh root@new_machine '/usr/sbin/zfs receive newpool/new_name'

-R creates a replication stream package that sends everything 
recursively up to the named snapshot, including properties, snapshots, 
descendent file systems and clones.

If you're not interested in older snapshots, use -rp instead of -R 
whereas -r creates a recursive stream package and -p tells zfs send 
to include properties.


3) Have a coffee or two and wait :-)


4) Optionally move the received zfs filesystems, volumes, ... one level up 
in your new pool:

root@newmachine:# zfs rename newpool/new_name/filesystem1 
newpool/filesystem1


I have done this several times in the past without problems. You could 
optionally compress the stream with gzip and uncompress it when receiving:

root@oldmachine:# zfs send -R oldpool@snapshot_name | gzip | \
ssh root@new_machine '/usr/bin/gunzip | /usr/sbin/zfs receive 
newpool/new_name'


In my experience with some 100GB of data this took far longer in our LAN 
than the same without compressing although the sending and receiving 
machines should have been fast enough...


If you don't want to send the whole pool, replace oldpool above with 
the file system / data set you're interested in; the commands stay the 
same.


Best regards

Thorsten
___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss


Re: [OpenIndiana-discuss] Antwort: Transferring everything in one zpool to be a subtree in another larger zpool

2015-05-04 Thread Hans J. Albertsson

Thanks, that was exactly it.

To add a few bits of info:

You first must get a pair of private/public keys

ssh-keyget -t rsa

for the OI user to run the zfs send.

Then copy the public key from ~/.ssh/id_rsa.pub  on OI to the user root 
(or other user with admin privs) on freenas.

In the web mgmt gui on freenas, click Account-view-users-root
and click edit at the bottom. Put the contents of your .pub key into 
the public key field of the edit form.


To test run

ssh root@freenas_address ls

When it works, run

pfexec zfs send -R Tank/dataset@snapshot | ssh root@freenas_address 
'/sbin/zfs recv Tank/Media/name'


on OI
zfs on freenas is in /sbin,

I suppose you could use password login for root on freenas, but freenas 
is set up to use rsa key controlled login for replication. So this way 
probably has been tested a lot more.



On 2015-05-04 10:25, Thorsten Heit wrote:

Hi,


I have this old machine running oi151a7 with a zpool containing a set of
filesystems used for media files of various types, and I want to

transfer

them all, in one go, safely but not unnecessarily slowly, to a new zpool

on

a different machine.

The old zpool is on a 2tb two-disk mirror on the oi151a7,  while the new
zpool is a freenas raidz2 of 12tb or so actual data space.

I want your advice on what is the correct, safest and/or most practical

way

to do it.

Both machines are on a GB network, and neither of them will be doing
anything else while transferring the data.

I'm guessing zfs send/recv over the network, using nc for buffering, but
I've never done this before, so please advice me.


Quite easy with zfs and a little time for the sending/receiving process:

1) Create a snapshot of the pool you intend to move:

root@oldmachine:# zfs snapshot -r oldpool@snapshot_name


2) Send the pool to your new machine:

root@oldmachine:# zfs send -R oldpool@snapshot_name | \
 ssh root@new_machine '/usr/sbin/zfs receive newpool/new_name'

-R creates a replication stream package that sends everything
recursively up to the named snapshot, including properties, snapshots,
descendent file systems and clones.

If you're not interested in older snapshots, use -rp instead of -R
whereas -r creates a recursive stream package and -p tells zfs send
to include properties.


3) Have a coffee or two and wait :-)


4) Optionally move the received zfs filesystems, volumes, ... one level up
in your new pool:

root@newmachine:# zfs rename newpool/new_name/filesystem1
newpool/filesystem1


I have done this several times in the past without problems. You could
optionally compress the stream with gzip and uncompress it when receiving:

root@oldmachine:# zfs send -R oldpool@snapshot_name | gzip | \
 ssh root@new_machine '/usr/bin/gunzip | /usr/sbin/zfs receive
newpool/new_name'


In my experience with some 100GB of data this took far longer in our LAN
than the same without compressing although the sending and receiving
machines should have been fast enough...


If you don't want to send the whole pool, replace oldpool above with
the file system / data set you're interested in; the commands stay the
same.


Best regards

Thorsten
___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss



___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss


Re: [OpenIndiana-discuss] Antwort: Transferring everything in one zpool to be a subtree in another larger zpool

2015-05-04 Thread Hans J. Albertsson

On 2015-05-04 10:25, Thorsten Heit wrote:
 Hi,
  ...
 3) Have a coffee or two and wait

More like leave overnight, for an HP N54L...


___
openindiana-discuss mailing list
openindiana-discuss@openindiana.org
http://openindiana.org/mailman/listinfo/openindiana-discuss