Re: OT: fastest way to copy one drive to another

[Joerg Lechner via users]

 Hi,
to avoid blocking of folders while copying from Your old system, You can make 
an installation on an usb flash medium and boot Your old laptop with this flash 
medium. For my copying processes (I have many of them) I use an old internal 
disk connected with an usb adapter to my laptop. Old internal tested disks 
(i.e. 1 TB) at ebay You can get for about 20,-€.
Kind regards




___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: finding damaged files on an XFS filesystem (Was Re: OT: fastest way to copy one drive to another)

[Francis . Montagnac]

Hi.

On Thu, 20 Sep 2018 19:06:08 -0500 Ranjan Maitra wrote:

> I am sorry to return to this, but I am still struggling to find out
> the answer to how to find files that had bad blocks in my old drive.

> To recap, in order to rescue a failing disk with an xfs filesystem,
> I used ddrescue to copy the contents to an identically sized, but
> healthy disk.

> The mapfile from ddrescue clearly indicated several bad sectors.

You can use "ddrescue --fill-mode" using this mapfile and the healthy
disk then "grep -rl" to find the files that had bad blocks.

For example:

echo BAD-SECTOR > BAD-SECTOR
ddrescue --fill-mode='l?*-' BAD-SECTOR /dev/Your_Healthy_Disk mapfile &

# mount /dev/Your_Healthy_Disk

grep -rl BAD-SECTOR Mount_Point

Adjust the value of the --fill-mode option according to your
mapfile. In my recent case I only had 

'?' non-tried block 
'*' failed block non-trimmed 

The l is to write also the location data.

Ref: https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html#Fill-mode

-- 
Francis
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: finding damaged files on an XFS filesystem (Was Re: OT: fastest way to copy one drive to another)

[Cameron Simpson]

On 22Sep2018 14:09, Roger Heflin  wrote:

If you have the original disk and can mount the filesystem then do this:

create a script like this and call it say /dir/testcatfile:
cat $1 > /dev/null
RC=$?
if [ ${RC} != 0 ] ; then
 echo "$1 is corrupt"
fi


Just a shell remark: because the shell considers any command to be a predicate, 
you can just go:


   cat "$1" >/dev/null || echo "$1 is corrupt"

i.e. assert the truth of the "cat" invocation (truth means success). Since "||" 
is short circuiting OR, just like C, the OR expression tries the first 
predicate (cat), and only tries the second if the first is false.


Truth or consequences.


chmod +x /dir/testcatfile

Then do this:
find /tmp -type f  -exec /dir/testcatfile {} \;

each file it gets an io error on will print out a message.


You can make this considerably faster via xargs:

 find /path/to/your/drive -type f -print0 \
 | xargs -0 cat -- >/dev/null 2>cat-errors.txt

which will try to cat everything in batches and report errors to 
"cat-errors.txt" for later review.


It should be substantially faster than the "find ... -exec script" version.

Cheers,
Cameron Simpson 
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: finding damaged files on an XFS filesystem (Was Re: OT: fastest way to copy one drive to another)

[Roger Heflin]

The find trick works on all filesystems, it only relies on the scsi
layer returning the io error back to the cat command.
On Sat, Sep 22, 2018 at 3:11 PM Ranjan Maitra  wrote:
>
> We do have the original disk. Just making sure before we do this: does it 
> matter that it is an xfs system?
>
> Many thanks,
> Ranjan
>
> On Sat, 22 Sep 2018 14:09:02 -0500 Roger Heflin  wrote:
>
> > If you have the original disk and can mount the filesystem then do this:
> >
> > create a script like this and call it say /dir/testcatfile:
> > cat $1 > /dev/null
> > RC=$?
> > if [ ${RC} != 0 ] ; then
> >   echo "$1 is corrupt"
> > fi
> >
> > chmod +x /dir/testcatfile
> >
> > Then do this:
> > find /tmp -type f  -exec /dir/testcatfile {} \;
> >
> > each file it gets an io error on will print out a message.
> >
> > This only works if you still have the original and can mount it as it
> > relies on the disk returning io errors on the original disk.
> >
> > On Sat, Sep 22, 2018 at 9:20 AM Tony Nelson
> >  wrote:
> > >
> > > On 18-09-22 00:44:17, Ranjan Maitra wrote:
> > >   ...
> > > > Is this the right thing to do?
> > >
> > > Probably.  I expect the affected files to contain NULs, which you could
> > > search for.  Then you'd know that you have the correct files and where
> > > the damage is.
> > >
> > > --
> > > 
> > > TonyN.:'   
> > >'  
> > > ___
> > > users mailing list -- users@lists.fedoraproject.org
> > > To unsubscribe send an email to users-le...@lists.fedoraproject.org
> > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives: 
> > > https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
> > ___
> > users mailing list -- users@lists.fedoraproject.org
> > To unsubscribe send an email to users-le...@lists.fedoraproject.org
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: 
> > https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
>
>
> --
> Important Notice: This mailbox is ignored: e-mails are set to be deleted on 
> receipt. Please respond to the mailing list if appropriate. For those needing 
> to send personal or professional e-mail, please use appropriate addresses.
> ___
> users mailing list -- users@lists.fedoraproject.org
> To unsubscribe send an email to users-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: finding damaged files on an XFS filesystem (Was Re: OT: fastest way to copy one drive to another)

[Ranjan Maitra]

We do have the original disk. Just making sure before we do this: does it 
matter that it is an xfs system?

Many thanks,
Ranjan

On Sat, 22 Sep 2018 14:09:02 -0500 Roger Heflin  wrote:

> If you have the original disk and can mount the filesystem then do this:
> 
> create a script like this and call it say /dir/testcatfile:
> cat $1 > /dev/null
> RC=$?
> if [ ${RC} != 0 ] ; then
>   echo "$1 is corrupt"
> fi
> 
> chmod +x /dir/testcatfile
> 
> Then do this:
> find /tmp -type f  -exec /dir/testcatfile {} \;
> 
> each file it gets an io error on will print out a message.
> 
> This only works if you still have the original and can mount it as it
> relies on the disk returning io errors on the original disk.
> 
> On Sat, Sep 22, 2018 at 9:20 AM Tony Nelson
>  wrote:
> >
> > On 18-09-22 00:44:17, Ranjan Maitra wrote:
> >   ...
> > > Is this the right thing to do?
> >
> > Probably.  I expect the affected files to contain NULs, which you could
> > search for.  Then you'd know that you have the correct files and where
> > the damage is.
> >
> > --
> > 
> > TonyN.:'   
> >'  
> > ___
> > users mailing list -- users@lists.fedoraproject.org
> > To unsubscribe send an email to users-le...@lists.fedoraproject.org
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: 
> > https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
> ___
> users mailing list -- users@lists.fedoraproject.org
> To unsubscribe send an email to users-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org


-- 
Important Notice: This mailbox is ignored: e-mails are set to be deleted on 
receipt. Please respond to the mailing list if appropriate. For those needing 
to send personal or professional e-mail, please use appropriate addresses.
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: finding damaged files on an XFS filesystem (Was Re: OT: fastest way to copy one drive to another)

[Roger Heflin]

If you have the original disk and can mount the filesystem then do this:

create a script like this and call it say /dir/testcatfile:
cat $1 > /dev/null
RC=$?
if [ ${RC} != 0 ] ; then
  echo "$1 is corrupt"
fi

chmod +x /dir/testcatfile

Then do this:
find /tmp -type f  -exec /dir/testcatfile {} \;

each file it gets an io error on will print out a message.

This only works if you still have the original and can mount it as it
relies on the disk returning io errors on the original disk.

On Sat, Sep 22, 2018 at 9:20 AM Tony Nelson
 wrote:
>
> On 18-09-22 00:44:17, Ranjan Maitra wrote:
>   ...
> > Is this the right thing to do?
>
> Probably.  I expect the affected files to contain NULs, which you could
> search for.  Then you'd know that you have the correct files and where
> the damage is.
>
> --
> 
> TonyN.:'   
>'  
> ___
> users mailing list -- users@lists.fedoraproject.org
> To unsubscribe send an email to users-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: finding damaged files on an XFS filesystem (Was Re: OT: fastest way to copy one drive to another)

[Tony Nelson]

On 18-09-22 00:44:17, Ranjan Maitra wrote:
 ...

Is this the right thing to do?


Probably.  I expect the affected files to contain NULs, which you could
search for.  Then you'd know that you have the correct files and where
the damage is.

--

TonyN.:'   
  '  
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: finding damaged files on an XFS filesystem (Was Re: OT: fastest way to copy one drive to another)

[Ranjan Maitra]

On Fri, 21 Sep 2018 14:34:06 +1000 Cameron Simpson  wrote:

> On 20Sep2018 21:31, Ranjan Maitra  wrote:
> >On Fri, 21 Sep 2018 12:12:26 +1000 Cameron Simpson  wrote:
> >> On 20Sep2018 19:06, Ranjan Maitra  wrote:
> >> >I am sorry to return to this, but I am still struggling to find out the 
> >> >answer
> >> >to how to find files that had bad blocks in my old drive.
> [...]
> >> >ddrescue on the new disk found no problems, but I know there are affected 
> >> >files (the source disk was over 95% full), and I would like to know what 
> >> >those files are called.
> >>
> >> See "man xfs_db". It looks like the "blockget -v -b block-number" command 
> >> may
> >> provide useful info.
> >
> >Thanks very much for this information: it is very helpful.
> >
> >I do know the bytes and can figure out the sectors that are damaged. But I 
> >can't figure out how that translates to the block number of an xfs 
> >filesystem. 
> >Is there any reference for how to do this?
> 
> I would presume that it counts from the start of the partition containing the 
> file system. The xfs_db command "sb" consults the filesystem superblock and 
> should tell you the block size (4KiB is typical I think).
> 
> So I would start by:
> 
> - find the start sector of the partition containing the filesystem (counting 
>   from zero); that is your base
> 
> - subtract that from the bad sector number; that should be the sector offset 
>   into the XFS partition
> 
> - divide by the number of sectors per block (eg 512 byte sectors and 4KiB 
>   blocks would mean 8 sectors per block); hopefully ddrescue tells you what 
>   sector size it is using
> 
> Look up that block with "blockget" and then see if the corresponding file 
> seems 
> to have corresponding damage in its data.
> 
> Disclaimer: I haven't tried this. But I can if you run into trouble, and I 
> also 
> know an XFS expert I might consult if we can't figure this out ourselves.
> 

I wanted to provide a bit more information on progress into my and my wife's 
efforts to rescue files from one of her failing hard drives to another of the 
same type, but in better health. 


Here's what we did to discover the files that were damaged on the broken drive. 
 I had used ddrescue (wonderful utility, very fast) to get the files off of the 
failing disk onto /dev/sdc.

I'm still not sure if what we did has worked because none of the files that 
were in damaged sectors appear to be damaged, but they are big text files and 
it is hard to know for sure.

(1) We located the start of the xfs partition in question:

$ sudo fdisk -l /dev/sdc
Disk /dev/sdc: 3.7 TiB, 4000787030016 bytes, 7814037168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: ED0AA573-BFBA-4223-9036-D1B7EB0E49B9

Device StartEndSectors  Size Type
/dev/sdc1   2048 7814023167 7814021120  3.7T Linux filesystem

It starts on byte partition_start_byte = 1048576.  I notice my block size is 
block_size = 4096 (and confirm it with xfs_db -c "sb" -c "print").

(2) My wife wrote a script to extract the affected bytes from the ddrescue 
mapfile and converted them into file system blocks (fsb), as used by xfs_db.  
Each ddrescue block has a start_byte and a byte_length.  The conversion is:
start_block = floor( (start_byte - partition_start_byte) / block_size )
block_length = ceil( byte_length / block_size)

We then pasted the results from all such sets of blocks together in a massive 
xfs_db command as follows:

xfs_db -c "blockget -n" -c "fsb start_block" -c "blockuse -n" -c "fsb 
start_block + 1" -c "blockuse -n" ... -c "fsb start_block + block_length" -c 
"blockuse -n" ...

We waited several minutes, then parsed the affected files from the stdout of 
the above command.  The python script to do the job is below:

Here's the resource that finally helped me grok xfs well enough to get this far:

https://righteousit.wordpress.com/2018/05/21/xfs-part-1-superblock/


 python script 

#!/usr/bin/python3

import math
import subprocess as sp

device = "/dev/sdc1"# my device
mapfile = "mapfile" # ddrescue mapfile
partition_byte_offset = 1048576 # the first byte of my partition
block_size = 4096   # my block size in bytes

with open(mapfile, "r") as fh:
cmd = ["xfs_db", "-c", "blockget -n"]
for line in fh:
if not line.startswith("#"):
words = line.split()
if words[2] == '-': # bad bytes
start = int(words[0], 16)
length = int(words[1], 16)
start_block = math.floor((start - 
partition_byte_offset) / block_size)
length_block = math.ceil(length / block_size)
print(str(start_block) + ", " + 
str(length_block))
for b

Re: OT: finding damaged files on an XFS filesystem (Was Re: OT: fastest way to copy one drive to another)

[Ranjan Maitra]

On Fri, 21 Sep 2018 14:34:06 +1000 Cameron Simpson  wrote:

> On 20Sep2018 21:31, Ranjan Maitra  wrote:
> >On Fri, 21 Sep 2018 12:12:26 +1000 Cameron Simpson  wrote:
> >> On 20Sep2018 19:06, Ranjan Maitra  wrote:
> >> >I am sorry to return to this, but I am still struggling to find out the 
> >> >answer
> >> >to how to find files that had bad blocks in my old drive.
> [...]
> >> >ddrescue on the new disk found no problems, but I know there are affected 
> >> >files (the source disk was over 95% full), and I would like to know what 
> >> >those files are called.
> >>
> >> See "man xfs_db". It looks like the "blockget -v -b block-number" command 
> >> may
> >> provide useful info.
> >
> >Thanks very much for this information: it is very helpful.
> >
> >I do know the bytes and can figure out the sectors that are damaged. But I 
> >can't figure out how that translates to the block number of an xfs 
> >filesystem. 
> >Is there any reference for how to do this?
> 
> I would presume that it counts from the start of the partition containing the 
> file system. The xfs_db command "sb" consults the filesystem superblock and 
> should tell you the block size (4KiB is typical I think).
> 
> So I would start by:
> 
> - find the start sector of the partition containing the filesystem (counting 
>   from zero); that is your base
> 
> - subtract that from the bad sector number; that should be the sector offset 
>   into the XFS partition
> 
> - divide by the number of sectors per block (eg 512 byte sectors and 4KiB 
>   blocks would mean 8 sectors per block); hopefully ddrescue tells you what 
>   sector size it is using
> 
> Look up that block with "blockget" and then see if the corresponding file 
> seems 
> to have corresponding damage in its data.
> 
> Disclaimer: I haven't tried this. But I can if you run into trouble, and I 
> also 
> know an XFS expert I might consult if we can't figure this out ourselves.
> 

Thanks very much! I have read the xfs_db manual, but it left me with more 
questions than 
answers.  Specifically blockget needs the "filesystem block number",
but what really is the filesystem block number?  Is that distinct from
the allocation group block number?  I issued one or two blockget -v -b
block-number commands, but they took so long and without knowing if I
was asking for the right block, I just gave up.  I feel like I need
someone experienced in xfs structure to give me a sense of how to navigate
through the disk using xfs_db.

Many thanks again for your help!

Best wishes,
Ranjan

___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: finding damaged files on an XFS filesystem (Was Re: OT: fastest way to copy one drive to another)

[Cameron Simpson]

On 20Sep2018 21:31, Ranjan Maitra  wrote:

On Fri, 21 Sep 2018 12:12:26 +1000 Cameron Simpson  wrote:

On 20Sep2018 19:06, Ranjan Maitra  wrote:
>I am sorry to return to this, but I am still struggling to find out the answer
>to how to find files that had bad blocks in my old drive.

[...]
>ddrescue on the new disk found no problems, but I know there are affected 
>files (the source disk was over 95% full), and I would like to know what 
>those files are called.


See "man xfs_db". It looks like the "blockget -v -b block-number" command may
provide useful info.


Thanks very much for this information: it is very helpful.

I do know the bytes and can figure out the sectors that are damaged. But I 
can't figure out how that translates to the block number of an xfs filesystem. 
Is there any reference for how to do this?


I would presume that it counts from the start of the partition containing the 
file system. The xfs_db command "sb" consults the filesystem superblock and 
should tell you the block size (4KiB is typical I think).


So I would start by:

- find the start sector of the partition containing the filesystem (counting 
 from zero); that is your base


- subtract that from the bad sector number; that should be the sector offset 
 into the XFS partition


- divide by the number of sectors per block (eg 512 byte sectors and 4KiB 
 blocks would mean 8 sectors per block); hopefully ddrescue tells you what 
 sector size it is using


Look up that block with "blockget" and then see if the corresponding file seems 
to have corresponding damage in its data.


Disclaimer: I haven't tried this. But I can if you run into trouble, and I also 
know an XFS expert I might consult if we can't figure this out ourselves.


Cheers,
Cameron Simpson 
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: finding damaged files on an XFS filesystem (Was Re: OT: fastest way to copy one drive to another)

[Ranjan Maitra]

On Fri, 21 Sep 2018 12:12:26 +1000 Cameron Simpson  wrote:

> On 20Sep2018 19:06, Ranjan Maitra  wrote:
> >I am sorry to return to this, but I am still struggling to find out the 
> >answer 
> >to how to find files that had bad blocks in my old drive.
> >
> >To recap, in order to rescue a failing disk with an xfs filesystem, I used 
> >ddrescue to copy the contents to an identically sized, but healthy disk.
> >The mapfile from ddrescue clearly indicated several bad sectors.
> >
> >ddrescue on the new disk found no problems, but I know there are affected 
> >files (the source disk was over 95% full), and I would like to know what 
> >those files are called.
> 
> See "man xfs_db". It looks like the "blockget -v -b block-number" command may 
> provide useful info.

Thanks very much for this information: it is very helpful.

I do know the bytes and can figure out the sectors that are damaged. But I 
can't figure out how that translates to the block number of an xfs filesystem. 
Is there any reference for how to do this?

Best wishes,
Ranjan
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: finding damaged files on an XFS filesystem (Was Re: OT: fastest way to copy one drive to another)

[Cameron Simpson]

On 20Sep2018 19:06, Ranjan Maitra  wrote:
I am sorry to return to this, but I am still struggling to find out the answer 
to how to find files that had bad blocks in my old drive.


To recap, in order to rescue a failing disk with an xfs filesystem, I used 
ddrescue to copy the contents to an identically sized, but healthy disk.
The mapfile from ddrescue clearly indicated several bad sectors.

ddrescue on the new disk found no problems, but I know there are affected files 
(the source disk was over 95% full), and I would like to know what those files 
are called.


See "man xfs_db". It looks like the "blockget -v -b block-number" command may 
provide useful info.


Cheers,
Cameron Simpson 
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

OT: finding damaged files on an XFS filesystem (Was Re: OT: fastest way to copy one drive to another)

[Ranjan Maitra]

On Wed, 12 Sep 2018 18:02:58 +1000 Cameron Simpson  wrote:

> On 12Sep2018 08:35, Cameron Simpson  wrote:
> >On 11Sep2018 15:11, Ranjan Maitra  wrote:
> >>There is also xfs_repair: will this utility find the affected files?
> >
> >Short answer: I don't actually know. It may be more useful to run it 
> >against the bad drive because if those blocks are file data blocks 
> >then on the bad drive it _may_ register them as bad and report; on the 
> >new drive they may just look like blocks of zero bytes because 
> >ddrescue will have filled them in.
> >
> >However, you _also_ need to run it against the new drive, because if 
> >the bad blocks were filesystem structure black (as opposed to file data 
> >blocks) then your filesystem will be corrupt anyway.
> 
> Just to followup to this, xfs_repair(8) has this interesting paragraph:
> 
>Disk Errors
>xfs_repair aborts on most disk I/O errors. Therefore, if you are trying
>to  repair  a  filesystem that was damaged due to a disk drive failure,
>steps should be taken to ensure that all blocks in the  filesystem  are
>readable and writable before attempting to use xfs_repair to repair the
>filesystem. A possible method is using dd(8) to copy the  data  onto  a
>good disk.
> 
> So it may abort on your old drive. I suspect that if it completes on the old 
> drive that just means it didn't access any bad sectors. That may not mean the 
> copies on the good drive are good if their _data_ sectors were bad (and 
> filled 
> with zeroes by dd_rescue).
> 
> You definitely need to run it against the good drive as well.
> 

I am sorry to return to this, but I am still struggling to find out the answer 
to how to find files that had bad blocks in my old drive.

To recap, in order to rescue a failing disk with an xfs filesystem, I used 
ddrescue to copy the contents to an identically sized, but healthy disk. 

The mapfile from ddrescue clearly indicated several bad sectors.

ddrescue on the new disk found no problems, but I know there are affected files 
(the source disk was over 95% full), and I would like to know what those files 
are called.

I have seen help for ntfs 
(https://superuser.com/questions/346381/how-can-i-generate-a-list-of-files-from-a-list-of-sectors)
 and various other filesystems 
(https://wiki.archlinux.org/index.php/Identify_damaged_files, 
https://sites.google.com/site/itmyshare/storage/storage-disk/bad-blocks-how-to),
 but nothing for xfs.

Does anyone know how to identify these files?

Many thanks for more pointers,
Ranjan
-- 
Important Notice: This mailbox is ignored: e-mails are set to be deleted on 
receipt. Please respond to the mailing list if appropriate. For those needing 
to send personal or professional e-mail, please use appropriate addresses.
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Michael D. Setzer II]

On 13 Sep 2018 at 14:43, Roberto Ragusa wrote:

Subject:Re: OT: fastest way to copy one drive to another
To: users@lists.fedoraproject.org
From:   Roberto Ragusa 
Date sent:  Thu, 13 Sep 2018 14:43:51 +0200
Send reply to:  Community support for Fedora users 


> On 09/08/2018 01:34 PM, Michael D. Setzer II wrote:
> 
> > It is a c program that was included with g4l when I took it over, but then 
> > I 
> > rewrote it to do only what was actually used by the project.
> > 
> > It's included with the complete source code on sourceforge with the project 
> > source.
> 
> This is useful and may be worth a separate standalone package.
> 
> I usually use "pv" for pipe progress output.
> It apparently has buffer size option too, I'm discovering just now.

Wasn't aware of that program, and it seems to work with dialog.
Only issue I have, is that it is over 60+K in size, while the jetcat-mod is 
about 
8.5K. Since it is only working in ram, testing didn't show improvement making 
buffers larger, so use a 1M with the dd. Since once the disk buffer is full, it 
stays that way for most of the process. Perhaps with newer disks a larger 
buffer might increase speed a little.


jetcat-mod outputs lines ever 5 seconds by the option used, and has the 
format used by the dialog progress bar. G4L is using the dialog menu system 
along with the script for processing.
An output line looks like this.

47.16%   4044.00MB of 8574.67MB time: 0:01:13  55.40MB/sec

For the progress bar the first number is the critical part, rest is just 
displayed 
to show progress so far of the total size, elapsed time so far, and the 
calculated speed so far. With options that have compression, this can greatly 
increase the effective speed if unused space has been zeroed out.

With Fedora Core 3 long ago. Did a clean install on an 80G disk, and then 
did an image and it was 12G in sized. Cleared the unused space and redid 
image, and it dropped size to 2.5G.  Unused space with random data is still 
backed up and doesn't compress well.

Thanks for the info.

> 
> Regards.
> 
> -- 
>Roberto Ragusamail at robertoragusa.it
> ___
> users mailing list -- users@lists.fedoraproject.org
> To unsubscribe send an email to users-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org


++
 Michael D. Setzer II - Computer Science Instructor (Retired) 
 mailto:mi...@guam.net
 mailto:msetze...@gmail.com
 Guam - Where America's Day Begins
 G4L Disk Imaging Project maintainer 
 http://sourceforge.net/projects/g4l/
++

http://setiathome.berkeley.edu (Original)
Number of Seti Units Returned:  19,471
Processing time:  32 years, 290 days, 12 hours, 58 minutes
(Total Hours: 287,489)

BOINC@HOME CREDITS

ROSETTA  65786414.038490 | ABC  16613838.513356
SETI109526605.084600 | EINSTEIN141382979.499240
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Roberto Ragusa]

On 09/08/2018 01:34 PM, Michael D. Setzer II wrote:

> It is a c program that was included with g4l when I took it over, but then I 
> rewrote it to do only what was actually used by the project.
> 
> It's included with the complete source code on sourceforge with the project 
> source.

This is useful and may be worth a separate standalone package.

I usually use "pv" for pipe progress output.
It apparently has buffer size option too, I'm discovering just now.

Regards.

-- 
   Roberto Ragusamail at robertoragusa.it
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Cameron Simpson]

On 12Sep2018 08:35, Cameron Simpson  wrote:

On 11Sep2018 15:11, Ranjan Maitra  wrote:

There is also xfs_repair: will this utility find the affected files?


Short answer: I don't actually know. It may be more useful to run it 
against the bad drive because if those blocks are file data blocks 
then on the bad drive it _may_ register them as bad and report; on the 
new drive they may just look like blocks of zero bytes because 
ddrescue will have filled them in.


However, you _also_ need to run it against the new drive, because if 
the bad blocks were filesystem structure black (as opposed to file data 
blocks) then your filesystem will be corrupt anyway.


Just to followup to this, xfs_repair(8) has this interesting paragraph:

  Disk Errors
  xfs_repair aborts on most disk I/O errors. Therefore, if you are trying
  to  repair  a  filesystem that was damaged due to a disk drive failure,
  steps should be taken to ensure that all blocks in the  filesystem  are
  readable and writable before attempting to use xfs_repair to repair the
  filesystem. A possible method is using dd(8) to copy the  data  onto  a
  good disk.

So it may abort on your old drive. I suspect that if it completes on the old 
drive that just means it didn't access any bad sectors. That may not mean the 
copies on the good drive are good if their _data_ sectors were bad (and filled 
with zeroes by dd_rescue).


You definitely need to run it against the good drive as well.

Cheers,
Cameron Simpson 
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Cameron Simpson]

On 11Sep2018 15:11, Ranjan Maitra  wrote:
I wanted to mention that as I mentioned earlier, I used ddrescue and was able 
to copy the drive. However, the old (source) drive was found to have a few 
blocks that were bad. From what I understand from reading online, one can find 
these files. However, my filesystems for these two drives are xfs but most of 
the documentation is for other kinds of filesystems. How do I do this for xfs 
filesystems?


There is also xfs_repair: will this utility find the affected files?


Short answer: I don't actually know. It may be more useful to run it against 
the bad drive because if those blocks are file data blocks then on the bad 
drive it _may_ register them as bad and report; on the new drive they may just 
look like blocks of zero bytes because ddrescue will have filled them in.


However, you _also_ need to run it against the new drive, because if the bad 
blocks were filesystem structure black (as opposed to file data blocks) then 
your filesystem will be corrupt anyway.


This is one of the advantages of a file based copy instead of and image based 
copy such as dd: if there are bad blocks within a file a file based copy will 
report the bad file (because it couldn't read it, and it knows the name of the 
file it is trying to copy).  Also, the new drive will have a clean filesystem.  
An image based copy will copy damaged data structures (or copy bad data blocks 
as zeroed blocks).


By bad data structure I'm considering damage done to the filesystem metadata by 
the drive failures. If these "rescued" blocks lie within the filesystem 
structure data then your new filesystem will have damage: you _need_ to run 
xfs_repair over it anyway.


With a file based copy you start with a _correct_ empty filesystem on the 
target and add correct data to it.


XFS is stable against reboots and crashes on the whole, by maintaining its data 
structures in a cohenrent fashion, and which is why its fsck command is a 
compatability shell script which does some option parsing and then does nothing 
because the "mount" journal replay does any repair.


However, "out of band" damage caused by a failing drive cannot be implicitly 
repaired because the filesystem doesn't know about it. You need xfs_repair for 
that.


The xfs_repair manual page ("man xfs_repair") is extensive and informative. Had 
a read.


Cheers,
Cameron Simpson 
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Ranjan Maitra]

Hi,

I wanted to mention that as I mentioned earlier, I used ddrescue and was able 
to copy the drive. However, the old (source) drive was found to have a few 
blocks that were bad. From what I understand from reading online, one can find 
these files. However, my filesystems for these two drives are xfs but most of 
the documentation is for other kinds of filesystems. How do I do this for xfs 
filesystems?

There is also xfs_repair: will this utility find the affected files?

Many thanks,
Ranjan 
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Michael D. Setzer II]

On 10 Sep 2018 at 19:05, wwp wrote:

Date sent:  Mon, 10 Sep 2018 19:05:13 +0200
From:   wwp 
To: users@lists.fedoraproject.org
Subject:Re: OT: fastest way to copy one drive to another
Send reply to:  Community support for Fedora users 


> Hello Michael,
> 
> 
> On Fri, 07 Sep 2018 13:00:17 +1000 "Michael D. Setzer II" 
>  wrote:
> 
> > A number of good replies already, but just to add a note:
> > I've been the maintainer of the g4l disk imaging project since 2004, and it 
> > basically uses dd for most operations.
> 
> I'm replying, not only OT to the ML but OT to this thread, because I've
> just experimented g4l for backing up several windows systems (for Linux
> boxes I run my own rsync-based scripts) and wanted to share that. The
> process was pretty simple: shutting them down, booting from the USB storage
> where g4l resides (either legacy or UEFI mode), configured the process
> to send to a FTP server over the LAN. The process is reliable and
> pretty fast, approx 2h for each 500GB disk (RAW copy mode), I though
> it would be longer. The only thing I added is generating a .md5 file on
> storage, in case, that should be done by g4l IMO.
For testing the image, since it is a compressed file and lzop and gzip both 
have a option to check the archive (-t) and it is a 32bit check, so don't know 
what adding a .md5 file to confirm the file versus checking the image. Did 
once have a brand new server that I was creating images on, and they were 
corrupt. Ran an memtest on system, and it turned out one of brand new 
512M modules had an error that passed the first 7 tests with no error, but 
failed on the 8 test pattern. So in the docs recommend always using the 
compression program test to validate the image.

> 
> My former experience were using a similar approach but with clonezilla
> (in that time, no FTP sending but local storage or at least I couldn't
> find it, and a slower process, less simple at least, partition by
> partition and a user interface that is prone to user error), which led
> me to back my Windows systems up way less often.
> 

Generally recommend a full disk image the first time, the ntfsclone option 
only backs up used data on ntfs partitions, so is usually faster. 
G4L can be booted from a windows disk using grub4dos, was easy to add 
with 95 and 98 and XP as a boot option, 7 and above require making the 
grub4dos the primary boot manager, and you have to do some tricks in 
renaming files. 

Thanks for the info. There is a doc file on sourceforge, but not many 
download it.

Working on version 0.55, and just did alpha 48. Just had a typhoon pass, and 
power was out for 23 hours, so just getting back online. Had 192 emails.



> IOW, Michael, thanks for providing g4l!
> 
> 
> Regards,
> 
> -- 
> wwp
> 


++
 Michael D. Setzer II - Computer Science Instructor (Retired) 
 mailto:mi...@guam.net
 mailto:msetze...@gmail.com
 Guam - Where America's Day Begins
 G4L Disk Imaging Project maintainer 
 http://sourceforge.net/projects/g4l/
++

http://setiathome.berkeley.edu (Original)
Number of Seti Units Returned:  19,471
Processing time:  32 years, 290 days, 12 hours, 58 minutes
(Total Hours: 287,489)

BOINC@HOME CREDITS

ROSETTA  65783064.026787 | ABC  16613838.513356
SETI109522169.740739 | EINSTEIN141379519.999240
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[wwp]

Hello Michael,


On Fri, 07 Sep 2018 13:00:17 +1000 "Michael D. Setzer II"  
wrote:

> A number of good replies already, but just to add a note:
> I've been the maintainer of the g4l disk imaging project since 2004, and it 
> basically uses dd for most operations.

I'm replying, not only OT to the ML but OT to this thread, because I've
just experimented g4l for backing up several windows systems (for Linux
boxes I run my own rsync-based scripts) and wanted to share that. The
process was pretty simple: shutting them down, booting from the USB storage
where g4l resides (either legacy or UEFI mode), configured the process
to send to a FTP server over the LAN. The process is reliable and
pretty fast, approx 2h for each 500GB disk (RAW copy mode), I though
it would be longer. The only thing I added is generating a .md5 file on
storage, in case, that should be done by g4l IMO.

My former experience were using a similar approach but with clonezilla
(in that time, no FTP sending but local storage or at least I couldn't
find it, and a slower process, less simple at least, partition by
partition and a user interface that is prone to user error), which led
me to back my Windows systems up way less often.

IOW, Michael, thanks for providing g4l!


Regards,

-- 
wwp


pgpps79xhf4L8.pgp
Description: OpenPGP digital signature
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Ranjan Maitra]

Dear friends,

I wanted to thank everyone for contributing and helping out. In the end, I 
decided to go for ddrescue (because I could not find jetcat-mod beyond some old 
posts of Michael online). In any case, ddrescue copied everything with a meter 
in around six hours. It also reported a few bad bits (in the old drive) and I 
am now trying to figure out how to identify the files with those bad bits.

Thank you again for all your help and discussion. I learnt a lot of new and 
cool way of doing things! I have said this in the past, but I love this 
discussion group for how helpful and free with time and suggestions people are. 
My father-in-law uses Ubuntu for historical reasons and they do not come even 
close in those discussion lists (in fact they are very far).

Best wishes,
Ranjan

On Sat, 8 Sep 2018 17:08:30 -0600 JD  wrote:

> Hello Cameron,
> +1
> 
> 
> On 09/08/2018 04:48 PM, Cameron Simpson wrote:
> > On 09Sep2018 06:45, Michael D. Setzer II  wrote:
> >> On 8 Sep 2018 at 17:10, Patrick O'Callaghan wrote:
> > [...]
> >>> The link in the footer points to g4l, but nothing in your message
> >>> mentioning jetcat-mod makes that connection. Only an off-list message
> >>> from you explained it, and you then repeated the information on-list.
> >>
> >> In my original message to the thread I specified that I had been the
> >> maintainer of the g4l disk imaging project since 2004, and made some
> >> comments. Later there was talk of using dd to do the copy, and I 
> >> pulled the
> >> one line from the 2000+ line script that specifically does the clone 
> >> copy
> >> process.
> >>
> >> The entire project is on sourceforge with the source code. Project is
> >> completely free, no paid options, no donations. Just an option that some
> >> might find useful.
> >>
> >> Wasn't trying to force it on anyone, and just sharing what I had seen 
> >> over the
> >> years.
> >
> > As the one on the tar/cp side of this discussion, I'd like to support 
> > Michael here.
> >
> > He did indeed say he was the g4l maintainer. And he was perfectly 
> > clear when he post his dd pipeline that it came from the g4l package. 
> > It isn't unreasonable at all that it uses a tool that is part of that 
> > same package.
> >
> > There's any number of "progress reporting cat" type commands out 
> > there; I've got one of my own called prcat [1] (just a perl script 
> > accepting a commandline buffer size, no fancy threading or parallelism).
> >
> > IMO there's little value in this bickering over an _example_ command 
> > line whose function is clear on inspection just because it has an odd 
> > but obviously equivalent to "cat" command in the middle. Particularly 
> > since nothing in this discussion is Fedora specific; it all applies to 
> > pretty much any Linux system and nearly applies to any UNIX system in 
> > general (such as the Mac I'm typing this on). Can we please go back to 
> > _technical_ discussion instead of this?
> >
> > To satisfy the "no link to the code" people:
> > [1] https://bitbucket.org/cameron_simpson/css/src/tip/bin/prcat
> >
> > Cheers,
> > Cameron Simpson 
> > ___
> > users mailing list -- users@lists.fedoraproject.org
> > To unsubscribe send an email to users-le...@lists.fedoraproject.org
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives: 
> > https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
> ___
> users mailing list -- users@lists.fedoraproject.org
> To unsubscribe send an email to users-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org


-- 
Important Notice: This mailbox is ignored: e-mails are set to be deleted on 
receipt. Please respond to the mailing list if appropriate. For those needing 
to send personal or professional e-mail, please use appropriate addresses.
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[JD]

Hello Cameron,
+1


On 09/08/2018 04:48 PM, Cameron Simpson wrote:

On 09Sep2018 06:45, Michael D. Setzer II  wrote:

On 8 Sep 2018 at 17:10, Patrick O'Callaghan wrote:

[...]

The link in the footer points to g4l, but nothing in your message
mentioning jetcat-mod makes that connection. Only an off-list message
from you explained it, and you then repeated the information on-list.


In my original message to the thread I specified that I had been the
maintainer of the g4l disk imaging project since 2004, and made some
comments. Later there was talk of using dd to do the copy, and I 
pulled the
one line from the 2000+ line script that specifically does the clone 
copy

process.

The entire project is on sourceforge with the source code. Project is
completely free, no paid options, no donations. Just an option that some
might find useful.

Wasn't trying to force it on anyone, and just sharing what I had seen 
over the

years.


As the one on the tar/cp side of this discussion, I'd like to support 
Michael here.


He did indeed say he was the g4l maintainer. And he was perfectly 
clear when he post his dd pipeline that it came from the g4l package. 
It isn't unreasonable at all that it uses a tool that is part of that 
same package.


There's any number of "progress reporting cat" type commands out 
there; I've got one of my own called prcat [1] (just a perl script 
accepting a commandline buffer size, no fancy threading or parallelism).


IMO there's little value in this bickering over an _example_ command 
line whose function is clear on inspection just because it has an odd 
but obviously equivalent to "cat" command in the middle. Particularly 
since nothing in this discussion is Fedora specific; it all applies to 
pretty much any Linux system and nearly applies to any UNIX system in 
general (such as the Mac I'm typing this on). Can we please go back to 
_technical_ discussion instead of this?


To satisfy the "no link to the code" people:
[1] https://bitbucket.org/cameron_simpson/css/src/tip/bin/prcat

Cheers,
Cameron Simpson 
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Cameron Simpson]

On 09Sep2018 06:45, Michael D. Setzer II  wrote:

On 8 Sep 2018 at 17:10, Patrick O'Callaghan wrote:

[...]

The link in the footer points to g4l, but nothing in your message
mentioning jetcat-mod makes that connection. Only an off-list message
from you explained it, and you then repeated the information on-list.


In my original message to the thread I specified that I had been the
maintainer of the g4l disk imaging project since 2004, and made some
comments. Later there was talk of using dd to do the copy, and I pulled the
one line from the 2000+ line script that specifically does the clone copy
process.

The entire project is on sourceforge with the source code. Project is
completely free, no paid options, no donations. Just an option that some
might find useful.

Wasn't trying to force it on anyone, and just sharing what I had seen over the
years.


As the one on the tar/cp side of this discussion, I'd like to support Michael 
here.


He did indeed say he was the g4l maintainer. And he was perfectly clear when he 
post his dd pipeline that it came from the g4l package. It isn't unreasonable 
at all that it uses a tool that is part of that same package.


There's any number of "progress reporting cat" type commands out there; I've 
got one of my own called prcat [1] (just a perl script accepting a commandline 
buffer size, no fancy threading or parallelism).


IMO there's little value in this bickering over an _example_ command line whose 
function is clear on inspection just because it has an odd but obviously 
equivalent to "cat" command in the middle. Particularly since nothing in this 
discussion is Fedora specific; it all applies to pretty much any Linux system 
and nearly applies to any UNIX system in general (such as the Mac I'm typing 
this on). Can we please go back to _technical_ discussion instead of this?


To satisfy the "no link to the code" people:
[1] https://bitbucket.org/cameron_simpson/css/src/tip/bin/prcat

Cheers,
Cameron Simpson 
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Patrick O'Callaghan]

On Sun, 2018-09-09 at 06:45 +1000, Michael D. Setzer II wrote:
> > The link in the footer points to g4l, but nothing in your message
> > mentioning jetcat-mod makes that connection. Only an off-list message
> > from you explained it, and you then repeated the information on-list.
> > 
> 
> In my original message to the thread I specified that I had been the 
> maintainer of the g4l disk imaging project since 2004, and made some 
> comments. Later there was talk of using dd to do the copy, and I pulled the 
> one line from the 2000+ line script that specifically does the clone copy 
> process. 

That's clear now, but it wasn't at all clear at the time as you didn't
make any reference to jetcat-mod being part of g4l, and this wasn't
easily discoverable except by actually examining the latter.

> The entire project is on sourceforge with the source code. Project is 
> completely free, no paid options, no donations. Just an option that some 
> might find useful. 

That's good, and I'm sure some will find it useful, but (as Ed says) it
would be much more accessible as an RPM in some package repo.

poc
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Michael D. Setzer II]

On 8 Sep 2018 at 17:10, Patrick O'Callaghan wrote:

Subject:    Re: OT: fastest way to copy one drive to another
From:   Patrick O'Callaghan 
To: users@lists.fedoraproject.org
Date sent:  Sat, 08 Sep 2018 17:10:05 +0100
Send reply to:  Community support for Fedora users 


> On Sat, 2018-09-08 at 22:43 +1000, Michael D. Setzer II wrote:
> > On 8 Sep 2018 at 20:23, Ed Greshko wrote:
> > 
> > From:   Ed Greshko 
> > Subject:Re: OT: fastest way to copy one drive to another
> > To: users@lists.fedoraproject.org
> > Date sent:  Sat, 8 Sep 2018 20:23:45 +0800
> > Send reply to:  Community support for Fedora users 
> > 
> > 
> > > On 9/8/18 6:50 PM, Patrick O'Callaghan wrote:
> > > > On Sat, 2018-09-08 at 08:38 +1000, Michael D. Setzer II wrote:
> > > > > The jetcat-mod is a filter that monitors the process of the copy, and 
> > > > > outputs the
> > > > > info every 5 seconds with this option. The dialog then displays a 
> > > > > running graph of
> > > > > the progress along with estimated speed. In the clone case there is no
> > > > > compression, so the rate is generally the physical speed of disk. 
> > > > > With image files
> > > > > that are compressed, the effective speed can vary greatly depending 
> > > > > on how much
> > > > > the data was compressed. Mostly just gives progress versus just 
> > > > > waiting for the dd
> > > > > command to finish. 
> > > > 
> > > > The only results I see from Google refer to actual jet engines, so I 
> > > > ask again,
> > > > where does this come from? 
> > > 
> > > I know what you mean.
> > > 
> > > I avoid using commands/tools/utilities that aren't part of the Fedora 
> > > distributions
> > > when answering queries on the mailing list.
> > > 
> > > The exception would be commands/tools/utilities which are easily 
> > > installed from other
> > > well-known repos.  I try and
> > > take care to mention the repo they can been installed from.
> > > 
> > > I try my best not to keep people guessing.  And if I do, I give the links 
> > > to where
> > > something can be found.
> > > 
> > 
> > The link is in the footer of every message I send???
> > Joined a Fedora Mentoring list long ago, but it must have been a dead list. 
> > Got a welcome message, but then message to list gave no responses.
> 
> The link in the footer points to g4l, but nothing in your message
> mentioning jetcat-mod makes that connection. Only an off-list message
> from you explained it, and you then repeated the information on-list.
> 

In my original message to the thread I specified that I had been the 
maintainer of the g4l disk imaging project since 2004, and made some 
comments. Later there was talk of using dd to do the copy, and I pulled the 
one line from the 2000+ line script that specifically does the clone copy 
process. 

The entire project is on sourceforge with the source code. Project is 
completely free, no paid options, no donations. Just an option that some 
might find useful. 

Wasn't trying to force it on anyone, and just sharing what I had seen over the 
years. 


> poc
> ___
> users mailing list -- users@lists.fedoraproject.org
> To unsubscribe send an email to users-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org


++
 Michael D. Setzer II - Computer Science Instructor (Retired) 
 mailto:mi...@guam.net
 mailto:msetze...@gmail.com
 Guam - Where America's Day Begins
 G4L Disk Imaging Project maintainer 
 http://sourceforge.net/projects/g4l/
++

http://setiathome.berkeley.edu (Original)
Number of Seti Units Returned:  19,471
Processing time:  32 years, 290 days, 12 hours, 58 minutes
(Total Hours: 287,489)

BOINC@HOME CREDITS

ROSETTA  65779701.987238 | ABC  16613838.513356
SETI109517999.255698 | EINSTEIN141376685.499240
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Patrick O'Callaghan]

On Sat, 2018-09-08 at 22:43 +1000, Michael D. Setzer II wrote:
> On 8 Sep 2018 at 20:23, Ed Greshko wrote:
> 
> From: Ed Greshko 
> Subject:  Re: OT: fastest way to copy one drive to another
> To:   users@lists.fedoraproject.org
> Date sent:Sat, 8 Sep 2018 20:23:45 +0800
> Send reply to:Community support for Fedora users 
> 
> 
> > On 9/8/18 6:50 PM, Patrick O'Callaghan wrote:
> > > On Sat, 2018-09-08 at 08:38 +1000, Michael D. Setzer II wrote:
> > > > The jetcat-mod is a filter that monitors the process of the copy, and 
> > > > outputs the
> > > > info every 5 seconds with this option. The dialog then displays a 
> > > > running graph of
> > > > the progress along with estimated speed. In the clone case there is no
> > > > compression, so the rate is generally the physical speed of disk. With 
> > > > image files
> > > > that are compressed, the effective speed can vary greatly depending on 
> > > > how much
> > > > the data was compressed. Mostly just gives progress versus just waiting 
> > > > for the dd
> > > > command to finish. 
> > > 
> > > The only results I see from Google refer to actual jet engines, so I ask 
> > > again,
> > > where does this come from? 
> > 
> > I know what you mean.
> > 
> > I avoid using commands/tools/utilities that aren't part of the Fedora 
> > distributions
> > when answering queries on the mailing list.
> > 
> > The exception would be commands/tools/utilities which are easily installed 
> > from other
> > well-known repos.  I try and
> > take care to mention the repo they can been installed from.
> > 
> > I try my best not to keep people guessing.  And if I do, I give the links 
> > to where
> > something can be found.
> > 
> 
> The link is in the footer of every message I send???
> Joined a Fedora Mentoring list long ago, but it must have been a dead list. 
> Got a welcome message, but then message to list gave no responses.

The link in the footer points to g4l, but nothing in your message
mentioning jetcat-mod makes that connection. Only an off-list message
from you explained it, and you then repeated the information on-list.

poc
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Ed Greshko]

On 9/8/18 8:43 PM, Michael D. Setzer II wrote:
> The link is in the footer of every message I send???


Do you think people read the footers?   If they did, we wouldn't get people 
sending
"unsubscribe" messages to the list.  :-)

And, if they did, do you think they would glean from that where and how to 
compile to
get that one item?

FWIW, I stop reading when I reach the footers of people's messages.  I didn't 
notice
the link.  I doubt that I'm unique.  Even if I did see the link, I'm rather 
dense and
I probably wouldn't have made the connection.




signature.asc
Description: OpenPGP digital signature
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Michael D. Setzer II]

On 8 Sep 2018 at 20:23, Ed Greshko wrote:

From:   Ed Greshko 
Subject:Re: OT: fastest way to copy one drive to another
To: users@lists.fedoraproject.org
Date sent:  Sat, 8 Sep 2018 20:23:45 +0800
Send reply to:  Community support for Fedora users 


> On 9/8/18 6:50 PM, Patrick O'Callaghan wrote:
> > On Sat, 2018-09-08 at 08:38 +1000, Michael D. Setzer II wrote:
> >> The jetcat-mod is a filter that monitors the process of the copy, and 
> >> outputs the
> >> info every 5 seconds with this option. The dialog then displays a running 
> >> graph of
> >> the progress along with estimated speed. In the clone case there is no
> >> compression, so the rate is generally the physical speed of disk. With 
> >> image files
> >> that are compressed, the effective speed can vary greatly depending on how 
> >> much
> >> the data was compressed. Mostly just gives progress versus just waiting 
> >> for the dd
> >> command to finish. 
> > The only results I see from Google refer to actual jet engines, so I ask 
> > again,
> > where does this come from? 
> 
> I know what you mean.
> 
> I avoid using commands/tools/utilities that aren't part of the Fedora 
> distributions
> when answering queries on the mailing list.
> 
> The exception would be commands/tools/utilities which are easily installed 
> from other
> well-known repos.  I try and
> take care to mention the repo they can been installed from.
> 
> I try my best not to keep people guessing.  And if I do, I give the links to 
> where
> something can be found.
> 

The link is in the footer of every message I send???
Joined a Fedora Mentoring list long ago, but it must have been a dead list. 
Got a welcome message, but then message to list gave no responses.

> One of the not so nice things I've seen happen is people, not the most 
> experienced,
> install non-standard stuff on their systems and then forget they've done it.  
> And
> then they complain when stuff that worked for them before no longer works as 
> an
> upgrade changes a library that is no longer compatible.
> 
> 
> 
> 
> 


++
 Michael D. Setzer II - Computer Science Instructor (Retired) 
 mailto:mi...@guam.net
 mailto:msetze...@gmail.com
 Guam - Where America's Day Begins
 G4L Disk Imaging Project maintainer 
 http://sourceforge.net/projects/g4l/
++

http://setiathome.berkeley.edu (Original)
Number of Seti Units Returned:  19,471
Processing time:  32 years, 290 days, 12 hours, 58 minutes
(Total Hours: 287,489)

BOINC@HOME CREDITS

ROSETTA  65778205.643098 | ABC  16613838.513356
SETI109514958.178814 | EINSTEIN141374481.499240
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Ed Greshko]

On 9/8/18 6:50 PM, Patrick O'Callaghan wrote:
> On Sat, 2018-09-08 at 08:38 +1000, Michael D. Setzer II wrote:
>> The jetcat-mod is a filter that monitors the process of the copy, and 
>> outputs the
>> info every 5 seconds with this option. The dialog then displays a running 
>> graph of
>> the progress along with estimated speed. In the clone case there is no
>> compression, so the rate is generally the physical speed of disk. With image 
>> files
>> that are compressed, the effective speed can vary greatly depending on how 
>> much
>> the data was compressed. Mostly just gives progress versus just waiting for 
>> the dd
>> command to finish. 
> The only results I see from Google refer to actual jet engines, so I ask 
> again,
> where does this come from? 

I know what you mean.

I avoid using commands/tools/utilities that aren't part of the Fedora 
distributions
when answering queries on the mailing list.

The exception would be commands/tools/utilities which are easily installed from 
other
well-known repos.  I try and
take care to mention the repo they can been installed from.

I try my best not to keep people guessing.  And if I do, I give the links to 
where
something can be found.

One of the not so nice things I've seen happen is people, not the most 
experienced,
install non-standard stuff on their systems and then forget they've done it.  
And
then they complain when stuff that worked for them before no longer works as an
upgrade changes a library that is no longer compatible.






signature.asc
Description: OpenPGP digital signature
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Michael D. Setzer II]

On 8 Sep 2018 at 11:50, Patrick O'Callaghan wrote:

Subject:    Re: OT: fastest way to copy one drive to another
From:   Patrick O'Callaghan 
To: users@lists.fedoraproject.org
Date sent:  Sat, 08 Sep 2018 11:50:08 +0100
Send reply to:  Community support for Fedora users 


> On Sat, 2018-09-08 at 08:38 +1000, Michael D. Setzer II wrote:
> > The jetcat-mod is a filter that monitors the process of the copy, and 
> > outputs 
> > the info every 5 seconds with this option. The dialog then displays a 
> > running 
> > graph of the progress along with estimated speed. In the clone case there 
> > is 
> > no compression, so the rate is generally the physical speed of disk. With 
> > image files that are compressed, the effective speed can vary greatly 
> > depending on how much the data was compressed. Mostly just gives 
> > progress versus just waiting for the dd command to finish.
> 
> The only results I see from Google refer to actual jet engines, so I
> ask again, where does this come from?
> 
It is a c program that was included with g4l when I took it over, but then I 
rewrote it to do only what was actually used by the project.

It's included with the complete source code on sourceforge with the project 
source.


> poc
> ___
> users mailing list -- users@lists.fedoraproject.org
> To unsubscribe send an email to users-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org


++
 Michael D. Setzer II - Computer Science Instructor (Retired) 
 mailto:mi...@guam.net
 mailto:msetze...@gmail.com
 Guam - Where America's Day Begins
 G4L Disk Imaging Project maintainer 
 http://sourceforge.net/projects/g4l/
++

http://setiathome.berkeley.edu (Original)
Number of Seti Units Returned:  19,471
Processing time:  32 years, 290 days, 12 hours, 58 minutes
(Total Hours: 287,489)

BOINC@HOME CREDITS

ROSETTA  65778205.643098 | ABC  16613838.513356
SETI109514958.178814 | EINSTEIN141374481.499240
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Jon Ingason]

Den 2018-09-08 kl. 12:50, skrev Patrick O'Callaghan:
> On Sat, 2018-09-08 at 08:38 +1000, Michael D. Setzer II wrote:
>> The jetcat-mod is a filter that monitors the process of the copy, and 
>> outputs 
>> the info every 5 seconds with this option. The dialog then displays a 
>> running 
>> graph of the progress along with estimated speed. In the clone case there is 
>> no compression, so the rate is generally the physical speed of disk. With 
>> image files that are compressed, the effective speed can vary greatly 
>> depending on how much the data was compressed. Mostly just gives 
>> progress versus just waiting for the dd command to finish.
> 
> The only results I see from Google refer to actual jet engines, so I
> ask again, where does this come from?

Found this when I googled "jetcat-mod":

ghost4linux/jetcat-mod at master · kevinneu/ghost4linux · GitHub
https://github.com/kevinneu/ghost4linux/blob/.../jetcat-mod

I guess he is refering to this. You can find "jetcat-mod" in the G4L
(Ghost4Linux) distribution.
> 
> poc
> ___
> users mailing list -- users@lists.fedoraproject.org
> To unsubscribe send an email to users-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org
> 


-- 
Regards

Jon Ingason
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Michael D. Setzer II]

On 8 Sep 2018 at 10:01, Alexander Dalloz wrote:

Subject:Re: OT: fastest way to copy one drive to another
To: users@lists.fedoraproject.org
From:   Alexander Dalloz 
Date sent:  Sat, 8 Sep 2018 10:01:11 +0200
Send reply to:  Community support for Fedora users 


> Am 08.09.2018 um 00:38 schrieb Michael D. Setzer II:
> > The jetcat-mod is a filter that monitors the process of the copy, and 
> > outputs
> > the info every 5 seconds with this option. The dialog then displays a 
> > running
> > graph of the progress along with estimated speed. In the clone case there is
> > no compression, so the rate is generally the physical speed of disk. With
> > image files that are compressed, the effective speed can vary greatly
> > depending on how much the data was compressed. Mostly just gives
> > progress versus just waiting for the dd command to finish.
> 
> Why no just dd parameter "status=progress"?
> 

Back in 2004, don't think status had progress as an option at all?
Second the format of the status is different, and reports on bytes and the 
output to screen comes at different rates, and is on the same line over and 
over.
Third, it doesn't have any relationship to the size of what is being copied in 
the data or time.

Did a test with an approximat 8G file

time dd bs=1M if=e-2009-02-22.lzo 2>/dev/null | /jetcat-mod -f 5000 -p 
8780465 | dd bs=1M of=testx.x 2>/dev/null
1.63% 140.00MB of 8574.67MB time: 0:00:03  46.67MB/sec
7.00% 600.00MB of 8574.67MB time: 0:00:08  75.00MB/sec
11.66%   1000.00MB of 8574.67MB time: 0:00:13  76.92MB/sec
16.93%   1452.00MB of 8574.67MB time: 0:00:18  80.67MB/sec
20.34%   1744.00MB of 8574.67MB time: 0:00:23  75.83MB/sec
23.65%   2028.00MB of 8574.67MB time: 0:00:28  72.43MB/sec
25.98%   2228.00MB of 8574.67MB time: 0:00:33  67.52MB/sec
29.16%   2500.00MB of 8574.67MB time: 0:00:38  65.79MB/sec
31.39%   2692.00MB of 8574.67MB time: 0:00:43  62.60MB/sec
33.96%   2912.00MB of 8574.67MB time: 0:00:48  60.67MB/sec
36.25%   3108.00MB of 8574.67MB time: 0:00:53  58.64MB/sec
39.51%   3388.00MB of 8574.67MB time: 0:00:58  58.41MB/sec
41.66%   3572.00MB of 8574.67MB time: 0:01:03  56.70MB/sec
45.02%   3860.00MB of 8574.67MB time: 0:01:08  56.76MB/sec
47.16%   4044.00MB of 8574.67MB time: 0:01:13  55.40MB/sec
49.96%   4284.00MB of 8574.67MB time: 0:01:18  54.92MB/sec
52.57%   4508.00MB of 8574.67MB time: 0:01:23  54.31MB/sec
55.23%   4736.00MB of 8574.67MB time: 0:01:28  53.82MB/sec
57.24%   4908.00MB of 8574.67MB time: 0:01:33  52.77MB/sec
59.62%   5112.00MB of 8574.67MB time: 0:01:38  52.16MB/sec
62.23%   5336.00MB of 8574.67MB time: 0:01:43  51.81MB/sec
64.94%   5568.00MB of 8574.67MB time: 0:01:48  51.56MB/sec
67.64%   5800.00MB of 8574.67MB time: 0:01:53  51.33MB/sec
70.72%   6064.00MB of 8574.67MB time: 0:01:58  51.39MB/sec
73.38%   6292.00MB of 8574.67MB time: 0:02:03  51.15MB/sec
76.18%   6532.00MB of 8574.67MB time: 0:02:08  51.03MB/sec
78.93%   6768.00MB of 8574.67MB time: 0:02:13  50.89MB/sec
81.26%   6968.00MB of 8574.67MB time: 0:02:18  50.49MB/sec
84.06%   7208.00MB of 8574.67MB time: 0:02:23  50.41MB/sec
86.81%   7444.00MB of 8574.67MB time: 0:02:28  50.30MB/sec
87.93%   7540.00MB of 8574.67MB time: 0:02:33  49.28MB/sec
90.31%   7744.00MB of 8574.67MB time: 0:02:38  49.01MB/sec
93.44%   8012.00MB of 8574.67MB time: 0:02:43  49.15MB/sec
95.16%   8160.00MB of 8574.67MB time: 0:02:48  48.57MB/sec
100.00%  8373.71MB of 8574.67MB time: 0:02:52  48.68MB/sec
100

real2m51.646s
user0m0.817s
sys 1m18.394s

time dd bs=1M if=e-2009-02-22.lzo of=testx.x status=progress
8767143936 bytes (8.8 GB, 8.2 GiB) copied, 158 s, 55.5 MB/s
8373+1 records in
8373+1 records out
8780465661 bytes (8.8 GB, 8.2 GiB) copied, 158.455 s, 55.4 MB/s

real2m38.507s
user0m0.110s
sys 0m47.135s

The dd with progress was about 13 seconds faster, but doesn't show  
percentages or time info. 

The dialog progress bar uses the lines created every 5 seconds to show info 
and bar. 
Note: I just put the file size in the line, which wasn't in the correct format, 
size 
I just dropped the last 3 digits of byte  instead of converting it to K value.

Tried to post a message with an example of bar, but it is currently on hold 
since the 28K image file result in message being larger than 60K, which is 
the list limit. 

The info doesn't make a difference in the image, and is only to give progress 
info.


> Alexander
> ___
> users mailing list -- users@lists.fedoraproject.org
> To u

Re: OT: fastest way to copy one drive to another

[Patrick O'Callaghan]

On Sat, 2018-09-08 at 08:38 +1000, Michael D. Setzer II wrote:
> The jetcat-mod is a filter that monitors the process of the copy, and outputs 
> the info every 5 seconds with this option. The dialog then displays a running 
> graph of the progress along with estimated speed. In the clone case there is 
> no compression, so the rate is generally the physical speed of disk. With 
> image files that are compressed, the effective speed can vary greatly 
> depending on how much the data was compressed. Mostly just gives 
> progress versus just waiting for the dd command to finish.

The only results I see from Google refer to actual jet engines, so I
ask again, where does this come from?

poc
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Alexander Dalloz]

Am 08.09.2018 um 00:38 schrieb Michael D. Setzer II:

The jetcat-mod is a filter that monitors the process of the copy, and outputs
the info every 5 seconds with this option. The dialog then displays a running
graph of the progress along with estimated speed. In the clone case there is
no compression, so the rate is generally the physical speed of disk. With
image files that are compressed, the effective speed can vary greatly
depending on how much the data was compressed. Mostly just gives
progress versus just waiting for the dd command to finish.


Why no just dd parameter "status=progress"?

Alexander
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Patrick O'Callaghan]

On Sat, 2018-09-08 at 08:10 +1000, Cameron Simpson wrote:
> On 07Sep2018 15:56, JD  wrote:
> > On 09/07/2018 03:34 PM, Michael D. Setzer II wrote:
> > > > On Fri, 7 Sep 2018 16:37:49 -0400 Tony Nelson 
> > > >  
> > > > wrote:
> > > 
> > > The line for the cloning process in my g4l is like this.
> > > dd bs=1M if=$clonesource 2>/dev/null |jetcat-mod -f 5000 -p $clonesize
> > > 2>$progout |dd bs=1M of=$clonetarget 2>/dev/null &
> 
> [...]
> > > Uses a dialog script to set the variables, and then runs the copy command 
> > > in
> > > background. The foreground script takes the data written to the progout 
> > > file
> > > and displays a progress bar via dialog.
> 
> [...]
> > Michael D. Setzer II,
> > What is the purpose of inserting the utility
> > jetcat-mod  between the 2 dd commands?
> 
> Apologies, I think I've lost track of the attributions.
> 
> I'd expect jetcat to (a) provide an agressive buffering beyond what a pipe 
> provides (letting the first dd get ahead more) and (b) report progress.

Never heard of jetcat and it appears to be unknown to dnf.

poc
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Ranjan Maitra]

On Sat, 8 Sep 2018 07:34:12 +1000 "Michael D. Setzer II"  
wrote:

> On 7 Sep 2018 at 15:54, Ranjan Maitra wrote:
> 
> Date sent:Fri, 7 Sep 2018 15:54:59 -0500
> From: Ranjan Maitra 
> To:   users@lists.fedoraproject.org
> Subject:          Re: OT: fastest way to copy one drive to another
> Organization: Mailbox Ignored
> Send reply to:users@lists.fedoraproject.org
> 
> > On Fri, 7 Sep 2018 16:37:49 -0400 Tony Nelson 
> >  wrote:
> > 
> > > On 18-09-07 15:48:23, jdow wrote:
> > > 
> > > > dd if=/dev/ of=/dev/ bs=1073741824  
> > > > conv=sparse,noerror & pid=$!
> > > 
> > > bs=1M
> > > 
> > 
> > Btw, do the drives have to be unmounted? Just making sure.
> 
> Drives should not be mounted at time, or changes could be made during the 
> copying process.
> 
> The line for the cloning process in my g4l is like this.
> dd bs=1M if=$clonesource 2>/dev/null |jetcat-mod -f 5000 -p $clonesize 
> 2>$progout |dd bs=1M of=$clonetarget 2>/dev/null &
> 
> Uses a dialog script to set the variables, and then runs the copy command in 
> background. The foreground script takes the data written to the progout file 
> and displays a progress bar via dialog. 
> 
> Be very careful to make sure you copy the correct drive to the correct drive, 
> since if you get it wrong you loose it all.  
> 

Thanks! I just wanted to ask: 

dd bs=1M if=$clonesource 2>/dev/null |jetcat-mod -f 5000 -p $clonesize > 
2>$progout |dd bs=1M of=$clonetarget 2>/dev/null &

I guess that the $clonesource is /dev/sdb1 (in my case) and $clonetarget is 
/dev/sdc1 but what is $clonesize and $progout?

Do I need more packages installed? What is the g4l utility?

Many thanks again!
Ranjan

 
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Michael D. Setzer II]

On 7 Sep 2018 at 15:56, JD wrote:

Date sent:  Fri, 07 Sep 2018 15:56:21 -0600
From:   JD 
To: users@lists.fedoraproject.org
Subject:Re: OT: fastest way to copy one drive to another
Send reply to:  Community support for Fedora users 


> 
> 
> On 09/07/2018 03:34 PM, Michael D. Setzer II wrote:
> > On 7 Sep 2018 at 15:54, Ranjan Maitra wrote:
> >
> > Date sent:  Fri, 7 Sep 2018 15:54:59 -0500
> > From:   Ranjan Maitra 
> > To: users@lists.fedoraproject.org
> > Subject:    Re: OT: fastest way to copy one drive to another
> > Organization:   Mailbox Ignored
> > Send reply to:  users@lists.fedoraproject.org
> >
> >> On Fri, 7 Sep 2018 16:37:49 -0400 Tony Nelson 
> >>  wrote:
> >>
> >>> On 18-09-07 15:48:23, jdow wrote:
> >>>
> >>>> dd if=/dev/ of=/dev/ bs=1073741824
> >>>> conv=sparse,noerror & pid=$!
> >>> bs=1M
> >>>
> >> Btw, do the drives have to be unmounted? Just making sure.
> > Drives should not be mounted at time, or changes could be made during the
> > copying process.
> >
> > The line for the cloning process in my g4l is like this.
> > dd bs=1M if=$clonesource 2>/dev/null |jetcat-mod -f 5000 -p $clonesize
> > 2>$progout |dd bs=1M of=$clonetarget 2>/dev/null &
> >
> > Uses a dialog script to set the variables, and then runs the copy command in
> > background. The foreground script takes the data written to the progout file
> > and displays a progress bar via dialog.
> >
> > Be very careful to make sure you copy the correct drive to the correct 
> > drive,
> > since if you get it wrong you loose it all.
> >
> Michael D. Setzer II,
> What is the purpose of inserting the utility
> jetcat-mod  between the 2 dd commands?
> 

The jetcat-mod is a filter that monitors the process of the copy, and outputs 
the info every 5 seconds with this option. The dialog then displays a running 
graph of the progress along with estimated speed. In the clone case there is 
no compression, so the rate is generally the physical speed of disk. With 
image files that are compressed, the effective speed can vary greatly 
depending on how much the data was compressed. Mostly just gives 
progress versus just waiting for the dd command to finish.



> ___
> users mailing list -- users@lists.fedoraproject.org
> To unsubscribe send an email to users-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org


++
 Michael D. Setzer II - Computer Science Instructor (Retired) 
 mailto:mi...@guam.net
 mailto:msetze...@gmail.com
 Guam - Where America's Day Begins
 G4L Disk Imaging Project maintainer 
 http://sourceforge.net/projects/g4l/
++

http://setiathome.berkeley.edu (Original)
Number of Seti Units Returned:  19,471
Processing time:  32 years, 290 days, 12 hours, 58 minutes
(Total Hours: 287,489)

BOINC@HOME CREDITS

ROSETTA  65778205.643098 | ABC  16613838.513356
SETI109514958.178814 | EINSTEIN141374481.499240
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Cameron Simpson]

On 07Sep2018 15:56, JD  wrote:

On 09/07/2018 03:34 PM, Michael D. Setzer II wrote:
On Fri, 7 Sep 2018 16:37:49 -0400 Tony Nelson  
wrote:

The line for the cloning process in my g4l is like this.
dd bs=1M if=$clonesource 2>/dev/null |jetcat-mod -f 5000 -p $clonesize
2>$progout |dd bs=1M of=$clonetarget 2>/dev/null &

[...]

Uses a dialog script to set the variables, and then runs the copy command in
background. The foreground script takes the data written to the progout file
and displays a progress bar via dialog.

[...]

Michael D. Setzer II,
What is the purpose of inserting the utility
jetcat-mod  between the 2 dd commands?


Apologies, I think I've lost track of the attributions.

I'd expect jetcat to (a) provide an agressive buffering beyond what a pipe 
provides (letting the first dd get ahead more) and (b) report progress.


Cheers,
Cameron Simpson 
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[JD]

On 09/07/2018 03:34 PM, Michael D. Setzer II wrote:

On 7 Sep 2018 at 15:54, Ranjan Maitra wrote:

Date sent:  Fri, 7 Sep 2018 15:54:59 -0500
From:   Ranjan Maitra 
To: users@lists.fedoraproject.org
Subject:Re: OT: fastest way to copy one drive to another
Organization:   Mailbox Ignored
Send reply to:  users@lists.fedoraproject.org


On Fri, 7 Sep 2018 16:37:49 -0400 Tony Nelson  
wrote:


On 18-09-07 15:48:23, jdow wrote:


dd if=/dev/ of=/dev/ bs=1073741824
conv=sparse,noerror & pid=$!

bs=1M


Btw, do the drives have to be unmounted? Just making sure.

Drives should not be mounted at time, or changes could be made during the
copying process.

The line for the cloning process in my g4l is like this.
dd bs=1M if=$clonesource 2>/dev/null |jetcat-mod -f 5000 -p $clonesize
2>$progout |dd bs=1M of=$clonetarget 2>/dev/null &

Uses a dialog script to set the variables, and then runs the copy command in
background. The foreground script takes the data written to the progout file
and displays a progress bar via dialog.

Be very careful to make sure you copy the correct drive to the correct drive,
since if you get it wrong you loose it all.


Michael D. Setzer II,
What is the purpose of inserting the utility
jetcat-mod  between the 2 dd commands?

___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Michael D. Setzer II]

On 7 Sep 2018 at 15:54, Ranjan Maitra wrote:

Date sent:  Fri, 7 Sep 2018 15:54:59 -0500
From:   Ranjan Maitra 
To: users@lists.fedoraproject.org
Subject:Re: OT: fastest way to copy one drive to another
Organization:   Mailbox Ignored
Send reply to:  users@lists.fedoraproject.org

> On Fri, 7 Sep 2018 16:37:49 -0400 Tony Nelson  
> wrote:
> 
> > On 18-09-07 15:48:23, jdow wrote:
> > 
> > > dd if=/dev/ of=/dev/ bs=1073741824  
> > > conv=sparse,noerror & pid=$!
> > 
> > bs=1M
> > 
> 
> Btw, do the drives have to be unmounted? Just making sure.

Drives should not be mounted at time, or changes could be made during the 
copying process.

The line for the cloning process in my g4l is like this.
dd bs=1M if=$clonesource 2>/dev/null |jetcat-mod -f 5000 -p $clonesize 
2>$progout |dd bs=1M of=$clonetarget 2>/dev/null &

Uses a dialog script to set the variables, and then runs the copy command in 
background. The foreground script takes the data written to the progout file 
and displays a progress bar via dialog. 

Be very careful to make sure you copy the correct drive to the correct drive, 
since if you get it wrong you loose it all.  


> 
> Thanks,
> Ranjan
> ___
> users mailing list -- users@lists.fedoraproject.org
> To unsubscribe send an email to users-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org


++
 Michael D. Setzer II - Computer Science Instructor (Retired) 
 mailto:mi...@guam.net
 mailto:msetze...@gmail.com
 Guam - Where America's Day Begins
 G4L Disk Imaging Project maintainer 
 http://sourceforge.net/projects/g4l/
++

http://setiathome.berkeley.edu (Original)
Number of Seti Units Returned:  19,471
Processing time:  32 years, 290 days, 12 hours, 58 minutes
(Total Hours: 287,489)

BOINC@HOME CREDITS

ROSETTA  65778205.643098 | ABC  16613838.513356
SETI109514958.178814 | EINSTEIN141374481.499240
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Ranjan Maitra]

On Fri, 7 Sep 2018 16:37:49 -0400 Tony Nelson  
wrote:

> On 18-09-07 15:48:23, jdow wrote:
> 
> > dd if=/dev/ of=/dev/ bs=1073741824  
> > conv=sparse,noerror & pid=$!
> 
> bs=1M
> 

Btw, do the drives have to be unmounted? Just making sure.

Thanks,
Ranjan
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Ranjan Maitra]

On Fri, 7 Sep 2018 13:22:35 -0700 Rick Stevens  wrote:

> On 09/07/2018 12:48 PM, jdow wrote:
> > On 20180907 11:49, Ranjan Maitra wrote:
> >> On Fri, 7 Sep 2018 14:26:32 -0400 Tim Evans  wrote:
> >>
> >>> On 09/07/2018 02:22 PM, Ranjan Maitra wrote:
> >>>
>  I was doing this and it is definitely faster than rsync:
> 
>  cd /drive1
>  tar cf - uncopieddir1 uncopieddir2 ... | ( cd /drive2 ; tar xf - )
> 
>  But, after about 16 hours, I am only 229G in (out of 3.7T). This is
>  much slower than the other thread with USB drives which did 400GB in
>  8 hours.
> >>>
> >>> Try this:
> >>>
> >>> # cd /drive1
> >>>
> >>> # find . -print -depth | cpio -pdm /drive2
> >>
> >> Thanks!
> >>
> >> So, should I cancel the other job and do this? I am not sure what to
> >> do, sorry.
> >>
> >> Ranjan
> > 
> > Unless the drive is sparsely filled plain old "dd" may be the quickest
> > end to end operation. And you get an exact duplicate so the OS and
> > filesystem on the disks won't give you any trouble if you have a mix of
> > filesystems.
> > 
> > dd if=/dev/ of=/dev/ bs=1073741824 conv=sparse,noerror &
> > pid=$!
> > Remember that "kill -USR1 $pid" will then tell you how much you've
> > copied if you get "is done yet" anxiety.
> > 
> > You suffer no time for directory accesses, file reads and writes, and
> > all that nonsense. If you do this fairly often keep it handy as a script
> > or at least an example. The only time this won't work as nicely is if
> > the source and destination disks are not on the same machine. The
> > slowness of the tar solution won't be a problem because the network will
> > as likely as not be your limiting factor.
> 
> Good solution, but I believe he's really hitting the USB write
> speed/cache flush/bus contention issue. I doubt that filesystem overhead
> has anything to do with his problem. IIRC, he's doing USB>USB
> transfers.
> 
> This is why, when possible, I use an ESATA external drive for backups.
> I try to buy laptops that have ESATA ports and I have ESATA PCIe cards
> for my desktops. It's not ideal, but I've seen far too many cases where
> it's simply the USB ports not keeping up. It'd be interesting for him
> to install gkrellm and monitor both of the drives involved to see which
> one is stalling. I'm willing to bet it's the target (write) drive (the
> source drive will stop doing I/O while the target drive has I/O out the
> yingyang).
> 
> But I've been wrong before (as any reader of this forum can attest to).

Hi, 

I have SATA drives, both internal. No USB involved. You are mixing up with the 
other thread.

Ranjan
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Tony Nelson]

On 18-09-07 15:48:23, jdow wrote:

dd if=/dev/ of=/dev/ bs=1073741824  
conv=sparse,noerror & pid=$!


bs=1M

--

TonyN.:'   
  '  
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Rick Stevens]

On 09/07/2018 12:48 PM, jdow wrote:
> On 20180907 11:49, Ranjan Maitra wrote:
>> On Fri, 7 Sep 2018 14:26:32 -0400 Tim Evans  wrote:
>>
>>> On 09/07/2018 02:22 PM, Ranjan Maitra wrote:
>>>
 I was doing this and it is definitely faster than rsync:

 cd /drive1
 tar cf - uncopieddir1 uncopieddir2 ... | ( cd /drive2 ; tar xf - )

 But, after about 16 hours, I am only 229G in (out of 3.7T). This is
 much slower than the other thread with USB drives which did 400GB in
 8 hours.
>>>
>>> Try this:
>>>
>>> # cd /drive1
>>>
>>> # find . -print -depth | cpio -pdm /drive2
>>
>> Thanks!
>>
>> So, should I cancel the other job and do this? I am not sure what to
>> do, sorry.
>>
>> Ranjan
> 
> Unless the drive is sparsely filled plain old "dd" may be the quickest
> end to end operation. And you get an exact duplicate so the OS and
> filesystem on the disks won't give you any trouble if you have a mix of
> filesystems.
> 
> dd if=/dev/ of=/dev/ bs=1073741824 conv=sparse,noerror &
> pid=$!
> Remember that "kill -USR1 $pid" will then tell you how much you've
> copied if you get "is done yet" anxiety.
> 
> You suffer no time for directory accesses, file reads and writes, and
> all that nonsense. If you do this fairly often keep it handy as a script
> or at least an example. The only time this won't work as nicely is if
> the source and destination disks are not on the same machine. The
> slowness of the tar solution won't be a problem because the network will
> as likely as not be your limiting factor.

Good solution, but I believe he's really hitting the USB write
speed/cache flush/bus contention issue. I doubt that filesystem overhead
has anything to do with his problem. IIRC, he's doing USB>USB
transfers.

This is why, when possible, I use an ESATA external drive for backups.
I try to buy laptops that have ESATA ports and I have ESATA PCIe cards
for my desktops. It's not ideal, but I've seen far too many cases where
it's simply the USB ports not keeping up. It'd be interesting for him
to install gkrellm and monitor both of the drives involved to see which
one is stalling. I'm willing to bet it's the target (write) drive (the
source drive will stop doing I/O while the target drive has I/O out the
yingyang).

But I've been wrong before (as any reader of this forum can attest to).
--
- Rick Stevens, Systems Engineer, AllDigitalri...@alldigital.com -
- AIM/Skype: therps2ICQ: 226437340   Yahoo: origrps2 -
--
-   When in doubt, mumble.   -
--
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Ranjan Maitra]

On Fri, 7 Sep 2018 12:48:23 -0700 jdow  wrote:

> On 20180907 11:49, Ranjan Maitra wrote:
> > On Fri, 7 Sep 2018 14:26:32 -0400 Tim Evans  wrote:
> > 
> >> On 09/07/2018 02:22 PM, Ranjan Maitra wrote:
> >>
> >>> I was doing this and it is definitely faster than rsync:
> >>>
> >>> cd /drive1
> >>> tar cf - uncopieddir1 uncopieddir2 ... | ( cd /drive2 ; tar xf - )
> >>>
> >>> But, after about 16 hours, I am only 229G in (out of 3.7T). This is much 
> >>> slower than the other thread with USB drives which did 400GB in 8 hours.
> >>
> >> Try this:
> >>
> >> # cd /drive1
> >>
> >> # find . -print -depth | cpio -pdm /drive2
> > 
> > Thanks!
> > 
> > So, should I cancel the other job and do this? I am not sure what to do, 
> > sorry.
> > 
> > Ranjan
> 
> Unless the drive is sparsely filled plain old "dd" may be the quickest end to 
> end operation. And you get an exact duplicate so the OS and filesystem on the 
> disks won't give you any trouble if you have a mix of filesystems.
> 
> dd if=/dev/ of=/dev/ bs=1073741824 conv=sparse,noerror & 
> pid=$!
> Remember that "kill -USR1 $pid" will then tell you how much you've copied if 
> you 
> get "is done yet" anxiety.
> 
> You suffer no time for directory accesses, file reads and writes, and all 
> that 
> nonsense. If you do this fairly often keep it handy as a script or at least 
> an 
> example. The only time this won't work as nicely is if the source and 
> destination disks are not on the same machine. The slowness of the tar 
> solution 
> won't be a problem because the network will as likely as not be your limiting 
> factor.

Thanks very much! The first drive is 96% full and they are on the same machine. 

So, I kill the first one and get this started then? Does it matter that the 
first drive may be a failing drive?

Btw, "kill -USR1 $pid" does not actually kill the job, right?

Many thanks,
Ranjan

___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[jdow]

On 20180907 11:49, Ranjan Maitra wrote:

On Fri, 7 Sep 2018 14:26:32 -0400 Tim Evans  wrote:


On 09/07/2018 02:22 PM, Ranjan Maitra wrote:


I was doing this and it is definitely faster than rsync:

cd /drive1
tar cf - uncopieddir1 uncopieddir2 ... | ( cd /drive2 ; tar xf - )

But, after about 16 hours, I am only 229G in (out of 3.7T). This is much slower 
than the other thread with USB drives which did 400GB in 8 hours.


Try this:

# cd /drive1

# find . -print -depth | cpio -pdm /drive2


Thanks!

So, should I cancel the other job and do this? I am not sure what to do, sorry.

Ranjan


Unless the drive is sparsely filled plain old "dd" may be the quickest end to 
end operation. And you get an exact duplicate so the OS and filesystem on the 
disks won't give you any trouble if you have a mix of filesystems.


dd if=/dev/ of=/dev/ bs=1073741824 conv=sparse,noerror & pid=$!
Remember that "kill -USR1 $pid" will then tell you how much you've copied if you 
get "is done yet" anxiety.


You suffer no time for directory accesses, file reads and writes, and all that 
nonsense. If you do this fairly often keep it handy as a script or at least an 
example. The only time this won't work as nicely is if the source and 
destination disks are not on the same machine. The slowness of the tar solution 
won't be a problem because the network will as likely as not be your limiting 
factor.


{^_^}
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Ranjan Maitra]

On Fri, 7 Sep 2018 14:26:32 -0400 Tim Evans  wrote:

> On 09/07/2018 02:22 PM, Ranjan Maitra wrote:
> 
> > I was doing this and it is definitely faster than rsync:
> > 
> > cd /drive1
> > tar cf - uncopieddir1 uncopieddir2 ... | ( cd /drive2 ; tar xf - )
> > 
> > But, after about 16 hours, I am only 229G in (out of 3.7T). This is much 
> > slower than the other thread with USB drives which did 400GB in 8 hours.
> 
> Try this:
> 
> # cd /drive1
> 
> # find . -print -depth | cpio -pdm /drive2

Thanks!

So, should I cancel the other job and do this? I am not sure what to do, sorry.

Ranjan
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Tim Evans]

On 09/07/2018 02:22 PM, Ranjan Maitra wrote:


I was doing this and it is definitely faster than rsync:

cd /drive1
tar cf - uncopieddir1 uncopieddir2 ... | ( cd /drive2 ; tar xf - )

But, after about 16 hours, I am only 229G in (out of 3.7T). This is much slower 
than the other thread with USB drives which did 400GB in 8 hours.


Try this:

# cd /drive1

# find . -print -depth | cpio -pdm /drive2


--
Tim Evans   |   5 Chestnut Court
|   Owings Mills, MD 21117
|   443-394-3864
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Ranjan Maitra]

> 
> Even faster is a tar pipeline:
> 
>   cd /drive1
>   tar cf - . | ( cd /drive2; tar xf - )
> 
> because both cp and rsync do one file at a time. There will inherently be 
> small 
> pauses at each file boundary. Actually, rsync might stream a little.
> 
> Using piped tars and many files, particularly many small files, the first tar 
> can get ahead of the second tar for better throughput - the data queued in 
> the 
> pipe (which has a buffer, and a generous one on Linux) allows the first tar 
> to 
> proceed until the pipe is full if the second tar is blocked.  (The second tar 
> will of course be blocked writing to drive2, but it won't be blocked reading 
> from drive1 because the first tar can read followon files from drive1 which 
> the 
> second tar reads from the pipe).

Hi,

I was doing this and it is definitely faster than rsync:

cd /drive1
tar cf - uncopieddir1 uncopieddir2 ... | ( cd /drive2 ; tar xf - )

But, after about 16 hours, I am only 229G in (out of 3.7T). This is much slower 
than the other thread with USB drives which did 400GB in 8 hours.

Is this a function of the health of the first disk? 

Also, I wanted to ask: if this job died and I restarted it, would it be 
possible to not have to start it from all over again? (Sort of like rsync can 
do.)

Best wishes,
Ranjan

 
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[wwp]

Hello Ranjan,


On Fri, 7 Sep 2018 05:35:16 -0500 Ranjan Maitra  wrote:

> On Fri, 7 Sep 2018 07:31:05 +0200 wwp  wrote:
> 
> > Hello Ranjan,
> > 
> > 
> > On Thu, 6 Sep 2018 19:20:37 -0500 Ranjan Maitra  wrote:
> >   
> > > Hi,
> > > 
> > > I have two drives mounted on a F28 system. Both are identical 4TB drives. 
> > > The second one is empty. I am concerned about the first one failing so 
> > > would like to copy the contents (which are around 3.7 TB) to the second. 
> > > 
> > > What is the fastest way to copy the contents of the first drive to the 
> > > second? I was using rsync, but is there a better way?  
> > 
> > You didn't say the most important thing.. is this a one-shot copy only?
> > A full copy of the disk to run periodically or do you want to keep the
> > second disk in sync w/ the first one as frequently as possible?
> >   
> 
> Thanks! 
> 
> Sorry, a one-time deal for now. Essentially transferring the first's contents 
> to the second.

If it's a one-time, IMO don't bother w/ technical details and thinking
of what has been said about incremental methods, do it simple and use
the method you do know, whatever it's cp, rsync or dd. Start it and
watch, the time wasted in learning more or hesitating won't save you
any minute during this one-time copy, speed/time is not your concern
here.


Regards,

-- 
wwp


pgpSwSj0aYb1J.pgp
Description: OpenPGP digital signature
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Ranjan Maitra]

On Fri, 7 Sep 2018 07:31:05 +0200 wwp  wrote:

> Hello Ranjan,
> 
> 
> On Thu, 6 Sep 2018 19:20:37 -0500 Ranjan Maitra  wrote:
> 
> > Hi,
> > 
> > I have two drives mounted on a F28 system. Both are identical 4TB drives. 
> > The second one is empty. I am concerned about the first one failing so 
> > would like to copy the contents (which are around 3.7 TB) to the second. 
> > 
> > What is the fastest way to copy the contents of the first drive to the 
> > second? I was using rsync, but is there a better way?
> 
> You didn't say the most important thing.. is this a one-shot copy only?
> A full copy of the disk to run periodically or do you want to keep the
> second disk in sync w/ the first one as frequently as possible?
> 

Thanks! 

Sorry, a one-time deal for now. Essentially transferring the first's contents 
to the second.

Ranjan
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[wwp]

Hello Ranjan,


On Thu, 6 Sep 2018 19:20:37 -0500 Ranjan Maitra  wrote:

> Hi,
> 
> I have two drives mounted on a F28 system. Both are identical 4TB drives. The 
> second one is empty. I am concerned about the first one failing so would like 
> to copy the contents (which are around 3.7 TB) to the second. 
> 
> What is the fastest way to copy the contents of the first drive to the 
> second? I was using rsync, but is there a better way?

You didn't say the most important thing.. is this a one-shot copy only?
A full copy of the disk to run periodically or do you want to keep the
second disk in sync w/ the first one as frequently as possible?


Regards,

-- 
wwp


pgp1w5dnC0yHt.pgp
Description: OpenPGP digital signature
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Joe Zeff]

On 09/06/2018 08:28 PM, Cameron Simpson wrote:
I've used Clonezilla. It's handy for imaging new machines from a master 
machine (I'm working for someone doing that right now) but otherwise 
it's rather tedious and is rather aimed at getting data from an 
arbitrary machine to another over a network, its core strength. It'll 
image multiple machines at once, too.


I suggested Clonezilla because I've used it several times to move a 
working system over to a new, larger disk.  There are, of course, many 
different ways to do this, and the best one is the one that works for you.

___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Michael D. Setzer II]

A number of good replies already, but just to add a note:
I've been the maintainer of the g4l disk imaging project since 2004, and it 
basically uses dd for most operations.

First thing, it will take a long time to do a clone of the disks.
Don't expect to get the speed that the drive reports, since that is the 
buffered 
speed. In doing an clone image, the buffer is filled almost immediately, and 
then you will be getting the physical speed.

On this machines I get the cached speed of about 100 times faster than the 
physical speed. So, that is what you will probable end up getting for the 
process.

hdparm -Tt /dev/sda

/dev/sda:
 Timing cached reads:   6104 MB in  2.00 seconds = 3053.12 MB/sec
 Timing buffered disk reads:  106 MB in  3.07 seconds =  34.53 MB/sec

As mentioned by others, the 2nd disk will need to be the same size or larger. 
Sometimes the exact same disks can have different sizes because of bad 
sectors. Had 20 new machines in classroom once, and drives all the same 
make and model, but one system reported smaller size. Was able to image 
that system to all the others with no issues.

dd_rescue is a good program if there are any bad sectors or problem 
sectors, since it will try to work around them. It is included on the g4l disk.

I generally add g4l to the grub boot by using the 40_custom file.
Just copy the latest kernel and ramdisk.lzma file into the /boot directory
and then make a new grub.cfg file, and it is an option on boot.
Also, can be run from cd or usb.

# !/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type 
the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
menuentry G4L {
linux /bz4x18.5 root=/dev/ram0 telnetd=yes
initrd /ramdisk.lzma
}

menuentry G4L_NOSMP {
linux /bz4x18.5 root=/dev/ram0 telnetd=yes nosmp
initrd /ramdisk.lzma
}

menuentry G4L_FailSafe {
linux /bz4x18.5 root=/dev/ram0 noapic noacpi pnpbios=off acpi=off 
pci=noacpi nosmp
initrd /ramdisk.lzma
}

Another issue that causes problems. If you make a clone image of the disks, 
you should not reboot the machine with both disks, since they will report the 
same blkids for the different disks since they are now identical. 

In the past, it wasn't a problem, since one disk would be /dev/sda and the 
other /dev/sdb, but most systems now use the blkid system, so have two 
disks with the same ids causes issues.

Well, just some things to mention.
Good Luck.



On 6 Sep 2018 at 19:20, Ranjan Maitra wrote:

Date sent:  Thu, 6 Sep 2018 19:20:37 -0500
From:   Ranjan Maitra 
To: Community support for Fedora users 

Subject:            OT: fastest way to copy one drive to another
Organization:   Mailbox Ignored
Send reply to:  users@lists.fedoraproject.org

> Hi,
> 
> I have two drives mounted on a F28 system. Both are identical 4TB drives. The 
> second one is empty. I am concerned about the first one failing so would like 
> to copy the contents (which are around 3.7 TB) to the second. 
> 
> What is the fastest way to copy the contents of the first drive to the 
> second? I was using rsync, but is there a better way?
> 
> Many thanks in advance for any advice,
> Ranjan
> 
> 
> -- 
> Important Notice: This mailbox is ignored: e-mails are set to be deleted on 
> receipt. Please respond to the mailing list if appropriate. For those needing 
> to send personal or professional e-mail, please use appropriate addresses.
> ___
> users mailing list -- users@lists.fedoraproject.org
> To unsubscribe send an email to users-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org


++
 Michael D. Setzer II - Computer Science Instructor (Retired) 
 mailto:mi...@guam.net
 mailto:msetze...@gmail.com
 Guam - Where America's Day Begins
 G4L Disk Imaging Project maintainer 
 http://sourceforge.net/projects/g4l/
++

http://setiathome.berkeley.edu (Original)
Number of Seti Units Returned:  19,471
Processing time:  32 years, 290 days, 12 hours, 58 minutes
(Total Hours: 287,489)

BOINC@HOME CREDITS

ROSETTA  65776394.242111 | ABC  16613838.513356
SETI109511779.106103 | EINSTEIN141373538.499240
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code 

Re: OT: fastest way to copy one drive to another

[Cameron Simpson]

On 06Sep2018 21:00, Ranjan Maitra  wrote:

On Thu, 6 Sep 2018 19:07:11 -0600 Joe Zeff  wrote:

On 09/06/2018 06:20 PM, Ranjan Maitra wrote:
> I have two drives mounted on a F28 system. Both are identical 4TB drives. 
> The second one is empty. I am concerned about the first one failing so 
> would like to copy the contents (which are around 3.7 TB) to the second.

>
> What is the fastest way to copy the contents of the first drive to the 
second? I was using rsync, but is there a better way?

I'd go with Clonezilla if I were in your position.


Thanks very much! I have no experience with Clonezilla and the tar seems to be 
doing the work so I will stick to it for now.


I've used Clonezilla. It's handy for imaging new machines from a master machine 
(I'm working for someone doing that right now) but otherwise it's rather 
tedious and is rather aimed at getting data from an arbitrary machine to 
another over a network, its core strength. It'll image multiple machines at 
once, too.


If you've got the drives mounted locally already it is probably not worth the 
pain.


Anecdote: I once did some work for a group where they insisted on making new 
VMs by using Ghost (similar use case to Clonezilla) to a new blank VM. Instead 
of, you know, just copying a drive image file :-(


Cheers,
Cameron Simpson 
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Cameron Simpson]

On 06Sep2018 21:25, Fred Smith  wrote:

On Fri, Sep 07, 2018 at 11:10:08AM +1000, Cameron Simpson wrote:

On 06Sep2018 19:20, Ranjan Maitra  wrote:
>I have two drives mounted on a F28 system. Both are identical 4TB
>drives. The second one is empty. I am concerned about the first
>one failing so would like to copy the contents (which are around
>3.7 TB) to the second.
>
>What is the fastest way to copy the contents of the first drive to
>the second? I was using rsync, but is there a better way?


while the other suggestions might be faster, I prefer to use dd
(or ddrescue) 'cause it gives a bit-for-bit copy of everything
on the drive, partitions and all. ddrescue is handy in case it
finds spots that are hard to read on the source drive, it goes
back and retries them til it gets a good read.


If the discs aren't full (or close to full) this should be slower. It does have 
the advantage of doing sequential drive I/O, so minimal drive head stepping.


Downsides: drives must be exactly the same size. If you're doing this with a 
partition (safer) the same size constraint applies (well, the target area can 
be bigger and you can resize the filesystem afterwards, depending on the 
filesystem).


If your drives are good where there's data, dd has some (small) risk of having 
an I/O error on nondata (unallocated areas). Tedious and annoying.


The other advantage of a file based copy (cp, rsync, tar etc) is that you can 
seize the opportunity to switch filesystems, eg upgrading from ext3 or ext4 to 
xfs.


Cheers,
Cameron Simpson 
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[fred roller]

On Thu, Sep 6, 2018 at 6:26 PM Fred Smith 
wrote:

On Fri, Sep 07, 2018 at 11:10:08AM +1000, Cameron Simpson wrote:
> On 06Sep2018 19:20, Ranjan Maitra  wrote:
> >I have two drives mounted on a F28 system. Both are identical 4TB
> >drives. The second one is empty. I am concerned about the first
> >one failing so would like to copy the contents (which are around
> >3.7 TB) to the second.
> >
> >What is the fastest way to copy the contents of the first drive to
> >the second? I was using rsync, but is there a better way?

while the other suggestions might be faster, I prefer to use dd
(or ddrescue) 'cause it gives a bit-for-bit copy of everything
on the drive, partitions and all. ddrescue is handy in case it
finds spots that are hard to read on the source drive, it goes
back and retries them til it gets a good read.


>
> I like "cp -a", it is faster than rsync. Rsync's strength is
> incremental update: make a sweep afterwards with rsync to convince
> yourself it is correct.
>
> Even faster is a tar pipeline:
>
>  cd /drive1
>  tar cf - . | ( cd /drive2; tar xf - )
>
> because both cp and rsync do one file at a time. There will
> inherently be small pauses at each file boundary. Actually, rsync
> might stream a little.
>
> Using piped tars and many files, particularly many small files, the
> first tar can get ahead of the second tar for better throughput -
> the data queued in the pipe (which has a buffer, and a generous one
> on Linux) allows the first tar to proceed until the pipe is full if
> the second tar is blocked.  (The second tar will of course be
> blocked writing to drive2, but it won't be blocked reading from
> drive1 because the first tar can read followon files from drive1
> which the second tar reads from the pipe).
>
> However, if you're already a significant way through your copy you
> may as well stick with rsync unless you can easily do things in
> chunks, as changing systems means wasting time copying already
> copied data. Do a "df" and make an estimate.
>
> If there are still hours to go you could consider switching methods
> and doing the uncopied subdirectories:
>
>  cd /drive1
>  tar cf - uncopieddir1 uncopieddir2 ... | ( cd /drive2 ; tar xf - )
>
> and then come back with rsync afterwards to clean up the rest:
>
>  rsync -iaP /drive1/ /drive2/
>
> Cheers,
> Cameron Simpson 
> ___
> users mailing list -- users@lists.fedoraproject.org
> To unsubscribe send an email to users-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

-- 
 Fred Smith -- fre...@fcshome.stoneham.ma.us
-
   But God demonstrates his own love for us in this:
 While we were still sinners,
  Christ died for us.
--- Romans 5:8 (niv)
--
___

users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org



If the second drive is not going to be removed and being used to b/u the
first would a raid mirror be a better option?
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Ranjan Maitra]

On Thu, 6 Sep 2018 19:07:11 -0600 Joe Zeff  wrote:

> On 09/06/2018 06:20 PM, Ranjan Maitra wrote:
> > Hi,
> > 
> > I have two drives mounted on a F28 system. Both are identical 4TB drives. 
> > The second one is empty. I am concerned about the first one failing so 
> > would like to copy the contents (which are around 3.7 TB) to the second.
> > 
> > What is the fastest way to copy the contents of the first drive to the 
> > second? I was using rsync, but is there a better way?
> 
> I'd go with Clonezilla if I were in your position.

Thanks very much! I have no experience with Clonezilla and the tar seems to be 
doing the work so I will stick to it for now.

Best,
Ranjan
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Ranjan Maitra]

On Fri, 7 Sep 2018 11:10:08 +1000 Cameron Simpson  wrote:

> On 06Sep2018 19:20, Ranjan Maitra  wrote:
> >I have two drives mounted on a F28 system. Both are identical 4TB drives. 
> >The 
> >second one is empty. I am concerned about the first one failing so would 
> >like 
> >to copy the contents (which are around 3.7 TB) to the second.
> >
> >What is the fastest way to copy the contents of the first drive to the 
> >second? 
> >I was using rsync, but is there a better way?
> 
> I like "cp -a", it is faster than rsync. Rsync's strength is incremental 
> update: make a sweep afterwards with rsync to convince yourself it is correct.
> 
> Even faster is a tar pipeline:
> 
>   cd /drive1
>   tar cf - . | ( cd /drive2; tar xf - )
> 
> because both cp and rsync do one file at a time. There will inherently be 
> small 
> pauses at each file boundary. Actually, rsync might stream a little.
> 
> Using piped tars and many files, particularly many small files, the first tar 
> can get ahead of the second tar for better throughput - the data queued in 
> the 
> pipe (which has a buffer, and a generous one on Linux) allows the first tar 
> to 
> proceed until the pipe is full if the second tar is blocked.  (The second tar 
> will of course be blocked writing to drive2, but it won't be blocked reading 
> from drive1 because the first tar can read followon files from drive1 which 
> the 
> second tar reads from the pipe).
> 
> However, if you're already a significant way through your copy you may as 
> well 
> stick with rsync unless you can easily do things in chunks, as changing 
> systems 
> means wasting time copying already copied data. Do a "df" and make an 
> estimate.
> 
> If there are still hours to go you could consider switching methods and doing 
> the uncopied subdirectories:
> 
>   cd /drive1
>   tar cf - uncopieddir1 uncopieddir2 ... | ( cd /drive2 ; tar xf - )
> 
> and then come back with rsync afterwards to clean up the rest:
> 
>   rsync -iaP /drive1/ /drive2/

Wow! What a fabulous approach: it has already done at least 10 times more in 10 
minutes using your tar set up than using rsync in over an hour. Very cool!

Thanks again!

Best wishes,
Ranjan
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Fred Smith]

On Fri, Sep 07, 2018 at 11:10:08AM +1000, Cameron Simpson wrote:
> On 06Sep2018 19:20, Ranjan Maitra  wrote:
> >I have two drives mounted on a F28 system. Both are identical 4TB
> >drives. The second one is empty. I am concerned about the first
> >one failing so would like to copy the contents (which are around
> >3.7 TB) to the second.
> >
> >What is the fastest way to copy the contents of the first drive to
> >the second? I was using rsync, but is there a better way?

while the other suggestions might be faster, I prefer to use dd
(or ddrescue) 'cause it gives a bit-for-bit copy of everything
on the drive, partitions and all. ddrescue is handy in case it
finds spots that are hard to read on the source drive, it goes
back and retries them til it gets a good read.


> 
> I like "cp -a", it is faster than rsync. Rsync's strength is
> incremental update: make a sweep afterwards with rsync to convince
> yourself it is correct.
> 
> Even faster is a tar pipeline:
> 
>  cd /drive1
>  tar cf - . | ( cd /drive2; tar xf - )
> 
> because both cp and rsync do one file at a time. There will
> inherently be small pauses at each file boundary. Actually, rsync
> might stream a little.
> 
> Using piped tars and many files, particularly many small files, the
> first tar can get ahead of the second tar for better throughput -
> the data queued in the pipe (which has a buffer, and a generous one
> on Linux) allows the first tar to proceed until the pipe is full if
> the second tar is blocked.  (The second tar will of course be
> blocked writing to drive2, but it won't be blocked reading from
> drive1 because the first tar can read followon files from drive1
> which the second tar reads from the pipe).
> 
> However, if you're already a significant way through your copy you
> may as well stick with rsync unless you can easily do things in
> chunks, as changing systems means wasting time copying already
> copied data. Do a "df" and make an estimate.
> 
> If there are still hours to go you could consider switching methods
> and doing the uncopied subdirectories:
> 
>  cd /drive1
>  tar cf - uncopieddir1 uncopieddir2 ... | ( cd /drive2 ; tar xf - )
> 
> and then come back with rsync afterwards to clean up the rest:
> 
>  rsync -iaP /drive1/ /drive2/
> 
> Cheers,
> Cameron Simpson 
> ___
> users mailing list -- users@lists.fedoraproject.org
> To unsubscribe send an email to users-le...@lists.fedoraproject.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

-- 
 Fred Smith -- fre...@fcshome.stoneham.ma.us -
   But God demonstrates his own love for us in this: 
 While we were still sinners, 
  Christ died for us.
--- Romans 5:8 (niv) --
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Cameron Simpson]

On 06Sep2018 19:20, Ranjan Maitra  wrote:
I have two drives mounted on a F28 system. Both are identical 4TB drives. The 
second one is empty. I am concerned about the first one failing so would like 
to copy the contents (which are around 3.7 TB) to the second.


What is the fastest way to copy the contents of the first drive to the second? 
I was using rsync, but is there a better way?


I like "cp -a", it is faster than rsync. Rsync's strength is incremental 
update: make a sweep afterwards with rsync to convince yourself it is correct.


Even faster is a tar pipeline:

 cd /drive1
 tar cf - . | ( cd /drive2; tar xf - )

because both cp and rsync do one file at a time. There will inherently be small 
pauses at each file boundary. Actually, rsync might stream a little.


Using piped tars and many files, particularly many small files, the first tar 
can get ahead of the second tar for better throughput - the data queued in the 
pipe (which has a buffer, and a generous one on Linux) allows the first tar to 
proceed until the pipe is full if the second tar is blocked.  (The second tar 
will of course be blocked writing to drive2, but it won't be blocked reading 
from drive1 because the first tar can read followon files from drive1 which the 
second tar reads from the pipe).


However, if you're already a significant way through your copy you may as well 
stick with rsync unless you can easily do things in chunks, as changing systems 
means wasting time copying already copied data. Do a "df" and make an estimate.


If there are still hours to go you could consider switching methods and doing 
the uncopied subdirectories:


 cd /drive1
 tar cf - uncopieddir1 uncopieddir2 ... | ( cd /drive2 ; tar xf - )

and then come back with rsync afterwards to clean up the rest:

 rsync -iaP /drive1/ /drive2/

Cheers,
Cameron Simpson 
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

Re: OT: fastest way to copy one drive to another

[Joe Zeff]

On 09/06/2018 06:20 PM, Ranjan Maitra wrote:

Hi,

I have two drives mounted on a F28 system. Both are identical 4TB drives. The 
second one is empty. I am concerned about the first one failing so would like 
to copy the contents (which are around 3.7 TB) to the second.

What is the fastest way to copy the contents of the first drive to the second? 
I was using rsync, but is there a better way?


I'd go with Clonezilla if I were in your position.
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org

OT: fastest way to copy one drive to another

[Ranjan Maitra]

Hi,

I have two drives mounted on a F28 system. Both are identical 4TB drives. The 
second one is empty. I am concerned about the first one failing so would like 
to copy the contents (which are around 3.7 TB) to the second. 

What is the fastest way to copy the contents of the first drive to the second? 
I was using rsync, but is there a better way?

Many thanks in advance for any advice,
Ranjan


-- 
Important Notice: This mailbox is ignored: e-mails are set to be deleted on 
receipt. Please respond to the mailing list if appropriate. For those needing 
to send personal or professional e-mail, please use appropriate addresses.
___
users mailing list -- users@lists.fedoraproject.org
To unsubscribe send an email to users-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/users@lists.fedoraproject.org