Re: Problem selecting options for cp command

[Lars Noodén]

On 09/05/2016 05:21 PM, Richard Owlett wrote:
> "archive" had brought up mental image of dusty back rooms where things
> were being preserved for posterity. Not making a copy to actually be used.
> 
> Someone had suggested rsync but that has too many options for me to
> bungle. "cp" may be slower but appears more comfortable.
> 
> Thanks to all.

For rsync, the only options that you need to worry about are -a and -v.
Even then the -v is optional.

rsync -av /media/richard/myrepo/ /media/richard/test/

Just remember the trailing slash on the directories and it will go fine.

Regards,
Lars

Re: Problem selecting options for cp command

[Richard Owlett]

On 9/5/2016 7:46 AM, Mark Fletcher wrote:



On Mon, Sep 5, 2016 at 9:30 PM Richard Owlett
> wrote:

I attempted to copy contents of one partition to another using
 cp -R /media/richard/myrepo /media/richard/test

/media/richard/myrepo is a hard disk partition
/media/richard/test   is a USB flash drive

I received "illegal operation" error messages as symbolic links
were encountered.
"man cp" was too terse to be illuminating.


I think cp -a is your best bet. It will copy everything,
shouldn't choke on links, and will also set permissions etc
correctly on the target device, assuming you have permissions to
do so (you should be doing the copy as root, always assuming you
want to keep permissions and access modes intact).

I recently used it to copy hard disk partitions to a replacement
SSD (some discussions about that are in the archives of this
list) and it worked just dandy.

Mark


"archive" had brought up mental image of dusty back rooms where 
things were being preserved for posterity. Not making a copy to 
actually be used.


Someone had suggested rsync but that has too many options for me 
to bungle. "cp" may be slower but appears more comfortable.


Thanks to all.

Re: Problem selecting options for cp command

[Jonathan Dowland]

On Mon, Sep 05, 2016 at 08:25:37AM -0500, Richard Owlett wrote:
> That may have been the problem. It's possible the flash drive HAD BEEN
> formatted FAT-32. No way to tell now as I brute forced wiped drive by having
> gparted format it as ext2.

If you are going to put an extX family filesystem on it, I'd recommend at
least ext3, and ideally ext4, which have journals and therefore are more
robust in the face of things like power cuts.


-- 
Jonathan Dowland
Please do not CC me, I am subscribed to the list.


signature.asc
Description: Digital signature

Re: Problem selecting options for cp command

[Richard Owlett]

On 9/5/2016 7:55 AM, Henrique de Moraes Holschuh wrote:

On Mon, 05 Sep 2016, Richard Owlett wrote:

I received "illegal operation" error messages as symbolic links were
encountered.


That means the target filesystem does not support symlinks.



That may have been the problem. It's possible the flash drive HAD 
BEEN formatted FAT-32. No way to tell now as I brute forced wiped 
drive by having gparted format it as ext2.

Re: Problem selecting options for cp command

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Mon, Sep 05, 2016 at 09:55:46AM -0300, Henrique de Moraes Holschuh wrote:
> On Mon, 05 Sep 2016, Richard Owlett wrote:
> > I received "illegal operation" error messages as symbolic links were
> > encountered.
> 
> That means the target filesystem does not support symlinks.

Oh, right. Good point -- forgot the most important part.

Luckily someone is reading carefully :-)

regards
- -- t
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlfNbNUACgkQBcgs9XrR2kZigACdHrkKUeZ2v0BEadC/GlOnOWys
lNgAnjil3HPOzEeuoqfG3n8JeG2UNNEe
=DUDz
-END PGP SIGNATURE-

Re: Problem selecting options for cp command

[Thomas Schmitt]

Hi,

Richard Owlett wrote:
> /media/richard/test   is a USB flash drive
> I received "illegal operation" error messages as symbolic links were
> encountered.

Maybe the filesystem is of a type which does not support symbolic links ?

  $ cp -R ./test/softlinks /mnt/fat
  cp: cannot create symbolic link ‘/mnt/fat/softlinks/irrelang’: Operation not 
permitted


Have a nice day :)

Thomas

Re: Problem selecting options for cp command

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Mon, Sep 05, 2016 at 08:43:44AM -0400, Patrick Wiseman wrote:
> On Mon, Sep 5, 2016 at 8:29 AM, Richard Owlett  wrote:
> 
> > I attempted to copy contents of one partition to another using
> >cp -R /media/richard/myrepo /media/richard/test
> >
> > /media/richard/myrepo is a hard disk partition
> > /media/richard/test   is a USB flash drive
> >
> 
> I think what you want is the magic tar command:
> 
> $tar cf - [files] | (cd /[dir]; tar xf - )
> 
> It's wise, just to make sure it's going to do what you expect, to do
> 
> $tar cf - [files] | (cd /[dir]; tar tvf - )
> 
> first!

While basically correct, this is antiquated.

It stems from the good ol'days "cp" had no -a option. With the
ubiquity of the Gnu tools (thanks, Gnu!), you have cp -a, which
is what you really want. Or rsync (especially if you expect having
to stop the copy operation mid-way and restart it, or if you have
an old version of the copy to start of, as in backups).

When copying across machine boundaries, use rsync. Much nicer.
As in "by orders of magnitude much nicer".

Regards
- -- t
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlfNa8QACgkQBcgs9XrR2kYI+QCfVEyF9/35j6+WLYEjDXqfUiWZ
AdsAn1w35Wxia/I1B6yaos+JF+zl+kaf
=dHpj
-END PGP SIGNATURE-

Re: Problem selecting options for cp command

[Henrique de Moraes Holschuh]

On Mon, 05 Sep 2016, Richard Owlett wrote:
> I received "illegal operation" error messages as symbolic links were
> encountered.

That means the target filesystem does not support symlinks.

-- 
  Henrique Holschuh

Re: Problem selecting options for cp command

[tomas]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On Mon, Sep 05, 2016 at 07:29:52AM -0500, Richard Owlett wrote:
> I attempted to copy contents of one partition to another using
>cp -R /media/richard/myrepo /media/richard/test
> 
> /media/richard/myrepo is a hard disk partition
> /media/richard/test   is a USB flash drive
> 
> I received "illegal operation" error messages as symbolic links were
> encountered.
> "man cp" was too terse to be illuminating.

First: what do you *want*?
  (a) if cp finds a link it should copy it as a (now possibly
dangling) link)
  (b) it should copy (recursively) the contents of the dir
the link points to.

Once you are clear, it's easier to decide

Hint: when doing backup, you almost always want cp -a (*not* -R).
It will copy the links as links. If they point to some place
outside the tree under copy, they are dangling links in the copy.
On backups you almost always want that.

Regards
- -- tomás
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.12 (GNU/Linux)

iEYEARECAAYFAlfNamQACgkQBcgs9XrR2kbzmACffqDzwcIiLQ0mUhNFgfHNj3pF
jQEAniafWEURYJx7Q1kDt8E+h+U0U2yJ
=O7Ig
-END PGP SIGNATURE-

Re: Problem selecting options for cp command

[Dan Purgert]

Richard Owlett wrote:
> I attempted to copy contents of one partition to another using
> cp -R /media/richard/myrepo /media/richard/test
>
> /media/richard/myrepo is a hard disk partition
> /media/richard/test   is a USB flash drive
>
> I received "illegal operation" error messages as symbolic links 
> were encountered.
> "man cp" was too terse to be illuminating.
> [snip]
>
> What option(s) should I be using?
> Should I be looking at "dd" rather than "cp"?
> [snip]

Using 'dd' may work, though it seems you need to simply add the -l
option to the 'cp' command you were using.

-- 
|_|O|_| Registered Linux user #585947
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| 

Re: Problem selecting options for cp command

[Mark Fletcher]

On Mon, Sep 5, 2016 at 9:30 PM Richard Owlett  wrote:

> I attempted to copy contents of one partition to another using
> cp -R /media/richard/myrepo /media/richard/test
>
> /media/richard/myrepo is a hard disk partition
> /media/richard/test   is a USB flash drive
>
> I received "illegal operation" error messages as symbolic links
> were encountered.
> "man cp" was too terse to be illuminating.
>
>
I think cp -a is your best bet. It will copy everything, shouldn't choke on
links, and will also set permissions etc correctly on the target device,
assuming you have permissions to do so (you should be doing the copy as
root, always assuming you want to keep permissions and access modes intact).

I recently used it to copy hard disk partitions to a replacement SSD (some
discussions about that are in the archives of this list) and it worked just
dandy.

Mark

Re: Problem selecting options for cp command

[Patrick Wiseman]

On Mon, Sep 5, 2016 at 8:29 AM, Richard Owlett  wrote:

> I attempted to copy contents of one partition to another using
>cp -R /media/richard/myrepo /media/richard/test
>
> /media/richard/myrepo is a hard disk partition
> /media/richard/test   is a USB flash drive
>

I think what you want is the magic tar command:

$tar cf - [files] | (cd /[dir]; tar xf - )

It's wise, just to make sure it's going to do what you expect, to do

$tar cf - [files] | (cd /[dir]; tar tvf - )

first!

Substitute 'u' for 'x' to update files; otherwise, old files will
overwrite newer files with the same name.  (I know; I did it!)

To copy a whole directory; create the parent dir if it doesn't already
exist, then:

$tar cf - [dir] | (cd /[parent_dir]; tar xf - )

I haven't had occasion to use it in a while, but I'm pretty sure it makes a
"perfect" copy.

HTH

Patrick

Problem selecting options for cp command

[Richard Owlett]

I attempted to copy contents of one partition to another using
   cp -R /media/richard/myrepo /media/richard/test

/media/richard/myrepo is a hard disk partition
/media/richard/test   is a USB flash drive

I received "illegal operation" error messages as symbolic links 
were encountered.

"man cp" was too terse to be illuminating.

https://www.gnu.org/software/coreutils/manual/html_node/cp-invocation.html#cp-invocation
[apparently relevant excerpts below] was not enlightening.

"...
By default, cp does not copy directories. However, the -R, -a, 
and -r options cause cp to copy recursively by descending into 
source directories and copying files to corresponding destination 
directories.


When copying from a symbolic link, cp normally follows the link 
only when not copying recursively or when --link (-l) is used. 
This default can be overridden with the --archive (-a), -d, 
--dereference (-L), --no-dereference (-P), and -H options. If 
more than one of these options is specified, the last one 
silently overrides the others.


When copying to a symbolic link, cp follows the link only when it 
refers to an existing regular file. However, when copying to a 
dangling symbolic link, cp refuses by default, and fails with a 
diagnostic, since the operation is inherently dangerous. This 
behavior is contrary to historical practice and to POSIX. Set 
POSIXLY_CORRECT to make cp attempt to create the target of a 
dangling destination symlink, in spite of the possible risk. 
Also, when an option like --backup or --link acts to rename or 
remove the destination before copying, cp renames or removes the 
symbolic link rather than the file it points to.

..."

"...
‘-d’Copy symbolic links as symbolic links rather than copying 
the files that they
point to, and preserve hard links between source files 
in the copies. Equivalent

to --no-dereference --preserve=links.
..."

"...
‘-H’If a command line argument specifies a symbolic link, 
then copy the file it
points to rather than the symbolic link itself. However, 
copy (preserving its
nature) any symbolic link that is encountered via 
recursive traversal.

..."

"...
‘-R’
‘-r’
‘--recursive’
Copy directories recursively. By default, do not follow 
symbolic links in the source unless used together with the --link 
(-l) option; see the --archive (-a), -d, --dereference (-L), 
--no-dereference (-P), and -H options. Special files are copied 
by creating a destination file of the same type as the source; 
see the --copy-contents option. It is not portable to use -r to 
copy symbolic links or special files. On some non-GNU systems, -r 
implies the equivalent of -L and --copy-contents for historical 
reasons. Also, it is not portable to use -R to copy symbolic 
links unless you also specify -P, as POSIX allows implementations 
that dereference symbolic links by default.

..."

What option(s) should I be using?
Should I be looking at "dd" rather than "cp"?
"man dd" and 
https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html#dd-invocation 
not illuminating.


TIA

Re: Speedup cp command?

[Thomas Schmitt]

Hi,

i wrote:
> >xorriso -osirrox on:sort_lba_on:auto_chmod_on:o_excl_off \

Richard Owlett wrote:
> That works nicely.

xorriso packs them up, xorriso packs them out.


> Advantage over cp is progress messages.

It's more entertaining than watching cp -r. :))

In case one prefers silent operation, it is possible to restrict verbosity
to events of severity "WARNING" or worse:

  xorriso -report_about warning \
  -osirrox on:sort_lba_on:auto_chmod_on:o_excl_off \
  ...


> 3 of the 13 DVD's had problems with reading files and gave a fatal error
> message [thought I had saved it but I didn't] all successfully copied on
> retry.

In case you need to really fight for a damaged ISO, there would be ways
to combine the successfully read blocks from multiple runs of

  xorriso ...  -check_media  ...

to a single ISO image on disk and to inquire the paths of data files
which stay damaged.


Have a nice day :)

Thomas

Re: Speedup cp command?

[Richard Owlett]

On 8/24/2016 6:09 AM, Thomas Schmitt wrote:

Hi,

the proposed xorriso -extract run fails on mounted media unless
the -osirrox parameter string contains

   ":o_excl_off"

I.e.:

   xorriso -osirrox on:sort_lba_on:auto_chmod_on:o_excl_off \
   -indev /dev/cdrom \
   -extract / /media/richard/myrepository/dvd_1
[snip]


That works nicely. Advantage over cp is progress messages.
3 of the 13 DVD's had problems with reading files and gave a 
fatal error message [thought I had saved it but I didn't] all 
successfully copied on retry.

Re: Speedup cp command?

[Thomas Schmitt]

Hi,

> for a DVD (but not a CD), I think just cat without isosize
> would work as well.

Only for DVD-R[W] written with write type DAO and even that depends
on the burn program used. DVD have a natural chunk size of 32 KiB.
Only DAO respects your wish if you send smaller chunks.

Overwritable media (DVD-RAM, DVD+RW, BD-RE, formatted DVD-RW) in
most cases yield their full capacity when being read.


> files in
> ISO 9660 filesystems were written in sequence and should not cause much
> seeking.

One should think so. At least if data of tree neighbors have similar
block addresses. (Which is not the case in current Debian ISOs but seems
not to be the reason for the miserable performance of cp -r.)


> Any idea how they did manage that?

Well, in part "they" is me as software heir of Vreixo Formoso Lopes
who implented a mapping from directory tree to data content which is
deterministic but quite chaotic. A red-black tree is involved which
tries to find identical input files.

In GNU xorriso-1.3.8 as used by debian-cd, this red-black tree is
traversed for data content mapping. Since 1.4.2, the ISO 9660 name tree,
which is alphabetically sorted by specification, gives the sequence for
this mapping.
Only pitfall is that ISO 9660 names are uppercase and possibly mangled
versions of the Rock Ridge names, which mount(8) uses by default.
So some hopping might still be needed when walking the mounted file
tree alphabetically.

But as demonstrated, the more reasonable mapping does not significantly
improve the performance of cp -r.


> rsync would speed things up since it establishes a list of all the
> files to copy before starting

Let's try with the non-chaoticly repacked ISO.
Being first time user of rsync i have to guess and RTFM.

  # mount /dev/sr1 /mnt/iso

  $ time rsync --no-p --no-g --no-o --chmod=ugo=rwX -r \
   /mnt/iso /dvdbuffer/debian_dvd_1_unpacked
  skipping non-regular file "iso/debian"
  ...

Noise is very promising. Few clonking.
  
  real6m43.584s

That's 404 seconds. Not too bad, compared to 333 seconds for sequentially
copying the ISO to hard disk.

My rsync options still hate softlinks. But i guess this is not significant
for performance measurement.

rsync on the original DVD 1 needs 730 seconds. With much clonking.
cp -r needed 1048 seconds, osirrox without sort_lba needed 661 seconds.

So:
cp -r must be doing something that does not go well with the ISO 9660
driver of Linux.


Have a nice day :)

Thomas

Re: Speedup cp command?

[Nicolas George]

L'octidi 8 fructidor, an CCXXIV, Thomas Schmitt a écrit :
>   dd if=/dev/cdrom bs=1M count=$blocks of=/media/richard/myisos/dvd_1.iso

Useless use of dd. head -c will perform as well, without the need for
arithmetic. And, for a DVD (but not a CD), I think just cat without isosize
would work as well.

> A discussion on reproducible-builds a year ago yielded that the file
> content sorting order by libisofs did not match the sorting order of
> directory records in the tree. This was fixed by release 1.4.2.

While reading the beginning of your mail, I was about to point that files in
ISO 9660 filesystems were written in sequence and should not cause much
seeking. Obviously, you already knew it and thought of it.

Any idea how they did manage that? Naively, I imagine that creating the
directory index and then creating the file entries is done from the same
in-memory data structure after a single sort.

> Nevertheless it turns out that the layers of Debian GNU/Linux 8 still
> do a poor job. I repacked the ISO by xorriso-1.4.5 and verified that
> the data extents are sorted according to the sorting of the ECMA-119
> and Rock Ridge tree. Simple tree traversal or alphabetically sorted
> tree traversal would yield smooth reading, but cp -r has different ideas
> about sequence.

You can use -v to easily know the order cp choses. AFAIK, cp -r does no
sorting and uses the order from the kernel, and the kernel, for ISO 9660
uses the order in the directory data, so that should work ok.

Actually, since rsync does its own sorting, so it could lead to worse
results in this case.

Still, we are speaking of a Debian install CD: the bulk of the data should
be made of a pool directory with only subdirectories on one level containing
plain files. All with file names from the almost-portable character set (+
and ~ are used), in lowercase. There is not much room for sorting
discrepancies.

But IIRC, ISO 9660 stores all the directory structure first and only then
the files' payload. That could be an explanation, since cp -r reads the
directories as they come (and even, apparently, subdirectories after it has
copied the plain files). That could explain seeking:

readdir pool
readdir pool/a [no seeking]
copy pool/a/a-1.deb [seeking over the rest of the directory structure]
copy pool/a/a.orig.tar.gz [no seeking]
readdir pool/b [seeking back]
...

In that case, running "find /media/cdrom > /dev/null" repeatedly to keep the
whole structure of the hierarchy in the inodes and dentries cache could
speed things up.

And also, rsync would speed things up since it establishes a list of all the
files to copy before starting, and its sort order should yield the same
result with these particular file names. I do not have an optical drive at
hand to check.


signature.asc
Description: Digital signature

Re: Speedup cp command?

[Thomas Schmitt]

Hi,

the proposed xorriso -extract run fails on mounted media unless
the -osirrox parameter string contains

  ":o_excl_off"

I.e.:

  xorriso -osirrox on:sort_lba_on:auto_chmod_on:o_excl_off \
  -indev /dev/cdrom \
  -extract / /media/richard/myrepository/dvd_1

Else you will see an error message

  libburn : SORRY : Cannot open busy device '/dev/sr1' : Device or resource busy


I do not get this message because i circumvent the /dev/sr global
ioctl mutex by using /dev/sg instead. So i can operate more than one
drive at a time by ioctl(SG_IO).
(The mutex seems to be totally unnecessary. It was installed 6 years
 ago by a drive-by programmer who removed the Big Kernel Lock.
 Some of my users hacked their kernels to get rid of it.)


Have a nice day :)

Thomas

Re: Speedup cp command?

[Thomas Schmitt]

Hi,

Richard Owlett wrote:
> I used
>   cp -R /media/cdrom0 /media/richard/myrepository/dvd_1
> It gave me what I wanted [*N.B.* I did not want dvd_1.iso]
> It was SLOW.

An average DVD+RW can be read at about 10 MB/s average speed.
That would be about 7 minutes.
Reading usually is slower in the inner area and faster outwards.

A major slowdown is caused scattered random access. The optical head
moves to a new position quite slowly and often loudly.
Copying the plain ISO image does not involve random access.
Random access on hard disk or in RAM is much faster.


So an intermediate .iso might be the fastest vanilla way to get the
data from medium to disk.
Depending on your disk speed and RAM luxury, the additional cp -r for
unpacking might still end before a plain cp -r would have ended.

  blocks=$(expr $(/sbin/isosize /dev/cdrom) / 1024 / 1024 + 1)
  mkdir /media/richard/myisos
  dd if=/dev/cdrom bs=1M count=$blocks of=/media/richard/myisos/dvd_1.iso
  mkdir /mnt/iso
  mount -o loop /media/richard/myisos/dvd_1.iso /mnt/iso
  cp -r /mnt/iso /media/richard/myrepository/dvd_1
  umount /mnt/iso
  rm /media/richard/myisos/dvd_1.iso


If you do not have buffer space for the ISO or want to avoid the
intermediate steps, try this:

  xorriso -osirrox on:sort_lba_on:auto_chmod_on \
  -indev /dev/cdrom \
  -extract / /media/richard/myrepository/dvd_1

My measurements with DVD+RW on drive LG GH24NSC0:

  dd to .iso:   333 s

  cp -r :  1084 s despite lots of RAM ! Miserable noises from drive.

  osirrox   :   342 s with "sort_lba_on" which lets it read with
  monotonically ascending block addresses.

661 s without "sort_lba_on". Clonks less than cp -r.


debian-cd could get a file arrangement which is more friendly to copiers
if it would use a newer version of xorriso.
A discussion on reproducible-builds a year ago yielded that the file
content sorting order by libisofs did not match the sorting order of
directory records in the tree. This was fixed by release 1.4.2.

Nevertheless it turns out that the layers of Debian GNU/Linux 8 still
do a poor job. I repacked the ISO by xorriso-1.4.5 and verified that
the data extents are sorted according to the sorting of the ECMA-119
and Rock Ridge tree. Simple tree traversal or alphabetically sorted
tree traversal would yield smooth reading, but cp -r has different ideas
about sequence.

  cp -r   :  998 s  still clonking terribly.

  osirrox without sort_lba:  356 s  working smoothly.

(xorriso-1.4.6 is planned to come soon with more changes proposed by
 reproducible-builds. So i do not prod debian-cd now.)


Have a nice day :)

Thomas

Re: Speedup cp command?

[David Wright]

On Tue 23 Aug 2016 at 17:23:31 (-0400), Greg Wooledge wrote:
> On Tue, Aug 23, 2016 at 04:16:42PM -0500, Richard Owlett wrote:
> > Thanks. I'll try it as soon as copy of DVD#2 ends.
> > What's special about a loop mount in this circumstance? As I read 
> > the rsync man page it was pretty similar to cp and it had 
> > accepted a plain automount [I'm on Jessie with Mate DE]]
> 
> rsync vs. cp won't make any difference if the destination directory
> is empty.  In either case, you have to read every byte of input and
> write every byte of output.

Does it make any difference if the DVD drive and the hard drive
are master and slave on the same IDE controller?

Cheers,
David.

Re: Speedup cp command?

[Nicolas George]

> On Tue, Aug 23, 2016 at 03:18:30PM -0500, Richard Owlett wrote:
> > I'm copying Debian distribution DVDs.
> > I used
> >   cp -R /media/cdrom0 /media/richard/myrepository/dvd_1
> > 
> > It gave me what I wanted [*N.B.* I did not want dvd_1.iso]
> > It was SLOW.

Optical media are « SLOW ». You can check for yourself: measure the actual
speed (using for example « iostat 1 », or maybe « df
/media/richard/myrepository/dvd_1; sleep 100; df
/media/richard/myrepository/dvd_1 » and a calculator), and compare to the
official speed of your drive.

> > The man page for rsync suggested that it could do it faster.
> > Can it?

No.

> > If so, what is correct syntax to get the same result as the command above?

« rsync -r » instead of « cp -r » (or -R, they are synonyms for cp but not
for rsync), nothing more. But in this particular case, I would advise -a
instead of -r for both.

Andrew M.A. Cater a écrit :
> Loop mount the DVD
> 
> mount -t iso /media/cdrom0 -o loop /mnt

That does not make any sense. In this situation, /media/cdrom0 is a
directory. You can not do loopback on a directory.

If the data is on an optical medium, it must be read from the optical
medium, at the speed of the optical medium, there is no way around. Even if
you manage to convince the kernel to do loopback on something that is
already a block device, it will not help.


signature.asc
Description: Digital signature

Re: Speedup cp command?

[Andrew M.A. Cater]

On Tue, Aug 23, 2016 at 04:16:42PM -0500, Richard Owlett wrote:
> On 8/23/2016 4:05 PM, Andrew M.A. Cater wrote:
> >On Tue, Aug 23, 2016 at 03:18:30PM -0500, Richard Owlett wrote:
> >>I'm copying Debian distribution DVDs.
> >>I used
> >>   cp -R /media/cdrom0 /media/richard/myrepository/dvd_1
> >>
> >>It gave me what I wanted [*N.B.* I did not want dvd_1.iso]
> >>It was SLOW.
> >>The man page for rsync suggested that it could do it faster.
> >>Can it?
> >>If so, what is correct syntax to get the same result as the command above?
> >>
> >>TIA
> >
> >Loop mount the DVD
> >
> >mount -t iso /media/cdrom0 -o loop /mnt
> >
> >cd /mnt
> >
> >and you should see all the files within the DVD.
> >
> >cd /media/richard/repository/dvd_1/
> >
> >rsync -pavz /mnt/ .
> >
> >Should do it. If you stop and restart rsync, it should start from the place 
> >it left off,more or less.
> >
> >Hope this helps,
> >
> >AndyC
> 
> Thanks. I'll try it as soon as copy of DVD#2 ends.
> What's special about a loop mount in this circumstance? As I read the rsync
> man page it was pretty similar to cp and it had accepted a plain automount
> [I'm on Jessie with Mate DE]]
> 

Loop mount allows you to "see inside" the mounted media, effectively.

I've just noticed a typo - if you want to copy and paste the instructions, then 
for you, at least,
that will need to be

cd /media/richard/myrepository/dvd_1/ ; rsync -pavz /mnt/ .

[I missed out the "my" in myrepository :) ]

As someone else pointed out: you still need to copy everything so it can be 
slow especially on 4G - essentially you're probably copying from one 
portion of the disk to another so lots of reads and writes on the same disk. 
Rsync just allows you to start/stop much more readily.

All the best,

AndyC

Re: Speedup cp command?

[Richard Owlett]

On 8/23/2016 4:23 PM, Greg Wooledge wrote:

On Tue, Aug 23, 2016 at 04:16:42PM -0500, Richard Owlett wrote:

Thanks. I'll try it as soon as copy of DVD#2 ends.
What's special about a loop mount in this circumstance? As I read
the rsync man page it was pretty similar to cp and it had
accepted a plain automount [I'm on Jessie with Mate DE]]


rsync vs. cp won't make any difference if the destination directory
is empty.  In either case, you have to read every byte of input and
write every byte of output.


I can see that. I was wondering if rsync was using a better 
buffering scheme. That might make a difference when copying 
between rotating media.




rsync is tremendously useful when you've already got a partial copy
of the input.  It uses heuristics to figure out what it actually needs
to copy, and skips the parts you already have.

Re: Speedup cp command?

[Greg Wooledge]

On Tue, Aug 23, 2016 at 04:16:42PM -0500, Richard Owlett wrote:
> Thanks. I'll try it as soon as copy of DVD#2 ends.
> What's special about a loop mount in this circumstance? As I read 
> the rsync man page it was pretty similar to cp and it had 
> accepted a plain automount [I'm on Jessie with Mate DE]]

rsync vs. cp won't make any difference if the destination directory
is empty.  In either case, you have to read every byte of input and
write every byte of output.

rsync is tremendously useful when you've already got a partial copy
of the input.  It uses heuristics to figure out what it actually needs
to copy, and skips the parts you already have.

Re: Speedup cp command?

[Richard Owlett]

On 8/23/2016 4:05 PM, Andrew M.A. Cater wrote:

On Tue, Aug 23, 2016 at 03:18:30PM -0500, Richard Owlett wrote:

I'm copying Debian distribution DVDs.
I used
   cp -R /media/cdrom0 /media/richard/myrepository/dvd_1

It gave me what I wanted [*N.B.* I did not want dvd_1.iso]
It was SLOW.
The man page for rsync suggested that it could do it faster.
Can it?
If so, what is correct syntax to get the same result as the command above?

TIA


Loop mount the DVD

mount -t iso /media/cdrom0 -o loop /mnt

cd /mnt

and you should see all the files within the DVD.

cd /media/richard/repository/dvd_1/

rsync -pavz /mnt/ .

Should do it. If you stop and restart rsync, it should start from the place it 
left off,more or less.

Hope this helps,

AndyC


Thanks. I'll try it as soon as copy of DVD#2 ends.
What's special about a loop mount in this circumstance? As I read 
the rsync man page it was pretty similar to cp and it had 
accepted a plain automount [I'm on Jessie with Mate DE]]

Re: Speedup cp command?

[Andrew M.A. Cater]

On Tue, Aug 23, 2016 at 03:18:30PM -0500, Richard Owlett wrote:
> I'm copying Debian distribution DVDs.
> I used
>   cp -R /media/cdrom0 /media/richard/myrepository/dvd_1
> 
> It gave me what I wanted [*N.B.* I did not want dvd_1.iso]
> It was SLOW.
> The man page for rsync suggested that it could do it faster.
> Can it?
> If so, what is correct syntax to get the same result as the command above?
> 
> TIA

Loop mount the DVD

mount -t iso /media/cdrom0 -o loop /mnt

cd /mnt

and you should see all the files within the DVD.

cd /media/richard/repository/dvd_1/

rsync -pavz /mnt/ .

Should do it. If you stop and restart rsync, it should start from the place it 
left off,more or less.

Hope this helps,

AndyC

Speedup cp command?

[Richard Owlett]

I'm copying Debian distribution DVDs.
I used
  cp -R /media/cdrom0 /media/richard/myrepository/dvd_1

It gave me what I wanted [*N.B.* I did not want dvd_1.iso]
It was SLOW.
The man page for rsync suggested that it could do it faster.
Can it?
If so, what is correct syntax to get the same result as the 
command above?


TIA

About the `-u' option of `cp' command

[Rodolfo Medina]

Hallo, Debian users.

In `cp' man it is said:

-u, --update
  copy only when the SOURCE file is  newer  than  the  destination
  file or when the destination file is missing


Now, it happens sometimes to me that, even with `-u', `cp' will copy the file
also when it isn't newer at all than the destination file, as here:

$ ls -lh ing.tex /mnt/pendrive2/ing.tex 
-rw-r--r-- 1 rodolfo rodolfo 163K 2011-08-31 18:44 ing.tex
-rwxr-xr-x 1 rodolfo rodolfo 163K 2011-08-31 18:44 /mnt/pendrive2/ing.tex
$ cp -viup ing.tex /mnt/pendrive2
cp: overwrite `/mnt/pendrive2/ing.tex'?

Why this, and how to avoid it?

Thanks for any help
Rodolfo


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87ehzyukew@gmail.com

Re: About the `-u' option of `cp' command

[Sven Joachim]

On 2011-09-03 09:58 +0200, Rodolfo Medina wrote:

 In `cp' man it is said:

 -u, --update
   copy only when the SOURCE file is  newer  than  the  destination
   file or when the destination file is missing


 Now, it happens sometimes to me that, even with `-u', `cp' will copy the file
 also when it isn't newer at all than the destination file, as here:

 $ ls -lh ing.tex /mnt/pendrive2/ing.tex 
 -rw-r--r-- 1 rodolfo rodolfo 163K 2011-08-31 18:44 ing.tex
 -rwxr-xr-x 1 rodolfo rodolfo 163K 2011-08-31 18:44 /mnt/pendrive2/ing.tex
 $ cp -viup ing.tex /mnt/pendrive2
 cp: overwrite `/mnt/pendrive2/ing.tex'?

 Why this, and how to avoid it?

Might be due to high resolution timestamps on the source filesystem, but
not on the target (the pendrive seems to have an FAT filesystem).  Use
the --full-time ls option to find out.

Sven


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87obz258x7@turtle.gmx.de

Re: About the `-u' option of `cp' command

[Ivan Shmakov]

 Sven Joachim svenj...@gmx.de writes:
 On 2011-09-03 09:58 +0200, Rodolfo Medina wrote:

[…]

  Now, it happens sometimes to me that, even with `-u', `cp' will copy
  the file also when it isn't newer at all than the destination file,
  as here:

  $ ls -lh ing.tex /mnt/pendrive2/ing.tex 
  -rw-r--r-- 1 rodolfo rodolfo 163K 2011-08-31 18:44 ing.tex
  -rwxr-xr-x 1 rodolfo rodolfo 163K 2011-08-31 18:44 /mnt/pendrive2/ing.tex
  $ cp -viup ing.tex /mnt/pendrive2
  cp: overwrite `/mnt/pendrive2/ing.tex'?

  Why this, and how to avoid it?

  Might be due to high resolution timestamps on the source filesystem,
  but not on the target (the pendrive seems to have an FAT filesystem).
  Use the --full-time ls option to find out.

Please note that the filesystems of the FAT family have 2 second
time resolution, while the usual Unix filesystems have 1 second
resolution at worst.  Therefore, the original file may be
18:44:55, and the destination is 18:44:54 (i. e., time gets
truncated.)

Using rsync(1) and --modify-window= will probably solve the
problem.  Consider, e. g.:

$ rsync -v -urt -O --modify-window=1 -- \
  ing.tex /mnt/pendrive2/ 

(The -v -urt Rsync options roughly correspond to the -v -urp
cp(1) ones.)

-- 
FSF associate member #7257  Coming soon: Software Freedom Day
http://mail.sf-day.org/lists/listinfo/ planning-ru (ru), sfd-discuss (en)


-- 
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/8639ge9f8m@gray.siamics.net

Re: About the `-u' option of `cp' command

[Andrew McGlashan]

Hello,

Rodolfo Medina wrote:

Hallo, Debian users.

In `cp' man it is said:

-u, --update
  copy only when the SOURCE file is  newer  than  the  destination
  file or when the destination file is missing


Why not use rsync for this?

Cheers

--
Kind Regards
AndrewM

Andrew McGlashan
Broadband Solutions now including VoIP


--
To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org 
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Archive: http://lists.debian.org/4e61f09b.2040...@affinityvision.com.au

Re: Using the cp command.

[Bob Proulx]

alex wrote:
 I'm trying to copy the contents of one partition to another with:
 cp -afv  (partition a)/*  (partition b)/
 [...]
 Is there a way to copy the entire contents of one partition  to another  
 partition that contains duplicates  without having to bang on y for each 
 duplicatedifferent options, for example?
 I checked out ' man cp' but don't see anything that might help.  Perhaps 
 there's a different way of doing the copy.

I really like using rsync for this.

  rsync -a (partition a)/*  (partition b)/

Or, since I have usually mounted the new disk under the old one and
need to prevent the recursion, the following:

  mkdir /new
  mount /dev/whatever /new
  rsync -a --exclude /new / /new/

Bob


pgpWWSw4Ul63n.pgp
Description: PGP signature

Re: Using the cp command.

[alex]

CW Harris wrote:
On Thu, Jun 10, 2004 at 10:21:18AM -0400, alex wrote:
 

I'm trying to copy the contents of one partition to another with:
   cp -afv  (partition a)/*  (partition b)/
Normally, the command works fine except when (partition b) already 
contains a large number of directories and files that are duplicates of 
those in (partition a) . 

The command works but each duplicate requires permission  to be 
overwritten so it's a tedious process, clicking on 'y' for each 
duplicate being copied.
   

What version of cp are you using?  I cannot duplicate this behavior
here.  Are you sure you don't have an -i set?  Are you using /bin/cp
or an alias?
cp (coreutils) 5.0.91  [here]
you might check out --reply=yes option (man cp)
 

That was it, Chris. I just updated cp and it works fine.  
Thanks-alex


--
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: Using the cp command.

[alex]

CW Harris wrote:
On Thu, Jun 10, 2004 at 10:21:18AM -0400, alex wrote:
 

I'm trying to copy the contents of one partition to another with:
   cp -afv  (partition a)/*  (partition b)/
Normally, the command works fine except when (partition b) already 
contains a large number of directories and files that are duplicates of 
those in (partition a) . 

The command works but each duplicate requires permission  to be 
overwritten so it's a tedious process, clicking on 'y' for each 
duplicate being copied.
   

What version of cp are you using?  I cannot duplicate this behavior
here.  Are you sure you don't have an -i set?  Are you using /bin/cp
or an alias?
cp (coreutils) 5.0.91  [here]
you might check out --reply=yes option (man cp)
 

After updating  my cp, I found that info cp provides a bit more 
information
about --reply=___  than is found in man cp.   I  then ran :

#   cp -afv --reply=yes  /mnt/hda1 /*  /mnt/hda2/
It worked perfectly.hands off, all the way
I used it on a friend's computer to make an updated backup
of WindowsXP on hda2 for him with a KNOPPIX CD.
hda1 and hda2 were mounted before doing the cp.

Thanks Chris---alex
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Using the cp command.

[alex]

I'm trying to copy the contents of one partition to another with:
cp -afv  (partition a)/*  (partition b)/
Normally, the command works fine except when (partition b) already 
contains a large number of directories and files that are duplicates of 
those in (partition a) . 

The command works but each duplicate requires permission  to be 
overwritten so it's a tedious process, clicking on 'y' for each 
duplicate being copied.

Is there a way to copy the entire contents of one partition  to another  
partition that contains duplicates  without having to bang on y for each 
duplicatedifferent options, for example?
I checked out ' man cp' but don't see anything that might help.  Perhaps 
there's a different way of doing the copy.

The command works fine if there aren't any duplicates in the partitions.
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: Using the cp command.

[Thomas Adam]

 --- alex [EMAIL PROTECTED] wrote: 

 I'm trying to copy the contents of one partition to another with:
  cp -afv  (partition a)/*  (partition b)/

Please see this:

http://www.hantslug.org.uk/cgi-bin/wiki.pl?LinuxHints/OneDiskToAnother

-- Thomas Adam

=
The Linux Weekend Mechanic -- http://linuxgazette.net
TAG Editor -- http://linuxgazette.net

shrug We'll just save up your sins, Thomas, and punish 
you for all of them at once when you get better. The 
experience will probably kill you. :)

 -- Benjamin A. Okopnik (Linux Gazette Technical Editor)





___ALL-NEW Yahoo! Messenger - 
so many all-new ways to express yourself http://uk.messenger.yahoo.com


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: Using the cp command.

[Daved Daly]

use tar or cpio?

(cd /partitiona; tar cvf - ./) | (cd /partitionb; tar xf -)

or something like that...

On Thu, 10 Jun 2004 10:21:18 -0400, alex [EMAIL PROTECTED] wrote:
 
 I'm trying to copy the contents of one partition to another with:
 cp -afv  (partition a)/*  (partition b)/
 
 Normally, the command works fine except when (partition b) already
 contains a large number of directories and files that are duplicates of
 those in (partition a) .
 
 The command works but each duplicate requires permission  to be
 overwritten so it's a tedious process, clicking on 'y' for each
 duplicate being copied.
 
 Is there a way to copy the entire contents of one partition  to another
 partition that contains duplicates  without having to bang on y for each
 duplicatedifferent options, for example?
 I checked out ' man cp' but don't see anything that might help.  Perhaps
 there's a different way of doing the copy.
 
 The command works fine if there aren't any duplicates in the partitions.
 
 
 --
 To UNSUBSCRIBE, email to [EMAIL PROTECTED]
 with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]
 



-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: Using the cp command.

[CW Harris]

On Thu, Jun 10, 2004 at 10:21:18AM -0400, alex wrote:
 I'm trying to copy the contents of one partition to another with:
 cp -afv  (partition a)/*  (partition b)/
 
 Normally, the command works fine except when (partition b) already 
 contains a large number of directories and files that are duplicates of 
 those in (partition a) . 
 
 The command works but each duplicate requires permission  to be 
 overwritten so it's a tedious process, clicking on 'y' for each 
 duplicate being copied.

What version of cp are you using?  I cannot duplicate this behavior
here.  Are you sure you don't have an -i set?  Are you using /bin/cp
or an alias?

cp (coreutils) 5.0.91  [here]

you might check out --reply=yes option (man cp)


-- 
Chris Harris [EMAIL PROTECTED]
---
GNU/Linux --- The best things in life are free.


-- 
To UNSUBSCRIBE, email to [EMAIL PROTECTED] 
with a subject of unsubscribe. Trouble? Contact [EMAIL PROTECTED]

Re: Using the cp command.

[Brad Sims]

On Thursday 10 June 2004 10:49 am, CW Harris wrote:
 What version of cp are you using?  I cannot duplicate this behavior
 here.  Are you sure you don't have an -i set?  Are you using /bin/cp
 or an alias?
 
 cp (coreutils) 5.0.91  [here]
 
 you might check out --reply=yes option (man cp)

heh or for geek value and an award for gratuitous use of pipe;
'/bin/yes | cp -$foo $options' might do the same thing.

IIRC /bin/yes dates back to when all cp and rm commands were
-i by default (back in the 70's as I understand it)

-- 
It could be, if the Microsoft way of handling the unzipped file was
safe and reasonable, but they've written their software to be too
smart (that is to say, brain-dead-eager-to-do-wrong-things), and the
bells and whistles are the problem. --  Werehatrack on opening zip files

Re: cp command

[Nathan E Norman]

On Tue, 15 Jul 1997, Bob Clark wrote:

:Syd Alsobrook wrote:
:
: Really easy
:
: The best way to copy a directory tree from one drive to another is
:
: cp -pr /usr /hd
:
: Thanks
:
: Syd
: 
:
:Hmm... The _best_ way?  I not so sure.  Like most things in
:*nix where there are lots of ways to get the job done there
:is usually not a _best_ way.  A major problem with cp -pr is
:that linked-files, both hard and soft, are broken and all
:links are copied individually as regular files.  This can
:cause MAJOR problems as well as consuming more disk space.

Also, the cp command doesn't always do so well with device files, fifos,
etc.  The tar command has some of these problems as well.

:A couple of ways to avoid this problem are:
:
:mkdir newDir
:cd oldDir
:find . -print | cpio -pdm newDir

This is my preferred method, except I usually do a cpio -padm ... makes
it a little more transparent that things have been moved.  Aethetics,
mostly. 

:
:OR
:
:mkdir newDir
:cd newDir
:(cd oldDir;tar cf - .) | tar xpf -

This won't work on the /dev directory, I don't think.

:I'm sure these are not the _best_ way, but as far as I know
:there are no hidden side-effects with links, permissions,
:etc.
:
:My $0.02
:--Bob


--
TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word unsubscribe to
[EMAIL PROTECTED] . 
Trouble?  e-mail to [EMAIL PROTECTED] .

Re: cp command

[Bob Clark]

Syd Alsobrook wrote:

 Really easy

 The best way to copy a directory tree from one drive to another is

 cp -pr /usr /hd

 Thanks

 Syd
 

Hmm... The _best_ way?  I not so sure.  Like most things in
*nix where there are lots of ways to get the job done there
is usually not a _best_ way.  A major problem with cp -pr is
that linked-files, both hard and soft, are broken and all
links are copied individually as regular files.  This can
cause MAJOR problems as well as consuming more disk space.

A couple of ways to avoid this problem are:

mkdir newDir
cd oldDir
find . -print | cpio -pdm newDir

OR

mkdir newDir
cd newDir
(cd oldDir;tar cf - .) | tar xpf -

I'm sure these are not the _best_ way, but as far as I know
there are no hidden side-effects with links, permissions,
etc.

My $0.02
--Bob---BeginMessage---
Syd Alsobrook wrote:
 
 Really easy
 
 The best way to copy a directory tree from one drive to another is
 
 cp -pr /usr /hd
 
 Thanks
 
 Syd
 

Hmm... The _best_ way?  I not so sure.  Like most things in
*nix where there are lots of ways to get the job done there
is usually not a _best_ way.  A major problem with cp -pr is
that linked-files, both hard and soft, are broken and all
links are copied individually as regular files.  This can
cause MAJOR problems as well as consuming more disk space.

A couple of ways to avoid this problem are:

mkdir newDir
cd oldDir
find . -print | cpio -pdm newDir

OR

mkdir newDir
cd newDir
(cd oldDir;tar cf - .) | tar xpf -

I'm sure these are not the _best_ way, but as far as I know
there are no hidden side-effects with links, permissions,
etc.

My $0.02
--Bob
---End Message---

Re: cp command

[Anthony Fok]

On Tue, 15 Jul 1997, Bob Clark wrote:

 Syd Alsobrook wrote:
  Really easy
  The best way to copy a directory tree from one drive to another is
  cp -pr /usr /hd
 Hmm... The _best_ way?  I not so sure.  Like most things in
 *nix where there are lots of ways to get the job done there
 is usually not a _best_ way.  A major problem with cp -pr is
 that linked-files, both hard and soft, are broken and all
 links are copied individually as regular files.  This can
 cause MAJOR problems as well as consuming more disk space.

Try:cp -a /usr /hd

where -a stands for --archive.  It *does* preserve links!  ^_^
(According to cp --help, it says -a is equivalent to -dpR)

Anthony

-- 
Anthony Fok Tung-Ling[EMAIL PROTECTED]
Civil Engineeringhttp://www.ualberta.ca/~foka/
University of Alberta, CanadaKeep smiling!  *^_^*


--
TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word unsubscribe to
[EMAIL PROTECTED] . 
Trouble?  e-mail to [EMAIL PROTECTED] .

cp command

[Syd Alsobrook]

Really easy

The best way to copy a directory tree from one drive to another is 

cp -pr /usr /hd

Thanks

Syd

http://www.uc.edu/~alsobrsp

How do you know you're having fun   
 if there's no one watching you have it.
Douglas Adams
-BEGIN PGP PUBLIC KEY BLOCK-
Version: 2.6.2

mQCNAzN3eq0AAAEEAMGqDqrPkUM0PEPT4UBXCVDylEkUabrkqq/yBk1koSqHWlxs
02wVIVl/2B33MuS1aF3XfjvQLH2J6VJTUtyOwH2yglfpyJ68/TaE7od2gT3V99ax
a/bENj3x5xnCxQFuKJCBUi4l5CoHEEn8Og6I0IS3vz+nJFkKF0AiZe6TgKrhAAUR
tCVTeWQgQWxzb2Jyb29rIDxhbHNvYnJzcEBlbWFpbC51Yy5lZHU+iQCVAwUQM3eI
00AiZe6TgKrhAQHMWAP/W/ZacyMwNrHF0LU3eMyzTbjeFotEws1FmmE2ALjTgVI2
emtsN7vynlZ17p4qPDPlO18G/PTQ2r61kIywmn3bApaTsij5wSXMYv+4VRUvs1wX
Ie8gHs6jGZBa3wnutCzaDDW/VPy/atpvLSWTc7Mku1BJzee5whRxxw+QUK2KjD0=
=EjnG
-END PGP PUBLIC KEY BLOCK-


--
TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word unsubscribe to
[EMAIL PROTECTED] . 
Trouble?  e-mail to [EMAIL PROTECTED] .