Re: No os.copy()? Why not?
On Thu, Apr 5, 2012 at 1:15 AM, Steve Howell wrote: > On Apr 4, 1:37 am, Chris Angelico wrote: >> And, as a subtle point: This method can't create the file "at size". I >> don't know how it'll end up allocating space, but certainly there's no >> opportunity to announce to the OS at file open/create time "please >> allocate X bytes for this file". That may be an utterly trivial point, >> or a crucially vital one. >> >> ChrisA > > FWIW shutil.py doesn't do anything particularly fancy with respect to > creating files "at size", unless I'm missing something: So it doesn't. However, that's current implementation, and it's entirely possible that a future version might add that feature - it wouldn't break any existing code, and would improve file system performance. Doing it yourself means you miss out on any such enhancements. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: No os.copy()? Why not?
On Wed, 04 Apr 2012 08:14:18 -0400, Roy Smith wrote: >> And sparse files are really hard to reproduce, at least on Unix: on >> Linux even the system's cp doesn't guarantee sparseness of the copy (the >> manual mentions a "crude heuristic"). > > I imagine the heuristic is to look for blocks of all zeros. Yes. Although it's not really accurate to describe it as a "heuristic". With --sparse=always, it will try to make the output sparse regardless of whether the input was sparse, replacing any all-zeros block with a hole. The default of --sparse=auto will only create a sparse file if the input itself is sparse, i.e. if the length of the file rounded up to the nearest block exceeds its disk usage. Regardless of the --sparse= setting and whether the input was sparse, if it tries to create a sparse file it will create holes wherever possible rather than attempting to preserve the exact pattern of holes in a sparse input file. -- http://mail.python.org/mailman/listinfo/python-list
Re: No os.copy()? Why not?
On Apr 4, 1:37 am, Chris Angelico wrote:
> On Wed, Apr 4, 2012 at 3:53 PM, Steven D'Aprano
>
> wrote:
> > On Tue, 03 Apr 2012 15:46:31 -0400, D'Arcy Cain wrote:
>
> >> def cp(infile, outfile):
> >> open(outfile, "w").write(open(infile).read())
>
> > Because your cp doesn't copy the FILE, it copies the file's CONTENTS,
> > which are not the same thing.
>
> And, as a subtle point: This method can't create the file "at size". I
> don't know how it'll end up allocating space, but certainly there's no
> opportunity to announce to the OS at file open/create time "please
> allocate X bytes for this file". That may be an utterly trivial point,
> or a crucially vital one.
>
> ChrisA
FWIW shutil.py doesn't do anything particularly fancy with respect to
creating files "at size", unless I'm missing something:
http://hg.python.org/cpython/file/2.7/Lib/shutil.py
Only one level away from copyfile, you have copyfileobj, which is a
read/write loop:
46 def copyfileobj(fsrc, fdst, length=16*1024):
47 """copy data from file-like object fsrc to file-like object
fdst"""
48 while 1:
49 buf = fsrc.read(length)
50 if not buf:
51 break
52 fdst.write(buf)
...and that gets called by copyfile, which only does a little bit of
"os"-related stuff:
66 def copyfile(src, dst):
67 """Copy data from src to dst"""
68 if _samefile(src, dst):
69 raise Error("`%s` and `%s` are the same file" % (src,
dst))
70
71 for fn in [src, dst]:
72 try:
73 st = os.stat(fn)
74 except OSError:
75 # File most likely does not exist
76 pass
77 else:
78 # XXX What about other special files? (sockets,
devices...)
79 if stat.S_ISFIFO(st.st_mode):
80 raise SpecialFileError("`%s` is a named pipe" %
fn)
81
82 with open(src, 'rb') as fsrc:
83 with open(dst, 'wb') as fdst:
84 copyfileobj(fsrc, fdst)
The "value add" vs. a simple read/write loop depends on whether you
want OSError suppressed. The _samefile guard is nice to have, but
probably unnecessary for many apps.
I'm sure shutil.copyfile() makes perfect sense for most use cases, and
it's nice that you can see what it does under the covers pretty
easily, but it's not rocket science.
--
http://mail.python.org/mailman/listinfo/python-list
Re: No os.copy()? Why not?
Am 03.04.2012 11:34 schrieb John Ladasky: I use subprocess.call() for quite a few other things. I just figured that I should use the tidier modules whenever I can. Of course. I only wanted to point out that os.system() is an even worse approach. shutils.copy() is by far better, of course. -- http://mail.python.org/mailman/listinfo/python-list
Re: No os.copy()? Why not?
On Wed, Apr 4, 2012 at 10:08 PM, Roy Smith wrote: > Slightly off-topic, but are there file systems these days which support > off-line copying? If I have a disk at the other end of a network link, > it would be nice to tell the disk to copy a file and tell me when it's > done. Depends on your network protocol. One of the coolest and oldest tricks with FTP is initiating a file transfer from one remote host to another; I've never done it but it ought to work with localhost (ie two sessions to the same host). ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: No os.copy()? Why not?
In article <87fwcj4zru@dpt-info.u-strasbg.fr>, Alain Ketterlin wrote: > And sparse files are really hard to reproduce, at least on Unix: on > Linux even the system's cp doesn't guarantee sparseness of the copy (the > manual mentions a "crude heuristic"). I imagine the heuristic is to look for blocks of all zeros. The problem is, unless you know the block size of the file system, you can only guess as to how many zeros in a row you need to look for. In the old days, dump/restore used to know about sparse files. But things like dump/restore really get inside the file system's kimono. In today's world of SANs, WANs, and all sorts of virtual file-system-ish things, I would expect that's less common. -- http://mail.python.org/mailman/listinfo/python-list
Re: No os.copy()? Why not?
On Tue, 03 Apr 2012 15:46:31 -0400, D'Arcy Cain wrote:
> > cp is not a system command, it's a shell command. Why not just use the
> > incredibly simple and portable
> >
> >>>>open("outfile", "w").write(open("infile").read())
In article <4f7be1e8$0$2$c3e8da3$54964...@news.astraweb.com>,
Steven D'Aprano wrote:
> Because your cp doesn't copy the FILE, it copies the file's CONTENTS,
> which are not the same thing.
Not to mention that this will read the entire contents of the file into
memory at once. Probably don't want to do that with 100 GB of data.
Slightly off-topic, but are there file systems these days which support
off-line copying? If I have a disk at the other end of a network link,
it would be nice to tell the disk to copy a file and tell me when it's
done. As opposed to dragging all that data over the network just so I
can buffer it in local memory and shove it right back out the network
port to the same disk. That kind of stuff used to be standard practice
in the neanderthalic days of IBM mainframes.
--
http://mail.python.org/mailman/listinfo/python-list
Re: No os.copy()? Why not?
Steven D'Aprano writes:
> On Tue, 03 Apr 2012 15:46:31 -0400, D'Arcy Cain wrote:
>
>> On 03/28/12 16:12, John Ladasky wrote:
>>> I'm looking for a Python (2.7) equivalent to the Unix "cp" command.
>>>>>open("outfile", "w").write(open("infile").read())
> Because your cp doesn't copy the FILE, it copies the file's CONTENTS,
> which are not the same thing.
> Consider:
> * permissions
> * access times
> * file ownership
> * other metadata
> * alternate streams and/or resource fork, on platforms that support them
> * sparse files
> By the time you finish supporting the concept of copying the file itself,
> rather than merely its content, you will have something similar to the
> shutil.copy command -- only less tested.
A minor point, but shutil.copy only "copies" contents and permissions
(no access times, etc.) You probably mean shutil.copy2.
And sparse files are really hard to reproduce, at least on Unix: on
Linux even the system's cp doesn't guarantee sparseness of the copy (the
manual mentions a "crude heuristic").
But of course shutil.copy is the best solution to mimic a raw cp.
-- Alain.
--
http://mail.python.org/mailman/listinfo/python-list
Re: No os.copy()? Why not?
On Wed, Apr 4, 2012 at 3:53 PM, Steven D'Aprano wrote: > On Tue, 03 Apr 2012 15:46:31 -0400, D'Arcy Cain wrote: > >> def cp(infile, outfile): >> open(outfile, "w").write(open(infile).read()) > > Because your cp doesn't copy the FILE, it copies the file's CONTENTS, > which are not the same thing. And, as a subtle point: This method can't create the file "at size". I don't know how it'll end up allocating space, but certainly there's no opportunity to announce to the OS at file open/create time "please allocate X bytes for this file". That may be an utterly trivial point, or a crucially vital one. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
Re: No os.copy()? Why not?
On Tue, 03 Apr 2012 15:46:31 -0400, D'Arcy Cain wrote:
> On 03/28/12 16:12, John Ladasky wrote:
>> I'm looking for a Python (2.7) equivalent to the Unix "cp" command.
>> Since the equivalents of "rm" and "mkdir" are in the os module, I
>> figured I look there. I haven't found anything in the documentation. I
>> am also looking through the Python source code in os.py and its child,
>> posixfile.py.
>
> cp is not a system command, it's a shell command. Why not just use the
> incredibly simple and portable
>
>>>>open("outfile", "w").write(open("infile").read())
>
> put it into a method if you find that too much to type:
>
> def cp(infile, outfile):
>open(outfile, "w").write(open(infile).read())
Because your cp doesn't copy the FILE, it copies the file's CONTENTS,
which are not the same thing.
Consider:
* permissions
* access times
* file ownership
* other metadata
* alternate streams and/or resource fork, on platforms that support them
* sparse files
By the time you finish supporting the concept of copying the file itself,
rather than merely its content, you will have something similar to the
shutil.copy command -- only less tested.
--
Steven
--
http://mail.python.org/mailman/listinfo/python-list
Re: Re: No os.copy()? Why not?
On 01/-10/-28163 01:59 PM, Tycho Andersen wrote: Note, though, that this reads the whole file into memory. As many others have said, shutil is the most idiomatic option. * most idiomatic * clearest in terms of showing intent * potentially fastest * hardest to screw up (unlike concatenating file names into a 'system' call) * has at least a snowball's chance of persisting metadata associated with the file (even if shutil doesn't now, it could theoretically change) * portable There's basically nothing NOT to like about shutil, and at least one significant problems with all the other suggestions. Evan -- http://mail.python.org/mailman/listinfo/python-list
Re: No os.copy()? Why not?
On Tue, Apr 03, 2012 at 03:46:31PM -0400, D'Arcy Cain wrote:
> On 03/28/12 16:12, John Ladasky wrote:
> >I'm looking for a Python (2.7) equivalent to the Unix "cp" command.
> >Since the equivalents of "rm" and "mkdir" are in the os module, I
> >figured I look there. I haven't found anything in the documentation.
> >I am also looking through the Python source code in os.py and its
> >child, posixfile.py.
>
> cp is not a system command, it's a shell command. Why not just use the
> incredibly simple and portable
>
> >>>open("outfile", "w").write(open("infile").read())
Note, though, that this reads the whole file into memory. As many
others have said, shutil is the most idiomatic option.
\t
--
http://mail.python.org/mailman/listinfo/python-list
Re: No os.copy()? Why not?
On 03/28/12 16:12, John Ladasky wrote:
I'm looking for a Python (2.7) equivalent to the Unix "cp" command.
Since the equivalents of "rm" and "mkdir" are in the os module, I
figured I look there. I haven't found anything in the documentation.
I am also looking through the Python source code in os.py and its
child, posixfile.py.
cp is not a system command, it's a shell command. Why not just use the
incredibly simple and portable
>>>open("outfile", "w").write(open("infile").read())
put it into a method if you find that too much to type:
def cp(infile, outfile):
open(outfile, "w").write(open(infile).read())
--
D'Arcy J.M. Cain | Democracy is three wolves
http://www.druid.net/darcy/| and a sheep voting on
+1 416 425 1212 (DoD#0082)(eNTP) | what's for dinner.
IM: da...@vex.net
--
http://mail.python.org/mailman/listinfo/python-list
Re: No os.copy()? Why not?
On Tue, Apr 3, 2012 at 12:24 AM, Thomas Rachel
wrote:
> Am 02.04.2012 23:11 schrieb HoneyMonster:
>
>
>> One way:
>> import os
>>
>> os.system ("cp src sink")
>
>
> Yes. The worst way you could imagine.
>
> Why not the much much better
>
> from subprocess
> subprocess.call(['cp', 'src', 'sink'])
In any case, either of these approaches will only work in UNIX,
whereas shutil is cross-platform.
--
http://mail.python.org/mailman/listinfo/python-list
Re: No os.copy()? Why not?
I use subprocess.call() for quite a few other things. I just figured that I should use the tidier modules whenever I can. -- http://mail.python.org/mailman/listinfo/python-list
Re: No os.copy()? Why not?
Am 02.04.2012 23:11 schrieb HoneyMonster:
One way:
import os
os.system ("cp src sink")
Yes. The worst way you could imagine.
Why not the much much better
from subprocess
subprocess.call(['cp', 'src', 'sink'])
?
Then you can call it with (really) arbitrary file names:
def call_cp(from, to):
from subprocess
subprocess.call(['cp', '--', from, to])
Try that with os.system() and from="That's my file"...
Thomas
--
http://mail.python.org/mailman/listinfo/python-list
Re: No os.copy()? Why not?
On Wed, 28 Mar 2012 13:12:30 -0700, John Ladasky wrote:
> I'm looking for a Python (2.7) equivalent to the Unix "cp" command.
> Since the equivalents of "rm" and "mkdir" are in the os module, I
> figured I look there. I haven't found anything in the documentation.
> I am also looking through the Python source code in os.py and its child,
> posixfile.py.
>
> Any help? Thanks.
One way:
import os
os.system ("cp src sink")
--
http://mail.python.org/mailman/listinfo/python-list
Re: No os.copy()? Why not?
On Wed, Mar 28, 2012 at 2:12 PM, John Ladasky wrote: > I'm looking for a Python (2.7) equivalent to the Unix "cp" command. > Since the equivalents of "rm" and "mkdir" are in the os module, I > figured I look there. I haven't found anything in the documentation. > I am also looking through the Python source code in os.py and its > child, posixfile.py. > > Any help? Thanks. The os module wraps system calls, not shell commands. You want the shutil module, not the os module. -- http://mail.python.org/mailman/listinfo/python-list
Re: No os.copy()? Why not?
On Mar 28, 9:50 pm, alex23 wrote: > On Mar 29, 6:12 am, John Ladasky wrote: > > > I'm looking for a Python (2.7) equivalent to the Unix "cp" command. > > Any help? Thanks. > > Try the shutil module:http://docs.python.org/library/shutil.html Many thanks! That's what I was looking for. -- http://mail.python.org/mailman/listinfo/python-list
