[issue8876] distutils should not assume that hardlinks will work

[Éric Araujo]

Éric Araujo added the comment:

Thanks for the patches folks!

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Antoine Pitrou]

Antoine Pitrou added the comment:

This should now be fixed. Thanks for your patience :)

--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Roundup Robot]

Roundup Robot added the comment:

New changeset 263395345aa7 by Antoine Pitrou in branch '2.7':
Issue #8876: distutils now falls back to copying files when hard linking 
doesn't work.
https://hg.python.org/cpython/rev/263395345aa7

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Roundup Robot]

Roundup Robot added the comment:

New changeset d94d8789e924 by Antoine Pitrou in branch '3.4':
Issue #8876: distutils now falls back to copying files when hard linking 
doesn't work.
https://hg.python.org/cpython/rev/d94d8789e924

New changeset ce484e0840e3 by Antoine Pitrou in branch 'default':
Issue #8876: distutils now falls back to copying files when hard linking 
doesn't work.
https://hg.python.org/cpython/rev/ce484e0840e3

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Antoine Pitrou]

Antoine Pitrou added the comment:

For the record, the only place where distutils seems to use hard-linking is 
distutils.command.sdist. The fallback on copying is safe there, since it's done 
manually if os.link doesn't exist.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Antoine Pitrou]

Antoine Pitrou added the comment:

Updated patch with tests.

--
stage: test needed -> patch review
Added file: http://bugs.python.org/file37069/copy_file_link_fail.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
assignee: eric.araujo -> 
stage: patch review -> test needed
versions: +Python 3.5 -Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Éric Araujo]

Éric Araujo added the comment:

I can’t make any commitment to core Python at this time.  I still read all 
email and can be available for guidance or questions about obscure parts of 
distutils.

(Ideally I would find someone willing to be mentored to take over maintenance.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Antoine Pitrou]

Antoine Pitrou added the comment:

I agree this deserves fixing, and the patch looks basically ok. It would be 
nice to add a test, though (by mocking os.link()).

Éric, are you still here, or should someone else take over?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Daniel Farrell]

Daniel Farrell added the comment:

> An emerging workflow is using Docker for a Python environment.

This applies to Vagrant environments as well (VirtualBox shared folders, tests 
fail).

--
nosy: +dfarrell07

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Matt Wright]

Matt Wright added the comment:

Here's another example of where this is a pain. An emerging workflow is using 
Docker for a Python environment. However, on OS X, its common to use 
boot2docker (a lightweight VM). With VirtualBox on OS X, its common to setup a 
shared folder between the host and boot2docker so one can mount volumes to 
Docker containers from the host to the container(s) on the VM. So when running 
something like `tox` or `python setup.py sdist` things fail because VirtualBox 
does not allow hard links on the shared folder filesystem. Changing this would 
be grand. Otherwise I have the, arguably, ugly `del os.link` in my setup.py.

--
nosy: +Matt.Wright

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Éric Araujo]

Éric Araujo added the comment:

Sorry for the lax definition of “soon”; I’m back from a busy January and will 
try to make some time for CPython contributions.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Fabian Kochem]

Fabian Kochem added the comment:

A dirty hack is to include this line at the top of your setup.py:

del os.link

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Fabian Kochem]

Fabian Kochem added the comment:

Éric said that he should be able to get to it soon.:
https://twitter.com/merwok_/status/425596183176704002

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Larry Hastings]

Larry Hastings added the comment:

Has this been fixed?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Fabian Kochem]

Changes by Fabian Kochem :


--
nosy: +Fabian.Kochem

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Benjamin Peterson]

Benjamin Peterson added the comment:

Not blocking on this old bug.

--
priority: release blocker -> normal

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Éric Araujo]

Éric Araujo added the comment:

I’ll get this in the next bugfix releases.

--
keywords:  -needs review
nosy: +benjamin.peterson, georg.brandl, larry
priority: normal -> release blocker
versions: +Python 3.4 -Python 3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[sorin]

sorin added the comment:

Can we have this merged, it prevents us form using distutil, especially in a 
continuous integration environment where you do not have control over the build 
server.

See: https://drone.io/github.com/pycontribs/tendo/1

--
nosy: +sorin

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Antoine Pitrou]

Antoine Pitrou  added the comment:

> AFAICT using hard links only serves to save up a little time and disk
> space; it seems to me that always copying would solve one or two bugs
> at a small cost

I agree with this.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Éric Araujo]

Éric Araujo  added the comment:

samtygier’s patch to implement the fallback-to-copy idea looks good.  Not sure 
if an automated test would be written, but samtygier says it was manually 
tested.

--
components:  -Distutils2
keywords: +needs review
stage: needs patch -> patch review
versions:  -3rd party

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Éric Araujo]

Éric Araujo  added the comment:

Of course the conservative fix would be to try linking as of now and do a copy 
when an exception is caught, as was proposed.

FTR python.org link to the thread mentioned: 
http://mail.python.org/pipermail/distutils-sig/2005-August/004954.html

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Éric Araujo]

Éric Araujo  added the comment:

AFAICT using hard links only serves to save up a little time and disk space; it 
seems to me that always copying would solve one or two bugs at a small cost 
(not so small for large projects, I don’t know).  Could this impact setup 
scripts?  Maybe if os.link were monkey-patched and expected to be called.  I 
could ask on distutils-sig.

In #15205 Ollie noted this: “distutils2 appears to always copy instead of 
hardlinking resolving all of these issues”.  That would be because d2 uses 
shutil.copytree and distutils.util.copy_file (yes, a function from distutils1 
because of some Windows issue :().

--
assignee: tarek -> eric.araujo
nosy: +lemburg, olliewalsh, pitrou
stage:  -> needs patch
versions: +Python 3.3 -Python 3.1

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Éric Araujo]

Changes by Éric Araujo :


--
type:  -> behavior
versions: +3rd party, Python 2.7, Python 3.1, Python 3.2 -Python 2.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Éric Araujo]

Éric Araujo  added the comment:

Distutils is a special case: Many third-party code relies on its undocumented 
quirks and bugs, so it’s basically frozen. Some non-disruptive bugfixes are 
accepted, in which case the normal Python rules apply (e.g., no new features in 
2.7 which is in beta). Work has been moved to Distutils2, which breaks 
compatibility in order to fix things and add needed features.

If Tarek (maintainer of both packages) agrees this can go into Distutils, 
Benjamin Peterson (release manager) will say whether he agrees or not. Patient 
a while for Tarek to see this bug report, and thanks for your work :)

--
nosy: +merwok

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[samtygier]

samtygier  added the comment:

here is a patch against http://svn.python.org/projects/python/trunk

it moves the return statements into the individual file copying sections, and 
takes the final call to _copy_file_contents() out of the else. this allows an 
error in hardlinking to fall through by passing the exception.

i have only tested on Linux.

when running it still prints out
"hard linking CHANGES -> pyzgoubi-0.4dev"
because of how the message is generated in line 130. do you think this needs to 
be changed. do we need a
"hardlinking failed, falling back to copy"
message? it might cause unnecessary worry (distutils already suppresses 
tracebacks to avoid scaring non-python-programmers).

i have a patch that applies to 2.6 as well, is there much chance of that being 
accepted?

--
keywords: +patch
Added file: http://bugs.python.org/file17522/distutil-hardlink-trunk.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[Éric Araujo]

Changes by Éric Araujo :


--
components: +Distutils2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue8876] distutils should not assume that hardlinks will work

[samtygier]

New submission from samtygier :

distutils will currently try to use hardlinks if os has a 'link' attribute, 
however sometimes os.link() will fail, for example the filesystem may not 
support it (see attached traceback).

in commands/sdist.py in make_release_tree() there is the test:
if hasattr(os, 'link'):# can make hard links on this system
link = 'hard'
msg = "making hard links in %s..." % base_dir

'link' is then passed to copy_file() in file_util.py, which trusts that if link 
== 'hard', then hardlinking will work.

there has been discussion in the past, but i dont think it has been fixed
http://thread.gmane.org/gmane.comp.python.distutils.devel/2076

--
assignee: tarek
components: Distutils
files: hardlink-traceback.txt
messages: 106881
nosy: samtygier, tarek
priority: normal
severity: normal
status: open
title: distutils should not assume that hardlinks will work
versions: Python 2.6
Added file: http://bugs.python.org/file17521/hardlink-traceback.txt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com