Re: New python build system merged

[Ludovic Courtès]

Hi!

Leo Famulari  skribis:

> On Tue, Nov 29, 2016 at 07:09:27PM +0100, Hartmut Goebel wrote:
>> Am 29.11.2016 um 15:27 schrieb Ludovic Courtès:
>> > Good.  When you fix it (and other failures, if any), we can start a new
>> > evaluation or merge directly in master (the sooner the better!).
>> 
>> Done.
>> 
>> I'm very glad the new python build system is now in master. Thanks to
>> everybody who helped finishing this!
>
> So, in retrospect, most of the problems that we initially thought were
> related to this were actually not. There were still a few broken
> packages but the hundreds of build failures were not a consequence of
> this new build system.
>
> And I think I never said, Hartmut, thank you! Thank you for helping
> improve this part of the distribution, and for taking the time to adjust
> hundreds of packages to take advantage of this change.

I concur, thank you Hartmut!

> Already we closed one bug, "Collision between khal and vdirsyncer
> 'bin/vdirsyncer'". [0]
>
> Can we also close the long-standing serious bug, "Python .egg files must
> not be compressed"? [1]

That would be great, I love closing bugs.  :-)

Ludo’.

Re: New python build system merged

[Leo Famulari]

On Tue, Nov 29, 2016 at 07:09:27PM +0100, Hartmut Goebel wrote:
> Am 29.11.2016 um 15:27 schrieb Ludovic Courtès:
> > Good.  When you fix it (and other failures, if any), we can start a new
> > evaluation or merge directly in master (the sooner the better!).
> 
> Done.
> 
> I'm very glad the new python build system is now in master. Thanks to
> everybody who helped finishing this!

So, in retrospect, most of the problems that we initially thought were
related to this were actually not. There were still a few broken
packages but the hundreds of build failures were not a consequence of
this new build system.

And I think I never said, Hartmut, thank you! Thank you for helping
improve this part of the distribution, and for taking the time to adjust
hundreds of packages to take advantage of this change.

Already we closed one bug, "Collision between khal and vdirsyncer
'bin/vdirsyncer'". [0]

Can we also close the long-standing serious bug, "Python .egg files must
not be compressed"? [1]

[0]
http://bugs.gnu.org/22458

[1]
http://bugs.gnu.org/20765

Re: New python build system merged

[Ludovic Courtès]

Leo Famulari  skribis:

> On Wed, Nov 30, 2016 at 12:26:00PM -0500, Leo Famulari wrote:
>> On Wed, Nov 30, 2016 at 11:59:43AM -0500, Leo Famulari wrote:
>> > This is the currently running evaluation (post-merge) compared with
>> > before the merge:
>> > 
>> > https://hydra.gnu.org/eval/109381?compare=109380&full=1#tabs-now-fail
>> > 
>> > Already there are several hundred new failures... I'm not sure what the
>> > "root" failures are so far.
>> 
>> xorg-server "failed", but I think it's unrelated to the
>> python-build-system changes:
>> 
>> https://hydra.gnu.org/build/1658774
>> 
>> The message at the end of the log:
>> 
>> guix build: error: failed to create GC root 
>> `/home/hydra/offload-20121227-hydra.gnu.org-8762': File exists
>
> I forgot to mention that I've restarted the xorg-server build in case
> it's a transient error.

It’s a transient error, a bug in the “old” offload implementation.

Ludo’.

Re: New python build system merged

[Leo Famulari]

On Wed, Nov 30, 2016 at 06:30:22PM +0100, Hartmut Goebel wrote:
> Fair point, but results on hydra seem to be quite unreliable. So I
> though this is okay, since Ludo asked me to merge ASAP. maybe I
> misunderstood the exact meaning. And have been too impatient.

Yes, the Hydra interface can be confusing, and the CI server itself can
fail in suprising or spurious ways, as you saw with xorg-server...

> Some errors I checked are like this one:
> 
> @ build-succeeded 
> /gnu/store/nzjcl8aviiz4643n13nn5vxfn540x1xb-xorg-server-1.18.4.drv -
> guix build: error: failed to create GC root 
> `/home/hydra/offload-20121227-hydra.gnu.org-8762': File exists
> /gnu/store/j2pj0lhwwd430y5vfc6whrr5p4fc2b5m-xorg-server-1.18.4

I think the vast majority of the "failed" packages are a result of this
failure. But, I got a substitute for that package from a master branch
HEAD today, so I think it's not a "real" failure. I asked Hydra to redo
that build.

> Or even like this one
> , which fail
> with no visible error:
> 
> @ build-succeeded /gnu/store/i4dl6i1rz6pvcqhh5g70vqhbpa0nkswj-swig-3.0.5.drv -
> /gnu/store/89rx3gby3b8f732alz3yrm0s78s0nnb8-swig-3.0.5

I noticed this on the python-build-system evaluation, and I restarted it
since it seemed like a spurious failure. I can get a substitute for it,
so I think it's okay, now.

> So hydra seems to have some problem, which makes it really hard to track
> down issues.

Yes, it's true. And since none of us are motivated to hack on it so far,
we'll probably put up with it until the new Hydra replacement server,
bayfront, is operational. The official ETA is "Real Soon" [0] ;)

http://lists.gnu.org/archive/html/guix-devel/2016-11/msg01253.html

Re: New python build system merged

[Leo Famulari]

On Wed, Nov 30, 2016 at 12:26:00PM -0500, Leo Famulari wrote:
> On Wed, Nov 30, 2016 at 11:59:43AM -0500, Leo Famulari wrote:
> > This is the currently running evaluation (post-merge) compared with
> > before the merge:
> > 
> > https://hydra.gnu.org/eval/109381?compare=109380&full=1#tabs-now-fail
> > 
> > Already there are several hundred new failures... I'm not sure what the
> > "root" failures are so far.
> 
> xorg-server "failed", but I think it's unrelated to the
> python-build-system changes:
> 
> https://hydra.gnu.org/build/1658774
> 
> The message at the end of the log:
> 
> guix build: error: failed to create GC root 
> `/home/hydra/offload-20121227-hydra.gnu.org-8762': File exists

I forgot to mention that I've restarted the xorg-server build in case
it's a transient error.

Re: New python build system merged

[Hartmut Goebel]

Am 30.11.2016 um 17:59 schrieb Leo Famulari:
> Fair points, but the master branch is not where we put unfinished things
> to be fixed.

Fair point, but results on hydra seem to be quite unreliable. So I
though this is okay, since Ludo asked me to merge ASAP. maybe I
misunderstood the exact meaning. And have been too impatient.

> Already there are several hundred new failures... I'm not sure what the
> "root" failures are so far.

Some errors I checked are like this one:

@ build-succeeded 
/gnu/store/nzjcl8aviiz4643n13nn5vxfn540x1xb-xorg-server-1.18.4.drv -
guix build: error: failed to create GC root 
`/home/hydra/offload-20121227-hydra.gnu.org-8762': File exists
/gnu/store/j2pj0lhwwd430y5vfc6whrr5p4fc2b5m-xorg-server-1.18.4

Or even like this one
, which fail
with no visible error:

@ build-succeeded /gnu/store/i4dl6i1rz6pvcqhh5g70vqhbpa0nkswj-swig-3.0.5.drv -
/gnu/store/89rx3gby3b8f732alz3yrm0s78s0nnb8-swig-3.0.5

So hydra seems to have some problem, which makes it really hard to track
down issues.

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: New python build system merged

[Leo Famulari]

On Wed, Nov 30, 2016 at 11:59:43AM -0500, Leo Famulari wrote:
> This is the currently running evaluation (post-merge) compared with
> before the merge:
> 
> https://hydra.gnu.org/eval/109381?compare=109380&full=1#tabs-now-fail
> 
> Already there are several hundred new failures... I'm not sure what the
> "root" failures are so far.

xorg-server "failed", but I think it's unrelated to the
python-build-system changes:

https://hydra.gnu.org/build/1658774

The message at the end of the log:

guix build: error: failed to create GC root 
`/home/hydra/offload-20121227-hydra.gnu.org-8762': File exists

Re: New python build system merged

[Leo Famulari]

On Wed, Nov 30, 2016 at 04:02:52PM +0100, Hartmut Goebel wrote:
> Am 30.11.2016 um 09:20 schrieb Danny Milosavljevic:
> > I think it depends on how much work fixing them is. If it were just five 
> > minutes then I'd say leave it in master and fix the packages that failed.
> >
> > Otherwise revert.
> 
> I strongly against reverting this! We already have a backlog of several
> python packages which are using "setuptools" as input. If reverting the
> new python build system, we need to clean them all up later.
> 
> We need to go forward instead of reverting. Fixing up the build issues
> should be easy since the changes are not fundamental.

Fair points, but the master branch is not where we put unfinished things
to be fixed.

This is the currently running evaluation (post-merge) compared with
before the merge:

https://hydra.gnu.org/eval/109381?compare=109380&full=1#tabs-now-fail

Already there are several hundred new failures... I'm not sure what the
"root" failures are so far.

Re: New python build system merged

[Hartmut Goebel]

Am 30.11.2016 um 09:20 schrieb Danny Milosavljevic:
> I think it depends on how much work fixing them is. If it were just five 
> minutes then I'd say leave it in master and fix the packages that failed.
>
> Otherwise revert.

I strongly against reverting this! We already have a backlog of several
python packages which are using "setuptools" as input. If reverting the
new python build system, we need to clean them all up later.

We need to go forward instead of reverting. Fixing up the build issues
should be easy since the changes are not fundamental.

> However, Hartmut, I think that Build 1637640 is not even done building all 
> the packages yet (1853 

I'm curious (I do not understand hydra yet), but AKAIU a "build" is
about a single package. And https://hydra.gnu.org/build/1637640 builds
calibre. There is no queue not other packages.

I assume you are talking about evaluation 109379
, which the above build is part of.
This evaluation is outdated. I already fixed some of the errors you
listed, others did not occur on my system when trying to reproduce.


-- 

Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: New python build system merged

[ng0]

Ludovic Courtès  writes:

> ng0  skribis:
>
>> Hartmut Goebel  writes:
>>
>>> Am 29.11.2016 um 15:27 schrieb Ludovic Courtès:
 Good.  When you fix it (and other failures, if any), we can start a new
 evaluation or merge directly in master (the sooner the better!).
>>>
>>> Done.
>>>
>>> I'm very glad the new python build system is now in master. Thanks to
>>> everybody who helped finishing this!
>>>
>>> -- 
>>> Regards
>>> Hartmut Goebel
>>>
>>> | Hartmut Goebel  | h.goe...@crazy-compilers.com   |
>>> | www.crazy-compilers.com | compilers which you thought are impossible |
>>
>> Great job!
>>
>> So should I fix up all my ~70 or how many there are python
>> packages which I've sent in since september to get someone (or
>> hopefully multiple someones) to review them?
>
> Please do ping people.  Most likely the patch series fell into the
> cracks and only you can remind people of it.

Most likely, but there were also other issues. Like Harmut wanted
to do gunicorn while I was working on it at some point, so I
dropped that the series was forgotten. But I need to fix
conflicts anyway. I guess I can send in most of them by sunday,
there were also issues in many of the packages I need to
address. When I do this before people start to work on them it's
a bit easier to review as old issues aren't repeated.

I think most of them were in version-control, python,
networking, and it included the patches for kallithea, searx, and
some other software.

> I think Hartmut would be a good candidate to review them, Hartmut is our
> Python expert now.  :-)
>
> Ludo’.
>

-- 
♥Ⓐ  ng0  | ng0.chaosnet.org

Re: New python build system merged

[Ludovic Courtès]

ng0  skribis:

> Hartmut Goebel  writes:
>
>> Am 29.11.2016 um 15:27 schrieb Ludovic Courtès:
>>> Good.  When you fix it (and other failures, if any), we can start a new
>>> evaluation or merge directly in master (the sooner the better!).
>>
>> Done.
>>
>> I'm very glad the new python build system is now in master. Thanks to
>> everybody who helped finishing this!
>>
>> -- 
>> Regards
>> Hartmut Goebel
>>
>> | Hartmut Goebel  | h.goe...@crazy-compilers.com   |
>> | www.crazy-compilers.com | compilers which you thought are impossible |
>
> Great job!
>
> So should I fix up all my ~70 or how many there are python
> packages which I've sent in since september to get someone (or
> hopefully multiple someones) to review them?

Please do ping people.  Most likely the patch series fell into the
cracks and only you can remind people of it.

I think Hartmut would be a good candidate to review them, Hartmut is our
Python expert now.  :-)

Ludo’.

Re: New python build system merged

[Ludovic Courtès]

Hartmut Goebel  skribis:

> Am 29.11.2016 um 15:27 schrieb Ludovic Courtès:
>> Good.  When you fix it (and other failures, if any), we can start a new
>> evaluation or merge directly in master (the sooner the better!).
>
> Done.
>
> I'm very glad the new python build system is now in master. Thanks to
> everybody who helped finishing this!

Woow, that was fast.  I was expecting a signal from you before we
trigger the merge button.

> However, Hartmut, I think that Build 1637640 is not even done building all 
> the packages yet (1853 packages are pending; and some "python-" packages are 
> in the queue), so it's anyone's guess which packages are affected.
>
> As for the Python-requiring packages (which do or don't have "python" in the 
> name), the failing ones I can spot are:
> - borg (can't import borg) 
> - calibre ("list index out of range" in setup.py)
> - kicad (because of python2-wxpython failure "option 
> --single-version-externally-managed not recognized")
> - python2-wxpython ("option --single-version-externally-managed not 
> recognized")
> - python-sympy (testing fails because some tests that were supposed to fail 
> passed instead)
> - python-ipython (Can't find "docs/build/texinfo/ipython.info")
> - python2-beautifulsoup4 (Tries to use "python3" in convert-py3k - why? After 
> all it's supposed to use Python 2)
>
> There are others.

Hartmut, can you look into these now?  I see Leo already fixed Borg and
hopefully the remaining issues are relatively easy to address.

Thanks!

Ludo’.

Re: New python build system merged

[Leo Famulari]

On Tue, Nov 29, 2016 at 04:12:00PM -0500, Leo Famulari wrote:
> There is still at least one new failure, borg.

Most of the test failures can be fixed by using the
(add-installed-pythonpath) procedure to ensure that the installed borg
can be found by the test suite.

But there are still 4 failures involving FUSE. They all fail as shown
below.

Is it expected for the build environment to have FUSE? They were skipped
with the old Python build system [0], so I've disabled them in
1d60f7c2b38733b031519a48771c44d20acb785d for
the time being.

[0] https://hydra.gnu.org/build/1648289/log#line-544

=== FAILURES ===
__ ArchiverTestCase.test_fuse __

self = 

@unittest.skipUnless(has_llfuse, 'llfuse not installed')
def test_fuse(self):
def has_noatime(some_file):
atime_before = os.stat(some_file).st_atime_ns
try:
os.close(os.open(some_file, flags_noatime))
except PermissionError:
return False
else:
atime_after = os.stat(some_file).st_atime_ns
noatime_used = flags_noatime != flags_normal
return noatime_used and atime_before == atime_after

self.cmd('init', self.repository_location)
self.create_test_files()
have_noatime = has_noatime('input/file1')
self.cmd('create', self.repository_location + '::archive', 'input')
self.cmd('create', self.repository_location + '::archive2', 'input')
if has_lchflags:
# remove the file we did not backup, so input and mount become equal
os.remove(os.path.join('input', 'flagfile'))
mountpoint = os.path.join(self.tmpdir, 'mountpoint')
# mount the whole repository, archive contents shall show up in 
archivename subdirs of mountpoint:
>   with self.fuse_mount(self.repository_location, mountpoint):

borg/testsuite/archiver.py:1042: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/gnu/store/alk9r3rir93pjmv8im20f8xrvv90219z-python-3.5.2/lib/python3.5/contextlib.py:59:
 in __enter__
return next(self.gen)
borg/testsuite/__init__.py:110: in fuse_mount
self.cmd(*args, fork=True)
borg/testsuite/archiver.py:235: in cmd
self.assert_equal(ret, exit_code)
E   AssertionError: 2 != 0
- Captured stdout call -
fuse: device not found, try 'modprobe fuse' first

Re: New python build system merged

[Danny Milosavljevic]

Hi,

> There is still at least one new failure, borg. Are there more? If so, we
> should revert the changes until they are ready.

I think it depends on how much work fixing them is. If it were just five 
minutes then I'd say leave it in master and fix the packages that failed.

Otherwise revert.

However, Hartmut, I think that Build 1637640 is not even done building all the 
packages yet (1853 packages are pending; and some "python-" packages are in the 
queue), so it's anyone's guess which packages are affected.

As for the Python-requiring packages (which do or don't have "python" in the 
name), the failing ones I can spot are:
- borg (can't import borg) 
- calibre ("list index out of range" in setup.py)
- kicad (because of python2-wxpython failure "option 
--single-version-externally-managed not recognized")
- python2-wxpython ("option --single-version-externally-managed not recognized")
- python-sympy (testing fails because some tests that were supposed to fail 
passed instead)
- python-ipython (Can't find "docs/build/texinfo/ipython.info")
- python2-beautifulsoup4 (Tries to use "python3" in convert-py3k - why? After 
all it's supposed to use Python 2)

There are others.

On the other hand, many Python packages that failed building before now are 
working.
 
In any case I think the new Python build-system is an improvement and I can 
help fix some of those packages. Just don't merge when it's not even done 
building them yet :P

Re: New python build system merged

[ng0]

Hartmut Goebel  writes:

> Am 29.11.2016 um 15:27 schrieb Ludovic Courtès:
>> Good.  When you fix it (and other failures, if any), we can start a new
>> evaluation or merge directly in master (the sooner the better!).
>
> Done.
>
> I'm very glad the new python build system is now in master. Thanks to
> everybody who helped finishing this!
>
> -- 
> Regards
> Hartmut Goebel
>
> | Hartmut Goebel  | h.goe...@crazy-compilers.com   |
> | www.crazy-compilers.com | compilers which you thought are impossible |

Great job!

So should I fix up all my ~70 or how many there are python
packages which I've sent in since september to get someone (or
hopefully multiple someones) to review them? They'll most likely
no longer apply to the master in most cases and do not use the
changed build system (no propagated inputs there etc).

-- 
♥Ⓐ  ng0  | ng0.chaosnet.org

Re: New python build system merged

[Leo Famulari]

On Tue, Nov 29, 2016 at 07:09:27PM +0100, Hartmut Goebel wrote:
> Am 29.11.2016 um 15:27 schrieb Ludovic Courtès:
> > Good.  When you fix it (and other failures, if any), we can start a new
> > evaluation or merge directly in master (the sooner the better!).
> 
> Done.

There is still at least one new failure, borg. Are there more? If so, we
should revert the changes until they are ready.