Re: [PATCH 0/3] Add a generalized `wrap-language-programs`

[Ludovic Courtès]

Alex Sassmannshausen  skribis:

> I have tested Perl’s wrap phase, and that seems to work fine.  I was
> unfortunately not able to test Python’s wrap phase, as I did not know of an
> appropriate test candidate (I tried to build offlineimap, but this failed
> because Guile@2.013 failed at least one of it’s tests).

Oops.  I’ve fixed the Guile issue in commit
8134dc1f74c2df2d42e1510fd3ac35234a4d89df.

Thanks,
Ludo’.

Re: [PATCH 0/3] Add a generalized `wrap-language-programs`

[Alex Sassmannshausen]

Hi Ludo,

Ludovic Courtès writes:

> Hi Alex,
>
> Alex Sassmannshausen  skribis:
>
>>> You may try e.g. scons, pip, sphinx, or youtube-dl.
>>
>> Hartmut was kind enough to suggest the above python applications to test
>> this `wrap-language-programs` patch set against, however I have been
>> stuck in "Rebuild the world"-hell for the last day and a half setting up
>> testcases.
>>
>> I end up feeling a bit desesperated and incompetent, as I'm sure it's
>> not supposed to be this hard to create an appropriate test scenario here
>> — so I was wondering whether anyone has any pointers in avoiding
>> constantly rebuilding the world?
>
> The patch set you posted modifies (guix build utils), which is a module
> every single derivation uses.  That’s why modifying it leads to a full
> rebuild.

That too, makes a whole load of sense.  Thank you for explaining this to
me!

> To test ‘wrap-language-programs’ while avoiding a full rebuild, you
> could for instance add that procedure to (guix build
> python-build-system), in which case you’d have to rebuild “only” Python
> packages (but not Python itself), or you could introduce a new module
> and use that selectively.

Right, that seems absolutely clear now with your explanation above :-)

> I hope this clarifies and helps a bit!

Indeed it does, thank you!

Alex

Re: [PATCH 0/3] Add a generalized `wrap-language-programs`

[Ludovic Courtès]

Hi Alex,

Alex Sassmannshausen  skribis:

>> You may try e.g. scons, pip, sphinx, or youtube-dl.
>
> Hartmut was kind enough to suggest the above python applications to test
> this `wrap-language-programs` patch set against, however I have been
> stuck in "Rebuild the world"-hell for the last day and a half setting up
> testcases.
>
> I end up feeling a bit desesperated and incompetent, as I'm sure it's
> not supposed to be this hard to create an appropriate test scenario here
> — so I was wondering whether anyone has any pointers in avoiding
> constantly rebuilding the world?

The patch set you posted modifies (guix build utils), which is a module
every single derivation uses.  That’s why modifying it leads to a full
rebuild.

To test ‘wrap-language-programs’ while avoiding a full rebuild, you
could for instance add that procedure to (guix build
python-build-system), in which case you’d have to rebuild “only” Python
packages (but not Python itself), or you could introduce a new module
and use that selectively.

I hope this clarifies and helps a bit!

Ludo’.

Re: [PATCH 0/3] Add a generalized `wrap-language-programs`

[Alex Sassmannshausen]

Leo Famulari writes:

> On Fri, Nov 18, 2016 at 09:02:10AM +0100, Alex Sassmannshausen wrote:
>> Leo Famulari writes:
>> > Changes like this will need to go on the core-updates branch, where we
>> > make changes to the core of the system.
>> 
>> Makes sense, I will push it to core-updates tonight.
>
> To clarify, I didn't read all the patches yet, and I'm not the best
> person to review them. I was just trying to explain why it might take a
> long time to test this kind of change.

OK, sorry, I misunderstood.  Also in light of Hartmut's latest email, I
will hold off so we can reach a plan of action.

Alex

Re: [PATCH 0/3] Add a generalized `wrap-language-programs`

[Hartmut Goebel]

Am 17.11.2016 um 22:15 schrieb Leo Famulari:
> So, changing the build systems for these languages will require you to
> rebuild almost everything.

We now have thee build-systems which are changed: python (branch
python-build-system), cmake (branch staging, a simple change only) and perl.

What about consolidating the efforts and avoid rebuilding the world
three times? (How long will it take? The Lord took seven days for one
world :-)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: [PATCH 0/3] Add a generalized `wrap-language-programs`

[Leo Famulari]

On Fri, Nov 18, 2016 at 09:02:10AM +0100, Alex Sassmannshausen wrote:
> Leo Famulari writes:
> > Changes like this will need to go on the core-updates branch, where we
> > make changes to the core of the system.
> 
> Makes sense, I will push it to core-updates tonight.

To clarify, I didn't read all the patches yet, and I'm not the best
person to review them. I was just trying to explain why it might take a
long time to test this kind of change.

Re: [PATCH 0/3] Add a generalized `wrap-language-programs`

[Alex Sassmannshausen]

Hi Leo,

Leo Famulari writes:

> On Thu, Nov 17, 2016 at 04:58:45PM +0100, Alex Sassmannshausen wrote:
>> 
>> Hello
>> 
>> > You may try e.g. scons, pip, sphinx, or youtube-dl.
>> 
>> Hartmut was kind enough to suggest the above python applications to test
>> this `wrap-language-programs` patch set against, however I have been
>> stuck in "Rebuild the world"-hell for the last day and a half setting up
>> testcases.
>> 
>> I end up feeling a bit desesperated and incompetent, as I'm sure it's
>> not supposed to be this hard to create an appropriate test scenario here
>> — so I was wondering whether anyone has any pointers in avoiding
>> constantly rebuilding the world?
>
> Most of the Guix distribution depends on Perl, and most of the rest
> depends on Python ;)
>
> Try `guix refresh -l perl`.
>
> So, changing the build systems for these languages will require you to
> rebuild almost everything.

Thank you for pointing this out.  As I took a walk yesterday evening it
slowly started to dawn on me that something like this might be going on!
It's super useful to have this confirmed as the cause though: it gave me
that satisfying feeling of pieces of a larger puzzle falling into place
in my head :-)

> Changes like this will need to go on the core-updates branch, where we
> make changes to the core of the system.

Makes sense, I will push it to core-updates tonight.

> You could create a separate perl-test-build-system and use it for your
> test packages.

I guess this would mean that 'the world' would still depend on the
unchanged build system, thus being substitutable, and only my test
packages needing to be rebuilt with the changed build system.  That
makes a lot of sense.  Cheers!

With a new sense of clarity,

Alex

Re: [PATCH 0/3] Add a generalized `wrap-language-programs`

[Leo Famulari]

On Thu, Nov 17, 2016 at 04:58:45PM +0100, Alex Sassmannshausen wrote:
> 
> Hello
> 
> > You may try e.g. scons, pip, sphinx, or youtube-dl.
> 
> Hartmut was kind enough to suggest the above python applications to test
> this `wrap-language-programs` patch set against, however I have been
> stuck in "Rebuild the world"-hell for the last day and a half setting up
> testcases.
> 
> I end up feeling a bit desesperated and incompetent, as I'm sure it's
> not supposed to be this hard to create an appropriate test scenario here
> — so I was wondering whether anyone has any pointers in avoiding
> constantly rebuilding the world?

Most of the Guix distribution depends on Perl, and most of the rest
depends on Python ;)

Try `guix refresh -l perl`.

So, changing the build systems for these languages will require you to
rebuild almost everything.

Changes like this will need to go on the core-updates branch, where we
make changes to the core of the system.

You could create a separate perl-test-build-system and use it for your
test packages.

Re: [PATCH 0/3] Add a generalized `wrap-language-programs`

[Hartmut Goebel]

Am 17.11.2016 um 18:09 schrieb Alex Sassmannshausen:
> In both cases, the programs work fine with the generalized patch.
Fine :-)

-- 
Regards
Hartmut Goebel

| Hartmut Goebel  | h.goe...@crazy-compilers.com   |
| www.crazy-compilers.com | compilers which you thought are impossible |

Re: [PATCH 0/3] Add a generalized `wrap-language-programs`

[Alex Sassmannshausen]

Fwiw, I have now succeeded at building offlineimap, and as a result, it
seems youtube-dl also was able to be built easily (presumably because
they share a great number of dependencies).

In both cases, the programs work fine with the generalized patch.

Alex

Alex Sassmannshausen writes:

> Hello
>
>> You may try e.g. scons, pip, sphinx, or youtube-dl.
>
> Hartmut was kind enough to suggest the above python applications to test
> this `wrap-language-programs` patch set against, however I have been
> stuck in "Rebuild the world"-hell for the last day and a half setting up
> testcases.
>
> I end up feeling a bit desesperated and incompetent, as I'm sure it's
> not supposed to be this hard to create an appropriate test scenario here
> — so I was wondering whether anyone has any pointers in avoiding
> constantly rebuilding the world?
>
> For details:
> - I run on i686
> - using GuixSD
> - I rebased my patch set on master 2 days ago and tried to test from
> there
> - this resulted in a bunch of "401 - Not Found" for a number of the
> binary substitute dependencies.
> - and from then I have been rebuilding those dependencies
>
> I guess part of the problem is that the substitute servers won't
> necessarily have substitutes for a particular program from master yet?
>
> I guess an additional problem is that some of the packages fail tests on
> my setup intermittently (they fail using
> `guix package -i youtube-dl --fallback`, for instance, but would then
> succeed if I build using `guix build $dependency`).
>
> But would people have recommendations to ensure some level of substitute
> stability or some such?
>
> Best wishes,
>
> Alex
>
> Hartmut Goebel writes:
>
>> Am 16.11.2016 um 14:27 schrieb Alex Sassmannshausen:
>>> as I did not know of an
>>> appropriate test candidate (I tried to build offlineimap, but this failed
>>> because Guile@2.013 failed at least one of it’s tests).
>>
>> You may try e.g. scons, pip, sphinx, or youtube-dl.

Re: [PATCH 0/3] Add a generalized `wrap-language-programs`

[Alex Sassmannshausen]

Hello

> You may try e.g. scons, pip, sphinx, or youtube-dl.

Hartmut was kind enough to suggest the above python applications to test
this `wrap-language-programs` patch set against, however I have been
stuck in "Rebuild the world"-hell for the last day and a half setting up
testcases.

I end up feeling a bit desesperated and incompetent, as I'm sure it's
not supposed to be this hard to create an appropriate test scenario here
— so I was wondering whether anyone has any pointers in avoiding
constantly rebuilding the world?

For details:
- I run on i686
- using GuixSD
- I rebased my patch set on master 2 days ago and tried to test from
there
- this resulted in a bunch of "401 - Not Found" for a number of the
binary substitute dependencies.
- and from then I have been rebuilding those dependencies

I guess part of the problem is that the substitute servers won't
necessarily have substitutes for a particular program from master yet?

I guess an additional problem is that some of the packages fail tests on
my setup intermittently (they fail using
`guix package -i youtube-dl --fallback`, for instance, but would then
succeed if I build using `guix build $dependency`).

But would people have recommendations to ensure some level of substitute
stability or some such?

Best wishes,

Alex

Hartmut Goebel writes:

> Am 16.11.2016 um 14:27 schrieb Alex Sassmannshausen:
>> as I did not know of an
>> appropriate test candidate (I tried to build offlineimap, but this failed
>> because Guile@2.013 failed at least one of it’s tests).
>
> You may try e.g. scons, pip, sphinx, or youtube-dl.

[PATCH 0/3] Add a generalized `wrap-language-programs`

[Alex Sassmannshausen]

Hello,

Following on from discussion earlier in the week, please find attached 3
patches that implement a generalized `wrap-language-programs` procedure that
can be used to add a `wrap` phase to build-systems.

The first patch adds the procedure, and the following 2 patches add `wrap`
phase implementations using that procedure.

I have tested Perl’s wrap phase, and that seems to work fine.  I was
unfortunately not able to test Python’s wrap phase, as I did not know of an
appropriate test candidate (I tried to build offlineimap, but this failed
because Guile@2.013 failed at least one of it’s tests).

Feedback welcome!

Ludovic: you mentioned pushing this patch series to a separate branch so we
can test the builds on the build farm.  What branch would you propose?

Best wishes,

Alex

Alex Sassmannshausen (3):
  build/utils: Add `wrap-language-programs`.
  build/perl-build-system: Add `wrap` phase.
  build/python-build-system: Refactor `wrap`.

 guix/build/perl-build-system.scm   | 15 +--
 guix/build/python-build-system.scm | 36 
 guix/build/utils.scm   | 36 
 3 files changed, 57 insertions(+), 30 deletions(-)

-- 
2.10.2