Re: Enabling builddir != srcdir by default in jhbuild

[Emmanuele Bassi]

Hi;

On 6 June 2016 at 12:23, Bastien Nocera  wrote:

> The "srcdir != builddir" implemented in jhbuild isn't the same one as
> the one implemented in Continuous[1].

I know, that's why I wrote:

"""
The main change is that jhbuild runs the autogen script from within
the build directory, instead of running it into the source directory
and then running the configure script from the build directory.
"""

Continuous does:

  $(srcdir): NOCONFIGURE=1 ./autogen.sh
  $(builddir): $(srcdir)/configure $(arguments)
  $(builddir): make

whereas jhbuild (and most of our build instructions) let the
autogen.sh script call the configure script directly.

> Which is a bit of a shame when the jhbuild implementation was done to
> avoid continuous breaking. And that might also have explained a number
> of the breakages you've found.

In general, autogen.sh needs to cope with non-srcdir invocation
because the usual way to build with jhbuild is not. A large number of
GNOME modules, including the ones still using gnome-autogen.sh,
support this out of the box. The breakages I've seen mostly come from
copy-pasting autogen.sh from other (older) sources, or incomplete
attempts to support non-srcdir builds.

Ciao,
 Emmanuele.

-- 
https://www.bassi.io
[@] ebassi [@gmail.com]
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Bastien Nocera]

On Fri, 2016-06-03 at 13:11 +0200, Bastien Nocera wrote:
> On Thu, 2016-06-02 at 23:50 +0100, Emmanuele Bassi wrote:
> > [ Picking this up again ]
> > 
> > I've been spending the last couple of days fixing modules on
> > git.gnome.org (you may have noticed a commit or two from me on your
> > modules fixing builddir != srcdir issues); submitting bugs/patches
> > to
> > modules that are hosted elsewhere; and disabling non-srcdir builds
> > directly in the jhbuild modulesets. I'm fairly confident that all
> > remaining issues are now limited to modules that are in gnome-apps
> > or
> > gnome-world, but I haven't tested things like all the C++ language
> > bindings modules.
> > 
> > If you maintain a module listed in jhbuild, *please* update your
> > jhbuild local repository and try build your module with the new
> > buildroot setting enabled. If your module fails to build, fixing it
> > would also be appreciated — as it would spare you and everybody
> > else
> > time and effort; alternatively, you can poke me on IRC, and I'll
> > either fix it for you, or help you out in fixing it.
> > 
> > As a last resource, we can mark modules that do not support non-
> > srcdir
> > builds in the various modulesets, but I'd rather avoid it as it
> > defeats the purpose.
> 
> Looks like this is uncovering more bugs.
> $ jhbuild buildone -afn gnome-documents
> *** Configuring gnome-documents *** [1/1]
> /home/hadess/Projects/jhbuild/gnome-documents/autogen.sh --prefix
> /home/hadess/Projects/gnome-install --disable-Werror  --enable-
> getting-
> started 
> /usr/bin/gnome-autogen.sh
> fatal: Not a git repository (or any parent up to mount point /home)
> Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not
> set).
> 
> 
> The autogen.sh tries to run:
> git submodule update --init --recursive
> 
> Which might, or might not be a good idea. In any case, that won't
> work
> if cwd isn't the git repo.
> 
> What's the proper fix for this then?

The "srcdir != builddir" implemented in jhbuild isn't the same one as
the one implemented in Continuous[1].

Which is a bit of a shame when the jhbuild implementation was done to
avoid continuous breaking. And that might also have explained a number
of the breakages you've found.

[1]: Continuous runs autogen.sh from srcdir, jhbuild from builddir.
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Juan A. Suarez Romero]

On Fri, 2016-06-03 at 12:31 +0100, Emmanuele Bassi wrote:
> > What's the proper fix for this then?
> 
> In general, everything that modifies the Git repository or the
> autotools files, like `git submodule update` or `gtkdocize` or
> `intltoolize` will need to be called inside the source directory;
> this
> means switching to $srcdir, calling those tools, and then switch back
> to the build directory.


You could also use "git -C $srcdir ".


J.A.
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Emmanuele Bassi]

Hi Bastien;

On 3 June 2016 at 12:11, Bastien Nocera  wrote:
> On Thu, 2016-06-02 at 23:50 +0100, Emmanuele Bassi wrote:
>> [ Picking this up again ]
>>
>> As a last resource, we can mark modules that do not support non-
>> srcdir
>> builds in the various modulesets, but I'd rather avoid it as it
>> defeats the purpose.
>
> Looks like this is uncovering more bugs.

Yes; right now I'm blocked on getting a WebKit build going because of
dependencies.

> $ jhbuild buildone -afn gnome-documents
> *** Configuring gnome-documents *** [1/1]
> /home/hadess/Projects/jhbuild/gnome-documents/autogen.sh --prefix
> /home/hadess/Projects/gnome-install --disable-Werror  --enable-getting-
> started
> /usr/bin/gnome-autogen.sh
> fatal: Not a git repository (or any parent up to mount point /home)
> Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not
> set).
> 
>
> The autogen.sh tries to run:
> git submodule update --init --recursive
>
> Which might, or might not be a good idea. In any case, that won't work
> if cwd isn't the git repo.
>
> What's the proper fix for this then?

In general, everything that modifies the Git repository or the
autotools files, like `git submodule update` or `gtkdocize` or
`intltoolize` will need to be called inside the source directory; this
means switching to $srcdir, calling those tools, and then switch back
to the build directory.

In this case, I just fixed gnome-documents with this commit:

  
https://git.gnome.org/browse/gnome-documents/commit/?id=7668513171a0e9812a3fc2fc396e8b8d960e9072

The main change is that jhbuild runs the autogen script from within
the build directory, instead of running it into the source directory
and then running the configure script from the build directory. This
means that the autogen script needs to cope with this case. Most of
the autogen scripts do, and gnome-autogen.sh does as well, assuming
the $srcdir variable is set appropriately.

The rest of the build failures I've seen mostly deal with:

 * Vala
 * the introspection scanner not being given the appropriate include paths
 * the assumption that you run a full build in the srcdir before
running in a separate builddir, which is fair (even if not really
appropriate) for `make distcheck`, but it obviously fails for
non-srcdir builds

Ciao,
 Emmanuele.

Ciao,
 Emmanuele.

-- 
https://www.bassi.io
[@] ebassi [@gmail.com]
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Bastien Nocera]

On Thu, 2016-06-02 at 23:50 +0100, Emmanuele Bassi wrote:
> [ Picking this up again ]
> 
> I've been spending the last couple of days fixing modules on
> git.gnome.org (you may have noticed a commit or two from me on your
> modules fixing builddir != srcdir issues); submitting bugs/patches to
> modules that are hosted elsewhere; and disabling non-srcdir builds
> directly in the jhbuild modulesets. I'm fairly confident that all
> remaining issues are now limited to modules that are in gnome-apps or
> gnome-world, but I haven't tested things like all the C++ language
> bindings modules.
> 
> If you maintain a module listed in jhbuild, *please* update your
> jhbuild local repository and try build your module with the new
> buildroot setting enabled. If your module fails to build, fixing it
> would also be appreciated — as it would spare you and everybody else
> time and effort; alternatively, you can poke me on IRC, and I'll
> either fix it for you, or help you out in fixing it.
> 
> As a last resource, we can mark modules that do not support non-
> srcdir
> builds in the various modulesets, but I'd rather avoid it as it
> defeats the purpose.

Looks like this is uncovering more bugs.
$ jhbuild buildone -afn gnome-documents
*** Configuring gnome-documents *** [1/1]
/home/hadess/Projects/jhbuild/gnome-documents/autogen.sh --prefix
/home/hadess/Projects/gnome-install --disable-Werror  --enable-getting-
started 
/usr/bin/gnome-autogen.sh
fatal: Not a git repository (or any parent up to mount point /home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not
set).


The autogen.sh tries to run:
git submodule update --init --recursive

Which might, or might not be a good idea. In any case, that won't work
if cwd isn't the git repo.

What's the proper fix for this then?

Cheers
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Emmanuele Bassi]

[ Picking this up again ]

I've been spending the last couple of days fixing modules on
git.gnome.org (you may have noticed a commit or two from me on your
modules fixing builddir != srcdir issues); submitting bugs/patches to
modules that are hosted elsewhere; and disabling non-srcdir builds
directly in the jhbuild modulesets. I'm fairly confident that all
remaining issues are now limited to modules that are in gnome-apps or
gnome-world, but I haven't tested things like all the C++ language
bindings modules.

If you maintain a module listed in jhbuild, *please* update your
jhbuild local repository and try build your module with the new
buildroot setting enabled. If your module fails to build, fixing it
would also be appreciated — as it would spare you and everybody else
time and effort; alternatively, you can poke me on IRC, and I'll
either fix it for you, or help you out in fixing it.

As a last resource, we can mark modules that do not support non-srcdir
builds in the various modulesets, but I'd rather avoid it as it
defeats the purpose.

Ciao,
 Emmanuele.


On 31 May 2016 at 17:51, Emmanuele Bassi  wrote:
> Hi;
>
> I already pushed the default change to master, as that will only
> affect new clones or updates. I'm also building locally the default
> gnome moduleset — but I can safely say that the core platform builds
> fine. I'm just worried about gnome-world, but for that I guess we'll
> have to wait until stuff breaks.
>
> Ciao,
>  Emmanuele.
>
> On 31 May 2016 at 17:47, Michael Catanzaro  wrote:
>> On Mon, 2016-05-30 at 23:44 +0100, Emmanuele Bassi wrote:
>>> So, it seems that the discussion died on these shores.
>>>
>>> In the meantime, GVfs is but the latest module that broke because
>>> people don't test under builddir != srcdir; I really, *really* don't
>>> want to deal with this kind of perfectly avoidable build breakages
>>> any
>>> more.
>>>
>>> Ciao,
>>>  Emmanuele.
>>
>> Emmanuele, I think you can feel free to change the default in jhbuild
>> provided that everything in the apps and core suites still builds after
>> doing so. i.e. you need to make sure to add exceptions in the jhbuild
>> modulesets for all modules that need it.
>>
>> Just please wait a couple days first to see if there are any
>> substantial objections (which I do not expect).
>>
>> Michael
>
>
>
> --
> https://www.bassi.io
> [@] ebassi [@gmail.com]



-- 
https://www.bassi.io
[@] ebassi [@gmail.com]
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Alberto Mardegan]

On 31/05/2016 19:04, Emmanuele Bassi wrote:
>  b) we don't have the resources to set up and maintain a try server
> running continuous

I've been recently playing with the CI service provided by gitlab.com
and I find it very convenient: you just have to create a single file in
your repository, containing the build targets and rules, and then gitlab
automatically builds your project for every git branch you push (even
before you propose a merge).

IIUC, it's based on travis CI, and I believe that github offers roughly
the same (I don't use github much, so I'm not sure).

Not suggesting anything, just saying. :-)

Ciao,
  Alberto

___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Sébastien Wilmet]

On Wed, Jun 01, 2016 at 01:12:08PM +0100, Emmanuele Bassi wrote:
> The same thing will happen if you have a master/master-stable split,
> because now people will commit to master, break it anyway, and nobody
> will look at 'master-stable' because it's a completely different
> workflow than what Git and various tools and organisations use.

I said that jhbuild would build master-stable by default. A developer
would use master only for a few modules, the modules he/she works on.

Currently, GContinuous builds and tests the set of the latest commits
pushed on master. It doesn't have the time to test thoroughly each
commit individually. So what I have in mind, is that when that set of
commits fails, GContinuous would do a binary search in the list of
commits sorted in chronological order, to determine which module to tag
(i.e. for which module, the master branch will not get merged into
master-stable). When a module is tagged, a mail is sent to the
maintainers listed in the doap file. And then GContinuous can continue
with the next set of commits coming from master, except that it uses
master-stable for the tagged modules.

The problem is that GContinuous takes currently a shortcut: it doesn't
rebuild everything AFAIK. It just rebuilds the modules where there are
new commits, and for the other modules it just runs the installed unit
tests, if there are any. But, a certain commit in module A might break
the _build_ in module B. But if module B is not rebuilt when there is
the new commit in module A, GContinuous will think that everything is
alright, but when there is a new commit in module B and GContinuous
tries to rebuild it, it will fail but at that point it is too late, if
we take all the master-stable branches it fails.

So, there could be a master-stable-next branch, and e.g. once a day a
complete build, test and whatnot is done on all the master-stable-next
branches to determine if they can become master-stable. :-)

To simplify things, untagging a module could be done manually. Although
it is possible to imagine an algo doing that automatically.

Yes, I don't plan to work on all of this, it's just random ideas. But I
think this would be necessary to achieve a continuous delivery in a more
automated way. Maybe it'll be implemented in Fedora one day, with the
Atomic Workstation project?

--
Sébastien
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Emmanuele Bassi]

On 1 June 2016 at 10:33, Sébastien Wilmet  wrote:

>> A try server is not a simple thing to set up; and if the cost of
>> setting it up is non-negligible, it simply pales compared to the cost
>> of keeping it up. We can barely keep Continuous building as it is —
>> somebody would need to be funded full time just to keep it going for
>> each bug/feature/patch series that we'd want to test.
>
> With master/master-stable, developers, translators etc continue to work
> exactly as before. But GContinuous would automatically sets
> master-stable to commits that work (from the GContinuous point of view).
> And jhbuild would pick up the master-stable branch by default.
>
> It doesn't need more servers,

Oh, I can assure you: it does need a separate build machine — and not a VM.

> it needs development time. Currently
> GContinuous needs maintenance: tag modules manually to a previous
> commit, untag, etc etc. A lot of that manual work could be replaced by
> an automatic way. In the short term it needs more time to develop that,
> but in the longer term, it saves time and we would have something that
> works better.

This is wishful thinking, at best.

Continuous would have a lot less issues if people actually built and
tested locally *already*, using a set up that maps to what Continuous
does — hence the default of switching jhbuild to out of tree builds.

> With master/master-stable, no need to manually revert commits.

The only way this is going to work is if very, very few people can
push to master on git.gnome.org — which is obviously not going to be
the case.

If the workflow is:

 * people push to master-next
 * Continuous builds all 'master-next' branches
 * on success: ${SOMETHING} merges 'master-next' into 'master'
 * on failure: ${SOMETHING} sends an email to ${SOMEBODY} to fix 'master-next'

I can assure that these things will happen:

 * some maintainers will keep pushing to 'master', sidestepping the
whole automated build
 * some maintainers will try pushing to 'master-next', go through a
couple of issues (by and large with the build system) and stop
 * 'master-next' will break and *stay broken*, and 'master' will break
at random points because nothing runs 'master'

Unless you add a new machine that builds on 'master' as well, in which
case you now build everything twice and you have to unbreak two
separate builds, and you just made my life as a volunteer build
sheriff twice as hard, and I have not seen you volunteering for the
job either.

The same thing will happen if you have a master/master-stable split,
because now people will commit to master, break it anyway, and nobody
will look at 'master-stable' because it's a completely different
workflow than what Git and various tools and organisations use.

Obviously, on top of that, we have to write all the tooling that
implements the ${SOMETHING} above, and keeps it running; so now we
have:

 * two separate build bots
 * a set of tools that deals with checking and merging branches on git.gnome.org
 * a set of tools that deals with sending notifications to people
pushing commits that break either of those builds, and maybe files
bugs about it

And I have not seen anybody spend some time to work on the *existing*
Continuous builder to send an email to the people listed on the DOAP
file, or to the committer, whenever a module breaks.

There *are* various alternatives/steps to keep Continuous working:

1. have an 'integration'/'release-team' branch that only release team
members can use; this is merged (daily/nightly, or whenever the build
succeeds and the smoketesting runs)
2. limit pushing to the 'master' branch to people listed as
maintainers on the DOAP file; translations go to a 'i18n' branch that
gets merged manually, and all features/bug fixes happen on topic
branches
3. have a try server that automatically builds all topic branches in a project
4. have a try server that builds patch series on request (maybe with
Bugzilla integration, so that we get results as Bugzilla comments)

Any of the above can be implemented separately, and it'd be a great
thing already; considering the amount of people that care about
building GNOME at the moment, and the amount of work necessary to get
this show on the road, you'll forgive me if I don't hold my breath.

In general, though, GNOME is *not* a simple system; it's made of many,
many moving parts that are developed on different cadences, and those
moving parts are interconnected in non-trivial ways. Even Firefox,
which is at least a couple of orders of magnitude simpler than GNOME,
and has try servers, build sheriffs, and automated testing running on
different platforms, breaks a build to the point of requiring a
revert, and that happens at least once a month. The only large
difference is that Firefox has a dedicated team of people that keep
the build and testing under control.

Ciao,
 Emmanuele.

-- 
https://www.bassi.io
[@] ebassi [@gmail.com]
___

Re: Enabling builddir != srcdir by default in jhbuild

[Sébastien Wilmet]

On Wed, Jun 01, 2016 at 11:33:41AM +0200, Sébastien Wilmet wrote:
> With master/master-stable, developers, translators etc continue to work
> exactly as before. But GContinuous would automatically sets
> master-stable to commits that work (from the GContinuous point of view).
> And jhbuild would pick up the master-stable branch by default.
> 
> It doesn't need more servers, it needs development time.

With a bit more thinking, the server would probably need more resources
since it has more work to do to determine which module to tag when there
is a failure, if the algorithm used to determine that is simple and
generic (no heuristics). And it has also more work to do to test if it
can untag a module.

--
Sébastien
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Michael Catanzaro]

On Tue, 2016-05-31 at 17:51 +0100, Emmanuele Bassi wrote:
> I'm just worried about gnome-world, but for that I guess we'll
> have to wait until stuff breaks.

Most everything in gnome-world is already broken, so don't worry about
it. We don't maintain gnome-world. Feel free to fix anything you care
about, of course.
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Florian Müllner]

On Tue, May 31, 2016 at 6:47 PM Michael Catanzaro 
wrote:

> Just please wait a couple days first to see if there are any
> substantial objections (which I do not expect).


Is this really necessary? This is a revived thread from February, so surely
there has been plenty of time to voice objections ...
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Emmanuele Bassi]

Hi;

I already pushed the default change to master, as that will only
affect new clones or updates. I'm also building locally the default
gnome moduleset — but I can safely say that the core platform builds
fine. I'm just worried about gnome-world, but for that I guess we'll
have to wait until stuff breaks.

Ciao,
 Emmanuele.

On 31 May 2016 at 17:47, Michael Catanzaro  wrote:
> On Mon, 2016-05-30 at 23:44 +0100, Emmanuele Bassi wrote:
>> So, it seems that the discussion died on these shores.
>>
>> In the meantime, GVfs is but the latest module that broke because
>> people don't test under builddir != srcdir; I really, *really* don't
>> want to deal with this kind of perfectly avoidable build breakages
>> any
>> more.
>>
>> Ciao,
>>  Emmanuele.
>
> Emmanuele, I think you can feel free to change the default in jhbuild
> provided that everything in the apps and core suites still builds after
> doing so. i.e. you need to make sure to add exceptions in the jhbuild
> modulesets for all modules that need it.
>
> Just please wait a couple days first to see if there are any
> substantial objections (which I do not expect).
>
> Michael



-- 
https://www.bassi.io
[@] ebassi [@gmail.com]
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Michael Catanzaro]

On Mon, 2016-05-30 at 23:44 +0100, Emmanuele Bassi wrote:
> So, it seems that the discussion died on these shores.
> 
> In the meantime, GVfs is but the latest module that broke because
> people don't test under builddir != srcdir; I really, *really* don't
> want to deal with this kind of perfectly avoidable build breakages
> any
> more.
> 
> Ciao,
>  Emmanuele.

Emmanuele, I think you can feel free to change the default in jhbuild
provided that everything in the apps and core suites still builds after
doing so. i.e. you need to make sure to add exceptions in the jhbuild
modulesets for all modules that need it.

Just please wait a couple days first to see if there are any
substantial objections (which I do not expect).

Michael
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Emmanuele Bassi]

Hi;

On 31 May 2016 at 16:01, Sébastien Wilmet  wrote:
> On Mon, May 30, 2016 at 11:44:48PM +0100, Emmanuele Bassi wrote:
>> So, it seems that the discussion died on these shores.
>>
>> In the meantime, GVfs is but the latest module that broke because
>> people don't test under builddir != srcdir; I really, *really* don't
>> want to deal with this kind of perfectly avoidable build breakages any
>> more.
>
> The root of the problem, or the elephant in the room, is that committers
> push the commits directly on master.

No, not really.

I mean: in an idea world, we'd have code reviews for *every* commit,
and we'd have try servers; topic branches for bug fixes and features;
bots that deal with validating against a test suite every time a pull
request is updated; and bots that deal with merging to master. We'd
likely also have a build system that does not behave like it's 1983.

In reality, we make do with what we have.

> If you want real continuous
> integration/delivery, a commit should be pushed on master only if it has
> the green light from GNOME Continuous. It's the only way to avoid
> chasing continuously developers breaking stuff.

This is the role of a try server; the problem is that:

 a) we have many interacting components
 b) we don't have the resources to set up and maintain a try server
running continuous
 c) unless you have a very limited subset of developers being able to
push to master, people will push to master

I mean: we have an automated tool for translations that pushes to
master by default — and does not have any validation system to prevent
pushing translations breaking the build. We don't even have automated
hooks to validate translations, and those are usually the obvious
culprit of build failures.

A try server is not a simple thing to set up; and if the cost of
setting it up is non-negligible, it simply pales compared to the cost
of keeping it up. We can barely keep Continuous building as it is —
somebody would need to be funded full time just to keep it going for
each bug/feature/patch series that we'd want to test.

> There could be a master-next branch that GNOME Continuous test and
> rebase/merge on top of master automatically if the build and tests
> succeed.

It'll be a cold, cold day in hell before we can actually follow this
process with a system as complex as GNOME.

I mean: I proposed to have build sheriffs ensuring that Continuous
keep building and I nearly got accused of killing the GNOME community
because I had the temerity of saying that the subset of GNOME
specified in the Continuous manifest should always build, at the cost
of reverting commits that break the build if the maintainer is not
responsive.

> Would it be too cumbersome for committers?

Yes; but, more importantly, it would strain our existing
infrastructure way, way beyond any breaking point.

Ciao,
 Emmanuele.

-- 
https://www.bassi.io
[@] ebassi [@gmail.com]
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Sébastien Wilmet]

On Tue, May 31, 2016 at 05:01:23PM +0200, Sébastien Wilmet wrote:
> There could be a master-next branch that GNOME Continuous test and
> rebase/merge on top of master automatically if the build and tests
> succeed.

Or instead of master-next/master, having master/master-stable. Only
GNOME Continuous would be able to change master-stable. By default
jhbuild would use master-stable. And we would continue to push commits
on master as we do now.

--
Sébastien
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Sébastien Wilmet]

On Mon, May 30, 2016 at 11:44:48PM +0100, Emmanuele Bassi wrote:
> So, it seems that the discussion died on these shores.
> 
> In the meantime, GVfs is but the latest module that broke because
> people don't test under builddir != srcdir; I really, *really* don't
> want to deal with this kind of perfectly avoidable build breakages any
> more.

The root of the problem, or the elephant in the room, is that committers
push the commits directly on master. If you want real continuous
integration/delivery, a commit should be pushed on master only if it has
the green light from GNOME Continuous. It's the only way to avoid
chasing continuously developers breaking stuff.

There could be a master-next branch that GNOME Continuous test and
rebase/merge on top of master automatically if the build and tests
succeed.

Would it be too cumbersome for committers?

Of course it's not implemented by GNOME Continuous, so it would need
development to integrate all that. But if it's done, GNOME Continuous
would need much less maintenance, and GNOME would be ready to ship at
any time, and some users could use bleeding-edge GNOME if they want.

--
Sébastien
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Ondrej Holy]

Sorry for that! I've added "buildroot = '~/gnome/build'" into my jhbuildrc
thanks to this thread... so +1 for changing the defaults.

2016-05-31 0:44 GMT+02:00 Emmanuele Bassi :

> So, it seems that the discussion died on these shores.
>
> In the meantime, GVfs is but the latest module that broke because
> people don't test under builddir != srcdir; I really, *really* don't
> want to deal with this kind of perfectly avoidable build breakages any
> more.
>
> Ciao,
>  Emmanuele.
>
>
> On 2 March 2016 at 10:29, Yanko Kaneti  wrote:
> > On Sun, 2016-02-28 at 09:54 -0600, Michael Catanzaro wrote:
> >> On Sun, 2016-02-28 at 14:33 +, Emmanuele Bassi wrote:
> >> >
> >> > My proposal is to enable this behaviour in the default jhbuildrc,
> >> > so
> >> > that all GNOME projects automatically build in a separate root.
> >> > This
> >> > change should have no, or minimal impact on the subset of the
> >> > moduleset that is covered by Continuous; it may require fixing the
> >> > build of various modules in the gnome-apps or gnome-world
> >> > modulesets.
> >> > The major upsides would be that:
> >> +1, I've been wanting to do this for a while.
> >>
> >> There is a problem though, in that Automake's vala support is pretty
> >> broken for git builds where builddir != srcdir, hence all (well,
> >> virtually all) vala projects only support builddir != srcdir when
> >> building from tarballs. Sometimes the only way to fix builddir !=
> >> srcdir issues is to introduce broken nonsensical changes to the
> >> Automake file [1]. So when you make this change, you should do a
> >> clean
> >> build of all the modules and add exceptions for all the modules that
> >> break, with the expectation that most Vala modules will be broken.
> >> (You
> >> were probably planning to do this anyway, so just pointing it out.)
> >>
> >> If anyone is interested in fixing this in Automake, that would be
> >> ideal.
> >
> > I was interested sometime ago and produced something minimal that works
> > for me but dropped the ball due to not being able to conjure the
> > testcases for the fix...
> >
> > http://lists.gnu.org/archive/html/automake-patches/2014-12/msg0.html
> >
> >
> > - Yanko
> >
> >
> >> [1] https://bugzilla.gnome.org/show_bug.cgi?id=760679
> >
> > ___
> > desktop-devel-list mailing list
> > desktop-devel-list@gnome.org
> > https://mail.gnome.org/mailman/listinfo/desktop-devel-list
>
>
>
> --
> https://www.bassi.io
> [@] ebassi [@gmail.com]
> ___
> desktop-devel-list mailing list
> desktop-devel-list@gnome.org
> https://mail.gnome.org/mailman/listinfo/desktop-devel-list
>



-- 
Regards

Ondrej
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Carlos Soriano Sanchez]

I would like to have this as default too.

As pointed out previously, maybe we can run a whole build and create exceptions 
for those that doesn't build and file bugs for them, but change already the 
default?
I think the sooner the better, if not we are not going to take it out of our 
chest.

Cheers,
Carlos Soriano

- Original Message -
| So, it seems that the discussion died on these shores.
| 
| In the meantime, GVfs is but the latest module that broke because
| people don't test under builddir != srcdir; I really, *really* don't
| want to deal with this kind of perfectly avoidable build breakages any
| more.
| 
| Ciao,
|  Emmanuele.
| 
| 
| On 2 March 2016 at 10:29, Yanko Kaneti  wrote:
| > On Sun, 2016-02-28 at 09:54 -0600, Michael Catanzaro wrote:
| >> On Sun, 2016-02-28 at 14:33 +, Emmanuele Bassi wrote:
| >> >
| >> > My proposal is to enable this behaviour in the default jhbuildrc,
| >> > so
| >> > that all GNOME projects automatically build in a separate root.
| >> > This
| >> > change should have no, or minimal impact on the subset of the
| >> > moduleset that is covered by Continuous; it may require fixing the
| >> > build of various modules in the gnome-apps or gnome-world
| >> > modulesets.
| >> > The major upsides would be that:
| >> +1, I've been wanting to do this for a while.
| >>
| >> There is a problem though, in that Automake's vala support is pretty
| >> broken for git builds where builddir != srcdir, hence all (well,
| >> virtually all) vala projects only support builddir != srcdir when
| >> building from tarballs. Sometimes the only way to fix builddir !=
| >> srcdir issues is to introduce broken nonsensical changes to the
| >> Automake file [1]. So when you make this change, you should do a
| >> clean
| >> build of all the modules and add exceptions for all the modules that
| >> break, with the expectation that most Vala modules will be broken.
| >> (You
| >> were probably planning to do this anyway, so just pointing it out.)
| >>
| >> If anyone is interested in fixing this in Automake, that would be
| >> ideal.
| >
| > I was interested sometime ago and produced something minimal that works
| > for me but dropped the ball due to not being able to conjure the
| > testcases for the fix...
| >
| > http://lists.gnu.org/archive/html/automake-patches/2014-12/msg0.html
| >
| >
| > - Yanko
| >
| >
| >> [1] https://bugzilla.gnome.org/show_bug.cgi?id=760679
| >
| > ___
| > desktop-devel-list mailing list
| > desktop-devel-list@gnome.org
| > https://mail.gnome.org/mailman/listinfo/desktop-devel-list
| 
| 
| 
| --
| https://www.bassi.io
| [@] ebassi [@gmail.com]
| ___
| desktop-devel-list mailing list
| desktop-devel-list@gnome.org
| https://mail.gnome.org/mailman/listinfo/desktop-devel-list
| 
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Emmanuele Bassi]

So, it seems that the discussion died on these shores.

In the meantime, GVfs is but the latest module that broke because
people don't test under builddir != srcdir; I really, *really* don't
want to deal with this kind of perfectly avoidable build breakages any
more.

Ciao,
 Emmanuele.


On 2 March 2016 at 10:29, Yanko Kaneti  wrote:
> On Sun, 2016-02-28 at 09:54 -0600, Michael Catanzaro wrote:
>> On Sun, 2016-02-28 at 14:33 +, Emmanuele Bassi wrote:
>> >
>> > My proposal is to enable this behaviour in the default jhbuildrc,
>> > so
>> > that all GNOME projects automatically build in a separate root.
>> > This
>> > change should have no, or minimal impact on the subset of the
>> > moduleset that is covered by Continuous; it may require fixing the
>> > build of various modules in the gnome-apps or gnome-world
>> > modulesets.
>> > The major upsides would be that:
>> +1, I've been wanting to do this for a while.
>>
>> There is a problem though, in that Automake's vala support is pretty
>> broken for git builds where builddir != srcdir, hence all (well,
>> virtually all) vala projects only support builddir != srcdir when
>> building from tarballs. Sometimes the only way to fix builddir !=
>> srcdir issues is to introduce broken nonsensical changes to the
>> Automake file [1]. So when you make this change, you should do a
>> clean
>> build of all the modules and add exceptions for all the modules that
>> break, with the expectation that most Vala modules will be broken.
>> (You
>> were probably planning to do this anyway, so just pointing it out.)
>>
>> If anyone is interested in fixing this in Automake, that would be
>> ideal.
>
> I was interested sometime ago and produced something minimal that works
> for me but dropped the ball due to not being able to conjure the
> testcases for the fix...
>
> http://lists.gnu.org/archive/html/automake-patches/2014-12/msg0.html
>
>
> - Yanko
>
>
>> [1] https://bugzilla.gnome.org/show_bug.cgi?id=760679
>
> ___
> desktop-devel-list mailing list
> desktop-devel-list@gnome.org
> https://mail.gnome.org/mailman/listinfo/desktop-devel-list



-- 
https://www.bassi.io
[@] ebassi [@gmail.com]
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Yanko Kaneti]

On Sun, 2016-02-28 at 09:54 -0600, Michael Catanzaro wrote:
> On Sun, 2016-02-28 at 14:33 +, Emmanuele Bassi wrote:
> > 
> > My proposal is to enable this behaviour in the default jhbuildrc,
> > so
> > that all GNOME projects automatically build in a separate root.
> > This
> > change should have no, or minimal impact on the subset of the
> > moduleset that is covered by Continuous; it may require fixing the
> > build of various modules in the gnome-apps or gnome-world
> > modulesets.
> > The major upsides would be that:
> +1, I've been wanting to do this for a while.
> 
> There is a problem though, in that Automake's vala support is pretty
> broken for git builds where builddir != srcdir, hence all (well,
> virtually all) vala projects only support builddir != srcdir when
> building from tarballs. Sometimes the only way to fix builddir !=
> srcdir issues is to introduce broken nonsensical changes to the
> Automake file [1]. So when you make this change, you should do a
> clean
> build of all the modules and add exceptions for all the modules that
> break, with the expectation that most Vala modules will be broken.
> (You
> were probably planning to do this anyway, so just pointing it out.)
> 
> If anyone is interested in fixing this in Automake, that would be
> ideal.

I was interested sometime ago and produced something minimal that works
for me but dropped the ball due to not being able to conjure the
testcases for the fix...

http://lists.gnu.org/archive/html/automake-patches/2014-12/msg0.html


- Yanko


> [1] https://bugzilla.gnome.org/show_bug.cgi?id=760679

___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Emmanuele Bassi]

I actually was thinking about putting the intermediate build state
inside `$XDG_CACHE_HOME/jhbuild/build` by default, like we put the
downloaded tarballs in `$XDG_CACHE_HOME/jhbuild/downloads` by default.

I can check if `$HOME/jhbuild` exists, and put a `build` directory under there.

Ciao,
 Emmanuele.


On 29 February 2016 at 13:41, Carlos Soriano Sanchez
 wrote:
> A big +1 to this idea.
>
> As a nitpick, change the buildroot to '~/jhbuild/build'
> because we use by default '~/jhbuild/checkout' and
> '~/jhbuild/install', or convince jhbuild devs to change
> default to '~/gnome/*'.
>
> Cheers,
> Carlos Soriano
>
> - Original Message -
> | Hi all;
> |
> | as you may know, automated build services – for instance Continuous,
> | OBS, or the autotools distcheck – use a build directory that is not
> | the same as the source directory. Our own jhbuild allows this, even if
> | it's disabled by default. This means that developers may work on a
> | feature or a fix in jhbuild, and build the project successfully – only
> | to introduce a build failure that either gets caught by the Continuous
> | system, or gets caught by the maintainer only during `make distcheck`
> | at release time.
> |
> | We've had many build failures that are usually the result of a broken
> | `builddir == srcdir` assumption; they usually break distcheck,
> | delaying releases; they also break Continuous, or builds from
> | distribution packagers. Various projects mandate a builddir != srcdir
> | default in their own build system, but it's not really mandatory.
> |
> | Jhbuild supports building projects in a separate root, which would
> | allow maintainers to catch build breakages; you just need to add:
> |
> | buildroot = '~/gnome/build'
> |
> | to your ~/.config/jhbuildrc. Jhbuild also allows projects to specify
> | that they cannot build if the build directory is not the source
> | directory, thus providing an escape hatch.
> |
> | My proposal is to enable this behaviour in the default jhbuildrc, so
> | that all GNOME projects automatically build in a separate root. This
> | change should have no, or minimal impact on the subset of the
> | moduleset that is covered by Continuous; it may require fixing the
> | build of various modules in the gnome-apps or gnome-world modulesets.
> | The major upsides would be that:
> |
> |  * automated builds of our software are possible without hacks
> |  * distchecking projects does not fail at the very last minute before
> | release but during development
> |  * we bring the development environment and the continuous deployment
> | environment closer
> |
> | What do maintainers think?
> |
> | Ciao,
> |  Emmanuele.
> |
> | --
> | https://www.bassi.io
> | [@] ebassi [@gmail.com]
> | ___
> | desktop-devel-list mailing list
> | desktop-devel-list@gnome.org
> | https://mail.gnome.org/mailman/listinfo/desktop-devel-list



-- 
https://www.bassi.io
[@] ebassi [@gmail.com]
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Carlos Soriano Sanchez]

A big +1 to this idea.

As a nitpick, change the buildroot to '~/jhbuild/build'
because we use by default '~/jhbuild/checkout' and
'~/jhbuild/install', or convince jhbuild devs to change
default to '~/gnome/*'.

Cheers,
Carlos Soriano

- Original Message -
| Hi all;
| 
| as you may know, automated build services – for instance Continuous,
| OBS, or the autotools distcheck – use a build directory that is not
| the same as the source directory. Our own jhbuild allows this, even if
| it's disabled by default. This means that developers may work on a
| feature or a fix in jhbuild, and build the project successfully – only
| to introduce a build failure that either gets caught by the Continuous
| system, or gets caught by the maintainer only during `make distcheck`
| at release time.
| 
| We've had many build failures that are usually the result of a broken
| `builddir == srcdir` assumption; they usually break distcheck,
| delaying releases; they also break Continuous, or builds from
| distribution packagers. Various projects mandate a builddir != srcdir
| default in their own build system, but it's not really mandatory.
| 
| Jhbuild supports building projects in a separate root, which would
| allow maintainers to catch build breakages; you just need to add:
| 
| buildroot = '~/gnome/build'
| 
| to your ~/.config/jhbuildrc. Jhbuild also allows projects to specify
| that they cannot build if the build directory is not the source
| directory, thus providing an escape hatch.
| 
| My proposal is to enable this behaviour in the default jhbuildrc, so
| that all GNOME projects automatically build in a separate root. This
| change should have no, or minimal impact on the subset of the
| moduleset that is covered by Continuous; it may require fixing the
| build of various modules in the gnome-apps or gnome-world modulesets.
| The major upsides would be that:
| 
|  * automated builds of our software are possible without hacks
|  * distchecking projects does not fail at the very last minute before
| release but during development
|  * we bring the development environment and the continuous deployment
| environment closer
| 
| What do maintainers think?
| 
| Ciao,
|  Emmanuele.
| 
| --
| https://www.bassi.io
| [@] ebassi [@gmail.com]
| ___
| desktop-devel-list mailing list
| desktop-devel-list@gnome.org
| https://mail.gnome.org/mailman/listinfo/desktop-devel-list
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Jens Georg]

On So, 2016-02-28 at 15:43 +0100, Jens Georg wrote:
> >  * automated builds of our software are possible without hacks
> >  * distchecking projects does not fail at the very last minute
> > before
> > release but during development
> >  * we bring the development environment and the continuous
> > deployment
> > environment closer
> > 
> > What do maintainers think?
> 
> +1

One thing to keep in mind, though: the Vala autotools support used to
be a bit daft here.
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Emmanuele Bassi]

Hi;

On 28 February 2016 at 15:54, Michael Catanzaro  wrote:
> On Sun, 2016-02-28 at 14:33 +, Emmanuele Bassi wrote:
>> My proposal is to enable this behaviour in the default jhbuildrc, so
>> that all GNOME projects automatically build in a separate root. This
>> change should have no, or minimal impact on the subset of the
>> moduleset that is covered by Continuous; it may require fixing the
>> build of various modules in the gnome-apps or gnome-world modulesets.
>> The major upsides would be that:
>
> +1, I've been wanting to do this for a while.
>
> There is a problem though, in that Automake's vala support is pretty
> broken for git builds where builddir != srcdir, hence all (well,
> virtually all) vala projects only support builddir != srcdir when
> building from tarballs. Sometimes the only way to fix builddir !=
> srcdir issues is to introduce broken nonsensical changes to the
> Automake file [1]. So when you make this change, you should do a clean
> build of all the modules and add exceptions for all the modules that
> break, with the expectation that most Vala modules will be broken. (You
> were probably planning to do this anyway, so just pointing it out.)
>
> If anyone is interested in fixing this in Automake, that would be
> ideal.
>
> [1] https://bugzilla.gnome.org/show_bug.cgi?id=760679

Given the amount of resources currently devoted to Vala, and
especially the interaction between the compiler and autotools, I'd say
that all Vala-based projects should use the
'supports-non-srcdir-builds="no"' attribute in their module definition
inside jhbuild modulesets.

Ciao,
 Emmanuele.

-- 
https://www.bassi.io
[@] ebassi [@gmail.com]
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Michael Catanzaro]

On Sun, 2016-02-28 at 14:33 +, Emmanuele Bassi wrote:
> My proposal is to enable this behaviour in the default jhbuildrc, so
> that all GNOME projects automatically build in a separate root. This
> change should have no, or minimal impact on the subset of the
> moduleset that is covered by Continuous; it may require fixing the
> build of various modules in the gnome-apps or gnome-world modulesets.
> The major upsides would be that:

+1, I've been wanting to do this for a while.

There is a problem though, in that Automake's vala support is pretty
broken for git builds where builddir != srcdir, hence all (well,
virtually all) vala projects only support builddir != srcdir when
building from tarballs. Sometimes the only way to fix builddir !=
srcdir issues is to introduce broken nonsensical changes to the
Automake file [1]. So when you make this change, you should do a clean
build of all the modules and add exceptions for all the modules that
break, with the expectation that most Vala modules will be broken. (You
were probably planning to do this anyway, so just pointing it out.)

If anyone is interested in fixing this in Automake, that would be
ideal.

[1] https://bugzilla.gnome.org/show_bug.cgi?id=760679
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Sébastien Wilmet]

On Sun, Feb 28, 2016 at 02:33:02PM +, Emmanuele Bassi wrote:
> buildroot = '~/gnome/build'

I agree it's a good idea, but I'll need to change a little my habits
when I run 'make' in only one sub-directory. Instead of being at the
same place in the git repo, I'll need to be somewhere in ~/gnome/build/.

And instead of 'git clean -xdf' or -Xdf, I can 'rm -rf ~/gnome/build/'
to start again with a clean build.

So, +1.

--
Sébastien
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Richard Hughes]

On 28 February 2016 at 14:33, Emmanuele Bassi  wrote:
> What do maintainers think?

Makes sense to me. The quicker we catch a destdir/srcdir issue the
easier it is to fix.

Richard
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Re: Enabling builddir != srcdir by default in jhbuild

[Jens Georg]

>  * automated builds of our software are possible without hacks
>  * distchecking projects does not fail at the very last minute before
> release but during development
>  * we bring the development environment and the continuous deployment
> environment closer
> 
> What do maintainers think?

+1
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list

Enabling builddir != srcdir by default in jhbuild

[Emmanuele Bassi]

Hi all;

as you may know, automated build services – for instance Continuous,
OBS, or the autotools distcheck – use a build directory that is not
the same as the source directory. Our own jhbuild allows this, even if
it's disabled by default. This means that developers may work on a
feature or a fix in jhbuild, and build the project successfully – only
to introduce a build failure that either gets caught by the Continuous
system, or gets caught by the maintainer only during `make distcheck`
at release time.

We've had many build failures that are usually the result of a broken
`builddir == srcdir` assumption; they usually break distcheck,
delaying releases; they also break Continuous, or builds from
distribution packagers. Various projects mandate a builddir != srcdir
default in their own build system, but it's not really mandatory.

Jhbuild supports building projects in a separate root, which would
allow maintainers to catch build breakages; you just need to add:

buildroot = '~/gnome/build'

to your ~/.config/jhbuildrc. Jhbuild also allows projects to specify
that they cannot build if the build directory is not the source
directory, thus providing an escape hatch.

My proposal is to enable this behaviour in the default jhbuildrc, so
that all GNOME projects automatically build in a separate root. This
change should have no, or minimal impact on the subset of the
moduleset that is covered by Continuous; it may require fixing the
build of various modules in the gnome-apps or gnome-world modulesets.
The major upsides would be that:

 * automated builds of our software are possible without hacks
 * distchecking projects does not fail at the very last minute before
release but during development
 * we bring the development environment and the continuous deployment
environment closer

What do maintainers think?

Ciao,
 Emmanuele.

-- 
https://www.bassi.io
[@] ebassi [@gmail.com]
___
desktop-devel-list mailing list
desktop-devel-list@gnome.org
https://mail.gnome.org/mailman/listinfo/desktop-devel-list