On 11.04.2013 18:43, Andrew Rist wrote:
On 4/11/2013 12:06 AM, Andre Fischer wrote:
On 10.04.2013 18:57, Andrew Rist wrote:
On 4/10/2013 1:33 AM, Herbert Dürr wrote:
On 2013/04/10 10:09 AM, Andre Fischer wrote:
tonight we had a build breaker in the windows build: a slot id
that is
used in SW had been removed in SVX. The reference in SW had also
been
removed, so this change should not be a problem.
But the windows build is still not a full build. Therefore the old SW
slot header files where used and the build broke.
There is an easy fix for situations like this: a clean build.
Incremental build are known to have problems thats why I suggested
[1] to default to a clean build. That didn't receive consensus
though and indeed there are good reasons against it:
The incremental build both tests the dependency system and it
reduces the load when building significantly.
On the already strained buildbot this means a factor of almost five
improvement as clean build takes about 4.5h whereas an incremental
build takes only 0.5-1.0h.
Andrew even had to reschedule the snapshot build away from the
weekly clean build because the buildbot load is a real problem.
[1] http://markmail.org/message/wmlhc5f5zaiiyu2o
[2] http://markmail.org/message/7q64ijlwygdqmwf3
Just to add here, that there are also issues with a clean build. The
clean build fails with some frequency on hung jobs and requires
manual attention.
That is one more reason to have more frequent clean builds so that we
can find the cause of these problems. They are not restricted to the
build bot. Others are affected, too. I would have tried to fix
this, but I am not able to reproduce this problem on my local machine.
Sorry, I think I was unclear there. Due to the complexity of our
build process and the interaction with the buildbot, there is a
reasonably high incidence of false positive failures on clean builds.
The Windows build ends up with hung processes and throws an
exception. If we were to switch to clean builds we could expect
several false positives a week, which would require manual
intervention. We have tests of clean builds daily on the linux boxes,
so in terms of coverage of the entire AOO buildbot setup, we have full
builds covered. I see the fact that some of our builds are
incremental and some are clean, as a feature, not as a shortcoming.
OK, I understand. I see two problems with this approach:
1. We will not find the build problems when we continue to hide them
with workarounds. Fixing them would benefit others as well, not just
the build bots.
2. The task of the build bots is not only to verify that our source code
is buildable but also to provide download sets for developers and QA.
In reality, breaking changes that require a clean build are pretty
rare. For me, the clean build on the weekend and incremental during
the week seems to be a good compromise.
I am not sure about that. Besides, it is sometimes a bit difficult
to judge 'how incompatible' a change really is. Change a slot
definition in SVX and its use in SW. Do we need anything more? With
a clean build we are on the safe side.
It's a trade off. How many false positives to you get, and how much
manual intervention does the system require. What I'm trying to
communicate is that my experience with 'this buildbot setup' suggests
that the current approach requires less of my time to keep healthy,
and produces less false positives.
Ah, again I understand. It is a matter of pragmatism. I can accept
that. I and I think all others on this list are thankful for your work
in this area. And maybe we can help to improve the current state. Is
there any documentation, a Wiki page maybe, to get an overview?
Maybe we can rededicate one of the daily builds of one of the branches
to debugging our windows build problems? Add a step to the build setup
to apply patches that provide us with more debug info?
Besides, the clean-build-on-weekend policy would require us to hold
all incompatible changes until Friday, or live with a broken build
during the week.
I really thing that we need a better solution. A switch for marking
a change as incompatible and that would be interpreted by the build
bot would be the absolute minimum. But even that would call for
trouble. At Sun we have been there and it did not really work so well.
This doesn't require waiting until the weekend, it requires a manual
clean run which can be kicked off easily. (I'm happy to show you, or
hit up Herbert - he has access, too)
It would be good to have this knowledge written down somewhere for easy
reference. Especially if it really remains our only way to get clean
builds.
I don't disagree with your general argument, I just see different
trade-offs, and I consider this type failure to have a fairly trivial
recovery (kicking off a manual clean build).
Again, I don't think that this is a trivial solution.
First I have to detect that my changes are incompatible. That is
sometimes easy but sometimes it is not. If it is not easy then there is
even the possibility that the incremental build will complete
successful. But the result might show some very subtle errors that we
may or may not find before a release.
I would rather not spend any time on the analysis.
Then we have to coordinate the triggering of incompatible builds. We
have developers working around the world at every time of the day. It
might be better to have an incompatible build to start a a fixed time
then have developer A trigger a clean build and developer B, who is one
hour late, has to do that again right after that. This might flood the
build servers with AOO builds.
-Andre
Andrew
This may become important in the coming weeks when we have to fix
some
bugs in the sidebar (which is about to be merged back into
trunk). The
sidebar is implemented in several modules. Without a clean windows
build we will run into build breakers very regularly.
It is possible to force a clean build manually.
I'm cleaning it up now and kicking off a build.
Thanks for taking care of this one.
-Andre
Andrew
Herbert
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org
For additional commands, e-mail: dev-h...@openoffice.apache.org
