On 12 April 2013 09:27, Andre Fischer <awf....@gmail.com> wrote: > On 11.04.2013 18:43, Andrew Rist wrote: > >> >> On 4/11/2013 12:06 AM, Andre Fischer wrote: >> >>> On 10.04.2013 18:57, Andrew Rist wrote: >>> >>>> >>>> On 4/10/2013 1:33 AM, Herbert Dürr wrote: >>>> >>>>> On 2013/04/10 10:09 AM, Andre Fischer wrote: >>>>> >>>>>> tonight we had a build breaker in the windows build: a slot id that is >>>>>> used in SW had been removed in SVX. The reference in SW had also been >>>>>> removed, so this change should not be a problem. >>>>>> But the windows build is still not a full build. Therefore the old SW >>>>>> slot header files where used and the build broke. >>>>>> >>>>>> There is an easy fix for situations like this: a clean build. >>>>>> >>>>> >>>>> Incremental build are known to have problems thats why I suggested [1] >>>>> to default to a clean build. That didn't receive consensus though and >>>>> indeed there are good reasons against it: >>>>> >>>>> The incremental build both tests the dependency system and it reduces >>>>> the load when building significantly. >>>>> >>>>> On the already strained buildbot this means a factor of almost five >>>>> improvement as clean build takes about 4.5h whereas an incremental build >>>>> takes only 0.5-1.0h. >>>>> >>>>> Andrew even had to reschedule the snapshot build away from the weekly >>>>> clean build because the buildbot load is a real problem. >>>>> >>>>> [1] >>>>> http://markmail.org/message/**wmlhc5f5zaiiyu2o<http://markmail.org/message/wmlhc5f5zaiiyu2o> >>>>> [2] >>>>> http://markmail.org/message/**7q64ijlwygdqmwf3<http://markmail.org/message/7q64ijlwygdqmwf3> >>>>> >>>>> Just to add here, that there are also issues with a clean build. The >>>> clean build fails with some frequency on hung jobs and requires manual >>>> attention. >>>> >>> >>> That is one more reason to have more frequent clean builds so that we >>> can find the cause of these problems. They are not restricted to the build >>> bot. Others are affected, too. I would have tried to fix this, but I am >>> not able to reproduce this problem on my local machine. >>> >> >> Sorry, I think I was unclear there. Due to the complexity of our build >> process and the interaction with the buildbot, there is a reasonably high >> incidence of false positive failures on clean builds. The Windows build >> ends up with hung processes and throws an exception. If we were to switch >> to clean builds we could expect several false positives a week, which would >> require manual intervention. We have tests of clean builds daily on the >> linux boxes, so in terms of coverage of the entire AOO buildbot setup, we >> have full builds covered. I see the fact that some of our builds are >> incremental and some are clean, as a feature, not as a shortcoming. >> > > OK, I understand. I see two problems with this approach: > > 1. We will not find the build problems when we continue to hide them with > workarounds. Fixing them would benefit others as well, not just the build > bots. > > 2. The task of the build bots is not only to verify that our source code > is buildable but also to provide download sets for developers and QA. > > >> >>> In reality, breaking changes that require a clean build are pretty >>>> rare. For me, the clean build on the weekend and incremental during the >>>> week seems to be a good compromise. >>>> >>> >>> I am not sure about that. Besides, it is sometimes a bit difficult to >>> judge 'how incompatible' a change really is. Change a slot definition in >>> SVX and its use in SW. Do we need anything more? With a clean build we are >>> on the safe side. >>> >> It's a trade off. How many false positives to you get, and how much >> manual intervention does the system require. What I'm trying to >> communicate is that my experience with 'this buildbot setup' suggests that >> the current approach requires less of my time to keep healthy, and produces >> less false positives. >> > > Ah, again I understand. It is a matter of pragmatism. I can accept that. > I and I think all others on this list are thankful for your work in this > area. And maybe we can help to improve the current state. Is there any > documentation, a Wiki page maybe, to get an overview? > > Maybe we can rededicate one of the daily builds of one of the branches to > debugging our windows build problems? Add a step to the build setup to > apply patches that provide us with more debug info? > If I somehow can get my hands on the windows build bot vm, I can let it run on my own server, and thereby free up the l10n build.
rgds Jan I. > > > >>> Besides, the clean-build-on-weekend policy would require us to hold all >>> incompatible changes until Friday, or live with a broken build during the >>> week. >>> I really thing that we need a better solution. A switch for marking a >>> change as incompatible and that would be interpreted by the build bot would >>> be the absolute minimum. But even that would call for trouble. At Sun we >>> have been there and it did not really work so well. >>> >> This doesn't require waiting until the weekend, it requires a manual >> clean run which can be kicked off easily. (I'm happy to show you, or hit >> up Herbert - he has access, too) >> > > It would be good to have this knowledge written down somewhere for easy > reference. Especially if it really remains our only way to get clean > builds. > > I don't disagree with your general argument, I just see different >> trade-offs, and I consider this type failure to have a fairly trivial >> recovery (kicking off a manual clean build). >> > > Again, I don't think that this is a trivial solution. > > First I have to detect that my changes are incompatible. That is > sometimes easy but sometimes it is not. If it is not easy then there is > even the possibility that the incremental build will complete successful. > But the result might show some very subtle errors that we may or may not > find before a release. > I would rather not spend any time on the analysis. > > Then we have to coordinate the triggering of incompatible builds. We have > developers working around the world at every time of the day. It might be > better to have an incompatible build to start a a fixed time then have > developer A trigger a clean build and developer B, who is one hour late, > has to do that again right after that. This might flood the build servers > with AOO builds. > > -Andre > > >> Andrew >> >>> >>> >>>> This may become important in the coming weeks when we have to fix some >>>>>> bugs in the sidebar (which is about to be merged back into trunk). >>>>>> The >>>>>> sidebar is implemented in several modules. Without a clean windows >>>>>> build we will run into build breakers very regularly. >>>>>> >>>>> >>>>> It is possible to force a clean build manually. >>>>> >>>> I'm cleaning it up now and kicking off a build. >>>> >>> >>> Thanks for taking care of this one. >>> >>> -Andre >>> >>> Andrew >>>> >>>> >>>> >>>>> Herbert >>>>> >>>>> >>>>> ------------------------------**------------------------------** >>>>> --------- >>>>> To unsubscribe, e-mail: >>>>> dev-unsubscribe@openoffice.**apache.org<dev-unsubscr...@openoffice.apache.org> >>>>> For additional commands, e-mail: dev-h...@openoffice.apache.org >>>>> >>>>> >>>> >>>> ------------------------------**------------------------------** >>>> --------- >>>> To unsubscribe, e-mail: >>>> dev-unsubscribe@openoffice.**apache.org<dev-unsubscr...@openoffice.apache.org> >>>> For additional commands, e-mail: dev-h...@openoffice.apache.org >>>> >>>> >>> >>> ------------------------------**------------------------------** >>> --------- >>> To unsubscribe, e-mail: >>> dev-unsubscribe@openoffice.**apache.org<dev-unsubscr...@openoffice.apache.org> >>> For additional commands, e-mail: dev-h...@openoffice.apache.org >>> >>> >> >> ------------------------------**------------------------------**--------- >> To unsubscribe, e-mail: >> dev-unsubscribe@openoffice.**apache.org<dev-unsubscr...@openoffice.apache.org> >> For additional commands, e-mail: dev-h...@openoffice.apache.org >> >> > > ------------------------------**------------------------------**--------- > To unsubscribe, e-mail: > dev-unsubscribe@openoffice.**apache.org<dev-unsubscr...@openoffice.apache.org> > For additional commands, e-mail: dev-h...@openoffice.apache.org > >
