Re: Build breaker and clean build
I have updated the buildbot wiki page [1] to reflect all I know about it. Maybe those of you, who know more, can add your knowledge. One of the more important missing parts is how to trigger a clean build on Windows. -Andre [1] http://wiki.openoffice.org/wiki/Buildbot - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Build breaker and clean build
On 15 April 2013 14:00, Andre Fischer awf@gmail.com wrote: I have updated the buildbot wiki page [1] to reflect all I know about it. Maybe those of you, who know more, can add your knowledge. One of the more important missing parts is how to trigger a clean build on Windows. -Andre [1] http://wiki.openoffice.org/**wiki/Buildbothttp://wiki.openoffice.org/wiki/Buildbot very informative, thanks. I think this is something we should reference in the notes for newcomers. rgds jan I --**--**- To unsubscribe, e-mail: dev-unsubscribe@openoffice.**apache.orgdev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Build breaker and clean build
On 12 April 2013 09:27, Andre Fischer awf@gmail.com wrote: On 11.04.2013 18:43, Andrew Rist wrote: On 4/11/2013 12:06 AM, Andre Fischer wrote: On 10.04.2013 18:57, Andrew Rist wrote: On 4/10/2013 1:33 AM, Herbert Dürr wrote: On 2013/04/10 10:09 AM, Andre Fischer wrote: tonight we had a build breaker in the windows build: a slot id that is used in SW had been removed in SVX. The reference in SW had also been removed, so this change should not be a problem. But the windows build is still not a full build. Therefore the old SW slot header files where used and the build broke. There is an easy fix for situations like this: a clean build. Incremental build are known to have problems thats why I suggested [1] to default to a clean build. That didn't receive consensus though and indeed there are good reasons against it: The incremental build both tests the dependency system and it reduces the load when building significantly. On the already strained buildbot this means a factor of almost five improvement as clean build takes about 4.5h whereas an incremental build takes only 0.5-1.0h. Andrew even had to reschedule the snapshot build away from the weekly clean build because the buildbot load is a real problem. [1] http://markmail.org/message/**wmlhc5f5zaiiyu2ohttp://markmail.org/message/wmlhc5f5zaiiyu2o [2] http://markmail.org/message/**7q64ijlwygdqmwf3http://markmail.org/message/7q64ijlwygdqmwf3 Just to add here, that there are also issues with a clean build. The clean build fails with some frequency on hung jobs and requires manual attention. That is one more reason to have more frequent clean builds so that we can find the cause of these problems. They are not restricted to the build bot. Others are affected, too. I would have tried to fix this, but I am not able to reproduce this problem on my local machine. Sorry, I think I was unclear there. Due to the complexity of our build process and the interaction with the buildbot, there is a reasonably high incidence of false positive failures on clean builds. The Windows build ends up with hung processes and throws an exception. If we were to switch to clean builds we could expect several false positives a week, which would require manual intervention. We have tests of clean builds daily on the linux boxes, so in terms of coverage of the entire AOO buildbot setup, we have full builds covered. I see the fact that some of our builds are incremental and some are clean, as a feature, not as a shortcoming. OK, I understand. I see two problems with this approach: 1. We will not find the build problems when we continue to hide them with workarounds. Fixing them would benefit others as well, not just the build bots. 2. The task of the build bots is not only to verify that our source code is buildable but also to provide download sets for developers and QA. In reality, breaking changes that require a clean build are pretty rare. For me, the clean build on the weekend and incremental during the week seems to be a good compromise. I am not sure about that. Besides, it is sometimes a bit difficult to judge 'how incompatible' a change really is. Change a slot definition in SVX and its use in SW. Do we need anything more? With a clean build we are on the safe side. It's a trade off. How many false positives to you get, and how much manual intervention does the system require. What I'm trying to communicate is that my experience with 'this buildbot setup' suggests that the current approach requires less of my time to keep healthy, and produces less false positives. Ah, again I understand. It is a matter of pragmatism. I can accept that. I and I think all others on this list are thankful for your work in this area. And maybe we can help to improve the current state. Is there any documentation, a Wiki page maybe, to get an overview? Maybe we can rededicate one of the daily builds of one of the branches to debugging our windows build problems? Add a step to the build setup to apply patches that provide us with more debug info? If I somehow can get my hands on the windows build bot vm, I can let it run on my own server, and thereby free up the l10n build. rgds Jan I. Besides, the clean-build-on-weekend policy would require us to hold all incompatible changes until Friday, or live with a broken build during the week. I really thing that we need a better solution. A switch for marking a change as incompatible and that would be interpreted by the build bot would be the absolute minimum. But even that would call for trouble. At Sun we have been there and it did not really work so well. This doesn't require waiting until the weekend, it requires a manual clean run which can be kicked off easily. (I'm happy to show you, or hit up Herbert - he has access, too) It would be good to have this knowledge written down somewhere for easy
Re: Build breaker and clean build
On 10.04.2013 18:57, Andrew Rist wrote: On 4/10/2013 1:33 AM, Herbert Dürr wrote: On 2013/04/10 10:09 AM, Andre Fischer wrote: tonight we had a build breaker in the windows build: a slot id that is used in SW had been removed in SVX. The reference in SW had also been removed, so this change should not be a problem. But the windows build is still not a full build. Therefore the old SW slot header files where used and the build broke. There is an easy fix for situations like this: a clean build. Incremental build are known to have problems thats why I suggested [1] to default to a clean build. That didn't receive consensus though and indeed there are good reasons against it: The incremental build both tests the dependency system and it reduces the load when building significantly. On the already strained buildbot this means a factor of almost five improvement as clean build takes about 4.5h whereas an incremental build takes only 0.5-1.0h. Andrew even had to reschedule the snapshot build away from the weekly clean build because the buildbot load is a real problem. [1] http://markmail.org/message/wmlhc5f5zaiiyu2o [2] http://markmail.org/message/7q64ijlwygdqmwf3 Just to add here, that there are also issues with a clean build. The clean build fails with some frequency on hung jobs and requires manual attention. That is one more reason to have more frequent clean builds so that we can find the cause of these problems. They are not restricted to the build bot. Others are affected, too. I would have tried to fix this, but I am not able to reproduce this problem on my local machine. In reality, breaking changes that require a clean build are pretty rare. For me, the clean build on the weekend and incremental during the week seems to be a good compromise. I am not sure about that. Besides, it is sometimes a bit difficult to judge 'how incompatible' a change really is. Change a slot definition in SVX and its use in SW. Do we need anything more? With a clean build we are on the safe side. Besides, the clean-build-on-weekend policy would require us to hold all incompatible changes until Friday, or live with a broken build during the week. I really thing that we need a better solution. A switch for marking a change as incompatible and that would be interpreted by the build bot would be the absolute minimum. But even that would call for trouble. At Sun we have been there and it did not really work so well. This may become important in the coming weeks when we have to fix some bugs in the sidebar (which is about to be merged back into trunk). The sidebar is implemented in several modules. Without a clean windows build we will run into build breakers very regularly. It is possible to force a clean build manually. I'm cleaning it up now and kicking off a build. Thanks for taking care of this one. -Andre Andrew Herbert - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Build breaker and clean build
On 4/11/2013 12:06 AM, Andre Fischer wrote: On 10.04.2013 18:57, Andrew Rist wrote: On 4/10/2013 1:33 AM, Herbert Dürr wrote: On 2013/04/10 10:09 AM, Andre Fischer wrote: tonight we had a build breaker in the windows build: a slot id that is used in SW had been removed in SVX. The reference in SW had also been removed, so this change should not be a problem. But the windows build is still not a full build. Therefore the old SW slot header files where used and the build broke. There is an easy fix for situations like this: a clean build. Incremental build are known to have problems thats why I suggested [1] to default to a clean build. That didn't receive consensus though and indeed there are good reasons against it: The incremental build both tests the dependency system and it reduces the load when building significantly. On the already strained buildbot this means a factor of almost five improvement as clean build takes about 4.5h whereas an incremental build takes only 0.5-1.0h. Andrew even had to reschedule the snapshot build away from the weekly clean build because the buildbot load is a real problem. [1] http://markmail.org/message/wmlhc5f5zaiiyu2o [2] http://markmail.org/message/7q64ijlwygdqmwf3 Just to add here, that there are also issues with a clean build. The clean build fails with some frequency on hung jobs and requires manual attention. That is one more reason to have more frequent clean builds so that we can find the cause of these problems. They are not restricted to the build bot. Others are affected, too. I would have tried to fix this, but I am not able to reproduce this problem on my local machine. Sorry, I think I was unclear there. Due to the complexity of our build process and the interaction with the buildbot, there is a reasonably high incidence of false positive failures on clean builds. The Windows build ends up with hung processes and throws an exception. If we were to switch to clean builds we could expect several false positives a week, which would require manual intervention. We have tests of clean builds daily on the linux boxes, so in terms of coverage of the entire AOO buildbot setup, we have full builds covered. I see the fact that some of our builds are incremental and some are clean, as a feature, not as a shortcoming. In reality, breaking changes that require a clean build are pretty rare. For me, the clean build on the weekend and incremental during the week seems to be a good compromise. I am not sure about that. Besides, it is sometimes a bit difficult to judge 'how incompatible' a change really is. Change a slot definition in SVX and its use in SW. Do we need anything more? With a clean build we are on the safe side. It's a trade off. How many false positives to you get, and how much manual intervention does the system require. What I'm trying to communicate is that my experience with 'this buildbot setup' suggests that the current approach requires less of my time to keep healthy, and produces less false positives. Besides, the clean-build-on-weekend policy would require us to hold all incompatible changes until Friday, or live with a broken build during the week. I really thing that we need a better solution. A switch for marking a change as incompatible and that would be interpreted by the build bot would be the absolute minimum. But even that would call for trouble. At Sun we have been there and it did not really work so well. This doesn't require waiting until the weekend, it requires a manual clean run which can be kicked off easily. (I'm happy to show you, or hit up Herbert - he has access, too) I don't disagree with your general argument, I just see different trade-offs, and I consider this type failure to have a fairly trivial recovery (kicking off a manual clean build). Andrew This may become important in the coming weeks when we have to fix some bugs in the sidebar (which is about to be merged back into trunk). The sidebar is implemented in several modules. Without a clean windows build we will run into build breakers very regularly. It is possible to force a clean build manually. I'm cleaning it up now and kicking off a build. Thanks for taking care of this one. -Andre Andrew Herbert - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Build breaker and clean build
On Apr 11, 2013 6:43 PM, Andrew Rist andrew.r...@oracle.com wrote: On 4/11/2013 12:06 AM, Andre Fischer wrote: On 10.04.2013 18:57, Andrew Rist wrote: On 4/10/2013 1:33 AM, Herbert Dürr wrote: On 2013/04/10 10:09 AM, Andre Fischer wrote: tonight we had a build breaker in the windows build: a slot id that is used in SW had been removed in SVX. The reference in SW had also been removed, so this change should not be a problem. But the windows build is still not a full build. Therefore the old SW slot header files where used and the build broke. There is an easy fix for situations like this: a clean build. Incremental build are known to have problems thats why I suggested [1] to default to a clean build. That didn't receive consensus though and indeed there are good reasons against it: The incremental build both tests the dependency system and it reduces the load when building significantly. On the already strained buildbot this means a factor of almost five improvement as clean build takes about 4.5h whereas an incremental build takes only 0.5-1.0h. Andrew even had to reschedule the snapshot build away from the weekly clean build because the buildbot load is a real problem. [1] http://markmail.org/message/wmlhc5f5zaiiyu2o [2] http://markmail.org/message/7q64ijlwygdqmwf3 Just to add here, that there are also issues with a clean build. The clean build fails with some frequency on hung jobs and requires manual attention. That is one more reason to have more frequent clean builds so that we can find the cause of these problems. They are not restricted to the build bot. Others are affected, too. I would have tried to fix this, but I am not able to reproduce this problem on my local machine. Sorry, I think I was unclear there. Due to the complexity of our build process and the interaction with the buildbot, there is a reasonably high incidence of false positive failures on clean builds. The Windows build ends up with hung processes and throws an exception. If we were to switch to clean builds we could expect several false positives a week, which would require manual intervention. We have tests of clean builds daily on the linux boxes, so in terms of coverage of the entire AOO buildbot setup, we have full builds covered. I see the fact that some of our builds are incremental and some are clean, as a feature, not as a shortcoming. In reality, breaking changes that require a clean build are pretty rare. For me, the clean build on the weekend and incremental during the week seems to be a good compromise. I am not sure about that. Besides, it is sometimes a bit difficult to judge 'how incompatible' a change really is. Change a slot definition in SVX and its use in SW. Do we need anything more? With a clean build we are on the safe side. It's a trade off. How many false positives to you get, and how much manual intervention does the system require. What I'm trying to communicate is that my experience with 'this buildbot setup' suggests that the current approach requires less of my time to keep healthy, and produces less false positives. Besides, the clean-build-on-weekend policy would require us to hold all incompatible changes until Friday, or live with a broken build during the week. I really thing that we need a better solution. A switch for marking a change as incompatible and that would be interpreted by the build bot would be the absolute minimum. But even that would call for trouble. At Sun we have been there and it did not really work so well. This doesn't require waiting until the weekend, it requires a manual clean run which can be kicked off easily. (I'm happy to show you, or hit up Herbert - he has access, too) I don't disagree with your general argument, I just see different trade-offs, and I consider this type failure to have a fairly trivial recovery (kicking off a manual clean build). just my little 2cent, andrew you are (in my opinion) the one who really have hands on with our build system (apologize to everyone else who has hands on, but I see andrew daily on #asftest). Can you please state your recommendation clearly, and you have my support. rgds jan I Andrew This may become important in the coming weeks when we have to fix some bugs in the sidebar (which is about to be merged back into trunk). The sidebar is implemented in several modules. Without a clean windows build we will run into build breakers very regularly. It is possible to force a clean build manually. I'm cleaning it up now and kicking off a build. Thanks for taking care of this one. -Andre Andrew Herbert - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org - To unsubscribe, e-mail:
Re: Build breaker and clean build
On 10 April 2013 10:09, Andre Fischer awf@gmail.com wrote: Hi, tonight we had a build breaker in the windows build: a slot id that is used in SW had been removed in SVX. The reference in SW had also been removed, so this change should not be a problem. But the windows build is still not a full build. Therefore the old SW slot header files where used and the build broke. There is an easy fix for situations like this: a clean build. This may become important in the coming weeks when we have to fix some bugs in the sidebar (which is about to be merged back into trunk). The sidebar is implemented in several modules. Without a clean windows build we will run into build breakers very regularly. As I see it, we have a real problem here. I am no expert in what we can do in buildbot, but isnt it possible to run a script before build, that removes all unxlngx6.pro directories and solver, that should secure a clean build ?? rgds Jan I Regards, Andre --**--**- To unsubscribe, e-mail: dev-unsubscribe@openoffice.**apache.orgdev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Build breaker and clean build
On 2013/04/10 10:09 AM, Andre Fischer wrote: tonight we had a build breaker in the windows build: a slot id that is used in SW had been removed in SVX. The reference in SW had also been removed, so this change should not be a problem. But the windows build is still not a full build. Therefore the old SW slot header files where used and the build broke. There is an easy fix for situations like this: a clean build. Incremental build are known to have problems thats why I suggested [1] to default to a clean build. That didn't receive consensus though and indeed there are good reasons against it: The incremental build both tests the dependency system and it reduces the load when building significantly. On the already strained buildbot this means a factor of almost five improvement as clean build takes about 4.5h whereas an incremental build takes only 0.5-1.0h. Andrew even had to reschedule the snapshot build away from the weekly clean build because the buildbot load is a real problem. [1] http://markmail.org/message/wmlhc5f5zaiiyu2o [2] http://markmail.org/message/7q64ijlwygdqmwf3 This may become important in the coming weeks when we have to fix some bugs in the sidebar (which is about to be merged back into trunk). The sidebar is implemented in several modules. Without a clean windows build we will run into build breakers very regularly. It is possible to force a clean build manually. Herbert - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Build breaker and clean build
On 2013/04/10 10:27 AM, janI wrote: I am no expert in what we can do in buildbot, but isnt it possible to run a script before build, that removes all unxlngx6.pro directories and solver, that should secure a clean build ?? Each nightly unxlngx6.pro buildbot run already does clean build. Only the nightly windows build is incremental whereas the weekly windows trunk and snapshot builds are done from scratch. I suggested to default to a better safe than sorry approach of a clean build but that was rejected at that time (2012/2) [1] [1] http://markmail.org/message/wmlhc5f5zaiiyu2o Herbert - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Build breaker and clean build
On 10.04.2013 10:33, Herbert Dürr wrote: On 2013/04/10 10:09 AM, Andre Fischer wrote: tonight we had a build breaker in the windows build: a slot id that is used in SW had been removed in SVX. The reference in SW had also been removed, so this change should not be a problem. But the windows build is still not a full build. Therefore the old SW slot header files where used and the build broke. There is an easy fix for situations like this: a clean build. Incremental build are known to have problems thats why I suggested [1] to default to a clean build. That didn't receive consensus though and indeed there are good reasons against it: The incremental build both tests the dependency system and it reduces the load when building significantly. On the already strained buildbot this means a factor of almost five improvement as clean build takes about 4.5h whereas an incremental build takes only 0.5-1.0h. Hm, we prefer a fast build over a correct build? I don't think that that is the right priority. Also, a build that breaks due to errors caused by not starting clean is wasted time, ours and that of the build server. Andrew even had to reschedule the snapshot build away from the weekly clean build because the buildbot load is a real problem. Would it help to do the windows build only every second day (night)? -Andre [1] http://markmail.org/message/wmlhc5f5zaiiyu2o [2] http://markmail.org/message/7q64ijlwygdqmwf3 This may become important in the coming weeks when we have to fix some bugs in the sidebar (which is about to be merged back into trunk). The sidebar is implemented in several modules. Without a clean windows build we will run into build breakers very regularly. It is possible to force a clean build manually. Herbert - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Build breaker and clean build
On 2013/04/10 11:15 AM, Andre Fischer wrote: On 10.04.2013 10:33, Herbert Dürr wrote: On 2013/04/10 10:09 AM, Andre Fischer wrote: tonight we had a build breaker in the windows build: a slot id that is used in SW had been removed in SVX. The reference in SW had also been removed, so this change should not be a problem. But the windows build is still not a full build. Therefore the old SW slot header files where used and the build broke. There is an easy fix for situations like this: a clean build. Incremental build are known to have problems thats why I suggested [1] to default to a clean build. That didn't receive consensus though and indeed there are good reasons against it: The incremental build both tests the dependency system and it reduces the load when building significantly. On the already strained buildbot this means a factor of almost five improvement as clean build takes about 4.5h whereas an incremental build takes only 0.5-1.0h. Hm, we prefer a fast build over a correct build? Some developers apparently do. Please see the mail thread at [1] [1] http://markmail.org/message/wmlhc5f5zaiiyu2o I don't think that that is the right priority. Also, a build that breaks due to errors caused by not starting clean is wasted time, ours and that of the build server. I agree. Andrew even had to reschedule the snapshot build away from the weekly clean build because the buildbot load is a real problem. Would it help to do the windows build only every second day (night)? Doing the regular build for trunk only every second night would only help if the nightly builds for branches such as l10n or ia2 would also be reduced. Do they still need nightly builds? Herbert - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Build breaker and clean build
On 4/10/2013 1:33 AM, Herbert Dürr wrote: On 2013/04/10 10:09 AM, Andre Fischer wrote: tonight we had a build breaker in the windows build: a slot id that is used in SW had been removed in SVX. The reference in SW had also been removed, so this change should not be a problem. But the windows build is still not a full build. Therefore the old SW slot header files where used and the build broke. There is an easy fix for situations like this: a clean build. Incremental build are known to have problems thats why I suggested [1] to default to a clean build. That didn't receive consensus though and indeed there are good reasons against it: The incremental build both tests the dependency system and it reduces the load when building significantly. On the already strained buildbot this means a factor of almost five improvement as clean build takes about 4.5h whereas an incremental build takes only 0.5-1.0h. Andrew even had to reschedule the snapshot build away from the weekly clean build because the buildbot load is a real problem. [1] http://markmail.org/message/wmlhc5f5zaiiyu2o [2] http://markmail.org/message/7q64ijlwygdqmwf3 Just to add here, that there are also issues with a clean build. The clean build fails with some frequency on hung jobs and requires manual attention. In reality, breaking changes that require a clean build are pretty rare. For me, the clean build on the weekend and incremental during the week seems to be a good compromise. This may become important in the coming weeks when we have to fix some bugs in the sidebar (which is about to be merged back into trunk). The sidebar is implemented in several modules. Without a clean windows build we will run into build breakers very regularly. It is possible to force a clean build manually. I'm cleaning it up now and kicking off a build. Andrew Herbert - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org
Re: Build breaker and clean build
On Apr 10, 2013 6:57 PM, Andrew Rist andrew.r...@oracle.com wrote: On 4/10/2013 1:33 AM, Herbert Dürr wrote: On 2013/04/10 10:09 AM, Andre Fischer wrote: tonight we had a build breaker in the windows build: a slot id that is used in SW had been removed in SVX. The reference in SW had also been removed, so this change should not be a problem. But the windows build is still not a full build. Therefore the old SW slot header files where used and the build broke. There is an easy fix for situations like this: a clean build. Incremental build are known to have problems thats why I suggested [1] to default to a clean build. That didn't receive consensus though and indeed there are good reasons against it: The incremental build both tests the dependency system and it reduces the load when building significantly. On the already strained buildbot this means a factor of almost five improvement as clean build takes about 4.5h whereas an incremental build takes only 0.5-1.0h. Andrew even had to reschedule the snapshot build away from the weekly clean build because the buildbot load is a real problem. [1] http://markmail.org/message/wmlhc5f5zaiiyu2o [2] http://markmail.org/message/7q64ijlwygdqmwf3 Just to add here, that there are also issues with a clean build. The clean build fails with some frequency on hung jobs and requires manual attention. In reality, breaking changes that require a clean build are pretty rare. For me, the clean build on the weekend and incremental during the week seems to be a good compromise. who whispered new build system :-) +1 please consider it also for l10n, I will shortly start making many makefile changes. rgd jan I This may become important in the coming weeks when we have to fix some bugs in the sidebar (which is about to be merged back into trunk). The sidebar is implemented in several modules. Without a clean windows build we will run into build breakers very regularly. It is possible to force a clean build manually. I'm cleaning it up now and kicking off a build. Andrew Herbert - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org - To unsubscribe, e-mail: dev-unsubscr...@openoffice.apache.org For additional commands, e-mail: dev-h...@openoffice.apache.org