Hi Andre,
where is the problem? (I know I asked this since months and do not get
any detailed answer) :-(
As I heard often in the past months that it isn't possible to compare
the test results in QUASTE of a Sun build with an OOo buildbot build.
As I know there are only a few differences and also differences exists
when running the automated tests with VCL TestTool in different test
environments. And I often told you and other QA members, that is will
not be possible to get the same result of the tests in all environments.
Therefore I said to you and also on the OOoCon that I want to introduce
TestBots. Then a CWS and Master builds are tested in the same
environment. Then one major root cause for differences in the test
results are eliminated. The other thing is to run the TestBots on builds
from a BuildBot. Then all results should be the same. That's the theorie
and some engineers are working on the solution TestBots.
If this isn't a solution for you and the OOo community, you should write
it to me and make other proposals. I want to work with you and the QA
community to eliminate the barriers. But I do not get any promised
feedback since months.
That the BuildBots can be more identical between SO and OOo builds is
another issue, I think. That this has to be addressed also is also clear
for me. But I want to separate this from the QA part.
Regards,
Thorsten
Andre Schnabel wrote:
Hi Nils,
-------- Original-Nachricht --------
Von: Nils Fuhrmann <nils.fuhrm...@sun.com>
Stephan Bergmann schrieb:
During FOSDEM, Mechtilde told me about a problem the QA community is
experiencing, namely that buildbot builds (of CWSs) are quite different
functionality-wise from the standard builds (of milestones and releases,
often done by Sun Hamburg Release Engineering). Those differences are
especially apparent in Base, Mechtilde told me. This problem in some
cases prevents easy testing of a CWS by the QA community, or even
thorough testing of a CWS in real life by replacing a standard OOo build
with a buildbot CWS build in (semi-)production use.
I know that there were some issue regarding QA'ing buildbot builds in
past. To get an idea what the real problem is, we should collect those
issues in detail when they occur to find the root cause for them
This is quite like going to the woods and look at each tree seperately to understand,
what "the wood" is.
(as we
always do). If those issues still are existent, I would await that this
list is already available somewhere (Mechtilde, do you have such list?).
There is no full list, as we would need to do several test runs on sun builds,
compare those to testruns done on equivalent buildbot builds - identify the
differencens ...
You will find differences due to:
- different configure settings (this is from my experience the biggest part, as
complete functional areas might be missing)
- differnt compilers
- different build environment
- different test environment
This would need to be done for at least all the major platforms and at least
all cat0 tests. This is a total of some weeks for running, analyzing and
comparing tests.
Really, we should investigate into the concrete list of issues before
thinking about any additional infrastructure.
Sorry, this is the totally wrong way of thinking.
the correct way was: How can we get more people helping in development (here
QA) by using existing infrastructure.
We do not need *additional* infrastructure. We just want to use existing buildbots to help with cws testing.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@openoffice.org
For additional commands, e-mail: dev-h...@openoffice.org
