Re: [Development] The qtbase CI should run the qtdeclarative tests
On 3/19/12 2:11 AM, ext Rohan McGovern rohan.mcgov...@nokia.com wrote: marius.storm-ol...@nokia.com said: While it's not normal to do this type of dependency lock for upwards dependencies, i think in this case we should indeed do the extra checking. Not just because QtDeclarative is such an important tech for Qt5, but also because it's a very good test case for QtBase. What would be the proposed workflow to resolve this situation? - I have a change in qtbase which I can't push because it breaks a qtdeclarative test. - I have a fix to the test in qtdeclarative but I can't push it because it doesn't pass without my qtbase change. One solution I can think of is to introduce a %reverse_dependencies in sync.profile, used for CI and nothing else, which can control the SHA1 of qtdeclarative used while testing qtbase. The easiest way is probably to do it in 3 stages: * Pin the SHA1 of qtbase in declarative * Submit the qtbase change * Unpin the sha1 and submit the declarative fix. Another is to skip the testing of qtdeclarative in qtbase branchname's CI if qtdeclarative's sync.profile does not contain qtbase = refs/heads/branchname. The extra time used for checking both repos will give much greater stability as a whole. Well, we don't really know that. Note that introducing qtdeclarative into the qtbase tests means that both directions of breakage are now possible (i.e. not only can qtbase changes block qtdeclarative's CI, but now qtdeclarative changes will be able to block qtbase's CI - though it should be unlikely). However it's certainly reasonable to expect that, and give it a try. Maybe the fast platforms can do the extra checking, while slow platforms still handle only QtBase? At least we will capture most compile issues? I think we'd start with adding new configurations rather than modifying existing ones. e.g. as there is currently a linux-g++-32 Ubuntu 10.04 x86 configuration for qtbase, there could be a new linux-g++-32 Ubuntu 10.04 x86 with declarative configuration. Doing it that way also means the with declarative configuration can run just the qtdeclarative autotests (we assert that there's no need to run the qtbase autotests again), theoretically not increasing the overall runtime. Unfortunately there's probably not enough resources to do that for some platforms (like Mac). If we can do it in a new configuration that only tests declarative that'll help the total runtime of the tests. Not having it for some platforms is IMO acceptable. No CI system will ever be perfect :) (The CI rules might get very complicated though.) Yes, in some ways I feel this is adding complexity to the test setup to work around a false simplicity in our source code setup. We claim that Qt is modular, but actually we know some parts of it are not really, so we add gates to enforce some level of de-modularization. The main problem here is that our test coverage of qtbase in itself is not good enough in some areas. So we have to cover this up by adding declarative tests in thus implicitly raising test coverage of qtbase. Rohan/Sergio, what will be the turn-around increase if we always test Declarative together with QtBase? The runtime should be somewhere between the current runtime of qtbase, and the qtbase runtime plus the qtdeclarative runtime. Anything more would be speculation ... but I don't think the runtime would be a big issue :) Anyway, overall I think it's worth a try, but we should accept that this could also create some new problems as well as solving some existing ones. Let's give it a try and see how it goes. Given the submit policies we now have in place this should work most of the times, and we'll simply need to deal with exceptions (where we need a change in both base and declarative) manually. I don't think this should happen too often anymore. Cheers, Lars ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] The qtbase CI should run the qtdeclarative tests
Knoll Lars (Nokia-MP/Oslo) said: On 3/19/12 2:11 AM, ext Rohan McGovern rohan.mcgov...@nokia.com wrote: marius.storm-ol...@nokia.com said: While it's not normal to do this type of dependency lock for upwards dependencies, i think in this case we should indeed do the extra checking. Not just because QtDeclarative is such an important tech for Qt5, but also because it's a very good test case for QtBase. What would be the proposed workflow to resolve this situation? - I have a change in qtbase which I can't push because it breaks a qtdeclarative test. - I have a fix to the test in qtdeclarative but I can't push it because it doesn't pass without my qtbase change. One solution I can think of is to introduce a %reverse_dependencies in sync.profile, used for CI and nothing else, which can control the SHA1 of qtdeclarative used while testing qtbase. The easiest way is probably to do it in 3 stages: * Pin the SHA1 of qtbase in declarative * Submit the qtbase change * Unpin the sha1 and submit the declarative fix. I don't understand this suggestion. If qtdeclarative's sync.profile has pinned to a particular qtbase SHA1, that shouldn't affect the with declarative qtbase CI configuration, right? So how does it resolve the deadlock? Or do you think declarative's sync.profile _should_ affect the with declarative qtbase CI config? That means pinning a qtbase SHA1 in qtdeclarative effectively disables this config, since you'll be testing qtdeclarative's requested qtbase SHA1 which does not include the incoming qtbase changes. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] The qtbase CI should run the qtdeclarative tests
On segunda-feira, 19 de março de 2012 14.28.35, Kent Hansen wrote: Den 19. mars 2012 10:32, skrev ext Thiago Macieira: On segunda-feira, 19 de março de 2012 08.01.41, lars.kn...@nokia.com wrote: Yes, in some ways I feel this is adding complexity to the test setup to work around a false simplicity in our source code setup. We claim that Qt is modular, but actually we know some parts of it are not really, so we add gates to enforce some level of de-modularization. The main problem here is that our test coverage of qtbase in itself is not good enough in some areas. So we have to cover this up by adding declarative tests in thus implicitly raising test coverage of qtbase. While the statement is true, I don't think it's the cause of the problem. qtdeclarative is well-known for depending on the internals of QtCore and QtGui. Internals are not unit-tested and will probably never be. None of the breakages I've seen in the last few weeks have been due to depending on qtbase internals. Where are the parts in qtdeclarative where depending on internals are likely to cause problems? Maybe we can clean that up some more. I expect the kernel: metatype, metaobject system and event delivery. Anyway, where *are* the breakages? Last time this happened, I asked you to tell us where the breakages are so we could be more careful with the changes. -- Thiago Macieira - thiago.macieira (AT) intel.com Software Architect - Intel Open Source Technology Center Intel Sweden AB - Registration Number: 556189-6027 Knarrarnäsgatan 15, 164 40 Kista, Stockholm, Sweden signature.asc Description: This is a digitally signed message part. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] The qtbase CI should run the qtdeclarative tests
On 03/19/2012 03:18 PM, ext Thiago Macieira wrote: On segunda-feira, 19 de março de 2012 14.28.35, Kent Hansen wrote: Den 19. mars 2012 10:32, skrev ext Thiago Macieira: On segunda-feira, 19 de março de 2012 08.01.41, lars.kn...@nokia.com wrote: Yes, in some ways I feel this is adding complexity to the test setup to work around a false simplicity in our source code setup. We claim that Qt is modular, but actually we know some parts of it are not really, so we add gates to enforce some level of de-modularization. The main problem here is that our test coverage of qtbase in itself is not good enough in some areas. So we have to cover this up by adding declarative tests in thus implicitly raising test coverage of qtbase. While the statement is true, I don't think it's the cause of the problem. qtdeclarative is well-known for depending on the internals of QtCore and QtGui. Internals are not unit-tested and will probably never be. None of the breakages I've seen in the last few weeks have been due to depending on qtbase internals. Where are the parts in qtdeclarative where depending on internals are likely to cause problems? Maybe we can clean that up some more. I expect the kernel: metatype, metaobject system and event delivery. Anyway, where *are* the breakages? Last time this happened, I asked you to tell us where the breakages are so we could be more careful with the changes. Indeed, it might be a good idea to start adding auto-tests in qtbase whenever there's an auto-test that fails in qtdeclarative etc. -- Samuel ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] The qtbase CI should run the qtdeclarative tests
On Mar 19, 2012, at 8:01 PM, ext kent.han...@nokia.com wrote: (Top-posting from webmail) Yes, we should try to add tests to qtbase when a bug is caught by a qtdeclarative test, just like we add tests for any other bug fix. http://codereview.qt-project.org/#change,19582 broke qquicktextinput. http://codereview.qt-project.org/#change,20286 fixed it, but no new test was added. qtbase already has a covering test for clipboard. The test is unfortunately skipped on Mac since the pasteboard server is not available during CI runs. We should be able to run some of it in-process like qquicktextinput does, I'll see what can be done the next time I touch the clipboard code. Morten ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] The qtbase CI should run the qtdeclarative tests
While it's not normal to do this type of dependency lock for upwards dependencies, i think in this case we should indeed do the extra checking. Not just because QtDeclarative is such an important tech for Qt5, but also because it's a very good test case for QtBase. The extra time used for checking both repos will give much greater stability as a whole. Maybe the fast platforms can do the extra checking, while slow platforms still handle only QtBase? At least we will capture most compile issues? (The CI rules might get very complicated though.) Rohan/Sergio, what will be the turn-around increase if we always test Declarative together with QtBase? Thanks! -- Sent from my Nokia N9 On 3/16/12 17:15 Hansen Kent (Nokia-MP/Oslo) wrote: Hi, Again the qtdeclarative CI has been blocked an entire day because of qtbase changes. It's great that the CI catches all these issues, but at the same time it's ridiculous how much time is spent suffering through failed qtdeclarative CI runs and fixing those failures up after the fact! If the turn-around time will increase by one hour or whatever by also testing qtdeclarative as part of the qtbase CI runs, so what? The increased stability and confidence in the results should easily make it a net win. No longer will there be a need to wonder whether a failed qtdeclarative CI run was due to one of the actual staged changed, or a recent qtbase change, and dreading the aftermath that comes from that. The current option of pinning the qtbase SHA1 used by qtdeclarative to some older, known good SHA1 _after_ a breaking qtbase change has gone through, is bad. BAD! Don't ask me to write a paper called Pinning of SHA1s considered harmful, because I will. Pinning just masks the failure. C'mon, after first locating a good qtbase SHA1 and being able to commit your own changes again, wouldn't you ideally want to go on with your own work instead of spending the rest of the day chasing down some unknown change in qtbase, contacting the author (if possible), waiting for the fix/revert, and babysitting that through the CI? But someone certainly needs to fix it, and while that's ongoing, new changes are blissfully making their way into qtbase, which means new qtdeclarative failures can appear, and it gets progressively harder to get out of the mess. (Happened twice this week.) Marking qtdeclarative tests as insignificant due to qtbase-introduced failures, just to get stuff through the CI again, is also a bit ho-humm. Who follows up that backlog of ignored tests? Yes, there are some qtbase changes that require qtdeclarative to be adapted (AKA intentional breakages), and then you need to run the qtbase CI in isolation. But that should be the exception, not the rule. It's in those cases one should first pin the qtbase SHA1 in qtdeclarative before staging the qtbase change, and afterwards adapt qtdeclarative (with a change already prepared and reviewed) and unpin the qtbase SHA1 at once. Takes away all the surprise. Hey, I know, we can just move qtdeclarative into qtbase. Bring back the -no-declarative configure option and we're golden. With best wishes for the weekend, Kent ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] The qtbase CI should run the qtdeclarative tests
marius.storm-ol...@nokia.com said: While it's not normal to do this type of dependency lock for upwards dependencies, i think in this case we should indeed do the extra checking. Not just because QtDeclarative is such an important tech for Qt5, but also because it's a very good test case for QtBase. What would be the proposed workflow to resolve this situation? - I have a change in qtbase which I can't push because it breaks a qtdeclarative test. - I have a fix to the test in qtdeclarative but I can't push it because it doesn't pass without my qtbase change. One solution I can think of is to introduce a %reverse_dependencies in sync.profile, used for CI and nothing else, which can control the SHA1 of qtdeclarative used while testing qtbase. Another is to skip the testing of qtdeclarative in qtbase branchname's CI if qtdeclarative's sync.profile does not contain qtbase = refs/heads/branchname. The extra time used for checking both repos will give much greater stability as a whole. Well, we don't really know that. Note that introducing qtdeclarative into the qtbase tests means that both directions of breakage are now possible (i.e. not only can qtbase changes block qtdeclarative's CI, but now qtdeclarative changes will be able to block qtbase's CI - though it should be unlikely). However it's certainly reasonable to expect that, and give it a try. Maybe the fast platforms can do the extra checking, while slow platforms still handle only QtBase? At least we will capture most compile issues? I think we'd start with adding new configurations rather than modifying existing ones. e.g. as there is currently a linux-g++-32 Ubuntu 10.04 x86 configuration for qtbase, there could be a new linux-g++-32 Ubuntu 10.04 x86 with declarative configuration. Doing it that way also means the with declarative configuration can run just the qtdeclarative autotests (we assert that there's no need to run the qtbase autotests again), theoretically not increasing the overall runtime. Unfortunately there's probably not enough resources to do that for some platforms (like Mac). (The CI rules might get very complicated though.) Yes, in some ways I feel this is adding complexity to the test setup to work around a false simplicity in our source code setup. We claim that Qt is modular, but actually we know some parts of it are not really, so we add gates to enforce some level of de-modularization. Rohan/Sergio, what will be the turn-around increase if we always test Declarative together with QtBase? The runtime should be somewhere between the current runtime of qtbase, and the qtbase runtime plus the qtdeclarative runtime. Anything more would be speculation ... but I don't think the runtime would be a big issue :) Anyway, overall I think it's worth a try, but we should accept that this could also create some new problems as well as solving some existing ones. ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
Re: [Development] The qtbase CI should run the qtdeclarative tests
Golf clap I relish modularity, but construing a hard dependency as a soft dependency is inviting a world of hurt for the CI underpant gnomes (/me makes a note to buy Kent flowers next time he is in Oslo) On Mar 16, 2012 3:14 PM, kent.han...@nokia.com wrote: Hi, Again the qtdeclarative CI has been blocked an entire day because of qtbase changes. It's great that the CI catches all these issues, but at the same time it's ridiculous how much time is spent suffering through failed qtdeclarative CI runs and fixing those failures up after the fact! If the turn-around time will increase by one hour or whatever by also testing qtdeclarative as part of the qtbase CI runs, so what? The increased stability and confidence in the results should easily make it a net win. No longer will there be a need to wonder whether a failed qtdeclarative CI run was due to one of the actual staged changed, or a recent qtbase change, and dreading the aftermath that comes from that. The current option of pinning the qtbase SHA1 used by qtdeclarative to some older, known good SHA1 _after_ a breaking qtbase change has gone through, is bad. BAD! Don't ask me to write a paper called Pinning of SHA1s considered harmful, because I will. Pinning just masks the failure. C'mon, after first locating a good qtbase SHA1 and being able to commit your own changes again, wouldn't you ideally want to go on with your own work instead of spending the rest of the day chasing down some unknown change in qtbase, contacting the author (if possible), waiting for the fix/revert, and babysitting that through the CI? But someone certainly needs to fix it, and while that's ongoing, new changes are blissfully making their way into qtbase, which means new qtdeclarative failures can appear, and it gets progressively harder to get out of the mess. (Happened twice this week.) Marking qtdeclarative tests as insignificant due to qtbase-introduced failures, just to get stuff through the CI again, is also a bit ho-humm. Who follows up that backlog of ignored tests? Yes, there are some qtbase changes that require qtdeclarative to be adapted (AKA intentional breakages), and then you need to run the qtbase CI in isolation. But that should be the exception, not the rule. It's in those cases one should first pin the qtbase SHA1 in qtdeclarative before staging the qtbase change, and afterwards adapt qtdeclarative (with a change already prepared and reviewed) and unpin the qtbase SHA1 at once. Takes away all the surprise. Hey, I know, we can just move qtdeclarative into qtbase. Bring back the -no-declarative configure option and we're golden. With best wishes for the weekend, Kent ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
[Development] The qtbase CI should run the qtdeclarative tests
Hi, Again the qtdeclarative CI has been blocked an entire day because of qtbase changes. It's great that the CI catches all these issues, but at the same time it's ridiculous how much time is spent suffering through failed qtdeclarative CI runs and fixing those failures up after the fact! If the turn-around time will increase by one hour or whatever by also testing qtdeclarative as part of the qtbase CI runs, so what? The increased stability and confidence in the results should easily make it a net win. No longer will there be a need to wonder whether a failed qtdeclarative CI run was due to one of the actual staged changed, or a recent qtbase change, and dreading the aftermath that comes from that. The current option of pinning the qtbase SHA1 used by qtdeclarative to some older, known good SHA1 _after_ a breaking qtbase change has gone through, is bad. BAD! Don't ask me to write a paper called Pinning of SHA1s considered harmful, because I will. Pinning just masks the failure. C'mon, after first locating a good qtbase SHA1 and being able to commit your own changes again, wouldn't you ideally want to go on with your own work instead of spending the rest of the day chasing down some unknown change in qtbase, contacting the author (if possible), waiting for the fix/revert, and babysitting that through the CI? But someone certainly needs to fix it, and while that's ongoing, new changes are blissfully making their way into qtbase, which means new qtdeclarative failures can appear, and it gets progressively harder to get out of the mess. (Happened twice this week.) Marking qtdeclarative tests as insignificant due to qtbase-introduced failures, just to get stuff through the CI again, is also a bit ho-humm. Who follows up that backlog of ignored tests? Yes, there are some qtbase changes that require qtdeclarative to be adapted (AKA intentional breakages), and then you need to run the qtbase CI in isolation. But that should be the exception, not the rule. It's in those cases one should first pin the qtbase SHA1 in qtdeclarative before staging the qtbase change, and afterwards adapt qtdeclarative (with a change already prepared and reviewed) and unpin the qtbase SHA1 at once. Takes away all the surprise. Hey, I know, we can just move qtdeclarative into qtbase. Bring back the -no-declarative configure option and we're golden. With best wishes for the weekend, Kent ___ Development mailing list Development@qt-project.org http://lists.qt-project.org/mailman/listinfo/development
