Re: TR : [onap-tsc] [IMPORTANT] OOM status update for RC0
Hi, On 24.03.2021 19:35, morgan.richo...@orange.com wrote: > *TL; DR:* > > xtesting dockers (tests) and xtesting-onap (test launchers) should have been > frozen before RC0..so next time we have to think to that.. > Integration shall guarantee a stable testing baseline to OOM to gate > confortably.. > > Full Text > --- > > As indicated by Sylvain, the tests in gate are used as a merge criteria > > so to avoid any turbulence, I should have frozen > - the xtesting dockers i.e the tests: no new test - I merged dcaemod > yesterday - it was false negative, the test was OK but was reported failed > for few hours (it was not declared in DB, I declared it this morning) > - xtesting-onap: I made an error this afternoon. I merged a MR dealing with > weekly rule for tern test on weekly which broke the testing CI part - I just > saw the approval..and saw the CI issue few minutes too late, I reverted it > quickly but we missed 2 cycles so ~ 4h. Regarding the numbers of patches to > be checked before RC0 and the gating resources, I should not have even tried. > (I oom_redeployed the 2 faulty gates). > > it means also that we have to sync on the chronology of the integration of > the tests in CI. > I assume that we should guarantee a stable test baseline to OOM between > before the RC0. Well in perfect world probably yes but we all know that tests also needs to be fixed and updated when teams are introducing new containers > > - Old tests (version N-1) used to verify that there is no regression are not > a real problem, they continue to work or need adaptations but we can detect > it. > - New tests (version N) can deal with > -- new features (needs the ONAP patch merges before being valid - we had a > misalignment with policy, the tests were updated but the patch not merged in > OOM for some weeks, the policy healthcheck was always FAIL) That's obviously a chicken and egg problem that can be solved in many ways but either of them would require some changes in the gating One of them that I can imagine is to keep the reference to xtesting docker version in then OOM repo. To updates xtesting dockers you need oom commit, as you need oom commit every change of those dockers would be gated. Additionally everyone who introduces or modifies a feature and needs to user newer version of xtesting docker may update it in the same commit as the functional change. Not sure if it's the best idea but still it's at least some idea... What do you think? > -- old features that were not covered so far by existing tests (usually not a > problem as it can be integrated in master eraly during the dev process) > > that is why I try to share early in time the CI evolution some weeks before > the RC0 => Honolulu CI evolution shared on the 17th of February: > https://protect2.fireeye.com/v1/url?k=c0ae9ad2-9f35a35d-c0af119d-0cc47a31307c-9d5162ca6ac8b403&q=1&e=ed62de06-8c17-4197-b0d6-b315c4e2df6d&u=https%3A%2F%2Fwiki.onap.org%2Fdownload%2Fattachments%2F6593670%2Fhonolulu_ci_evolution_08022021.pdf%3Fversion%3D1%26modificationDate%3D1612772982000%26api%3Dv2 > > but it is not easy to plan for new tests when the feature will be available > > Regarding the tests for Honolulu > - dcaemod (indicated by KK during the February meeting) is fully integrated > in CI > - tern is under integration > - pnf-macro under test (no new ONAP feature, covering SO macro flow + > multicloud + simulator management from onaptests) > - basic-clamp under test, we are not far but the lack of stability on master > prevented to finalize the test > - CDS regression tests: as discussed I wonder if it would not make sense to > add in in CSIT test first > - stability tests: need also a stable weekly master. For Honolulu we will > probably not integrate it in CI (problematic to launch long duration tests > from CI currently under investigation with the tern test) > > and very recently due to the SO stuck requests, Michal and myself started > working on basic_vm_macro and basic_cnf_macro (old feature/new tests). > Note this behavior has been observed in pnf-registrate which is using the > macro mode > > conclusions Personally I still believe that main fault is on us as OOM committers. We've been really to gentle with the review and allow patches to be merged trying to persuade ourselfs that we know the root cause of the issue, not taking into account that this one issue that we know may hide tons of other issues. > I would recommend for next releases > - to freeze the tests/CI Test launcher at RC0-n weeks..n to be defined I'd say that the deadline here would be same as end of coding for projects. > - to integrate new tests based on new features only on master when the branch > of the new version is available (RC0) Well I'm not convinced here. I'd allow those to be added at any time but maybe in the beginning we should have a separate category for them like "testing" or sth so that they are there but we
TR : [onap-tsc] [IMPORTANT] OOM status update for RC0
*TL; DR:* xtesting dockers (tests) and xtesting-onap (test launchers) should have been frozen before RC0..so next time we have to think to that.. Integration shall guarantee a stable testing baseline to OOM to gate confortably.. Full Text --- As indicated by Sylvain, the tests in gate are used as a merge criteria so to avoid any turbulence, I should have frozen - the xtesting dockers i.e the tests: no new test - I merged dcaemod yesterday - it was false negative, the test was OK but was reported failed for few hours (it was not declared in DB, I declared it this morning) - xtesting-onap: I made an error this afternoon. I merged a MR dealing with weekly rule for tern test on weekly which broke the testing CI part - I just saw the approval..and saw the CI issue few minutes too late, I reverted it quickly but we missed 2 cycles so ~ 4h. Regarding the numbers of patches to be checked before RC0 and the gating resources, I should not have even tried. (I oom_redeployed the 2 faulty gates). it means also that we have to sync on the chronology of the integration of the tests in CI. I assume that we should guarantee a stable test baseline to OOM between before the RC0. - Old tests (version N-1) used to verify that there is no regression are not a real problem, they continue to work or need adaptations but we can detect it. - New tests (version N) can deal with -- new features (needs the ONAP patch merges before being valid - we had a misalignment with policy, the tests were updated but the patch not merged in OOM for some weeks, the policy healthcheck was always FAIL) -- old features that were not covered so far by existing tests (usually not a problem as it can be integrated in master eraly during the dev process) that is why I try to share early in time the CI evolution some weeks before the RC0 => Honolulu CI evolution shared on the 17th of February: https://wiki.onap.org/download/attachments/6593670/honolulu_ci_evolution_08022021.pdf?version=1&modificationDate=1612772982000&api=v2 but it is not easy to plan for new tests when the feature will be available Regarding the tests for Honolulu - dcaemod (indicated by KK during the February meeting) is fully integrated in CI - tern is under integration - pnf-macro under test (no new ONAP feature, covering SO macro flow + multicloud + simulator management from onaptests) - basic-clamp under test, we are not far but the lack of stability on master prevented to finalize the test - CDS regression tests: as discussed I wonder if it would not make sense to add in in CSIT test first - stability tests: need also a stable weekly master. For Honolulu we will probably not integrate it in CI (problematic to launch long duration tests from CI currently under investigation with the tern test) and very recently due to the SO stuck requests, Michal and myself started working on basic_vm_macro and basic_cnf_macro (old feature/new tests). Note this behavior has been observed in pnf-registrate which is using the macro mode conclusions I would recommend for next releases - to freeze the tests/CI Test launcher at RC0-n weeks..n to be defined - to integrate new tests based on new features only on master when the branch of the new version is available (RC0) e.g. if we would consider such case today integrate in Master when Honolulu is available (we can always run the tests manually on the staging/daily and prepare the MR for the CI integration) - to allow integration of new tests (old features) before the frozen period then after the RC0 what is your view? De : onap-tsc@lists.onap.org [onap-tsc@lists.onap.org] de la part de Sylvain Desbureaux via lists.onap.org [sylvain.desbureaux=orange@lists.onap.org] Envoyé : mercredi 24 mars 2021 18:34 À : Krzysztof Opasiak Cc : dmcbr...@linuxfoundation.org; Lefevre, Catherine; Seshu m; TIMONEY, DAN; dengyuanh...@chinamobile.com; 徐冉; HAHN III, JIM; Fiachra Corcoran; onap-disc...@lists.onap.org; onap-tsc Objet : Re: [onap-tsc] [IMPORTANT] OOM status update for RC0 Also note that after the break of OOM master branch these last weeks and in order to not break it again, we (the OOM committers) have decided to merge a patch only if the code is OK and if the following rules on gate result is met: * changed component must have its pods running * if the patch changes one of « core » ONAP component (AAI, DMAAP, SDC, SDNC, SO and VID): we mandate 100% healthchecks and 100% e2e tests to pass * for other components, we allow one healthcheck and one e2e test to fail As the predictability of an ONAP deployment is not perfect, that a gate takes ~200min, that we have only 4 system gating right now (azure systems tends to fails fast these last days so I’ve decided to remove them for now), try to push your changes as soon as possible if you want to have these changes in Honolulu release. Best regards, Sylvain > Le 24 mars 2021 à 18:17, Krzysztof Op
[onap-tsc] ArchCom: Weekly Meeting Agenda - Mach 30, 2021
Team, The Agenda for the March 30, 2021 ArchCom weekly meeting can be found here: https://wiki.onap.org/display/DW/2021-Mar-30+ONAP+Architecture+Subcommittee+Meeting If you have any topic(s) that you would like to discuss please send it to me so I can add it to the agenda. Please note: The issue with the Architecture Subcommittee weekly meeting not showing up on the community calendar has now been fixed. Regards, Chaker -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#7652): https://lists.onap.org/g/onap-tsc/message/7652 Mute This Topic: https://lists.onap.org/mt/81583192/21656 Group Owner: onap-tsc+ow...@lists.onap.org Unsubscribe: https://lists.onap.org/g/onap-tsc/leave/2743226/21656/1412191262/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [onap-tsc] [IMPORTANT] OOM status update for RC0
Sylvain, Krzysztof: FYI - I found the root cause for the issues with this review : https://gerrit.onap.org/r/c/oom/+/118284 I'm in the process now of creating updated release builds (the bug is in a library shared by a number of repos, so I need to re-release all the dependent repos). Once that's done, I'll push a new changeset to my repo to bump the versions of the impacted images (I'll rebase locally before I push that changeset so we don't have to do a second gating build for the rebase) Dan -- Dan Timoney dtimo...@att.com Lead Member of Tech Staff ONAP Project Technical Lead : CCSDK and SDNC projects On 3/24/21, 1:35 PM, "onap-tsc@lists.onap.org on behalf of Sylvain Desbureaux via lists.onap.org" wrote: Also note that after the break of OOM master branch these last weeks and in order to not break it again, we (the OOM committers) have decided to merge a patch only if the code is OK and if the following rules on gate result is met: * changed component must have its pods running * if the patch changes one of « core » ONAP component (AAI, DMAAP, SDC, SDNC, SO and VID): we mandate 100% healthchecks and 100% e2e tests to pass * for other components, we allow one healthcheck and one e2e test to fail As the predictability of an ONAP deployment is not perfect, that a gate takes ~200min, that we have only 4 system gating right now (azure systems tends to fails fast these last days so I’ve decided to remove them for now), try to push your changes as soon as possible if you want to have these changes in Honolulu release. Best regards, Sylvain > Le 24 mars 2021 à 18:17, Krzysztof Opasiak a écrit : > > Team! > > *TL; DR:* > > OOM patch submitters, please fix your patches/respond to our comments > ASAP. Less than 24h has left till RC0 and functional changes not merged > before RC0 will be postponed to Istanbul. > > This is call to all OOM patch submitter but especially to projects: SO, > SDNC, VFC, UUI, Policy, DMAAP. > > *Full text:* > > Heads-up, there is less than 24 hours left until proposed RC0 date so > let me give you the status update for RC0. Please treat it as a reminder > for projects who would like to have their functional changes to be > included in the Honolulu release. > > We finally managed to process all 3 categories (bugfix, beforeM3 and > afterM3) that we had previously. All patches that are still in our > pipeline have been divided into 2 categories using tags: > > honoluluCandidates - for patches that could be merged before RC0 > https://urldefense.com/v3/__https://gerrit.onap.org/r/q/hashtag:*22honoluluCandidate*22*status:open__;JSUr!!BhdT!1tbw10e3gm6y8IKWHpmKxgrO9qVMapwrUcP0OIeTl0mM-KW1YoqNGD7i8fyR$ > > istanbul - for patches that we want to delay till Istanbul (effectively > until we branch out honolulu). > https://urldefense.com/v3/__https://gerrit.onap.org/r/q/hashtag:*22istanbul*22*status:open__;JSUr!!BhdT!1tbw10e3gm6y8IKWHpmKxgrO9qVMapwrUcP0OIeTl0mM-KW1YoqNGJIqm4iL$ > > So for Honolulu we currently have 18 candidates to be included. > > Out of those, below patches are waiting for the OOM team to review and gate: > https://urldefense.com/v3/__https://gerrit.onap.org/r/c/oom/*/119125__;Kw!!BhdT!1tbw10e3gm6y8IKWHpmKxgrO9qVMapwrUcP0OIeTl0mM-KW1YoqNGNFnqZHg$ > https://urldefense.com/v3/__https://gerrit.onap.org/r/c/oom/*/117808__;Kw!!BhdT!1tbw10e3gm6y8IKWHpmKxgrO9qVMapwrUcP0OIeTl0mM-KW1YoqNGNE_L_4v$ > https://urldefense.com/v3/__https://gerrit.onap.org/r/c/oom/*/118925__;Kw!!BhdT!1tbw10e3gm6y8IKWHpmKxgrO9qVMapwrUcP0OIeTl0mM-KW1YoqNGGdJG-_j$ > https://urldefense.com/v3/__https://gerrit.onap.org/r/c/oom/*/119488__;Kw!!BhdT!1tbw10e3gm6y8IKWHpmKxgrO9qVMapwrUcP0OIeTl0mM-KW1YoqNGG0ARglV$ > https://urldefense.com/v3/__https://gerrit.onap.org/r/c/oom/*/118248__;Kw!!BhdT!1tbw10e3gm6y8IKWHpmKxgrO9qVMapwrUcP0OIeTl0mM-KW1YoqNGH6kXUSs$ > https://urldefense.com/v3/__https://gerrit.onap.org/r/c/oom/*/117395__;Kw!!BhdT!1tbw10e3gm6y8IKWHpmKxgrO9qVMapwrUcP0OIeTl0mM-KW1YoqNGAgUdiaE$ > > It seems to be save to say that till tomorrow TSC meeting we should be > able to handle most of them unless some unexpected failures occur and we > will need submitters to fix their patches. > > Apart from that, below patches are waiting for authors to fix > them/respond to our review: > > https://urldefense.com/v3/__https://gerrit.onap.org/r/c/oom/*/113414__;Kw!!BhdT!1tbw10e3gm6y8IKWHpmKxgrO9qVMapwrUcP0OIeTl0mM-KW1YoqNGDAb4Dee$ > https://urldefense.com/v3/__https://gerrit.onap.org/r/c/oom/*/114380__;Kw!!BhdT!1tbw10e3gm6y8IKWHpmKxgrO9qVMapwrUcP0OIeTl0mM-KW1YoqNGJDt1T3I$ > https://urldefense.com/v3/__https://gerrit.onap.org/r/c/oom/*/118995__;Kw!!BhdT!1tbw10e3gm6y8IKWHpmKxgrO9qVMapwrUcP0OIeTl0mM-KW1YoqNGOmAq0ki$
[onap-tsc] ArchCom: Weekly Meeting Minutes - Mach 23, 2021
Team, The agenda , the meeting recording and any meeting minutes for the March 23, 2021 ArchCom weekly meeting can be found here: https://wiki.onap.org/display/DW/2021-Mar-23+ONAP+Architecture+Subcommittee+Meeting Regards, Chaker -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#7650): https://lists.onap.org/g/onap-tsc/message/7650 Mute This Topic: https://lists.onap.org/mt/81582932/21656 Group Owner: onap-tsc+ow...@lists.onap.org Unsubscribe: https://lists.onap.org/g/onap-tsc/leave/2743226/21656/1412191262/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [onap-tsc] [IMPORTANT] OOM status update for RC0
Also note that after the break of OOM master branch these last weeks and in order to not break it again, we (the OOM committers) have decided to merge a patch only if the code is OK and if the following rules on gate result is met: * changed component must have its pods running * if the patch changes one of « core » ONAP component (AAI, DMAAP, SDC, SDNC, SO and VID): we mandate 100% healthchecks and 100% e2e tests to pass * for other components, we allow one healthcheck and one e2e test to fail As the predictability of an ONAP deployment is not perfect, that a gate takes ~200min, that we have only 4 system gating right now (azure systems tends to fails fast these last days so I’ve decided to remove them for now), try to push your changes as soon as possible if you want to have these changes in Honolulu release. Best regards, Sylvain > Le 24 mars 2021 à 18:17, Krzysztof Opasiak a écrit : > > Team! > > *TL; DR:* > > OOM patch submitters, please fix your patches/respond to our comments > ASAP. Less than 24h has left till RC0 and functional changes not merged > before RC0 will be postponed to Istanbul. > > This is call to all OOM patch submitter but especially to projects: SO, > SDNC, VFC, UUI, Policy, DMAAP. > > *Full text:* > > Heads-up, there is less than 24 hours left until proposed RC0 date so > let me give you the status update for RC0. Please treat it as a reminder > for projects who would like to have their functional changes to be > included in the Honolulu release. > > We finally managed to process all 3 categories (bugfix, beforeM3 and > afterM3) that we had previously. All patches that are still in our > pipeline have been divided into 2 categories using tags: > > honoluluCandidates - for patches that could be merged before RC0 > https://gerrit.onap.org/r/q/hashtag:%22honoluluCandidate%22+status:open > > istanbul - for patches that we want to delay till Istanbul (effectively > until we branch out honolulu). > https://gerrit.onap.org/r/q/hashtag:%22istanbul%22+status:open > > So for Honolulu we currently have 18 candidates to be included. > > Out of those, below patches are waiting for the OOM team to review and gate: > https://gerrit.onap.org/r/c/oom/+/119125 > https://gerrit.onap.org/r/c/oom/+/117808 > https://gerrit.onap.org/r/c/oom/+/118925 > https://gerrit.onap.org/r/c/oom/+/119488 > https://gerrit.onap.org/r/c/oom/+/118248 > https://gerrit.onap.org/r/c/oom/+/117395 > > It seems to be save to say that till tomorrow TSC meeting we should be > able to handle most of them unless some unexpected failures occur and we > will need submitters to fix their patches. > > Apart from that, below patches are waiting for authors to fix > them/respond to our review: > > https://gerrit.onap.org/r/c/oom/+/113414 > https://gerrit.onap.org/r/c/oom/+/114380 > https://gerrit.onap.org/r/c/oom/+/118995 > https://gerrit.onap.org/r/c/oom/+/118331 > https://gerrit.onap.org/r/c/oom/+/118284 > https://gerrit.onap.org/r/c/oom/+/118510 > https://gerrit.onap.org/r/c/oom/+/119102 > https://gerrit.onap.org/r/c/oom/+/119012 > https://gerrit.onap.org/r/c/oom/+/119124 > https://gerrit.onap.org/r/c/oom/+/118602 > https://gerrit.onap.org/r/c/oom/+/119309 > > This means that everything is in your hands now. You need to fix your > review ASAP in order to give us time to gate and merge your patch before > reaching RC0 as according to new release cadence all OOM reviews that > contain functional changes and cannot be merged before RC0 are going to > be delayed until Istanbul (we'll just merge it to master as soon as you > make it work). > > Special heads up for: > SO > SDNC > VFC > UUI > Policy > DMAAP > > Major Honolulu container updates for those projects are still in the > review and waiting to be fixed. Especially for DMAAP which still > contains the hardcoded cert and if we don't get a working version of > your review soon then we'll have a certificate issue in master within > few days. > > Best regards, > -- > Krzysztof Opasiak > Samsung R&D Institute Poland > Samsung Electronics _ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is no
[onap-tsc] [IMPORTANT] OOM status update for RC0
Team! *TL; DR:* OOM patch submitters, please fix your patches/respond to our comments ASAP. Less than 24h has left till RC0 and functional changes not merged before RC0 will be postponed to Istanbul. This is call to all OOM patch submitter but especially to projects: SO, SDNC, VFC, UUI, Policy, DMAAP. *Full text:* Heads-up, there is less than 24 hours left until proposed RC0 date so let me give you the status update for RC0. Please treat it as a reminder for projects who would like to have their functional changes to be included in the Honolulu release. We finally managed to process all 3 categories (bugfix, beforeM3 and afterM3) that we had previously. All patches that are still in our pipeline have been divided into 2 categories using tags: honoluluCandidates - for patches that could be merged before RC0 https://gerrit.onap.org/r/q/hashtag:%22honoluluCandidate%22+status:open istanbul - for patches that we want to delay till Istanbul (effectively until we branch out honolulu). https://gerrit.onap.org/r/q/hashtag:%22istanbul%22+status:open So for Honolulu we currently have 18 candidates to be included. Out of those, below patches are waiting for the OOM team to review and gate: https://gerrit.onap.org/r/c/oom/+/119125 https://gerrit.onap.org/r/c/oom/+/117808 https://gerrit.onap.org/r/c/oom/+/118925 https://gerrit.onap.org/r/c/oom/+/119488 https://gerrit.onap.org/r/c/oom/+/118248 https://gerrit.onap.org/r/c/oom/+/117395 It seems to be save to say that till tomorrow TSC meeting we should be able to handle most of them unless some unexpected failures occur and we will need submitters to fix their patches. Apart from that, below patches are waiting for authors to fix them/respond to our review: https://gerrit.onap.org/r/c/oom/+/113414 https://gerrit.onap.org/r/c/oom/+/114380 https://gerrit.onap.org/r/c/oom/+/118995 https://gerrit.onap.org/r/c/oom/+/118331 https://gerrit.onap.org/r/c/oom/+/118284 https://gerrit.onap.org/r/c/oom/+/118510 https://gerrit.onap.org/r/c/oom/+/119102 https://gerrit.onap.org/r/c/oom/+/119012 https://gerrit.onap.org/r/c/oom/+/119124 https://gerrit.onap.org/r/c/oom/+/118602 https://gerrit.onap.org/r/c/oom/+/119309 This means that everything is in your hands now. You need to fix your review ASAP in order to give us time to gate and merge your patch before reaching RC0 as according to new release cadence all OOM reviews that contain functional changes and cannot be merged before RC0 are going to be delayed until Istanbul (we'll just merge it to master as soon as you make it work). Special heads up for: SO SDNC VFC UUI Policy DMAAP Major Honolulu container updates for those projects are still in the review and waiting to be fixed. Especially for DMAAP which still contains the hardcoded cert and if we don't get a working version of your review soon then we'll have a certificate issue in master within few days. Best regards, -- Krzysztof Opasiak Samsung R&D Institute Poland Samsung Electronics -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#7648): https://lists.onap.org/g/onap-tsc/message/7648 Mute This Topic: https://lists.onap.org/mt/81582486/21656 Group Owner: onap-tsc+ow...@lists.onap.org Unsubscribe: https://lists.onap.org/g/onap-tsc/leave/2743226/21656/1412191262/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
