Hello, It's been:
* 3 months since we asked for KaaS resources during DDF in Stockholm (and the approval from TSC and PTLs) * 2 months since we (re)asked LFN for the KaaS resources * 3 weeks since first technical discussion with Andrew * a week and a half since last email from LFN on this topic (which was addressing one of the many questions we have and not the most important one: when do we have the resources). We already missed El Alto (and Dublin…) priority on testing, I don’t want to miss also Frankfurt one. So, as it seems that you have issues to provide the solution you proposed 3 weeks ago, can we go back to the initial request (credentials to a KaaS provider) so we can move forward ASAP? For your information, CNCF, your sister foundation, is able to give credentials for hosted project (Kubernetes, Network Service Mesh (NSM), ...) on a lot of different platforms (NSM is deploying on GKE, AWS, Azure, Packet, …) so where there's a will there's a way. Hoping that have credentials very soon, --- Sylvain Desbureaux De : RICHOMME Morgan TGI/OLN <morgan.richo...@orange.com> Date : mardi 20 août 2019 à 10:44 À : "kp...@linuxfoundation.org" <kp...@linuxfoundation.org>, "jba...@linuxfoundation.org" <jba...@linuxfoundation.org> Cc : "mike.elli...@amdocs.com" <mike.elli...@amdocs.com>, DEBEAU Eric TGI/OLN <eric.deb...@orange.com>, DESBUREAUX Sylvain TGI/OLN <sylvain.desbure...@orange.com>, "hkirk...@linuxfoundation.org" <hkirk...@linuxfoundation.org>, "bf1...@att.com" <bf1...@att.com>, "catherine.lefe...@intl.att.com" <catherine.lefe...@intl.att.com> Objet : [Fwd: Re: [ONAP] Gating evolution] Hi Kenny as discussed yesterday, you will find hereafter the mail thread on the ONAP gating evolution sent on the 20th of June. this thread was initiated before the DDF in June and presented during PTL meeting (8th of July: https://wiki.onap.org/display/DW/PTL+2019-07-08) for official approval. A slide deck had been shared with PTLs and TSC: https://wiki.onap.org/download/attachments/64010469/OOM_CD_Gating_ptl.pdf?version=1&modificationDate=1562598568000&api=v2 A first estimation of the APEX cost was mentioned in the mail below and discussed during ad hoc meetings. The last mail on the mailing list dealing with the topic can be found here https://lists.onap.org/g/onap-discuss/message/17927?p=,,,20,0,0,0::Created,,Gating+evolution,20,2,0,32393410 It corresponds to a real and immediate community need. Regards Morgan ND: In parallel there are general discussions on mid term evolution of the CI/CD at LFN level (reporting done last week). I already mentioned during TAC meeting that I am a bit puzzled on the way LF manages the hardware resources (I asked several times for the list of LF HW community resources accross the different projects but never got an answer) and the CI/CD. If we consider OPNFV CI for instance, we can see that lots of the runs today correspond to projects that are sometimes finished since several months (old installers still running) or even years, run on old versions (almost no run on iruya...except functest), very high error rate on some jobs meaning that the jobs are clearly not maintained. There is probably a big waste of resources (and by consequence money). It shows also the difficulty and the challenges of a centralized model. Once community contributors stop their activity, they do not clean their job as the CI is out of their scope, the jobs remain because the centralized organization in charged of the CI has no idea that it is finished. The minutes of CI do not reflect the reality of the community. (same syndrome for the repos, some should have been archived since a long time to reflect the real activity of the community) See https://opnfv.biterg.io/app/kibana#/dashboard/00862e30-bbf7-11e8-8aac-ef7fd4d8cbad?_g=()&_a=(description:'Jenkins%20Jobs%20panel%20by%20Bitergia',filters:!(),fullScreenMode:!f,options:(darkTheme:!f,useMargins:!t),panels:!((gridData:(h:4,i:'3',w:5,x:7,y:0),id:JenkinsResults,panelIndex:'3',title:Results,type:visualization,version:'6.1.0-3'),(gridData:(h:2,i:'8',w:7,x:0,y:0),id:jenkins_main_numbers,panelIndex:'8',title:Summary,type:visualization,version:'6.1.0-3'),(gridData:(h:7,i:'11',w:12,x:0,y:7),id:a44d0f70-bbf7-11e8-8aac-ef7fd4d8cbad,panelIndex:'11',title:Jobs,type:visualization,version:'6.1.0-1'),(gridData:(h:2,i:'13',w:7,x:0,y:2),id:e0236790-bc12-11e8-8aac-ef7fd4d8cbad,panelIndex:'13',title:Jobs,type:visualization,version:'6.1.0-1'),(gridData:(h:3,i:'15',w:5,x:7,y:4),id:bedd3e00-bcab-11e8-8aac-ef7fd4d8cbad,panelIndex:'15',title:'Success%2FFailures%20in%20percentage',type:visualization,version:'6.1.0-1'),(gridData:(h:3,i:'16',w:7,x:0,y:4),id:f5a4cdb0-bcae-11e8-8aac-ef7fd4d8cbad,panelIndex:'16',title:'Duration%20Trend',type:visualization,version:'6.1.0-1')),query:(language:lucene,query:(query_string:(analyze_wildcard:!t,default_field:'*',query:'*'))),timeRestore:!f,title:'Jenkins%20Jobs',uiState:(P-10:(title:Buids),P-11:(spy:(mode:(fill:!f,name:!n)),title:'Export%20Builds',vis:(params:(config:(searchKeyword:''),sort:(columnIndex:!n,direction:!n)))),P-12:(spy:(mode:(fill:!f,name:!n)),vis:(legendOpen:!f)),P-13:(vis:(legendOpen:!f)),P-3:(title:Results),P-4:(title:Nodes,vis:(params:(config:(searchKeyword:''),sort:(columnIndex:!n,direction:!n)))),P-5:(title:'Builds%20over%20time',vis:(legendOpen:!f)),P-8:(spy:(mode:(fill:!f,name:!n)),title:Summary),P-9:(spy:(mode:(fill:!f,name:!n)),title:'Active%20nodes%20over%20time',vis:(legendOpen:!f))),viewMode:view) -------- Message transféré -------- De: DESBUREAUX Sylvain TGI/OLN <sylvain.desbure...@orange.com<mailto:DESBUREAUX%20Sylvain%20TGI/oln%20%3csylvain.desbure...@orange.com%3e>> À: jwagant...@linuxfoundation.org <jwagant...@linuxfoundation.org<mailto:%22jwagant...@linuxfoundation.org%22%20%3cjwagant...@linuxfoundation.org%3e>>, jba...@linuxfoundation.org <jba...@linuxfoundation.org<mailto:%22jba...@linuxfoundation.org%22%20%3cjba...@linuxfoundation.org%3e>>, hkirk...@linuxfoundation.org <hkirk...@linuxfoundation.org<mailto:%22hkirk...@linuxfoundation.org%22%20%3chkirk...@linuxfoundation.org%3e>>, kp...@linuxfoundation.org <kp...@linuxfoundation.org<mailto:%22kp...@linuxfoundation.org%22%20%3ckp...@linuxfoundation.org%3e>> Cc: DEBEAU Eric TGI/OLN <eric.deb...@orange.com<mailto:DEBEAU%20Eric%20TGI/oln%20%3ceric.deb...@orange.com%3e>>, mike.elli...@amdocs.com <mike.elli...@amdocs.com<mailto:%22mike.elli...@amdocs.com%22%20%3cmike.elli...@amdocs.com%3e>>, seshu.kuma...@huawei.com <seshu.kuma...@huawei.com<mailto:%22seshu.kuma...@huawei.com%22%20%3cseshu.kuma...@huawei.com%3e>>, gary.i...@huawei.com <gary.i...@huawei.com<mailto:%22gary.i...@huawei.com%22%20%3cgary.i...@huawei.com%3e>>, alexis.de_talho...@bell.ca <alexis.de_talho...@bell.ca<mailto:%22alexis.de_talho...@bell.ca%22%20%3calexis.de_talho...@bell.ca%3e>>, DANNO Vincent TGI/OLN <vincent.da...@orange.com<mailto:DANNO%20Vincent%20TGI/oln%20%3cvincent.da...@orange.com%3e>>, yang....@huawei.com <yang....@huawei.com<mailto:%22yang....@huawei.com%22%20%3cyang....@huawei.com%3e>>, bf1...@att.com <bf1...@att.com<mailto:%22bf1...@att.com%22%20%3cbf1...@att.com%3e>>, RICHOMME Morgan TGI/OLN <morgan.richo...@orange.com<mailto:RICHOMME%20Morgan%20TGI/oln%20%3cmorgan.richo...@orange.com%3e>>, catherine.lefe...@intl.att.com <catherine.lefe...@intl.att.com<mailto:%22catherine.lefe...@intl.att.com%22%20%3ccatherine.lefe...@intl.att.com%3e>> Objet: Re: [ONAP] Gating evolution Date: Thu, 04 Jul 2019 09:15:24 +0200 Hello, Do you have any news on this topic? El Alto release is due for September 30th and first tests are expected end of July. If we want to (try to) start automated gating for first commits, we need credentials as soon as possible (before July 9th) and it’s not _at all_ guaranteed. As summer is the holiday season, if we don’t have soon access to a lab, nothing will be done before mid/end august so expect first tests beginning/mid September. So this will be done only for the last session of test at best, and there would be an high risk to delay El Alto release (as we have seen in Casablanca and Dublin release). El Alto release is supposed to be focused on Test Automation & CI/CD pipeline as stated in the wiki but I’ve not seen anything on this since Stockholm DDF… Regards, --- Sylvain Desbureaux De : RICHOMME Morgan TGI/OLN <morgan.richo...@orange.com> Date : jeudi 20 juin 2019 à 18:43 À : "jwagant...@linuxfoundation.org" <jwagant...@linuxfoundation.org>, "jba...@linuxfoundation.org" <jba...@linuxfoundation.org>, "hkirk...@linuxfoundation.org" <hkirk...@linuxfoundation.org>, "catherine.lefe...@intl.att.com" <catherine.lefe...@intl.att.com>, "kp...@linuxfoundation.org" <kp...@linuxfoundation.org> Cc : DEBEAU Eric TGI/OLN <eric.deb...@orange.com>, "mike.elli...@amdocs.com" <mike.elli...@amdocs.com>, "seshu.kuma...@huawei.com" <seshu.kuma...@huawei.com>, "gary.i...@huawei.com" <gary.i...@huawei.com>, DESBUREAUX Sylvain TGI/OLN <sylvain.desbure...@orange.com>, "alexis.de_talho...@bell.ca" <alexis.de_talho...@bell.ca>, DANNO Vincent TGI/OLN <vincent.da...@orange.com>, "yang....@huawei.com" <yang....@huawei.com>, "bf1...@att.com" <bf1...@att.com> Objet : [ONAP] Gating evolution Hi as discussed during LFN DDF in Stockholm, we come back to you to discuss the possibility to get additional resources for community activities. As you know, the gating feature now allows to detect regression before merge on the installer repository. It has been introduced for Dublin release and is highly appreciated by the community. This light Openstack-zuul like feature requires some infrastructure in order to deploy a full fresh ONAP and test it within a CI chain. So far all the resources used to provide this service to the community are fully provided by Orange. In fact the on-demand deployments are done in Orange lab, operated by Orange team. In order to get geographical redundancy and share the load, it would fully make sense to add additional resources. Orange team is OK to participate to the setup/monitoring of the gating chain on these additional resources. Several options are possible: another community member hosting platforms, Cloud providers... We calculated some time ago that the dimensioning for the installer gating lead to 9 platforms in parallel at peak (for Dublin). Today we manage the gating with 1 or 2 platforms, as a consequence we added a queuing mechanism. Assuming that a full gating processing lasts about 1h30, if your patch is queued #5 and you have only 1 platform available, you have to wait 6x1h30 = 9h before getting the feedback on your patch. With 9 platforms, the max would have been 1h30. This dimensioning deals only with the gating of 1 ONAP sub-project: the installer project OOM. During DDF and in various presentations before, it was indicated that gating on key components would be also a good improvement. the SO component would be candidate for such gating. The gating of the SO would be relatively similar to the OOM gating. It requires some evolution on the docker generation (currently mainly managed by LF IT team) especiallyon docker generation and versioning. It does not look very complex but some work is needed. Of course if we add component gating, the dimensioning will change dramatically. As we can start we only 1 subcomponent, it is possible to scale as we grow. But in any case 2 // platform will be too light. In Orange for the gating we are deploying a kubernetes solution on top of our DC (Openstack). A full ONAP means 18 Kubernetes nodes - 3 controller nodes: each node= 4 cores / 4 Go RAM / 40 Go Disk - 3 etc nodes: each node= 4 cores / 4Go RAM / 40 Go Disk - 12 compute nodes: each node= 16 cores / 32 Go RAM / 40 Go Disk total for 1 platform = 216 cores / 408 Go RAM / 720 Go Disk we may notice that is is still very big and probably an obstacle for adoption...but it is another debate... Let's do the hypothesis we setup 2 new platforms for gating in public cloud the first evaluations of the costs are (just using quickly online calculator of the different providers based on a 100% usage per month, which is probably not the case; need to contact them directly for more accuracy) - Google GKE: ~10K$ (https://cloud.google.com/products/calculator/#tab=container) - Amazon EKS ~ 10K$ / month (https://aws.amazon.com/eks/pricing/?nc1=h_ls) - Azure: ~ 18K$ / month (https://azure.microsoft.com/en-gb/pricing/calculator/) - VMWare: ~10K€ (https://cloud.vmware.com/vmware-cloud-pks/pricing) - DigitalOcean: 8 K$/ month (https://www.digitalocean.com/pricing/) - OVH: ~ 15K$ (https://www.ovh.com/fr/public-cloud/prices/) - IBM: did not find the pricing page In a first stage, it would maybe make sense to ask community members (Microsoft, VMWare, IBM...) offering Kubernetes as a service offers if it would be possible to have credits for such community work. If TSC is OK and if we got the cloud credentials, Orange team would be pleased to contribute with integration/CI/CD team setting up CI chains for gating on these new resources. Regards Morgan _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#5430): https://lists.onap.org/g/onap-tsc/message/5430 Mute This Topic: https://lists.onap.org/mt/34185316/21656 Group Owner: onap-tsc+ow...@lists.onap.org Unsubscribe: https://lists.onap.org/g/onap-tsc/leave/2743226/1412191262/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
