Team, The issue with nexus3.onap.info was likely a routing change Sunday. For those requiring Casablanca 3.0.0-ONAP tagged images for new deployments I will have a 2nd proxy for public consumption up later today. This will free up the LF to address the RC in the new year.
Bottom line is a pull that takes 45 sec on the proxy takes 80-160 min from nexus3.onap.org only for this week - increasing pull times from 25 min to 35h - I retested an hour ago at 1730EST. Linux Foundation - I have your back - the quiet period started on the 17th for yourselves, for the rest of us on the 24th. I prototyped a proxy on AWS nexus3.onap.info - 80-100X faster image downloads. I will put up a public one on azure that is on a larger VM for the rest of the community to use (will need a cert - I will add to the jira) - if you use nexus3.onap.cloud please leave a note on the jira See details on the testing on clean VMs in and out of windriver against both nexus3.onap.org and nexus3.onap.info. I will post details as soon as I get it up about the larger azure proxy - it will take at least 35 hours to saturate the nexus3 proxy with 40G of Casablanca images via the docker_preload.sh script - I will do master later and advise when the proxy is ready. Details on prepping you host VMs to use the nexus3 proxy temporarily. https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-NexusProxy and https://jira.onap.org/browse/TSC-79 # on each host obrienbiometrics:nexus michaelobrien$ scp ubu...@nexus3.onap.info:~/certs/domain.crt . domain.crt obrienbiometrics:nexus michaelobrien$ scp domain.crt ubu...@ld3.onap.info:~/ domain.crt 100% 2114 15.7KB/s 00:00 # to avoid ubuntu@ip-172-31-33-46:~$ sudo docker login -u docker -p docker nexus3.onap.info:5000 Error response from daemon: Get https://nexus3.onap.info:5000/v1/users/: x509: certificate signed by unknown authority # cp cert ubuntu@ip-172-31-33-46:~$ sudo mkdir /etc/docker/certs.d ubuntu@ip-172-31-33-46:~$ sudo mkdir /etc/docker/certs.d/nexus3.onap.info:5000 ubuntu@ip-172-31-33-46:~$ sudo cp domain.crt /etc/docker/certs.d/nexus3.onap.info:5000/ca.crt ubuntu@ip-172-31-33-46:~$ sudo systemctl restart docker ubuntu@ip-172-31-33-46:~$ sudo docker login -u docker -p docker nexus3.onap.info:5000 Login Succeeded # testing # vm with the image existing - 2 sec ubuntu@ip-172-31-33-46:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8 2.1.8: Pulling from onap/aaf/aaf_agent Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8 # vm with layers existing except for last 5 - 5 sec ubuntu@a-cd-master:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8 2.1.8: Pulling from onap/aaf/aaf_agent 18d680d61657: Already exists .. 20 49e90af50c7d: Already exists 396607166153: Pull complete 8023c56d0234: Pull complete 440441b9ca5e: Pull complete 4376cc11267c: Pull complete acb05d09ff6e: Pull complete Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8 # clean AWS VM (clean install of docker) - no pulls yet - 45 sec for everything ubuntu@ip-172-31-14-34:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8 2.1.8: Pulling from onap/aaf/aaf_agent 18d680d61657: Pulling fs layer 0addb6fece63: Pulling fs layer 78e58219b215: Pulling fs layer eb6959a66df2: Pulling fs layer 321bd3fd2d0e: Pull complete 819d6de9e493: Pull complete 9c5cbae584e0: Pull complete 6b8abae643fc: Pull complete 595cdf95b083: Pull complete d5f35f05364b: Pull complete 33422549438c: Pull complete 507242d06459: Pull complete 797c42cd7990: Pull complete 55cb180bd7a7: Pull complete 8f791662f04f: Pull complete b21eb3946af1: Pull complete 36d0f286a7ec: Pull complete 2150f5ecf4e4: Pull complete 5b1a5109ab06: Pull complete 880ff0160341: Pull complete 14bbbdc46f89: Pull complete 03b5e4771470: Pull complete fdd6eab46e6c: Pull complete 80ff62223f5f: Pull complete e0633204c627: Pull complete 1b3a93e91ad9: Pull complete 29b72f79b735: Pull complete a2cb53787c38: Pull complete 49e90af50c7d: Pull complete 396607166153: Pull complete 8023c56d0234: Pull complete 440441b9ca5e: Pull complete 4376cc11267c: Pull complete acb05d09ff6e: Pull complete Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8 ubuntu@ip-172-31-14-34:~$ sudo docker images REPOSITORY TAG IMAGE ID CREATED SIZE nexus3.onap.info:5000/onap/aaf/aaf_agent 2.1.8 090b326a7f11 5 weeks ago 1.14 GB # going to test a same size image directly from the LF - with minimal common layers nexus3.onap.org:10001/onap/testsuite 1.3.2 c4b58baa95e8 3 weeks ago 1.13 GB # 5 min in we are still at 3% - numbers below are a min old ubuntu@ip-172-31-14-34:~$ sudo docker pull nexus3.onap.org:10001/onap/testsuite:1.3.2 1.3.2: Pulling from onap/testsuite 32802c0cfa4d: Downloading [=============> ] 8.416 MB/32.1 MB da1315cffa03: Download complete fa83472a3562: Download complete f85999a86bef: Download complete 3eca7452fe93: Downloading [=======================> ] 8.517 MB/17.79 MB 9f002f13a564: Downloading [=========================================> ] 8.528 MB/10.24 MB 02682cf43e5c: Waiting dfa9878b26c8: Waiting fdb3a2e7127c: Waiting adab76b27695: Waiting 3e1db7307fd3: Waiting d18a8cc3d4cd: Waiting 973f5d70b3c4: Waiting eb19d05de017: Waiting ac1f3947b000: Waiting 01ae31c1279e: Waiting 24ac662c0884: Waiting 268c36718aa7: Waiting 98559383536b: Waiting 856a1ffe236e: Waiting 80cbab20328c: Waiting caed9c28b75f: Waiting e3c00d7d86d4: Waiting 3109ae4e798e: Waiting 2dc71030fc1d: Waiting f1d763192c3d: Waiting adceeebb9b2b: Waiting 419a6a0c4acd: Waiting 048d8fd47ec5: Waiting 63fc3f31cfdc: Waiting 5441f94df8b4: Waiting 754645df4601: Waiting # in 5 min we get 3% 35/1130Mb - which comes out to 162 min for 1.13G for .org as opposed to 45 sec for .info - which is a 200X slowdown - some of this is due to the fact my nexus3.onap.info is on the same VPC as my test VM - testing on openlab # openlab - 2 min 40 sec which is 3.6 times slower - expected than in AWS - (25 min pulls vs 90min in openlab) - this makes nexus.onap.org 60 times slower in openlab than a proxy running from AWS (2 vCore/16G/ssd VM) ubuntu@onap-oom-obrien-rancher-e4:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8 2.1.8: Pulling from onap/aaf/aaf_agent 18d680d61657: Pull complete ... acb05d09ff6e: Pull complete Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8 #pulling smaller from nexus3.onap.info 2 min 20 - for 36Mb = 0.23Mb/sec - extrapolated to 1.13Gb for above is 5022 sec or 83 min - half the rough calculation above ubuntu@onap-oom-obrien-rancher-e4:~$ sudo docker pull nexus3.onap.org:10001/onap/aaf/sms:3.0.1 3.0.1: Pulling from onap/aaf/sms c67f3896b22c: Pull complete 134fac3103be: Pull complete c7fdcaf7f439: Pull complete 590a963725eb: Pull complete 4df2c296b645: Pull complete 15dd1c6ebaf1: Pull complete 76eeb922b789: Pull complete Digest: sha256:d5b64947edb93848acacaa9820234aa29e58217db9f878886b7bafae00fdb436 Status: Downloaded newer image for nexus3.onap.org:10001/onap/aaf/sms:3.0.1 # conclusion - nexus3.onap.org is experiencing a routing issue from their DC outbound causing a 80-100x slowdown over a proxy nexus3 - since 20181217 - as local jenkins.onap.org builds complete faster # workaround is to use a nexus3 proxy above and adding to values.yaml global: #repository: nexus3.onap.org:10001 repository: nexus3.onap.info:5000 repositoryCred: user: docker password: docker This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service> -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#4349): https://lists.onap.org/g/onap-tsc/message/4349 Mute This Topic: https://lists.onap.org/mt/28797521/21656 Mute #nexus3: https://lists.onap.org/mk?hashtag=nexus3&subid=2743226 Group Owner: onap-tsc+ow...@lists.onap.org Unsubscribe: https://lists.onap.org/g/onap-tsc/leave/2743226/1412191262/xyzzy [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
