LF, Hi, we have been seeing burst behavior for the last week. This morning in the integration lab session we see this on multiple machines. Downloads are fast when they are running - but completely stall every min or so for 30-60 sec.
For example pulling the dmaap container manually - a single pull took 3 min on one system - 18 on antoher and had sections where the job ran about 1 to 5 with bursts to 25 Mb/sec (max is usually 10Mb/sec per stream - up to 5 in parallel - I have seen 25) - but there were times 30-60 sec where the image download completely paused at 0Mb/sec. This behavior is different or is in combination with the traditional slowdown we used to see - very slow pulls - this issue is new in that it works fine during the burst - but has sections where all pulls pause. To be frank - I did see this pause behavior briefly for a couple seconds back even in Dec. I propose we meet again to test downloads again and get some volumetrics. Some details on this and previous issues in https://jira.onap.org/browse/OOM-933 Time 1130 EST 20180430 see slowdown again after our last meeting - a burst behaviour (burst/hold in 30-60 sec intervals) This in an azure system (clean) running a manual docker download - see attachment on OOM-716<https://jira.onap.org/browse/OOM-716> root@obrienondemand20180430:/var/lib/waagent/custom-script/download/0# history 1 cd /var/lib/waagent/custom-script/download/0/ 2 ls 3 tail -f stdout 4 wget https://jira.onap.org/secure/attachment/11444/prepull_docker_master.sh 6 sudo chmod 777 prepull_docker_master.sh 7 vi prepull_docker_master.sh 8 docker login -u docker -p docker nexus3.onap.org:10001 9 docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4 10 vi prepull_docker_master.sh 11 docker pull nexus3.onap.org:10001/onap/vid:1.2.1 12 docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4 root@obrienondemand20180430:/var/lib/waagent/custom-script/download/0# docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4 1.1.4: Pulling from onap/dmaap/dmaap-mr 0c5a750448f5: Pull complete e2356a247318: Pull complete 12f105c650c3: Pull complete 46d4cec7ab72: Pull complete f5b0369aa7fc: Pull complete a93519a09c8b: Pull complete 73ffcbe01622: Waiting 071f37eca411: Waiting dc33933d1e9b: Waiting cc3a1fce1c35: Waiting ab292f5d0d16: Waiting 9cd00aac030f: Waiting 3 min root@obrienondemand20180430:/var/lib/waagent/custom-script/download/0# docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4 1.1.4: Pulling from onap/dmaap/dmaap-mr 0c5a750448f5: Pull complete e2356a247318: Pull complete 12f105c650c3: Pull complete 46d4cec7ab72: Pull complete f5b0369aa7fc: Pull complete a93519a09c8b: Pull complete 73ffcbe01622: Pull complete 071f37eca411: Pull complete dc33933d1e9b: Pull complete cc3a1fce1c35: Pull complete ab292f5d0d16: Pull complete 9cd00aac030f: Pull complete Digest: sha256:8dff567ab978965f545fc1cadc8060cd34a2599ba586e786795469b32fc670ff Status: Downloaded newer image for nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4 testing full predeploy - started 1148 1149 ubuntu@obrienondemand20180430:~$ sudo ps -ef | grep docker | grep pull | wc -l 77 10 sec below 32eda82f66ec: Downloading [================> ] 96.86 MB/298.4 MB 32eda82f66ec: Downloading [====================> ] 124 MB/298.4 MB 1153 ubuntu@obrienondemand20180430:~$ sudo ps -ef | grep docker | grep pull | wc -l 75 This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement, you may review at https://www.amdocs.com/about/email-disclaimer <https://www.amdocs.com/about/email-disclaimer>
_______________________________________________ ONAP-TSC mailing list ONAP-TSC@lists.onap.org https://lists.onap.org/mailman/listinfo/onap-tsc
