On 08/02/2016 01:14 AM, Peretz, Ravit wrote: > Hi Jamo, > > I Uploaded another patch addressing your review [0] and rerun the updated > test [1] BTW, I added the public-bridge > parameter but I can't see it in the parameters list in the job like other > params, only in the scripts - is that expected?
did you re-push the job to the sandbox after made that br-ex addition? that's the only thing I can think of for why it wouldn't show up. > Regarding the testplan, I reviewed your patch[2] and I have a few questions: > 1. Why there's a difference between > netvirt-1node-openstack.txt and netvirt-vpnservice-1node-openstack.txt ? good question. the two jobs have two basic functionalities so far. tempest and connectivity (robot suites) the netvirt-1node (old netvirt impl) has that and now (I just noticed) it seems that someone is adding some JVM metrics suite at the end. This is not a functional suite, afaik, but will help visualize/debug down the road. The netvirt-vpnservice (new netvirt impl) needs to have the tunnels created manually, so there is one robot suite there to do that. There is also a new suite (basic_vpnservice) suite there that is being developed by Mahendar. After that, its' the core tempest and connectivity suites. > 2. If there should be a difference, why are you > now creating the netvirt-vpnservice-3node-openstack.txt to be exactly the > same as the netvirt-3node-openstack.txt ? 3. if > there shouldn't be a difference (and that was what I expected) why > duplicating the testplan? We don't have tempest yet in our 3node jobs. we have connectivity and clustering suites. looks like the JVM metrics stuff is there in the 3node though. For now, the netvirt-3node and netvirt-vpnservice-3node can be the same. They will probably diverge like the 1node jobs as new/more tests are given for the new impl we are planning to ship with boron. > Regarding the test run #3: I can see the run was completed but we have only > 47.3% pass rate[3] in comparison to 78% in > ovsdb feature[4]. Vpnservice "exclusive" failures are around: - Tap devices - > vm instances creation - subnet creations - > jvm plot > I think we should ignore the results for now. There was infra issues for a week or more that we *think* should be fixed just now. Venkat narrowed it down (with help from Thanh and Luis) to an MTU setting. But, now these openstack jobs should start functioning like they were. Give it another run and let's see. However, I think both your .yaml patch and my testplan .txt patch are right at this point and we can just merge. If you agree I can take care of that. Thanks, JamO > [0] https://git.opendaylight.org/gerrit/#/c/42907/2 [1] > https://jenkins.opendaylight.org/sandbox/job/netvirt-vpnservice-csit-3node-openstack-mitaka-openstack-boron/4/ > [2] > https://git.opendaylight.org/gerrit/#/c/42935/1 [3] > https://jenkins.opendaylight.org/sandbox/job/netvirt-vpnservice-csit-3node-openstack-mitaka-openstack-boron/3/robot/ > [4] > https://jenkins.opendaylight.org/releng/view/netvirt/job/netvirt-csit-3node-openstack-mitaka-openstack-boron/89/robot/ > > Ravit. > > -----Original Message----- From: Jamo Luhrsen [mailto:jluhr...@gmail.com] > Sent: יום ג 02 אוגוסט 2016 02:38 To: > Venkatrangan G - ERS, HCL Tech <venkatrang...@hcl.com>; Peretz, Ravit > <ravit.per...@hpe.com>; Tom Pantelis > <tompante...@gmail.com> Cc: odl netvirt dev > <netvirt-...@lists.opendaylight.org>; controller-dev@lists.opendaylight.org > Subject: Re: [controller-dev] ODL 3node CPU Usage > > Venkat, > > I have seen some jobs in the past fail because br-int was not created. I > think we have a bug there somewhere, but it's > not consistent so we need to do some debugging work. This is with the > netvirt-vpnservice impl, which only recently > starting auto creating that br-int. > > Ravit, > > I ran your job again [0] and it made it to the part where it needs to run > robot, which is good. We need to add a testplan > .txt file now. I did that here [1]. Then, in these jobs, when you "build > with parameters" we usually have an option for > PATCHREFSPEC and in there you put the refspec for the Int/Test gerrit and it > will use that. I did that here [2] and we'll > see what happens :) > > I also made two comments on your job .yaml gerrit. > > Thanks, JamO > > > [0] > https://jenkins.opendaylight.org/sandbox/job/netvirt-vpnservice-csit-3node-openstack-mitaka-openstack-boron/2/consoleFull > > [1] https://git.opendaylight.org/gerrit/42935 [2] > https://jenkins.opendaylight.org/sandbox/job/netvirt-vpnservice-csit-3node-openstack-mitaka-openstack-boron/3/ > > On 08/01/2016 08:24 AM, Venkatrangan G - ERS, HCL Tech wrote: >> Hi, >> >> The bridge (br-int) did not get created. Please check if this bridge gets >> created. >> >> ------ 2016-08-01 15:05:56.551 | ++ functions-common:die:186 >> : local exitcode=0 2016-08-01 >> 15:05:56.553 | [Call Trace] 2016-08-01 15:05:56.553 | >> ./stack.sh:1158:run_phase 2016-08-01 15:05:56.553 | >> /opt/stack/devstack/functions-common:1878:run_plugins 2016-08-01 >> 15:05:56.553 | >> /opt/stack/devstack/functions-common:1845:source 2016-08-01 15:05:56.553 | >> /opt/stack/networking-odl/devstack/plugin.sh:79:wait_for_active_bridge >> 2016-08-01 15:05:56.553 | >> /opt/stack/networking-odl/devstack/functions:82:test_with_retry 2016-08-01 >> 15:05:56.553 | >> /opt/stack/devstack/functions-common:2293:die 2016-08-01 15:05:56.555 | >> [ERROR] >> /opt/stack/devstack/functions-common:2293 br-int did not become available in >> 600 seconds. 2016-08-01 15:05:57.557 | >> Error on exit --------- >> >> Regards, Venkat G >> >> >> -----Original Message----- From: Peretz, Ravit [mailto:ravit.per...@hpe.com] >> Sent: Monday, August 1, 2016 8:13 AM To: >> Jamo Luhrsen <jluhr...@gmail.com>; Venkatrangan G - ERS, HCL Tech >> <venkatrang...@hcl.com>; Tom Pantelis >> <tompante...@gmail.com> Cc: odl netvirt dev >> <netvirt-...@lists.opendaylight.org>; controller-dev@lists.opendaylight.org >> Subject: RE: [controller-dev] ODL 3node CPU Usage >> >> Hey Jamo, >> >> Here is the first run for my new job: >> https://jenkins.opendaylight.org/sandbox/job/netvirt-vpnservice-csit-3 >> node-openstack-mitaka-openstack-boron/1/console >> >> seems like the devstack installation was not completed successfully. If you >> or anyone else can take a look it would be >> great. Will continue tomorrow. >> >> Ravit. >> >> -----Original Message----- From: Peretz, Ravit Sent: יום ב 01 אוגוסט 2016 >> 15:44 To: 'Jamo Luhrsen' <jluhr...@gmail.com>; >> 'Venkatrangan G - ERS, HCL Tech' <venkatrang...@hcl.com>; 'Tom Pantelis' >> <tompante...@gmail.com> Cc: 'odl netvirt dev' >> <netvirt-...@lists.opendaylight.org>; >> 'controller-dev@lists.opendaylight.org' >> <controller-dev@lists.opendaylight.org> >> Subject: RE: [controller-dev] ODL 3node CPU Usage >> >> Can someone please add me to the sandbox users group? I can't login :-( >> >> -----Original Message----- From: Peretz, Ravit Sent: יום ב 01 אוגוסט 2016 >> 15:22 To: 'Jamo Luhrsen' <jluhr...@gmail.com>; >> Venkatrangan G - ERS, HCL Tech <venkatrang...@hcl.com>; Tom Pantelis >> <tompante...@gmail.com> Cc: odl netvirt dev >> <netvirt-...@lists.opendaylight.org>; controller-dev@lists.opendaylight.org >> Subject: RE: [controller-dev] ODL 3node CPU >> Usage >> >> Hi Jamo, >> >> I added the relevant yaml (still need to verify it is working in the sandbox >> - doing it now and will update). >> https://git.opendaylight.org/gerrit/#/c/42907/ >> >> please take a look when you can, Ravit. >> >> -----Original Message----- From: Jamo Luhrsen [mailto:jluhr...@gmail.com] >> Sent: יום ה 28 יולי 2016 19:00 To: Peretz, >> Ravit <ravit.per...@hpe.com>; Venkatrangan G - ERS, HCL Tech >> <venkatrang...@hcl.com>; Tom Pantelis >> <tompante...@gmail.com> Cc: odl netvirt dev >> <netvirt-...@lists.opendaylight.org>; controller-dev@lists.opendaylight.org >> Subject: Re: [controller-dev] ODL 3node CPU Usage >> >> No, Ravit, I have not started. I am thinking it will be somewhat easy to >> get it running. You can look at the existing >> jobs for the original netvirt csit which has 1node and 3node and try to >> replicate that for netvirt-vpnservice 3node. >> >> it's probably just 1 yaml file to add in the builder repo. >> >> Thanks, JamO >> >> >> On 07/28/2016 07:39 AM, Peretz, Ravit wrote: >>> Thanks Jamo. >>> >>> Will pitch in on the 3node netvirt-vpnservice jobs. >>> >>> Can you please point me to a patch you are working on or should I start >>> from scratch? >>> >>> BTW - I was able to test simple ping between VMs on different OVS with >>> 3-node cluster (here locally). >>> >>> Thanks, Ravit. >>> >>> -----Original Message----- From: Jamo Luhrsen [mailto:jluhr...@gmail.com] >>> Sent: יום ד 27 יולי 2016 04:03 To: Peretz, >>> Ravit <ravit.per...@hpe.com>; Venkatrangan G - ERS, HCL Tech >>> <venkatrang...@hcl.com>; Tom Pantelis >>> <tompante...@gmail.com> Cc: odl netvirt dev >>> <netvirt-...@lists.opendaylight.org>; >>> controller-dev@lists.opendaylight.org Subject: Re: [controller-dev] ODL >>> 3node CPU Usage >>> >>> Ravit, >>> >>> I'm on PTO for the week. One thing I know we need done is to get the 3node >>> netvirt-vpnservice jobs running. >>> >>> you could take a look at that maybe? >>> >>> or you could take a look at all the "Legacy" netvirt bugs in bugzilla and >>> see if they are ok in netvirt-vpnservice or >>> not. >>> >>> just some ideas. >>> >>> Thanks, JamO >>> >>> On 07/24/2016 04:50 AM, Peretz, Ravit wrote: >>>> Hi guys, >>>> >>>> I'm starting to get into the netvirt-vpnservice clustering testing/bugs. >>>> >>>> Can you please point me to a known issue/testing effort I can start with? >>>> >>>> Thanks, Ravit. >>>> >>>> -----Original Message----- From: >>>> netvirt-dev-boun...@lists.opendaylight.org >>>> [mailto:netvirt-dev-boun...@lists.opendaylight.org] On Behalf Of >>>> Venkatrangan G - ERS, HCL Tech Sent: יום ה 21 יולי >>>> 2016 00:04 To: Jamo Luhrsen <jluhr...@gmail.com>; Tom Pantelis >>>> <tompante...@gmail.com> Cc: odl netvirt dev >>>> <netvirt-...@lists.opendaylight.org>; >>>> controller-dev@lists.opendaylight.org Subject: Re: [netvirt-dev] >>>> [controller-dev] ODL 3node CPU Usage >>>> >>>> I connected 3 Openstack nodes to the ODL cluster setting the ODLs as >>>> Managers and then the bridge got created in the >>>> OVS nodes. >>>> >>>> Then the script runs to observe the top and jstack, as I try some >>>> operations on the same. >>>> >>>> Regards, Venkat G >>>> >>>> -----Original Message----- From: Jamo Luhrsen [mailto:jluhr...@gmail.com] >>>> Sent: Wednesday, July 20, 2016 1:58 PM To: >>>> Tom Pantelis <tompante...@gmail.com>; Venkatrangan G - ERS, HCL Tech >>>> <venkatrang...@hcl.com> Cc: odl netvirt dev >>>> <netvirt-...@lists.opendaylight.org>; >>>> controller-dev@lists.opendaylight.org Subject: Re: [controller-dev] ODL >>>> 3node >>>> CPU Usage >>>> >>>> Venkat, >>>> >>>> I'm trying to reproduce this locally as well. I am setting up a 3 node >>>> cluster and installing ovsdb-openstack, but >>>> not seeing the CPU spikes. Using latest Be (4.3) distro from nexus. >>>> >>>> I'm missing something. Is this with a real openstack integration? I >>>> don't have that. Or maybe some ovs >>>> connections? >>>> >>>> I can connect a debugger and try to help figure out where the problem is, >>>> but I need to first reproduce it. >>>> >>>> JamO >>>> >>>> >>>> On 07/20/2016 01:42 PM, Tom Pantelis wrote: >>>>> There must be more to the stack than that. It would also help if you can >>>>> paste it so it's formatted correctly. >>>>> >>>>> On Wed, Jul 20, 2016 at 4:05 PM, Venkatrangan G - ERS, HCL Tech >>>>> <venkatrang...@hcl.com >>>>> <mailto:venkatrang...@hcl.com>> wrote: >>>>> >>>>> Hi,____ >>>>> >>>>> __ __ >>>>> >>>>> Today the script captured this information on the thread that was using >>>>> the high CPU%. Can you please suggest >>>>> which operation is this referring to., so that we can look at the >>>>> implementation for that operation.____ >>>>> >>>>> __ __ >>>>> >>>>> opendaylight-cluster-data-akka.remote.default-remote-dispatcher-8" >>>>> prio=10 tid=0x00007f59fc007800 nid=0x6f5 >>>>> runnable [0x00007f59e75f3000] java.lang.Thread.State: RUNNABLE at >>>>> com.google.common.cache.LocalCache$Segment.getEntry(LocalCache.java:2700) >>>>> at >>>>> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2180) at >>>>> com.google.common.cache.LocalCache.get(LocalCache.java:3937) at >>>>> com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) at >>>>> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) >>>>> at >>>>> com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830) >>>>> at >>>>> org.opendaylight.controller.cluster.datastore.node.utils.QNameFactory.create(QNameFactory.java:34) >>>>> at >>>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.readQName(NormalizedNodeInputStreamReader.java:257) >>>>> >>>>> at >>>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.readNormalizedNodeInternal(NormalizedNodeInputStreamReader.java:159) >>>>> >>>>> at >>>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.addDataContainerChildren(NormalizedNodeInputStreamReader.java:426) >>>>> >>>>> at >>>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.readNormalizedNodeInternal(NormalizedNodeInputStreamReader.java:155) >>>>> >>>>> at >>>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.addDataContainerChildren(NormalizedNodeInputStreamReader.java:426) >>>>> >>>>> at >>>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.readNodeIdentifierDependentNode(NormalizedNodeInputStreamReader.java:196) >>>>> >>>>> at >>>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.readNormalizedNodeInternal(NormalizedNodeInputStreamReader.java:159) >>>>> >>>>> at >>>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.addDataContainerChildren(NormalizedNodeInputStreamReader.java:422) >>>>> >>>>> at >>>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.readNodeIdentifierDependentNode(NormalizedNodeInputStreamReader.java:221) >>>>> >>>>> at >>>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.readNormalizedNodeInternal(NormalizedNodeInputStreamReader.java:159) >>>>> >>>>> at >>>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.addDataContainerChildren(NormalizedNodeInputStreamReader.java:422) >>>>> >>>>> at >>>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.readNodeIdentifierDependentNode(NormalizedNodeInputStreamReader.java:201) >>>>> >>>>> at >>>>> >>>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.Nor m a l >>>>> izedNodeInputStreamReader.readNormalizedNodeInternal(NormalizedNode I n p >>>>> utStreamReader.java:159)____ >>>>> >>>>> ------------____ >>>>> >>>>> __ __ >>>>> >>>>> __ __ >>>>> >>>>> Regards,____ >>>>> >>>>> Venkat G____ >>>>> >>>>> __ __ >>>>> >>>>> *From:*Venkatrangan G - ERS, HCL Tech *Sent:* Wednesday, July 20, 2016 >>>>> 12:46 AM *To:* 'Tom Pantelis' >>>>> <tompante...@gmail.com <mailto:tompante...@gmail.com>> *Cc:* >>>>> controller-dev@lists.opendaylight.org >>>>> <mailto:controller-dev@lists.opendaylight.org>; odl netvirt dev >>>>> <netvirt-...@lists.opendaylight.org >>>>> <mailto:netvirt-...@lists.opendaylight.org>> *Subject:* RE: >>>>> [controller-dev] ODL 3node CPU Usage____ >>>>> >>>>> __ __ >>>>> >>>>> Hi Tom,____ >>>>> >>>>> __ __ >>>>> >>>>> Thanks for response. I am modifying the script to run both commands in >>>>> parallel. Is there any clue about this >>>>> named thread so that we can investigate some particular implementation, >>>>> Is is related to remote notifiations?____ >>>>> >>>>> __ __ >>>>> >>>>> __ __ >>>>> >>>>> __ __ >>>>> >>>>> Regards,____ >>>>> >>>>> Venkat G____ >>>>> >>>>> __ __ >>>>> >>>>> *From:*Tom Pantelis [mailto:tompante...@gmail.com] *Sent:* Tuesday, July >>>>> 19, 2016 8:38 PM *To:* Venkatrangan G - >>>>> ERS, HCL Tech <venkatrang...@hcl.com <mailto:venkatrang...@hcl.com>> >>>>> *Cc:* controller-dev@lists.opendaylight.org >>>>> <mailto:controller-dev@lists.opendaylight.org>; odl netvirt dev >>>>> <netvirt-...@lists.opendaylight.org >>>>> <mailto:netvirt-...@lists.opendaylight.org>> *Subject:* Re: >>>>> [controller-dev] ODL 3node CPU Usage____ >>>>> >>>>> __ __ >>>>> >>>>> That is a parked/idle thread waiting for work. ____ >>>>> >>>>> __ __ >>>>> >>>>> On Tue, Jul 19, 2016 at 9:38 PM, Venkatrangan G - ERS, HCL Tech >>>>> <venkatrang...@hcl.com >>>>> <mailto:venkatrang...@hcl.com>> wrote:____ >>>>> >>>>> Hi,____ >>>>> >>>>> ____ >>>>> >>>>> I was testing the ODL 3node with the feature “odl-ovsdb-openstack” >>>>> installed in the nodes. Had a monitoring script >>>>> to observe the cpu usage when connected with a few OVS nodes.____ >>>>> >>>>> ____ >>>>> >>>>> a. Observed that the cpu usage was reaching beyond 100% very >>>>> frequently, could be due to frequent updates >>>>> from OVS.____ >>>>> >>>>> When I tried to identify the thread with the max CPU using jstack, It was >>>>> pointing me to this thread in all the 3 >>>>> nodes.____ >>>>> >>>>> ____ >>>>> >>>>> >>>>> "opendaylight-cluster-data-akka.remote.default-remote-dispatcher-8" >>>>> prio=10 tid=0x0000000002616000 nid=0x6eb >>>>> runnable [0x00007f2f2dee5000] java.lang.Thread.State: TIMED_WAITING >>>>> (parking) at sun.misc.Unsafe.park(Native >>>>> Method) - parking to wait for <0x0000000784630970> (a >>>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool) >>>>> at >>>>> >>>>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.j a >>>>> va:2135) at >>>>> >>>>> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067) at >>>>> >>>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java: 1 9 7 >>>>> 9) at >>>>> >>>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerTh r e a >>>>> d.java:107)____ >>>>> >>>>> ____ >>>>> >>>>> ____ >>>>> >>>>> ____ >>>>> >>>>> Kindly let us know if this needs to be reported as a bug.____ >>>>> >>>>> ____ >>>>> >>>>> ____ >>>>> >>>>> Regards,____ >>>>> >>>>> Venkat G____ >>>>> >>>>> ____ >>>>> >>>>> ::DISCLAIMER:: The contents of this e-mail and any attachment(s) are >>>>> confidential and intended for the named >>>>> recipient(s) only. E-mail transmission is not guaranteed to be secure or >>>>> error-free as information could be >>>>> intercepted, corrupted, lost, destroyed, arrive late or incomplete, or >>>>> may contain viruses in transmission. The e >>>>> mail and its contents (with or without referred errors) shall therefore >>>>> not attach any liability on the originator >>>>> or HCL or its affiliates. Views or opinions, if any, presented in this >>>>> email are solely those of the author and >>>>> may not necessarily reflect the views or opinions of HCL or its >>>>> affiliates. Any form of reproduction, >>>>> dissemination, copying, disclosure, modification, distribution and / or >>>>> publication of this message without the >>>>> prior written consent of authorized representative of HCL is strictly >>>>> prohibited. If you have received this email >>>>> in error please delete it and notify the sender immediately. Before >>>>> opening any email and/or attachments, please >>>>> check them for viruses and other defects.. ____ >>>>> >>>>> >>>>> _______________________________________________ controller-dev mailing >>>>> list controller-dev@lists.opendaylight.org >>>>> <mailto:controller-dev@lists.opendaylight.org> >>>>> >>>>> https://lists.opendaylight.org/mailman/listinfo/controller-dev____ >>>>> >>>>> __ __ >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ controller-dev mailing >>>>> list controller-dev@lists.opendaylight.org >>>>> https://lists.opendaylight.org/mailman/listinfo/controller-dev >>>>> >>>> _______________________________________________ netvirt-dev mailing list >>>> netvirt-...@lists.opendaylight.org >>>> https://lists.opendaylight.org/mailman/listinfo/netvirt-dev >>>> _______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/controller-dev
