Hi Jamo, I Uploaded another patch addressing your review [0] and rerun the updated test [1] BTW, I added the public-bridge parameter but I can't see it in the parameters list in the job like other params, only in the scripts - is that expected?
Regarding the testplan, I reviewed your patch[2] and I have a few questions: 1. Why there's a difference between netvirt-1node-openstack.txt and netvirt-vpnservice-1node-openstack.txt ? 2. If there should be a difference, why are you now creating the netvirt-vpnservice-3node-openstack.txt to be exactly the same as the netvirt-3node-openstack.txt ? 3. if there shouldn't be a difference (and that was what I expected) why duplicating the testplan? Regarding the test run #3: I can see the run was completed but we have only 47.3% pass rate[3] in comparison to 78% in ovsdb feature[4]. Vpnservice "exclusive" failures are around: - Tap devices - vm instances creation - subnet creations - jvm plot [0] https://git.opendaylight.org/gerrit/#/c/42907/2 [1] https://jenkins.opendaylight.org/sandbox/job/netvirt-vpnservice-csit-3node-openstack-mitaka-openstack-boron/4/ [2] https://git.opendaylight.org/gerrit/#/c/42935/1 [3] https://jenkins.opendaylight.org/sandbox/job/netvirt-vpnservice-csit-3node-openstack-mitaka-openstack-boron/3/robot/ [4] https://jenkins.opendaylight.org/releng/view/netvirt/job/netvirt-csit-3node-openstack-mitaka-openstack-boron/89/robot/ Ravit. -----Original Message----- From: Jamo Luhrsen [mailto:jluhr...@gmail.com] Sent: יום ג 02 אוגוסט 2016 02:38 To: Venkatrangan G - ERS, HCL Tech <venkatrang...@hcl.com>; Peretz, Ravit <ravit.per...@hpe.com>; Tom Pantelis <tompante...@gmail.com> Cc: odl netvirt dev <netvirt-...@lists.opendaylight.org>; controller-dev@lists.opendaylight.org Subject: Re: [controller-dev] ODL 3node CPU Usage Venkat, I have seen some jobs in the past fail because br-int was not created. I think we have a bug there somewhere, but it's not consistent so we need to do some debugging work. This is with the netvirt-vpnservice impl, which only recently starting auto creating that br-int. Ravit, I ran your job again [0] and it made it to the part where it needs to run robot, which is good. We need to add a testplan .txt file now. I did that here [1]. Then, in these jobs, when you "build with parameters" we usually have an option for PATCHREFSPEC and in there you put the refspec for the Int/Test gerrit and it will use that. I did that here [2] and we'll see what happens :) I also made two comments on your job .yaml gerrit. Thanks, JamO [0] https://jenkins.opendaylight.org/sandbox/job/netvirt-vpnservice-csit-3node-openstack-mitaka-openstack-boron/2/consoleFull [1] https://git.opendaylight.org/gerrit/42935 [2] https://jenkins.opendaylight.org/sandbox/job/netvirt-vpnservice-csit-3node-openstack-mitaka-openstack-boron/3/ On 08/01/2016 08:24 AM, Venkatrangan G - ERS, HCL Tech wrote: > Hi, > > The bridge (br-int) did not get created. > Please check if this bridge gets created. > > ------ > 2016-08-01 15:05:56.551 | ++ functions-common:die:186 : > local exitcode=0 > 2016-08-01 15:05:56.553 | [Call Trace] > 2016-08-01 15:05:56.553 | ./stack.sh:1158:run_phase > 2016-08-01 15:05:56.553 | > /opt/stack/devstack/functions-common:1878:run_plugins > 2016-08-01 15:05:56.553 | > /opt/stack/devstack/functions-common:1845:source > 2016-08-01 15:05:56.553 | > /opt/stack/networking-odl/devstack/plugin.sh:79:wait_for_active_bridge > 2016-08-01 15:05:56.553 | > /opt/stack/networking-odl/devstack/functions:82:test_with_retry > 2016-08-01 15:05:56.553 | > /opt/stack/devstack/functions-common:2293:die > 2016-08-01 15:05:56.555 | [ERROR] /opt/stack/devstack/functions-common:2293 > br-int did not become available in 600 seconds. > 2016-08-01 15:05:57.557 | Error on exit > --------- > > Regards, > Venkat G > > > -----Original Message----- > From: Peretz, Ravit [mailto:ravit.per...@hpe.com] > Sent: Monday, August 1, 2016 8:13 AM > To: Jamo Luhrsen <jluhr...@gmail.com>; Venkatrangan G - ERS, HCL Tech > <venkatrang...@hcl.com>; Tom Pantelis <tompante...@gmail.com> > Cc: odl netvirt dev <netvirt-...@lists.opendaylight.org>; > controller-dev@lists.opendaylight.org > Subject: RE: [controller-dev] ODL 3node CPU Usage > > Hey Jamo, > > Here is the first run for my new job: > https://jenkins.opendaylight.org/sandbox/job/netvirt-vpnservice-csit-3 > node-openstack-mitaka-openstack-boron/1/console > > seems like the devstack installation was not completed successfully. > If you or anyone else can take a look it would be great. Will continue > tomorrow. > > Ravit. > > -----Original Message----- > From: Peretz, Ravit > Sent: יום ב 01 אוגוסט 2016 15:44 > To: 'Jamo Luhrsen' <jluhr...@gmail.com>; 'Venkatrangan G - ERS, HCL > Tech' <venkatrang...@hcl.com>; 'Tom Pantelis' <tompante...@gmail.com> > Cc: 'odl netvirt dev' <netvirt-...@lists.opendaylight.org>; > 'controller-dev@lists.opendaylight.org' > <controller-dev@lists.opendaylight.org> > Subject: RE: [controller-dev] ODL 3node CPU Usage > > Can someone please add me to the sandbox users group? > I can't login :-( > > -----Original Message----- > From: Peretz, Ravit > Sent: יום ב 01 אוגוסט 2016 15:22 > To: 'Jamo Luhrsen' <jluhr...@gmail.com>; Venkatrangan G - ERS, HCL > Tech <venkatrang...@hcl.com>; Tom Pantelis <tompante...@gmail.com> > Cc: odl netvirt dev <netvirt-...@lists.opendaylight.org>; > controller-dev@lists.opendaylight.org > Subject: RE: [controller-dev] ODL 3node CPU Usage > > Hi Jamo, > > I added the relevant yaml (still need to verify it is working in the sandbox > - doing it now and will update). > https://git.opendaylight.org/gerrit/#/c/42907/ > > please take a look when you can, > Ravit. > > -----Original Message----- > From: Jamo Luhrsen [mailto:jluhr...@gmail.com] > Sent: יום ה 28 יולי 2016 19:00 > To: Peretz, Ravit <ravit.per...@hpe.com>; Venkatrangan G - ERS, HCL > Tech <venkatrang...@hcl.com>; Tom Pantelis <tompante...@gmail.com> > Cc: odl netvirt dev <netvirt-...@lists.opendaylight.org>; > controller-dev@lists.opendaylight.org > Subject: Re: [controller-dev] ODL 3node CPU Usage > > No, Ravit, I have not started. I am thinking it will be somewhat easy to get > it running. You can look at the existing jobs for the original netvirt csit > which has 1node and 3node and try to replicate that for netvirt-vpnservice > 3node. > > it's probably just 1 yaml file to add in the builder repo. > > Thanks, > JamO > > > On 07/28/2016 07:39 AM, Peretz, Ravit wrote: >> Thanks Jamo. >> >> Will pitch in on the 3node netvirt-vpnservice jobs. >> >> Can you please point me to a patch you are working on or should I start from >> scratch? >> >> BTW - I was able to test simple ping between VMs on different OVS with >> 3-node cluster (here locally). >> >> Thanks, >> Ravit. >> >> -----Original Message----- >> From: Jamo Luhrsen [mailto:jluhr...@gmail.com] >> Sent: יום ד 27 יולי 2016 04:03 >> To: Peretz, Ravit <ravit.per...@hpe.com>; Venkatrangan G - ERS, HCL >> Tech <venkatrang...@hcl.com>; Tom Pantelis <tompante...@gmail.com> >> Cc: odl netvirt dev <netvirt-...@lists.opendaylight.org>; >> controller-dev@lists.opendaylight.org >> Subject: Re: [controller-dev] ODL 3node CPU Usage >> >> Ravit, >> >> I'm on PTO for the week. One thing I know we need done is to get the 3node >> netvirt-vpnservice jobs running. >> >> you could take a look at that maybe? >> >> or you could take a look at all the "Legacy" netvirt bugs in bugzilla and >> see if they are ok in netvirt-vpnservice or not. >> >> just some ideas. >> >> Thanks, >> JamO >> >> On 07/24/2016 04:50 AM, Peretz, Ravit wrote: >>> Hi guys, >>> >>> I'm starting to get into the netvirt-vpnservice clustering testing/bugs. >>> >>> Can you please point me to a known issue/testing effort I can start with? >>> >>> Thanks, >>> Ravit. >>> >>> -----Original Message----- >>> From: netvirt-dev-boun...@lists.opendaylight.org >>> [mailto:netvirt-dev-boun...@lists.opendaylight.org] On Behalf Of >>> Venkatrangan G - ERS, HCL Tech >>> Sent: יום ה 21 יולי 2016 00:04 >>> To: Jamo Luhrsen <jluhr...@gmail.com>; Tom Pantelis >>> <tompante...@gmail.com> >>> Cc: odl netvirt dev <netvirt-...@lists.opendaylight.org>; >>> controller-dev@lists.opendaylight.org >>> Subject: Re: [netvirt-dev] [controller-dev] ODL 3node CPU Usage >>> >>> I connected 3 Openstack nodes to the ODL cluster setting the ODLs as >>> Managers and then the bridge got created in the OVS nodes. >>> >>> Then the script runs to observe the top and jstack, as I try some >>> operations on the same. >>> >>> Regards, >>> Venkat G >>> >>> -----Original Message----- >>> From: Jamo Luhrsen [mailto:jluhr...@gmail.com] >>> Sent: Wednesday, July 20, 2016 1:58 PM >>> To: Tom Pantelis <tompante...@gmail.com>; Venkatrangan G - ERS, HCL >>> Tech <venkatrang...@hcl.com> >>> Cc: odl netvirt dev <netvirt-...@lists.opendaylight.org>; >>> controller-dev@lists.opendaylight.org >>> Subject: Re: [controller-dev] ODL 3node CPU Usage >>> >>> Venkat, >>> >>> I'm trying to reproduce this locally as well. I am setting up a 3 node >>> cluster and installing ovsdb-openstack, but not seeing the CPU spikes. >>> Using latest Be (4.3) distro from nexus. >>> >>> I'm missing something. Is this with a real openstack integration? I don't >>> have that. Or maybe some ovs connections? >>> >>> I can connect a debugger and try to help figure out where the problem is, >>> but I need to first reproduce it. >>> >>> JamO >>> >>> >>> On 07/20/2016 01:42 PM, Tom Pantelis wrote: >>>> There must be more to the stack than that. It would also help if you can >>>> paste it so it's formatted correctly. >>>> >>>> On Wed, Jul 20, 2016 at 4:05 PM, Venkatrangan G - ERS, HCL Tech >>>> <venkatrang...@hcl.com <mailto:venkatrang...@hcl.com>> wrote: >>>> >>>> Hi,____ >>>> >>>> __ __ >>>> >>>> Today the script captured this information on the thread that was >>>> using the high CPU%. Can you please suggest which >>>> operation is this referring to., so that we can look at the >>>> implementation for that operation.____ >>>> >>>> __ __ >>>> >>>> opendaylight-cluster-data-akka.remote.default-remote-dispatcher-8" >>>> prio=10 tid=0x00007f59fc007800 nid=0x6f5 runnable >>>> [0x00007f59e75f3000] java.lang.Thread.State: RUNNABLE at >>>> >>>> com.google.common.cache.LocalCache$Segment.getEntry(LocalCache.java:2700) >>>> at >>>> com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2180) at >>>> com.google.common.cache.LocalCache.get(LocalCache.java:3937) at >>>> com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) at >>>> >>>> com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) >>>> at >>>> >>>> com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830) >>>> at >>>> >>>> org.opendaylight.controller.cluster.datastore.node.utils.QNameFactory.create(QNameFactory.java:34) >>>> at >>>> >>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.readQName(NormalizedNodeInputStreamReader.java:257) >>>> at >>>> >>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.readNormalizedNodeInternal(NormalizedNodeInputStreamReader.java:159) >>>> at >>>> >>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.addDataContainerChildren(NormalizedNodeInputStreamReader.java:426) >>>> at >>>> >>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.readNormalizedNodeInternal(NormalizedNodeInputStreamReader.java:155) >>>> at >>>> >>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.addDataContainerChildren(NormalizedNodeInputStreamReader.java:426) >>>> at >>>> >>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.readNodeIdentifierDependentNode(NormalizedNodeInputStreamReader.java:196) >>>> at >>>> >>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.readNormalizedNodeInternal(NormalizedNodeInputStreamReader.java:159) >>>> at >>>> >>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.addDataContainerChildren(NormalizedNodeInputStreamReader.java:422) >>>> at >>>> >>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.readNodeIdentifierDependentNode(NormalizedNodeInputStreamReader.java:221) >>>> at >>>> >>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.readNormalizedNodeInternal(NormalizedNodeInputStreamReader.java:159) >>>> at >>>> >>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.addDataContainerChildren(NormalizedNodeInputStreamReader.java:422) >>>> at >>>> >>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.NormalizedNodeInputStreamReader.readNodeIdentifierDependentNode(NormalizedNodeInputStreamReader.java:201) >>>> at >>>> >>>> org.opendaylight.controller.cluster.datastore.node.utils.stream.Nor >>>> m >>>> a >>>> l >>>> izedNodeInputStreamReader.readNormalizedNodeInternal(NormalizedNode >>>> I >>>> n >>>> p >>>> utStreamReader.java:159)____ >>>> >>>> ------------____ >>>> >>>> __ __ >>>> >>>> __ __ >>>> >>>> Regards,____ >>>> >>>> Venkat G____ >>>> >>>> __ __ >>>> >>>> *From:*Venkatrangan G - ERS, HCL Tech >>>> *Sent:* Wednesday, July 20, 2016 12:46 AM >>>> *To:* 'Tom Pantelis' <tompante...@gmail.com >>>> <mailto:tompante...@gmail.com>> >>>> *Cc:* controller-dev@lists.opendaylight.org >>>> <mailto:controller-dev@lists.opendaylight.org>; odl netvirt dev >>>> <netvirt-...@lists.opendaylight.org >>>> <mailto:netvirt-...@lists.opendaylight.org>> >>>> *Subject:* RE: [controller-dev] ODL 3node CPU Usage____ >>>> >>>> __ __ >>>> >>>> Hi Tom,____ >>>> >>>> __ __ >>>> >>>> Thanks for response. I am modifying the script to run both commands in >>>> parallel. Is there any clue about this named >>>> thread so that we can investigate some particular >>>> implementation, Is is related to remote notifiations?____ >>>> >>>> __ __ >>>> >>>> __ __ >>>> >>>> __ __ >>>> >>>> Regards,____ >>>> >>>> Venkat G____ >>>> >>>> __ __ >>>> >>>> *From:*Tom Pantelis [mailto:tompante...@gmail.com] >>>> *Sent:* Tuesday, July 19, 2016 8:38 PM >>>> *To:* Venkatrangan G - ERS, HCL Tech <venkatrang...@hcl.com >>>> <mailto:venkatrang...@hcl.com>> >>>> *Cc:* controller-dev@lists.opendaylight.org >>>> <mailto:controller-dev@lists.opendaylight.org>; odl netvirt dev >>>> <netvirt-...@lists.opendaylight.org >>>> <mailto:netvirt-...@lists.opendaylight.org>> >>>> *Subject:* Re: [controller-dev] ODL 3node CPU Usage____ >>>> >>>> __ __ >>>> >>>> That is a parked/idle thread waiting for work. ____ >>>> >>>> __ __ >>>> >>>> On Tue, Jul 19, 2016 at 9:38 PM, Venkatrangan G - ERS, HCL Tech >>>> <venkatrang...@hcl.com <mailto:venkatrang...@hcl.com>> >>>> wrote:____ >>>> >>>> Hi,____ >>>> >>>> ____ >>>> >>>> I was testing the ODL 3node with the feature “odl-ovsdb-openstack” >>>> installed in the nodes. Had a monitoring script to >>>> observe the cpu usage when connected with a few OVS >>>> nodes.____ >>>> >>>> ____ >>>> >>>> a. Observed that the cpu usage was reaching beyond 100% >>>> very frequently, could be due to frequent updates >>>> from OVS.____ >>>> >>>> When I tried to identify the thread with the max CPU using jstack, >>>> It was pointing me to this thread in all the 3 >>>> nodes.____ >>>> >>>> ____ >>>> >>>> >>>> >>>> "opendaylight-cluster-data-akka.remote.default-remote-dispatcher-8" >>>> prio=10 tid=0x0000000002616000 nid=0x6eb runnable >>>> [0x00007f2f2dee5000] java.lang.Thread.State: TIMED_WAITING >>>> (parking) at sun.misc.Unsafe.park(Native Method) - parking >>>> to wait for <0x0000000784630970> (a >>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool) at >>>> >>>> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.j >>>> a >>>> va:2135) at >>>> >>>> scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067) >>>> at >>>> >>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java: >>>> 1 >>>> 9 >>>> 7 >>>> 9) at >>>> >>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerTh >>>> r >>>> e >>>> a >>>> d.java:107)____ >>>> >>>> ____ >>>> >>>> ____ >>>> >>>> ____ >>>> >>>> Kindly let us know if this needs to be reported as a >>>> bug.____ >>>> >>>> ____ >>>> >>>> ____ >>>> >>>> Regards,____ >>>> >>>> Venkat G____ >>>> >>>> ____ >>>> >>>> ::DISCLAIMER:: The contents of this e-mail and any attachment(s) >>>> are confidential and intended for the named >>>> recipient(s) only. E-mail transmission is not guaranteed to be >>>> secure or error-free as information could be >>>> intercepted, corrupted, lost, destroyed, arrive late or >>>> incomplete, or may contain viruses in transmission. The e >>>> mail and its contents (with or without referred errors) shall >>>> therefore not attach any liability on the originator or >>>> HCL or its affiliates. Views or opinions, if any, presented in >>>> this email are solely those of the author and may not >>>> necessarily reflect the views or opinions of HCL or its >>>> affiliates. Any form of reproduction, dissemination, copying, >>>> disclosure, modification, distribution and / or publication of >>>> this message without the prior written consent of >>>> authorized representative of HCL is strictly prohibited. If you >>>> have received this email in error please delete it >>>> and notify the sender immediately. Before opening any email and/or >>>> attachments, please check them for viruses and >>>> other defects.. ____ >>>> >>>> >>>> _______________________________________________ >>>> controller-dev mailing list >>>> controller-dev@lists.opendaylight.org >>>> <mailto:controller-dev@lists.opendaylight.org> >>>> >>>> https://lists.opendaylight.org/mailman/listinfo/controller-dev____ >>>> >>>> __ __ >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> controller-dev mailing list >>>> controller-dev@lists.opendaylight.org >>>> https://lists.opendaylight.org/mailman/listinfo/controller-dev >>>> >>> _______________________________________________ >>> netvirt-dev mailing list >>> netvirt-...@lists.opendaylight.org >>> https://lists.opendaylight.org/mailman/listinfo/netvirt-dev >>> _______________________________________________ controller-dev mailing list controller-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/controller-dev
