Hi Jamo, As per the log -> Connection closed by device, Device:/127.0.0.1:49130, NodeId:null, I am seeing there are some tcp connections being established and ended from the localhost ip to the localhost of controller node. So if there is a tcpdump available for loopback ip, we can try to find which application/process trying to establish the tcp connection to 6653 port. I have checked below tcp dumps but I am not seeing any loopback traces.
Regards, Somashekhar -----Original Message----- From: Jamo Luhrsen [mailto:jluhr...@gmail.com] Sent: Tuesday, November 13, 2018 11:45 PM To: SOMASHEKHAR MANOHARA JAVALAGI <somashekhar.manohara.javal...@ericsson.com> Cc: D Arunprakash <d.arunprak...@ericsson.com>; openflowplugin-dev <openflowplugin-dev@lists.opendaylight.org>; odl netvirt dev <netvirt-...@lists.opendaylight.org> Subject: Re: [netvirt-dev] problems with openflowplugin/ovs connections Somashekhar, are the pcaps we already take not good enough? I'm not sure if you need something extra, or if you missed the links to them from below. here they are again: https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-oxygen/86/control_1/ https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-oxygen/86/compute_1/ https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-csit-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntrack-oxygen/86/compute_2/ Thanks, JamO On 11/12/18 8:54 PM, SOMASHEKHAR MANOHARA JAVALAGI wrote: > Hi Jamo, > > I am able to see from the logs that some tcp connection is getting > established on port 6653 and as no openflow handshaking is not happening, it > is disconnecting. The same thing is repeating for every 2 seconds. Can you > please capture the wireshark traces on controller node for loopback messages. > > Regards, > Somashekhar > > -----Original Message----- > From: D Arunprakash > Sent: Tuesday, November 06, 2018 8:47 AM > To: Jamo Luhrsen <jluhr...@gmail.com>; openflowplugin-dev > <openflowplugin-dev@lists.opendaylight.org>; odl netvirt dev > <netvirt-...@lists.opendaylight.org> > Subject: RE: [netvirt-dev] problems with openflowplugin/ovs > connections > > We will look into this and log a jira and add the relevant links. > > Regards, > Arun > > -----Original Message----- > From: Jamo Luhrsen [mailto:jluhr...@gmail.com] > Sent: Tuesday, November 06, 2018 5:56 AM > To: D Arunprakash <d.arunprak...@ericsson.com>; openflowplugin-dev > <openflowplugin-dev@lists.opendaylight.org>; odl netvirt dev > <netvirt-...@lists.opendaylight.org> > Subject: Re: [netvirt-dev] problems with openflowplugin/ovs > connections > > Hi Arun, > > any update on this one? do we need a jira to track it? These logs are coming > a lot and makes it very cumbersome to look at. Not to mention, they may be > indicating some problem internally. > > JamO > > On 10/29/18 4:29 PM, Jamo Luhrsen wrote: >> Hi Arun, >> >> sorry for the late response. We don't have packet captures on the odl >> nodes, but we do have them on the ovs nodes (2 computes and 1 control). >> >> they are done on a per suite basis, but I would just look at the >> captures during the elan suite since I know for sure that we are >> seeing all these messages when that suite is running. >> >> just look here: >> >> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-cs >> i >> t-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntr >> a >> ck-oxygen/86/control_1/ >> >> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-cs >> i >> t-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntr >> a >> ck-oxygen/86/compute_1/ >> >> https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netvirt-cs >> i >> t-3node-0cmb-1ctl-2cmp-openstack-queens-upstream-stateful-snat-conntr >> a >> ck-oxygen/86/compute_2/ >> >> >> in those folders you'll find a file like: >> >> tcpdump_port_6653__09_elan__10.30.170.134.pcap.xz >> >> >> Thanks, >> JamO >> >> >> >> On 10/17/18 10:22 PM, D Arunprakash wrote: >>> Hi Jamo, >>> Do we have packet capture at odl node level, there seems to be connection >>> established on 6653 port in a repeated manner. >>> >>> Regards, >>> Arun >>> >>> -----Original Message----- >>> From: netvirt-dev-boun...@lists.opendaylight.org >>> [mailto:netvirt-dev-boun...@lists.opendaylight.org] On Behalf Of >>> Jamo Luhrsen >>> Sent: Wednesday, October 17, 2018 11:06 PM >>> To: openflowplugin-dev <openflowplugin-dev@lists.opendaylight.org>; >>> odl netvirt dev <netvirt-...@lists.opendaylight.org> >>> Subject: [netvirt-dev] problems with openflowplugin/ovs connections >>> >>> Hi OFP, >>> >>> we are debugging different failures in netvirt CSIT and it's >>> starting to feel like we have some problems coming from OFP. To start, >>> could someone take a look at this one: >>> >>> https://jira.opendaylight.org/projects/OPNFLWPLUG/issues/OPNFLWPLUG- >>> 1 >>> 039?filter=allopenissues >>> >>> I think this might only be happening in our clustering jobs, but it >>> seems like the high level problem is that we can't set the connected node >>> to SLAVE and cancel the ovs connect (after 20s). Then it just repeats. >>> >>> >>> Also, we see a ton of logs like the below. what is this 127.0.0.1 >>> nodeid:null thing? Since this is in the karaf.log, it seems that 127.0.0.1 >>> would be on the host running ODL and that's not a node that we are running >>> ovs (that I know of): >>> >>> >>> 2018-10-07T21:54:38,904 | INFO | epollEventLoopGroup-11-7 | >>> AbstractConnectionAdapter | 391 - >>> org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl >>> - >>> 0.6.4 | The channel outbound queue size:1024 >>> 2018-10-07T21:54:38,904 | INFO | epollEventLoopGroup-9-6 | >>> SystemNotificationsListenerImpl | 382 - >>> org.opendaylight.openflowplugin.impl - 0.6.4 | ConnectionEvent: >>> Connection closed by device, Device:/127.0.0.1:52538, NodeId:null >>> 2018-10-07T21:54:38,905 | INFO | epollEventLoopGroup-11-7 | >>> SystemNotificationsListenerImpl | 382 - >>> org.opendaylight.openflowplugin.impl - 0.6.4 | ConnectionEvent: >>> Connection closed by device, Device:/127.0.0.1:49130, NodeId:null >>> 2018-10-07T21:54:38,906 | INFO | epollEventLoopGroup-11-6 | >>> SystemNotificationsListenerImpl | 382 - >>> org.opendaylight.openflowplugin.impl - 0.6.4 | ConnectionEvent: >>> Connection closed by device, Device:/127.0.0.1:49126, NodeId:null >>> 2018-10-07T21:54:38,906 | INFO | epollEventLoopGroup-9-8 | >>> SystemNotificationsListenerImpl | 382 - >>> org.opendaylight.openflowplugin.impl - 0.6.4 | ConnectionEvent: >>> Connection closed by device, Device:/127.0.0.1:52546, NodeId:null >>> 2018-10-07T21:54:38,906 | INFO | epollEventLoopGroup-9-7 | >>> SystemNotificationsListenerImpl | 382 - >>> org.opendaylight.openflowplugin.impl - 0.6.4 | ConnectionEvent: >>> Connection closed by device, Device:/127.0.0.1:52542, NodeId:null >>> 2018-10-07T21:54:40,912 | INFO | epollEventLoopGroup-9-9 | >>> AbstractConnectionAdapter | 391 - >>> org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl >>> - >>> 0.6.4 | The channel outbound queue size:1024 >>> 2018-10-07T21:54:40,913 | INFO | epollEventLoopGroup-9-10 | >>> AbstractConnectionAdapter | 391 - >>> org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl >>> - >>> 0.6.4 | The channel outbound queue size:1024 >>> 2018-10-07T21:54:40,913 | INFO | epollEventLoopGroup-11-8 | >>> AbstractConnectionAdapter | 391 - >>> org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl >>> - >>> 0.6.4 | The channel outbound queue size:1024 >>> 2018-10-07T21:54:40,914 | INFO | epollEventLoopGroup-9-11 | >>> AbstractConnectionAdapter | 391 - >>> org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl >>> - >>> 0.6.4 | The channel outbound queue size:1024 >>> 2018-10-07T21:54:40,915 | INFO | epollEventLoopGroup-11-9 | >>> AbstractConnectionAdapter | 391 - >>> org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl >>> - >>> 0.6.4 | The channel outbound queue size:1024 >>> 2018-10-07T21:54:40,915 | INFO | epollEventLoopGroup-11-10 | >>> AbstractConnectionAdapter | 391 - >>> org.opendaylight.openflowplugin.openflowjava.openflow-protocol-impl >>> - >>> 0.6.4 | The channel outbound queue size:1024 >>> >>> >>> >>> Thanks, >>> JamO >>> _______________________________________________ >>> netvirt-dev mailing list >>> netvirt-...@lists.opendaylight.org >>> https://lists.opendaylight.org/mailman/listinfo/netvirt-dev >>> _______________________________________________ openflowplugin-dev mailing list openflowplugin-dev@lists.opendaylight.org https://lists.opendaylight.org/mailman/listinfo/openflowplugin-dev
