Testing latest Ubuntu ppa kernel (4.6.0rc5), looks like the issues have been fixed! Tahnks!.
(Hopefully that hits mainline 16.04 kernel too). -----Original Message----- From: Jon Maloy [mailto:jon.ma...@ericsson.com] Sent: Wednesday, April 06, 2016 1:16 PM To: Jon Maloy; Rune Torgersen; 'Jon Maloy'; tipc-discussion@lists.sourceforge.net Cc: GUNA (gbala...@gmail.com); Ying Xue Subject: RE: [tipc-discussion] tipc nametable update problem > -----Original Message----- > From: Jon Maloy [mailto:jon.ma...@ericsson.com] > Sent: Wednesday, 06 April, 2016 14:07 > To: Rune Torgersen; 'Jon Maloy'; tipc-discussion@lists.sourceforge.net > Cc: GUNA (gbala...@gmail.com); Ying Xue > Subject: Re: [tipc-discussion] tipc nametable update problem > > Hi Rune, > I finally found what it is, - a missing buffer linearization, and it turns > out to be a > problem that has already been solved. I checked it in last November, and is > present in kernel 4.5, but not in 4.4. > The reason I didn't realize this right away was that found and solved this as > a > UDP-bearer specific problem, and posted the patch as such. Since UDP support > is > relatively new in TIPC, I didn't realize the need to have this correction > applied > further back. The solution as such was in generic code, so it really is solved even for your Ethernet based system. Sorry for not being clear enough about this. ///j > I will create a new patch and try to get it applied on the "stable" branch. > > Regards > ///jon > > > > -----Original Message----- > > From: Rune Torgersen [mailto:ru...@innovsys.com] > > Sent: Tuesday, 05 April, 2016 18:10 > > To: Jon Maloy; 'Jon Maloy'; tipc-discussion@lists.sourceforge.net > > Cc: erik.hu...@gmail.com; Richard Alpe; Parthasarathy Bhuvaragan; Xue Ying > > (ying.x...@gmail.com); Ying Xue > > Subject: RE: [tipc-discussion] tipc nametable update problem > > > > So after trying various things to get it to fail again, I finally got it. > > Even got an > > corrupted update... > > > > SO I rebooted 1.1.1, and started TIPC when it was up and running. > > Only 1.1.1 ports show in tipc-config -nt (see log_1_1_1.txt). > > Then I rebooted 1.1.2 (ended up doing that twice I think). > > > > Normally I could not talk from 1.1.1 to 1.1.2, but this time that worked > > fine, but > > 1.1.2 got bad updates after reboot, and 1.1.2 does NOT see the ports open on > > 1.1.1. > > (see log_1_1_2.txt and 1_1_1.txt. Last tipc-config -nt is take within 30 > > second of > > each other) > > > > The port I was testing was 104,65537 and 104,131073. > > > > This was done using Ubuntu 4.4.0-15 kernel (4.4.0-15-generic) > > > > -----Original Message----- > > From: Jon Maloy [mailto:jon.ma...@ericsson.com] > > Sent: Tuesday, April 05, 2016 11:27 AM > > To: Rune Torgersen; 'Jon Maloy'; tipc-discussion@lists.sourceforge.net > > Cc: erik.hu...@gmail.com; Richard Alpe; Parthasarathy Bhuvaragan; Xue Ying > > (ying.x...@gmail.com); Ying Xue > > Subject: RE: [tipc-discussion] tipc nametable update problem > > > > > > > > > -----Original Message----- > > > From: Rune Torgersen [mailto:ru...@innovsys.com] > > > Sent: Tuesday, 05 April, 2016 12:12 > > > To: Jon Maloy; 'Jon Maloy'; tipc-discussion@lists.sourceforge.net > > > Cc: erik.hu...@gmail.com; Richard Alpe; Parthasarathy Bhuvaragan; Xue Ying > > > (ying.x...@gmail.com); Ying Xue > > > Subject: RE: [tipc-discussion] tipc nametable update problem > > > > > > I should be able to do more testing. > > > > > > I do not know for sure that the mappings were missing or not before > > > reboot. > > > If I had restarted applications, then the mappings would be there before > > reboot. > > > > > > I do know that they are definitely missing after reboot. That is how I > > > first > > > discovered it, namely by not seeing the application registration from > > > 1.1.2 > after > > > reboot. > > > > > > Looked to me like most mappings were not present, but I'll recheck. > > > > > > I'll reboot both with a kernel that I know have a problem; > > > then start wireshark on 1, restart applications on 1.1.2, and make sure > > > they > can > > > talk. > > > print out nametable on both > > > Then reboot 1.1.2 and see. > > > > > > Anything else you'd want to see (short of running diag code)? > > > > That sounds like a plan. What I am most interested in right now is if it > > is only the > > "bulk" (pre-establishment) bindings that are lost, or if it is all of them. > > If we can confirm that this is the case we will have a very specific packet > > (#2) to > > trace on, and it should be possible to find out what happens to it and its > contents. > > > > ///jon > > > > > > > > > > -----Original Message----- > > > From: Jon Maloy [mailto:jon.ma...@ericsson.com] > > > Sent: Monday, April 04, 2016 3:47 PM > > > To: Rune Torgersen; 'Jon Maloy'; tipc-discussion@lists.sourceforge.net > > > Cc: erik.hu...@gmail.com; Richard Alpe; Parthasarathy Bhuvaragan; Xue Ying > > > (ying.x...@gmail.com); Ying Xue > > > Subject: RE: [tipc-discussion] tipc nametable update problem > > > > > > Hi Rune, > > > I don't get any further with this without more input. > > > > > > First a question: were *all* bindings from 1.1.2 missing after the > > > reboot, or > only > > > the first 6 ones (the ones in the "bulk" publication of message #12646 > > > (in the > big > > > pcap file)?). > > > If the latter is the case, then we know that it is the content of this > > > particular > > > message that is not being applied. This message is special, because it > > > contains > > all > > > bindings that were made on 1.1.2 prior to the link being established. This > > > message is always sent with sequence #2, and we can see from the dump > that > > it > > > was received (after a couple of retransmissions) and acknowledged by > > > 1.1.1, > > > which means it was delivered (?) up to the binding table. > > > > > > If the bindings were missing in 1.1.1 before the reboot, but not after > > > (which > > > seems to be contrary to what you state) my theory may still be valid. The > > > Wireshark dump does not go far enough back to see what happened to the > > > original publications; only that they were missing when you tried to > > > remove > > > them. I wonder if you (or anybody else who is able to reproduce the > problem) > > > could still make the effort to apply our patches and see what happens. > > > But of > > > course, if you are 100% sure that the bindings were missing even after the > > reboot > > > run you sent me, then the problem must be something else, and I don't see > > how > > > I can get further without instrumenting the code. > > > > > > Regards > > > ///jon > > > > > > > -----Original Message----- > > > > From: Rune Torgersen [mailto:ru...@innovsys.com] > > > > Sent: Monday, 04 April, 2016 13:23 > > > > To: Jon Maloy; 'Jon Maloy'; tipc-discussion@lists.sourceforge.net > > > > Cc: erik.hu...@gmail.com; Richard Alpe; Parthasarathy Bhuvaragan > > > > Subject: RE: [tipc-discussion] tipc nametable update problem > > > > > > > > They were not in there after the reboot, might not have been there > > > > before > > > > either. > > > > Only way to actually get it working was to restart whichever > > > > application has > > the > > > > missing registration on 1.1.2. > > > > > > > > > > > > -----Original Message----- > > > > From: Jon Maloy [mailto:jon.ma...@ericsson.com] > > > > Sent: Monday, April 04, 2016 11:44 AM > > > > To: Rune Torgersen; 'Jon Maloy'; tipc-discussion@lists.sourceforge.net > > > > Cc: erik.hu...@gmail.com; Richard Alpe; Parthasarathy Bhuvaragan > > > > Subject: RE: [tipc-discussion] tipc nametable update problem > > > > > > > > Thank you Rune, > > > > I think my theory was wrong. I can now see that the dropped items > > > > actually > > > were > > > > withdrawals, not publications, that were sent out just before the 1.1.2 > > > rebooted, > > > > of course because the server application was being killed at that > > > > moment. > > > > They were probably queued because the corresponding publications could > > not > > > > be found in the table. Were those entries visible in the table of 1.1.1 > > > > before > > you > > > > rebooted? My guess is not... > > > > > > > > ///jon > > > > > > > > > > > > > -----Original Message----- > > > > > From: Rune Torgersen [mailto:ru...@innovsys.com] > > > > > Sent: Monday, 04 April, 2016 11:11 > > > > > To: Jon Maloy; 'Jon Maloy'; tipc-discussion@lists.sourceforge.net > > > > > Cc: erik.hu...@gmail.com; Richard Alpe; Parthasarathy Bhuvaragan > > > > > Subject: RE: [tipc-discussion] tipc nametable update problem > > > > > > > > > > Here is the full capture. > > > > > (If this is too big, I'll make it available on a dropbox share). > > > > > > > > > > Reboot happened approx 21:31:48, 2016-03-30 UTC. > > > > > > > > > > -----Original Message----- > > > > > From: Jon Maloy [mailto:jon.ma...@ericsson.com] > > > > > Sent: Monday, April 04, 2016 9:57 AM > > > > > To: Rune Torgersen; 'Jon Maloy'; tipc-discussion@lists.sourceforge.net > > > > > Cc: erik.hu...@gmail.com; Richard Alpe; Parthasarathy Bhuvaragan > > > > > Subject: RE: [tipc-discussion] tipc nametable update problem > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: Rune Torgersen [mailto:ru...@innovsys.com] > > > > > > Sent: Monday, 04 April, 2016 09:53 > > > > > > To: Jon Maloy; 'Jon Maloy'; tipc-discussion@lists.sourceforge.net > > > > > > Cc: erik.hu...@gmail.com; Richard Alpe; Parthasarathy Bhuvaragan > > > > > > Subject: RE: [tipc-discussion] tipc nametable update problem > > > > > > > > > > > > The test set up I have are two servers with SuperMicro X10DRL-i > > > > motherboards, > > > > > > each having two Xeon E5-2630V3 8 core CPU's, and 64GB of memory. > > > > > > I am running Ubuntu 16.04 (beta). Each server also have 10 1G > > > > > > ethernet > > > > > > interfaces, but only one was active in this case, and I only use > > > > > > one as a > > > > bearer. > > > > > > > > > > > > There are other server pairs running on the same subnet with > > > > > > different > > > > netids. > > > > > > > > > > > > This particular issue happens when I reboot one of the two servers. > > > > > > The > > > > reboot > > > > > > (full cold reboot) takes almost 5 minutes because of POST with 10 > > > > > > NICS > > > trying > > > > to > > > > > > do PXE boot. > > > > > > > > > > My guess is that in this particular run you rebooted node 1.1.2? > > > > > > > > > > If so, it doesn't contradict my theory. The dropped entries may quite > > > > > well > > > have > > > > > been lingering in 1.1.1's backlog during the five minutes it took to > > > > > reboot > > the > > > > > peer, it there otherwise was no activity (bind/unbind) on 1.1.1 > > > > > during that > > > > period. > > > > > It is first when we try to make an additional insertion (probably the > > > > > one of > > the > > > > link > > > > > going up) that the expired backlog items are discovered and purged. > > > > > So, I am still very interested in what happened before the reboot, > > > > > since I > > > > believe > > > > > that the dropped entries is just a late symptom of a problem that > > manifested > > > > > itself much earlier. > > > > > > > > > > ///jon > > > > > > > > > > > > > > > > > So far, I had to go back to Ubuntu's 3.14.1 kernel (from their PPA, > > > > > > http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.14.1-trusty/) to > find > > > one > > > > > > that does not have a problem. > > > > > > All other I have tried (4.2, 4.4 and 4.5) have shown this problem. > > > > > > > > > > > > I should be able to compile a kernel and try. > > > > > > > > > > > > -----Original Message----- > > > > > > From: Jon Maloy [mailto:jon.ma...@ericsson.com] > > > > > > Sent: Friday, April 01, 2016 7:07 PM > > > > > > To: Rune Torgersen; 'Jon Maloy'; > > > > > > tipc-discussion@lists.sourceforge.net > > > > > > Cc: erik.hu...@gmail.com; Richard Alpe; Parthasarathy Bhuvaragan > > > > > > Subject: RE: [tipc-discussion] tipc nametable update problem > > > > > > > > > > > > Hi Rune, > > > > > > I am totally unable to reproduce your scenario, and the Wireshark > > > > > > dump > > > does > > > > > not > > > > > > in itself show anything wrong. > > > > > > > > > > > > But by comparing the dropped publications with ones arriving in the > > > > messages, I > > > > > > could quickly confirm that those are not the same, although they > partially > > > > > contain > > > > > > the same port names. > > > > > > Those of type 101 that are dropped has instance numbers that are not > the > > > > > same > > > > > > as in the arriving messages (you are probably generating new ones > > > > > > for > > each > > > > > > session), while those which are the same (103, 105..) have different > > > > publication > > > > > > key numbers. > > > > > > > > > > > > The conclusion can only be that those are leftovers from a previous > > session > > > > > which > > > > > > have not been purged when the contact with node 1.1.2 was lost. A > quick > > > > check > > > > > > of the code confirms this; entries in the name table backlog are > > > > > > only > > purged > > > > > > based on expiration time, and not on loss of contact with their > originating > > > > code, > > > > > > as they should be. This is clearly a bug that somebody has to fix > > > > > > (Partha, > > > > > > Richard?). > > > > > > > > > > > > Remains to understand how those entries got into the backlog in the > first > > > > place. > > > > > > Something must have happened in the previous session that prevented > > > them > > > > > > from being applied. Since you never use instance sequences, > overlapping > > > > > > sequences cannot be the problem. If it were a memory allocation > problem > > > > this > > > > > > would be visible in the log. One possibility I see is that we have > > > > > > a race > > > > condition > > > > > > between the purging of binding table from the pre-previous session > > > > > > and > > > the > > > > > > previous one. The call to the purging function tipc_publ_notify() > > > > > > is done > > > > outside > > > > > > any lock protection, so it is fully possible that a link that > > > > > > quickly goes > down > > > and > > > > > > comes back may be able to deliver a new batch of publications before > the > > > > > purging > > > > > > action is finished. This becomes particularly likely if the number > > > > > > of > > > publications > > > > > is > > > > > > large, and we are running in a multi-VM or multi-namespace > environment > > > on > > > > > the > > > > > > same host. (Can you confirm, Rune?) > > > > > > If only the interface or link is cycled, while the same application > > > > > > server > > > > > continues > > > > > > running on 1.1.2, and 1.1.1 still is intact, this is a possible > > > > > > scenario. > > > > > > The newly delivered publications will find a set of exactly equal > > publications > > > > > from > > > > > > the previous session in the name table, and hence be put in the > > > > > > backlog. > > > > > > > > > > > > How do we resolve this? My first idea was to just run a > process_backlog() > > > on > > > > > the > > > > > > flank of tipc_publ_notify(). But unfortunately we first need to run > > > > > > a > purge > > > on > > > > > the > > > > > > backlog, according to the above, and this purge would be unable to > > > > distinguish > > > > > > between "old" and "new" backlog items, and would have to purge them > > all. > > > > > > > > > > > > A better, but maybe not so neat solution would be to use a similar > > solution > > > as > > > > > we > > > > > > do with socket wakeup. We create a pseudo message with a new > > message > > > > type > > > > > > PURGER, and append that to the tail of the node's namedq when we > lose > > > > > contact > > > > > > with a node, but this time *before* we release the node write lock. > > > > > > We > > > could > > > > > > then test for this type, in addition to the PUBLICATION and > WITHDRAWAL > > > > > types, > > > > > > inside tipc_update_nametbl(), and call tipc_publ_notify(), still > > > > > > inside the > > > > name > > > > > > table lock, whenever this message type is encountered. This would > > > guarantee > > > > > > that things happen in sequential order, since any new publications > would > > > end > > > > > up > > > > > > behind the PURGER message in the node's namedq. > > > > > > > > > > > > Who has time to implement this? > > > > > > > > > > > > Also, do you Rune build your own kernel, so you could try out a > > > > > > patch > > from > > > us > > > > > and > > > > > > confirm my theory before we deliver such a solution upstream? > > > > > > > > > > > > Regards > > > > > > ///jon > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Rune Torgersen [mailto:ru...@innovsys.com] > > > > > > > Sent: Thursday, 31 March, 2016 14:56 > > > > > > > To: 'Jon Maloy'; tipc-discussion@lists.sourceforge.net > > > > > > > Subject: Re: [tipc-discussion] tipc nametable update problem > > > > > > > > > > > > > > Have not been able to capture a corrupted update yet, but did > manage > > to > > > > get > > > > > > > one where it dropped the updates. > > > > > > > > > > > > > > Here is the dmesg output (times are in CST). > > > > > > > Mar 30 16:31:48 testserv218 kernel: Dropping name table update > > > > > > > (1) of > > > {103, > > > > > > > 1003, 1003} from <1.1.2> key=4271114002 > > > > > > > Mar 30 16:31:48 testserv218 kernel: Dropping name table update > > > > > > > (1) of > > > {103, > > > > > 3, > > > > > > 3} > > > > > > > from <1.1.2> key=3675117576 > > > > > > > Mar 30 16:31:48 testserv218 kernel: Dropping name table update > > > > > > > (1) of > > > {101, > > > > > > > 133330, 133330} from <1.1.2> key=2005280282 > > > > > > > Mar 30 16:31:48 testserv218 kernel: Dropping name table update > > > > > > > (1) of > > > > > > > {13762562, 0, 0} from <1.1.2> key=3568185108 > > > > > > > Mar 30 16:31:48 testserv218 kernel: Dropping name table update > > > > > > > (1) of > > > {206, > > > > > 9, > > > > > > 9} > > > > > > > from <1.1.2> key=3641103006 > > > > > > > Mar 30 16:31:48 testserv218 kernel: Dropping name table update > > > > > > > (1) of > > > {101, > > > > > > > 133398, 133398} from <1.1.2> key=2675546830 > > > > > > > Mar 30 16:31:48 testserv218 kernel: Dropping name table update > > > > > > > (1) of > > > {101, > > > > > > > 133138, 133138} from <1.1.2> key=2939408752 > > > > > > > Mar 30 16:31:48 testserv218 kernel: Dropping name table update > > > > > > > (1) of > > > {105, > > > > > > 104, > > > > > > > 104} from <1.1.2> key=140803529 > > > > > > > Mar 30 16:31:48 testserv218 kernel: Dropping name table update > > > > > > > (1) of > > > {105, > > > > > 4, > > > > > > 4} > > > > > > > from <1.1.2> key=3695579549 > > > > > > > Mar 30 16:31:48 testserv218 kernel: Dropping name table update > > > > > > > (1) of > > > {101, > > > > > > > 133386, 133386} from <1.1.2> key=808970575 > > > > > > > > > > > > > > Attached it the tipc packets received on 1.1.1. (where the log > > > > > > > is from > > > during > > > > > the > > > > > > > same time period). > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Jon Maloy [mailto:ma...@donjonn.com] > > > > > > > Sent: Saturday, March 26, 2016 8:51 AM > > > > > > > To: tipc-discussion@lists.sourceforge.net > > > > > > > Subject: Re: [tipc-discussion] tipc nametable update problem > > > > > > > > > > > > > > Hi Rune, > > > > > > > I assume you are still using Ethernet/L2 a bearer, and not UDP? > > > > > > > UDP is > > > > > > > relatively new as supported bearer, and may incur new problems. > > > > > > > Anyway, name table updates are never fragmented. > > > > > > > As far as I can see you have five bindings on each node, and that > > > > > > > amounts to either one "bulk" update message with all bindings in > > > > > > > one > > > > > > > message (100 bytes in your case) > > > > > > > or five individual update messages of 20 bytes each. All > > > > > > > depending on > > > > > > > whether you application was started, and the bindings made before > > > > > > > or > > > > > > > after the link between the nodes are established. > > > > > > > > > > > > > > To me it looks like the dropped bindings are severely corrupted, > > > > > > > and > > > > > > > that may be a starting point for our trouble shooting. Could you > > > > > > > start > > > > > > > Wireshark and have a look at the messages being exchanged when > this > > > > > > > happens? If you only look for NAME_DISTRIBUTOR messages, the > > number > > > > of > > > > > > > messages to analyze should be very limited, and we can at least > > > > > > > see if > > > > > > > our bug is on the sending or the receiving side. > > > > > > > > > > > > > > God påske > > > > > > > ///jon > > > > > > > > > > > > > > > > > > > > > On 03/25/2016 12:05 PM, Rune Torgersen wrote: > > > > > > > > Is it possible for the update messages to be greater than 1 MTU? > > > > > > > > > > > > > > > > Because were doing a lot of video multicast, we’re turning on > > > > > > > > UDP > RSS > > > > > > hashing > > > > > > > to get messages to differnet receive queue (via ethtool -N ethN > > > > > > > rx- > flow- > > > > hash > > > > > > > udp4 sdfn) > > > > > > > > Because of that, there is a kernel warning per interface, and I > > > > > > > > am > > curious > > > if > > > > > > that > > > > > > > is what is causing this: > > > > > > > > > > > > > > > > igb 0000:07:00.0: enabling UDP RSS: fragmented packets may > > > > > > > > arrive > > out > > > of > > > > > > order > > > > > > > to the stack above > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > From: Erik Hugne [mailto:erik.hu...@gmail.com] > > > > > > > > Sent: Wednesday, March 23, 2016 12:07 PM > > > > > > > > To: Rune Torgersen > > > > > > > > Cc: tipc-discussion@lists.sourceforge.net > > > > > > > > Subject: Re: [tipc-discussion] tipc nametable update problem > > > > > > > > > > > > > > > > > > > > > > > > When an update is received, bit cannot immediately be applied to > the > > > > local > > > > > > > nametable, we retain it for a few seconds in a backlog queue. > > > > > > > > Then for each subsequent update received (that may have cleared > up > > > the > > > > > > > conflict) we try to apply any update stored in the backlog. > > > > > > > > The timeout can be set with sysctl -w tipc.named_timeout=xxx > > > > > > > > Default is 2000ms. > > > > > > > > > > > > > > > > So clock drift does not matter. > > > > > > > > > > > > > > > > I'm guessing that the nametable updates are dropped on the > sending > > > > side. > > > > > > > > Are there any interface renaming going on after tipc is enabled? > > > > > > > > > > > > > > > > //E > > > > > > > > On Mar 23, 2016 17:04, "Rune Torgersen" > > > > > > > <ru...@innovsys.com<mailto:ru...@innovsys.com>> wrote: > > > > > > > > How much clock drift between units does the nametable update > > allow? > > > > > > > > > > > > > > > > On one of the test units, the clock was off by about a second > between > > > > > them. > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Rune Torgersen > > > > > > > [mailto:ru...@innovsys.com<mailto:ru...@innovsys.com>] > > > > > > > > Sent: Tuesday, March 22, 2016 10:58 AM > > > > > > > > To: tipc-discussion@lists.sourceforge.net<mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net> > > > > > > > > Subject: Re: [tipc-discussion] tipc nametable update problem > > > > > > > > > > > > > > > > Still having nametable update problems (Using > TIPC_CLUSTER_SCOPE) > > > > > > > > > > > > > > > > Here is an except of tipc-config -nt on both systems: > > > > > > > > address 1.1.1: > > > > > > > > > > > > > > > > 104 1025 1025 <1.1.1:3540751351> > > > > > > > > 3540751351 cluster > > > > > > > > 104 65537 65537 <1.1.1:4046699456> > > > > > > > > 4046699456 cluster > > > > > > > > 104 131073 131073 <1.1.2:59828181> > > > > > > > > 59828181 cluster > > > > > > > > 104 16777984 16777984 <1.1.1:3135589675> > > > > > > > > 3135589675 > > cluster > > > > > > > > 104 33555200 33555200 <1.1.2:2193437365> > > > > > > > > 2193437365 > > cluster > > > > > > > > > > > > > > > > Address 1.1.2: > > > > > > > > 104 131073 131073 <1.1.2:59828181> > > > > > > > > 59828181 cluster > > > > > > > > 104 33555200 33555200 <1.1.2:2193437365> > > > > > > > > 2193437365 > > cluster > > > > > > > > > > > > > > > > So in this case 1 sees all address 2 has published, while 2 is > > > > > > > > not seeing > > > the > > > > > > > addesses from 1. > > > > > > > > 2 was rebooted to make this happen. > > > > > > > > > > > > > > > > Is tere a possibility I'm calling tipc-config too early, and > > > > > > > > the interface > is > > > not > > > > > yet > > > > > > > up, or is this still the same roblem I saw before. > > > > > > > > > > > > > > > > There is nome dropped nametable update messages in kernel: > > > > > > > > > > > > > > > > Mar 22 10:34:34 mitchelltelctrl2 kernel: Dropping name table > > > > > > > > update > > (0) > > > of > > > > > {0, > > > > > > 0, > > > > > > > 0} from <1.1.1> key=0 > > > > > > > > Mar 22 10:34:34 mitchelltelctrl2 kernel: Dropping name table > > > > > > > > update > > (0) > > > of > > > > > {0, > > > > > > 0, > > > > > > > 0} from <1.1.1> key=0 > > > > > > > > Mar 22 10:34:34 mitchelltelctrl2 kernel: Dropping name table > > > > > > > > update > > (0) > > > of > > > > > {0, > > > > > > 0, > > > > > > > 0} from <1.1.1> key=0 > > > > > > > > Mar 22 10:34:34 mitchelltelctrl2 kernel: Dropping name table > > > > > > > > update > > (0) > > > of > > > > > {0, > > > > > > 0, > > > > > > > 0} from <1.1.1> key=0 > > > > > > > > Mar 22 10:34:34 mitchelltelctrl2 kernel: Dropping name table > > > > > > > > update > > (0) > > > of > > > > > {0, > > > > > > 0, > > > > > > > 0} from <1.1.1> key=0 > > > > > > > > Mar 22 10:34:34 mitchelltelctrl2 kernel: Dropping name table > > > > > > > > update > > (0) > > > of > > > > > {0, > > > > > > 0, > > > > > > > 0} from <1.1.1> key=0 > > > > > > > > Mar 22 10:34:34 mitchelltelctrl2 kernel: Dropping name table > > > > > > > > update > > (0) > > > of > > > > > {0, > > > > > > 0, > > > > > > > 0} from <1.1.1> key=0 > > > > > > > > Mar 22 10:34:34 mitchelltelctrl2 kernel: Dropping name table > > > > > > > > update > > (0) > > > of > > > > > {0, > > > > > > 0, > > > > > > > 0} from <1.1.1> key=0 > > > > > > > > Mar 22 10:34:34 mitchelltelctrl2 kernel: Dropping name table > > > > > > > > update > > (0) > > > of > > > > > {0, > > > > > > 0, > > > > > > > 0} from <1.1.1> key=0 > > > > > > > > Mar 22 10:34:34 mitchelltelctrl2 kernel: Dropping name table > > > > > > > > update > > (0) > > > of > > > > > {0, > > > > > > 0, > > > > > > > 16600} from <1.1.1> key=4294915584 > > > > > > > > > > > > > > > > but they do not mention port 104. > > > > > > > > > > > > > > > > If I restart the application on 1 having 104:1025 open, it > > > > > > > > shows up on > 2. > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Rune Torgersen > > > > > > > [mailto:ru...@innovsys.com<mailto:ru...@innovsys.com>] > > > > > > > > Sent: Monday, March 21, 2016 12:17 AM > > > > > > > > To: Jon Maloy; Erik Hugne > > > > > > > > Cc: tipc-discussion@lists.sourceforge.net<mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net> > > > > > > > > Subject: Re: [tipc-discussion] tipc nametable update problem > > > > > > > > > > > > > > > > Using TIPC_CLUSTER_SCOPE will work. > > > > > > > > This was new system bring-up, and code was ported from older > > system, > > > > > > which > > > > > > > used TIPC 1.7.7 driver. > > > > > > > > A quick search and replace of TIPC_ZONE_SCOPE is not a bad > > > workaround. > > > > > > > > ________________________________________ > > > > > > > > From: Jon Maloy > > > > > > [jon.ma...@ericsson.com<mailto:jon.ma...@ericsson.com>] > > > > > > > > Sent: Saturday, March 19, 2016 10:57 AM > > > > > > > > To: Erik Hugne > > > > > > > > Cc: tipc-discussion@lists.sourceforge.net<mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net> > > > > > > > > Subject: Re: [tipc-discussion] tipc nametable update problem > > > > > > > > > > > > > > > > Maybe not completely trivial, but not very complex either. I > > > > > > > > know I > > > failed > > > > to > > > > > > > describe this verbally to you at one moment, but I can put it on > > > > > > > paper, > > and > > > > > you > > > > > > > will realize it is not a big deal. > > > > > > > > If you or anybody else are interested I can make an effort to > describe > > > this > > > > > > next > > > > > > > week. I don't have time to implement it myself at the moment. > > > > > > > > > > > > > > > > ///jon > > > > > > > > > > > > > > > > > > > > > > > > From: Erik Hugne > > > > > > > [mailto:erik.hu...@gmail.com<mailto:erik.hu...@gmail.com>] > > > > > > > > Sent: Friday, 18 March, 2016 12:38 > > > > > > > > To: Jon Maloy > > > > > > > > Subject: RE: [tipc-discussion] tipc nametable update problem > > > > > > > > > > > > > > > > > > > > > > > > Agree. > > > > > > > > But implementing a new lookup mechanism is not trivial.. :) > > > > > > > > > > > > > > > > @Rune afaik there is no functional limitation on using cluster > > > > > > > > scoped > > > > > > > publications, so i hope that's an acceptable workaround for you. > > > > > > > > > > > > > > > > //E > > > > > > > > On Mar 18, 2016 16:46, "Jon Maloy" > > > > > > > > > > > > > > > > <jon.ma...@ericsson.com<mailto:jon.ma...@ericsson.com><mailto:jon.maloy > > > > > > > @ericsson.com<mailto:jon.ma...@ericsson.com>>> wrote: > > > > > > > > Still weird that this starts happening now, when this issue is > supposed > > to > > > > be > > > > > > > remedied, and not earlier, when it wasn't. > > > > > > > > We really need that "permit overlapping publications" solution > > > > > > > > I > have > > > > been > > > > > > > preaching about. > > > > > > > > > > > > > > > > Br > > > > > > > > ///jon > > > > > > > > > > > > > > > > > > > > > > > >> -----Original Message----- > > > > > > > >> From: Rune Torgersen > > > > > > > > > > > > > > > > > > > > > > > > > > > > [mailto:ru...@innovsys.com<mailto:ru...@innovsys.com><mailto:runet@innov > > > > > > > sys.com<mailto:ru...@innovsys.com>>] > > > > > > > >> Sent: Friday, 18 March, 2016 10:25 > > > > > > > >> To: 'Erik Hugne' > > > > > > > >> Cc: tipc-discussion@lists.sourceforge.net<mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net><mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net<mailto:tipc- > > > > > > discuss...@lists.sourceforge.net>> > > > > > > > >> Subject: Re: [tipc-discussion] tipc nametable update problem > > > > > > > >> > > > > > > > >> Yes I have. > > > > > > > >> There are quite a few at the same time like this: > > > > > > > >> > > > > > > > >> Mar 17 20:08:58 restarttv kernel: Dropping name table update > > > > > > > >> (0) of > > > > > > > {1853110816, > > > > > > > >> 1952998688, 1801810542} from <1.1.1> key=1633905523 > > > > > > > >> Mar 17 20:08:58 restarttv kernel: Dropping name table update > > > > > > > >> (0) of > > > > > > > {542000723, > > > > > > > >> 544613732, 544437616} from <1.1.1> key=167800175 > > > > > > > >> Mar 17 20:08:58 restarttv kernel: Dropping name table update > > > > > > > >> (0) of > > > > > > > {544239474, > > > > > > > >> 1953325424, 543582572} from <1.1.1> key=1930035237 > > > > > > > >> Mar 17 20:08:58 restarttv kernel: Dropping name table update > > > > > > > >> (0) of > > > > > > > {1933189232, > > > > > > > >> 1869771885, 1634738291} from <1.1.1> key=1768843040 > > > > > > > >> Mar 17 20:08:58 restarttv kernel: Dropping name table update > > > > > > > >> (0) of > > > > > > > {1717660012, > > > > > > > >> 1701054976, 628308512} from <1.1.1> key=1869881446 > > > > > > > >> Mar 17 20:08:58 restarttv kernel: Dropping name table update > > > > > > > >> (0) of > > > > > {16397, > > > > > > > >> 1073741824, 16397} from <1.1.1> key=29285 > > > > > > > >> Mar 17 20:08:58 restarttv kernel: Dropping name table update > > > > > > > >> (0) of > > > > > > > {1633943667, > > > > > > > >> 1752134260, 544367969} from <1.1.1> key=1679834144 > > > > > > > >> Mar 17 20:08:58 restarttv kernel: Dropping name table update > > > > > > > >> (0) of > > > > > > > {1869771808, > > > > > > > >> 2003986804, 1698300018} from <1.1.1> key=4294915584 > > > > > > > >> Mar 17 20:08:58 restarttv kernel: Dropping name table update > > > > > > > >> (0) of > > > > > > > {1073741824, > > > > > > > >> 65279, 4294902016} from <1.1.1> key=1073741824 > > > > > > > >> Mar 17 20:08:58 restarttv kernel: Dropping name table update > > > > > > > >> (0) of > > > > > {65279, > > > > > > > >> 4294901760, 59154} from <1.1.1> key=65023 > > > > > > > >> > > > > > > > >> > > > > > > > >> From: Erik Hugne > > > > > > > > > > > > > > > > > > > > > > > > > > > > [mailto:erik.hu...@gmail.com<mailto:erik.hu...@gmail.com><mailto:erik.hugn > > > > > > > e...@gmail.com<mailto:erik.hu...@gmail.com>>] > > > > > > > >> Sent: Friday, March 18, 2016 1:48 AM > > > > > > > >> To: Rune Torgersen > > > > > > > >> Cc: tipc-discussion@lists.sourceforge.net<mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net><mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net<mailto:tipc- > > > > > > discuss...@lists.sourceforge.net>> > > > > > > > >> Subject: Re: [tipc-discussion] tipc nametable update problem > > > > > > > >> > > > > > > > >> > > > > > > > >> Hi Rune. > > > > > > > >> When the problem occurs, have you seen any traces like "tipc: > > > Dropping > > > > > > name > > > > > > > >> table update...." ? > > > > > > > >> > > > > > > > >> //E > > > > > > > >> On Mar 18, 2016 02:11, "Rune Torgersen" > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > <ru...@innovsys.com<mailto:ru...@innovsys.com><mailto:ru...@innovsys.co > > > > > > > > > > > > > > > > > > > > > > > > > > > > m<mailto:ru...@innovsys.com>><mailto:ru...@innovsys.com<mailto:runet@in > > > > > > > > > > > > novsys.com><mailto:ru...@innovsys.com<mailto:ru...@innovsys.com>>>> > > > > > > > wrote: > > > > > > > >> More info. > > > > > > > >> The failing ports are all opened as TIPC_ZONE_SCOPE. > > > > > > > >> Addresses of the two computers are 1.1.1 and 1.1.2. > > > > > > > >> > > > > > > > >> If I change the open param to TIPC_CLUSTER_SCOPE, the > nametable > > > > > seems > > > > > > to > > > > > > > >> update correctly. > > > > > > > >> > > > > > > > >> > > > > > > > >> -----Original Message----- > > > > > > > >> From: Rune Torgersen > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > [mailto:ru...@innovsys.com<mailto:ru...@innovsys.com><mailto:runet@innov > > > > > > > > > > > > > > > > > > > > > > > > > > > > sys.com<mailto:ru...@innovsys.com>><mailto:ru...@innovsys.com<mailto:run > > > > > > > > > > > > > > > > > > > > > > > > > > > > e...@innovsys.com><mailto:ru...@innovsys.com<mailto:ru...@innovsys.com>> > > > > > > > >] > > > > > > > >> Sent: Thursday, March 17, 2016 7:06 PM > > > > > > > >> To: 'tipc-discussion@lists.sourceforge.net<mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net><mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net<mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net>><mailto:tipc-<mailto:tipc- > > > ><mailto:tipc- > > > > > > > <mailto:tipc->> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > discuss...@lists.sourceforge.net<mailto:discuss...@lists.sourceforge.net><mail > > > > > > > > > > > > > > > > > > > > > > > > > > > > to:discuss...@lists.sourceforge.net<mailto:discuss...@lists.sourceforge.net>>>' > > > > > > > >> Subject: [tipc-discussion] tipc nametable update problem > > > > > > > >> > > > > > > > >> Hi. > > > > > > > >> > > > > > > > >> The product I work on uses TIPC for communication between > > different > > > > > > > >> computers on a network. We've actually been using older version > > > (1.7.7 > > > > > and > > > > > > > older > > > > > > > >> ) for nearly 10 years. > > > > > > > >> > > > > > > > >> On a new product, we're using the latest Ubuntu server (16.04, > > > > > > > >> still > in > > > > > beta) > > > > > > > using > > > > > > > >> kernel 4.4.0. > > > > > > > >> > > > > > > > >> On several occasions now, after boot, programs that open TIPC > > sockets > > > > > > during > > > > > > > >> the boot process, have ports that does not show in the > > > > > > > >> nametable > on > > > > the > > > > > > > other > > > > > > > >> computer. This of course causes the programs to not being able > > > > > > > >> to > > talk. > > > > > > > >> If we restart the program, reopening the TIPC port, then it > > > > > > > >> shows > up > > > on > > > > > both > > > > > > > >> sides. > > > > > > > >> > > > > > > > >> > > > > > > > >> I know this is somewhat sparse info, but I am not sure where to > start > > to > > > > > look > > > > > > at > > > > > > > >> this. > > > > > > > >> > > > > > > > >> One piece of info that might be useful, is that we kind of > > > > > > > >> require > the > > > old > > > > > > > interface > > > > > > > >> naming on our interfaces, so we have turned off systemd's > ethernet > > > > > > naming > > > > > > > >> scheme, and use udev to name the devices. > > > > > > > >> > > > > > > > >> This should be done well before we initializer the tipc driver > module > > > and > > > > > > give > > > > > > > it a > > > > > > > >> netid and address and enable the bearer links. > > > > > > > >> > > > > > > > >> ------------------------------------------------------------------------------ > > > > > > > >> Transform Data into Opportunity. > > > > > > > >> Accelerate data analysis in your applications with > > > > > > > >> Intel Data Analytics Acceleration Library. > > > > > > > >> Click to learn more. > > > > > > > >> > > http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140 > > > > > > > >> _______________________________________________ > > > > > > > >> tipc-discussion mailing list > > > > > > > >> tipc-discussion@lists.sourceforge.net<mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net><mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net<mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net>><mailto:tipc-<mailto:tipc- > > > ><mailto:tipc- > > > > > > > <mailto:tipc->> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > discuss...@lists.sourceforge.net<mailto:discuss...@lists.sourceforge.net><mail > > > > > > > > > > > > > > > > > > > > > > > > > > > > to:discuss...@lists.sourceforge.net<mailto:discuss...@lists.sourceforge.net>>> > > > > > > > >> https://lists.sourceforge.net/lists/listinfo/tipc-discussion > > > > > > > >> > > > > > > > >> ------------------------------------------------------------------------------ > > > > > > > >> Transform Data into Opportunity. > > > > > > > >> Accelerate data analysis in your applications with > > > > > > > >> Intel Data Analytics Acceleration Library. > > > > > > > >> Click to learn more. > > > > > > > >> > > http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140 > > > > > > > >> _______________________________________________ > > > > > > > >> tipc-discussion mailing list > > > > > > > >> tipc-discussion@lists.sourceforge.net<mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net><mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net<mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net>><mailto:tipc-<mailto:tipc- > > > ><mailto:tipc- > > > > > > > <mailto:tipc->> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > discuss...@lists.sourceforge.net<mailto:discuss...@lists.sourceforge.net><mail > > > > > > > > > > > > > > > > > > > > > > > > > > > > to:discuss...@lists.sourceforge.net<mailto:discuss...@lists.sourceforge.net>>> > > > > > > > >> https://lists.sourceforge.net/lists/listinfo/tipc-discussion > > > > > > > >> ------------------------------------------------------------------------------ > > > > > > > >> Transform Data into Opportunity. > > > > > > > >> Accelerate data analysis in your applications with > > > > > > > >> Intel Data Analytics Acceleration Library. > > > > > > > >> Click to learn more. > > > > > > > >> > > http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140 > > > > > > > >> _______________________________________________ > > > > > > > >> tipc-discussion mailing list > > > > > > > >> tipc-discussion@lists.sourceforge.net<mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net><mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net<mailto:tipc- > > > > > > discuss...@lists.sourceforge.net>> > > > > > > > >> https://lists.sourceforge.net/lists/listinfo/tipc-discussion > > > > > > > > ------------------------------------------------------------------------------ > > > > > > > > Transform Data into Opportunity. > > > > > > > > Accelerate data analysis in your applications with > > > > > > > > Intel Data Analytics Acceleration Library. > > > > > > > > Click to learn more. > > > > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140 > > > > > > > > _______________________________________________ > > > > > > > > tipc-discussion mailing list > > > > > > > > tipc-discussion@lists.sourceforge.net<mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net> > > > > > > > > https://lists.sourceforge.net/lists/listinfo/tipc-discussion > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > > > > > > Transform Data into Opportunity. > > > > > > > > Accelerate data analysis in your applications with > > > > > > > > Intel Data Analytics Acceleration Library. > > > > > > > > Click to learn more. > > > > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140 > > > > > > > > _______________________________________________ > > > > > > > > tipc-discussion mailing list > > > > > > > > tipc-discussion@lists.sourceforge.net<mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net> > > > > > > > > https://lists.sourceforge.net/lists/listinfo/tipc-discussion > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > > > > > > Transform Data into Opportunity. > > > > > > > > Accelerate data analysis in your applications with > > > > > > > > Intel Data Analytics Acceleration Library. > > > > > > > > Click to learn more. > > > > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140 > > > > > > > > _______________________________________________ > > > > > > > > tipc-discussion mailing list > > > > > > > > tipc-discussion@lists.sourceforge.net<mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net> > > > > > > > > https://lists.sourceforge.net/lists/listinfo/tipc-discussion > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > > > > > > Transform Data into Opportunity. > > > > > > > > Accelerate data analysis in your applications with > > > > > > > > Intel Data Analytics Acceleration Library. > > > > > > > > Click to learn more. > > > > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140 > > > > > > > > _______________________________________________ > > > > > > > > tipc-discussion mailing list > > > > > > > > tipc-discussion@lists.sourceforge.net<mailto:tipc- > > > > > > > discuss...@lists.sourceforge.net> > > > > > > > > https://lists.sourceforge.net/lists/listinfo/tipc-discussion > > > > > > > > ------------------------------------------------------------------------------ > > > > > > > > Transform Data into Opportunity. > > > > > > > > Accelerate data analysis in your applications with > > > > > > > > Intel Data Analytics Acceleration Library. > > > > > > > > Click to learn more. > > > > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140 > > > > > > > > _______________________________________________ > > > > > > > > tipc-discussion mailing list > > > > > > > > tipc-discussion@lists.sourceforge.net > > > > > > > > https://lists.sourceforge.net/lists/listinfo/tipc-discussion > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > > > > > Transform Data into Opportunity. > > > > > > > Accelerate data analysis in your applications with > > > > > > > Intel Data Analytics Acceleration Library. > > > > > > > Click to learn more. > > > > > > > http://pubads.g.doubleclick.net/gampad/clk?id=278785351&iu=/4140 > > > > > > > _______________________________________________ > > > > > > > tipc-discussion mailing list > > > > > > > tipc-discussion@lists.sourceforge.net > > > > > > > https://lists.sourceforge.net/lists/listinfo/tipc-discussion > ------------------------------------------------------------------------------ > _______________________________________________ > tipc-discussion mailing list > tipc-discussion@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/tipc-discussion ------------------------------------------------------------------------------ Find and fix application performance issues faster with Applications Manager Applications Manager provides deep performance insights into multiple tiers of your business applications. It resolves application problems quickly and reduces your MTTR. Get your free trial! https://ad.doubleclick.net/ddm/clk/302982198;130105516;z _______________________________________________ tipc-discussion mailing list tipc-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/tipc-discussion
