On 02/27/2017 04:07 PM, Butler, Peter wrote: > Partha et al, > > Thanks for the heads-up. Also note that our user-space applications were > built against headers (i.e. /usr/include/...) that came from the 4.4.0 > kernel. Are there any backward-compatibility issues running these binaries > against the newly built 4.9.11-based TIPC kernel module? It is my > understanding that kernel code is meant to be backward-compatible in > principle... > The kernel ABI (/usr/include) is always backward compatible. If you see issues with it, then they are bugs. /Partha > Peter > > > > -----Original Message----- > From: Parthasarathy Bhuvaragan [mailto:parthasarathy.bhuvara...@ericsson.com] > Sent: February-27-17 7:37 AM > To: Butler, Peter <pbut...@sonusnet.com> > Cc: Jon Maloy <jon.ma...@ericsson.com>; tipc-discussion@lists.sourceforge.net > Subject: Re: TIPC Oops in tipc_sk_recv > > On 02/24/2017 04:15 PM, Butler, Peter wrote: >> Hi Partha, >> >> >> >> In our situation we do not need to support the delivery of RPMs in any >> way. Literally the only thing we are changing on the target systems >> is the tipc.ko file. That is, the original 4.4.0 kernel and all other >> 4.4.0-specific kernel modules will be left untouched. >> >> >> >> I am doing this actually removing the include/uapi/linux/tipc* and >> net/tipc/* files from within our 4.4.0 kernel source tree, and >> replacing them with the files from kernel 4.9.11. (Note that kernel >> 4.9.11 actually has a couple more TIPC-related files than the 4.4.0 >> kernel.) To accomplish I had to make a few changes (as per the email >> thread between Jon and myself) to get it to compile. >> >> >> >> Then, when I kick off a 'make' (no 'make clean' is performed) at the >> top level of the kernel source tree the build process detects that >> everything TIPC-related requires building, and a new tipc.ko is >> generated. This tipc.ko is literally taken and installed onto the >> existing 4.4.0 systems without any other changes (e.g. no new bzImage >> is installed - the original kernel file is left untouched). >> >> >> >> We're not concerned about maintainability for now, as we plan on doing >> a full upgrade of the entire kernel at some point in the next few months. >> The hybrid of a 4.4.0 kernel running a TIPC source from 4.9.11 is only >> a stop-gap measure for an emergency fix needed asap. >> >> >> >> If you can foresee any issues with our short-term plan here let me >> know. As it stands I have the module built and running - but that of >> course doesn't mean that run-time issues won't occur. >> > Hi Peter, > > If you are taking the latest tipc, please patch yours with these two fixes > for the socket. > [PATCH net v1 1/2] tipc: fix socket flow control errors [PATCH net v1 2/2] > tipc: Fix missing connection request handling > > A word of caution: My users still face stability issues (connection are > permanently congested) while running load over several connections > (~1000+) and i am yet to find the root cause. > > /Partha >> >> >> /Peter >> >> >> >> *From:*Parthasarathy Bhuvaragan >> [mailto:parthasarathy.bhuvara...@ericsson.com] >> *Sent:* February-24-17 5:21 AM >> *To:* Butler, Peter <pbut...@sonusnet.com> >> *Cc:* Jon Maloy <jon.ma...@ericsson.com>; >> tipc-discussion@lists.sourceforge.net >> *Subject:* Re: TIPC Oops in tipc_sk_recv >> >> >> >> Hi Peter, >> >> >> >> The backporting strategy varies depending on: >> >> 1. Supporting upgrades of rpm's. Ex: can you deliver a new tipc rpm >> and update it on an existing kernel. >> >> 2. Delivering / Upgrading the entire kernel. No individual rpm updates >> are delivered. >> >> >> >> If its option 2, then you may be allowed to update tipc ABI i.e >> include the commits which touch include/uapi/linux/tipc*. >> >> I have to support option 1, so I cannot include any commit which >> touches files outside net/tipc/ without manual intervention. >> >> >> >> The way I do it is to using git and walk all the commits from a >> specific version upto say v4.9 and follow these rules: >> >> 1. Skip commits which are not tipc specific, i.e its introduced as a >> part of core net cleanup. They usually break the ABI. >> >> 2. If you skip a commit, the subsequent commit needs to be amended to >> apply cleanly. >> >> 3. When cherry-picking commits, use option "-x" to record the upstream >> commit id. This way you can do git-blame and find out the history. >> >> This is a slow process, but you will be sure of the commits you pick >> and its history. >> >> >> >> If you copy the tipc source from a later kernel to say v4.4.x, then >> you loose the history. This will hinder maintainability in the long run. >> >> >> >> /Partha >> >> ---------------------------------------------------------------------- >> -- >> >> *From:*Butler, Peter <pbut...@sonusnet.com >> <mailto:pbut...@sonusnet.com>> >> *Sent:* Thursday, February 23, 2017 9:29 PM >> *To:* Jon Maloy; tipc-discussion@lists.sourceforge.net >> <mailto:tipc-discussion@lists.sourceforge.net>; Parthasarathy >> Bhuvaragan >> *Cc:* Butler, Peter >> *Subject:* RE: TIPC Oops in tipc_sk_recv >> >> >> >> I have made the following final change: this change works around the >> different function signature for udp_tunnel6_xmit_skb() in udp_media.c >> (function is defined in net/ipv6/ip6_udp_tunnel.c): >> >> Change: >> err = udp_tunnel6_xmit_skb(ndst, ub->ubsock->sk, skb, >> ndst->dev, &src->ipv6, >> &dst->ipv6, 0, ttl, 0, src->port, >> dst->port, false); >> >> To be: >> err = udp_tunnel6_xmit_skb(ndst, ub->ubsock->sk, skb, >> ndst->dev, &src->ipv6, >> &dst->ipv6, 0, ttl, src->port, >> dst->port, false); >> >> That is, simply remove the '0' parameter (which comes immediately >> after the ttl parameter). In 4.9.11 this is a variable called 'label' >> and is being passed as '0', while in 4.4.0 it appears to be explicitly >> set to 0 directly within the udp_tunnel6_xmit_skb() function anyway. >> >> With that last change in effect, everything now compiles. (I have not >> tested anything, mind you.) >> >> Note that I did not come across any errors regarding the iov handling in >> msg_build() that you mentioned. Were you expecting compilation to fail >> there? Or were you expecting it to succeed, but the resulting TIPC >> functionality to simply be erroneous at run-time? >> >> Peter >> >> >> >> >> >> -----Original Message----- >> From: Butler, Peter >> Sent: February-23-17 2:48 PM >> To: Jon Maloy <jon.ma...@ericsson.com >> <mailto:jon.ma...@ericsson.com>>; >> tipc-discussion@lists.sourceforge.net >> <mailto:tipc-discussion@lists.sourceforge.net>; Parthasarathy >> Bhuvaragan <parthasarathy.bhuvara...@ericsson.com >> <mailto:parthasarathy.bhuvara...@ericsson.com>> >> Cc: Butler, Peter <pbut...@sonusnet.com <mailto:pbut...@sonusnet.com>> >> Subject: RE: TIPC Oops in tipc_sk_recv >> >> I have made the following change so as to work around the missing >> skwq_has_sleeper() function in our 4.4.0 kernel source stream (as >> required for the 4.9.11 TIPC source). This change was based on a >> comparison of 4.4.0 and 4.9.11 kernel code (include/net/sock.h and >> include/linux/wait.h). >> >> Change: >> if (skwq_has_sleeper(wq)) >> >> To be: >> if (wq && wq_has_sleeper(wq)) >> >> Let me know if that seems reasonable to you. >> >> With this change in effect, my compilation now proceeds further (see >> below). As always, any insight is much appreciated. >> >> CHK include/config/kernel.release >> CHK include/generated/uapi/linux/version.h >> CHK include/generated/utsrelease.h >> CHK include/generated/bounds.h >> CHK include/generated/timeconst.h >> CHK include/generated/asm-offsets.h >> CALL scripts/checksyscalls.sh >> LD net/tipc/built-in.o >> CC [M] net/tipc/addr.o >> CC [M] net/tipc/bcast.o >> CC [M] net/tipc/bearer.o >> CC [M] net/tipc/core.o >> CC [M] net/tipc/link.o >> CC [M] net/tipc/discover.o >> CC [M] net/tipc/msg.o >> CC [M] net/tipc/name_distr.o >> CC [M] net/tipc/subscr.o >> CC [M] net/tipc/monitor.o >> CC [M] net/tipc/name_table.o >> CC [M] net/tipc/net.o >> CC [M] net/tipc/netlink.o >> CC [M] net/tipc/netlink_compat.o >> CC [M] net/tipc/node.o >> CC [M] net/tipc/socket.o >> CC [M] net/tipc/eth_media.o >> CC [M] net/tipc/server.o >> CC [M] net/tipc/udp_media.o >> net/tipc/udp_media.c: In function 'tipc_udp_xmit': >> net/tipc/udp_media.c:199:9: error: too many arguments to function >> 'udp_tunnel6_xmit_skb' >> include/net/udp_tunnel.h:87:5: note: declared here >> make[1]: *** [net/tipc/udp_media.o] Error 1 >> make: *** [net/tipc/] Error 2 >> >> -----Original Message----- >> From: Butler, Peter >> Sent: February-23-17 2:14 PM >> To: Jon Maloy <jon.ma...@ericsson.com >> <mailto:jon.ma...@ericsson.com>>; >> tipc-discussion@lists.sourceforge.net >> <mailto:tipc-discussion@lists.sourceforge.net>; Parthasarathy >> Bhuvaragan <parthasarathy.bhuvara...@ericsson.com >> <mailto:parthasarathy.bhuvara...@ericsson.com>> >> Cc: Butler, Peter <pbut...@sonusnet.com <mailto:pbut...@sonusnet.com>> >> Subject: RE: TIPC Oops in tipc_sk_recv >> >> I have changed TIPC_DEF_MON_THRESHOLD (in core.h) from 32 to 100 as >> suggested. >> >> I still (of course) had to comment all functionality within >> __tipc_nl_add_monitor_peer() so as to get around the undefined >> nla_put_u64_64bit() function call. As such, >> __tipc_nl_add_monitor_peer() is now reduced to nothing more than a >> "return 0" statement. >> >> Note that I did not bother to similarly comment out other >> netlink-monitoring-related functions in monitor.c, since I assume that >> monitoring is now explicitly disabled (as per your suggestion to >> change >> TIPC_DEF_MON_THRESHOLD) - correct? >> >> As such my compilation now makes it this far (see below). I will look >> at this error but as always am open to (more enlightened) insight. >> >> CHK include/config/kernel.release >> CHK include/generated/uapi/linux/version.h >> CHK include/generated/utsrelease.h >> CHK include/generated/bounds.h >> CHK include/generated/timeconst.h >> CHK include/generated/asm-offsets.h >> CALL scripts/checksyscalls.sh >> LD net/tipc/built-in.o >> CC [M] net/tipc/addr.o >> CC [M] net/tipc/bcast.o >> CC [M] net/tipc/bearer.o >> CC [M] net/tipc/core.o >> CC [M] net/tipc/link.o >> CC [M] net/tipc/discover.o >> CC [M] net/tipc/msg.o >> CC [M] net/tipc/name_distr.o >> CC [M] net/tipc/subscr.o >> CC [M] net/tipc/monitor.o >> CC [M] net/tipc/name_table.o >> CC [M] net/tipc/net.o >> CC [M] net/tipc/netlink.o >> CC [M] net/tipc/netlink_compat.o >> CC [M] net/tipc/node.o >> CC [M] net/tipc/socket.o >> net/tipc/socket.c: In function 'tipc_write_space': >> net/tipc/socket.c:1492:2: error: implicit declaration of function >> 'skwq_has_sleeper' [-Werror=implicit-function-declaration] >> cc1: some warnings being treated as errors >> make[1]: *** [net/tipc/socket.o] Error 1 >> make: *** [net/tipc/] Error 2 >> >> -----Original Message----- >> From: Butler, Peter >> Sent: February-23-17 1:45 PM >> To: Jon Maloy <jon.ma...@ericsson.com >> <mailto:jon.ma...@ericsson.com>>; >> tipc-discussion@lists.sourceforge.net >> <mailto:tipc-discussion@lists.sourceforge.net>; Parthasarathy >> Bhuvaragan <parthasarathy.bhuvara...@ericsson.com >> <mailto:parthasarathy.bhuvara...@ericsson.com>> >> Cc: Butler, Peter <pbut...@sonusnet.com <mailto:pbut...@sonusnet.com>> >> Subject: RE: TIPC Oops in tipc_sk_recv >> >> I definitely don't want to be moving into dangerous waters, so I'll >> take your suggestion right now and start over.... >> >> -----Original Message----- >> From: Jon Maloy [mailto:jon.ma...@ericsson.com] >> Sent: February-23-17 1:43 PM >> To: Butler, Peter <pbut...@sonusnet.com >> <mailto:pbut...@sonusnet.com>>; tipc-discussion@lists.sourceforge.net >> <mailto:tipc-discussion@lists.sourceforge.net>; Parthasarathy >> Bhuvaragan <parthasarathy.bhuvara...@ericsson.com >> <mailto:parthasarathy.bhuvara...@ericsson.com>> >> Subject: RE: TIPC Oops in tipc_sk_recv >> >> >> >>> -----Original Message----- >>> From: Butler, Peter [mailto:pbut...@sonusnet.com] >>> Sent: Thursday, February 23, 2017 01:23 PM >>> To: Jon Maloy <jon.ma...@ericsson.com >>> <mailto:jon.ma...@ericsson.com>>; tipc- >>> discuss...@lists.sourceforge.net >> <mailto:discuss...@lists.sourceforge.net>; Parthasarathy Bhuvaragan >>> <parthasarathy.bhuvara...@ericsson.com >> <mailto:parthasarathy.bhuvara...@ericsson.com>> >>> Cc: Butler, Peter <pbut...@sonusnet.com >>> <mailto:pbut...@sonusnet.com>> >>> Subject: RE: TIPC Oops in tipc_sk_recv >>> >>> That might be a possibility - I know the customer is close to 32 >>> nodes however, so it might not be. >>> >>> I'm also looking at porting the required functionality from >>> include/net/netlink.h and lib/nlattr.c directly into the TIPC >>> monitor.c file (as opposed to changing any code directly in include/net and >>> lib/..... >> >> I think you are moving into dangerous waters here, unless you only >> want the code to compile. >> A simpler and safer option: change #define TIPC_DEF_MON_THRESHOLD in >> core.h from 32 to e.g. 100, and the hierarchical monitoring will be >> disabled. This is the way we have been running forever until 4.7, so >> this is a safe bet. >> >> //jon >> >>> >>> >>> >>> -----Original Message----- >>> From: Jon Maloy [mailto:jon.ma...@ericsson.com] >>> Sent: February-23-17 1:19 PM >>> To: Butler, Peter <pbut...@sonusnet.com >>> <mailto:pbut...@sonusnet.com>>; tipc- >>> discuss...@lists.sourceforge.net >> <mailto:discuss...@lists.sourceforge.net>; Parthasarathy Bhuvaragan >>> <parthasarathy.bhuvara...@ericsson.com >> <mailto:parthasarathy.bhuvara...@ericsson.com>> >>> Subject: RE: TIPC Oops in tipc_sk_recv >>> >>> >>> >>>> -----Original Message----- >>>> From: Butler, Peter [mailto:pbut...@sonusnet.com] >>>> Sent: Thursday, February 23, 2017 01:09 PM >>>> To: Jon Maloy <jon.ma...@ericsson.com >>>> <mailto:jon.ma...@ericsson.com>>; tipc- >>>> discuss...@lists.sourceforge.net >> <mailto:discuss...@lists.sourceforge.net>; Parthasarathy Bhuvaragan >>>> <parthasarathy.bhuvara...@ericsson.com >> <mailto:parthasarathy.bhuvara...@ericsson.com>> >>>> Cc: Butler, Peter <pbut...@sonusnet.com >>>> <mailto:pbut...@sonusnet.com>> >>>> Subject: RE: TIPC Oops in tipc_sk_recv >>>> >>>> Partha - an update for you >>>> >>>> I've ported all the TIPC code from 4.9.11 into our 4.4.0 kernel >>>> code base. By this I mean I have completely removed all the >>>> existing TIPC files in their entirety from: >>>> >>>> include/uapi/linux/tipc* >>>> net/tipc/* >>>> >>>> in our 4.4.0 kernel source tree, and replaced these with all the >>>> files from 4.9.11. >>>> >>>> As Jon indeed forewarned me, there will be a hurdle or two to >>>> integrate this with the 4.4.0 kernel's internal API. As it stands >>>> this is where the compilation first fails. I can certainly look >>>> into this myself >>> but am told you are the expert. >>>> (I am far from a kernel expert myself.) >>>> >>>> LD net/tipc/built-in.o >>>> CC [M] net/tipc/addr.o >>>> CC [M] net/tipc/bcast.o >>>> CC [M] net/tipc/bearer.o >>>> CC [M] net/tipc/core.o >>>> CC [M] net/tipc/link.o >>>> CC [M] net/tipc/discover.o >>>> CC [M] net/tipc/msg.o >>>> CC [M] net/tipc/name_distr.o >>>> CC [M] net/tipc/subscr.o >>>> CC [M] net/tipc/monitor.o >>>> net/tipc/monitor.c: In function '__tipc_nl_add_monitor_peer': >>> >>> Unless you are running a cluster > 32 nodes and need the hierarchical >>> neighbor monitoring feature, you can just comment out the contents of >>> this function and other monitor-related netlink function. >>> >>> ///jon >>> >>>> net/tipc/monitor.c:707:3: error: implicit declaration of function >>>> 'nla_put_u64_64bit' [-Werror=implicit-function-declaration] >>>> cc1: some warnings being treated as errors >>>> make[2]: *** [net/tipc/monitor.o] Error 1 >>>> make[1]: *** [net/tipc] Error 2 >>>> make: *** [net] Error 2 >>>> >>>> >>>> >>>> -----Original Message----- >>>> From: Butler, Peter >>>> Sent: February-23-17 10:56 AM >>>> To: Jon Maloy <jon.ma...@ericsson.com >>>> <mailto:jon.ma...@ericsson.com>>; tipc- >>>> discuss...@lists.sourceforge.net >> <mailto:discuss...@lists.sourceforge.net>; Parthasarathy Bhuvaragan >>>> <parthasarathy.bhuvara...@ericsson.com >> <mailto:parthasarathy.bhuvara...@ericsson.com>> >>>> Cc: Butler, Peter <pbut...@sonusnet.com >>>> <mailto:pbut...@sonusnet.com>> >>>> Subject: RE: TIPC Oops in tipc_sk_recv >>>> >>>> Hi Partha, >>>> >>>> I'll give you the short version here to save you the time of >>>> reading this entire thread. >>>> >>>> Basically I need to port the latest and greatest TIPC code (i.e. >>>> from the latest longterm kernel release, namely 4.9.11) into a >>>> 4.4.0 kernel source base. (I know that sounds ugly but it's for an >>>> emergency quick-fix and upgrading the entire kernel is not an >>>> option at this >>>> time...) >>>> >>>> Jon has said this is entirely doable but that you are the expert, >>>> and that there will be at least one minor hurdle in doing so, >>>> namely in iov handling in msg_build(). >>>> >>>> Thanks, >>>> >>>> Peter >>>> >>>> >>>> >>>> -----Original Message----- >>>> From: Jon Maloy [mailto:jon.ma...@ericsson.com] >>>> Sent: February-23-17 10:45 AM >>>> To: Butler, Peter <pbut...@sonusnet.com >>>> <mailto:pbut...@sonusnet.com>>; tipc- >>>> discuss...@lists.sourceforge.net >> <mailto:discuss...@lists.sourceforge.net>; Parthasarathy Bhuvaragan >>>> <parthasarathy.bhuvara...@ericsson.com >> <mailto:parthasarathy.bhuvara...@ericsson.com>> >>>> Subject: RE: TIPC Oops in tipc_sk_recv >>>> >>>> >>>> >>>>> -----Original Message----- >>>>> From: Butler, Peter [mailto:pbut...@sonusnet.com] >>>>> Sent: Thursday, February 23, 2017 10:25 AM >>>>> To: Jon Maloy <jon.ma...@ericsson.com >>>>> <mailto:jon.ma...@ericsson.com>>; tipc- >>>>> discuss...@lists.sourceforge.net >>>>> <mailto:discuss...@lists.sourceforge.net> >>>>> Cc: Butler, Peter <pbut...@sonusnet.com >>>>> <mailto:pbut...@sonusnet.com>> >>>>> Subject: RE: TIPC Oops in tipc_sk_recv >>>>> >>>>> Hi Jon, >>>>> >>>>> Thanks for the info. The solution we are considering (to give >>>>> the customer an emergency patch) is backport the TIPC code from >>>>> kernel >>>>> 4.4.50 into our 4.4.0 kernel source tree. From what I can see, I >>>>> should be able to do so with little effort. I am assuming (?) >>>>> that since 4.4.x is a longterm kernel release that the >>>>> 4.4.50 TIPC code is considered stable and devoid of the original >>>>> bug associated with this section of code in tipc_sk_rcv() - am I >>>>> wrong to assume that? >>>> >>>> Unfortunately yes. The only safe solution to the deadlock problem >>>> is the one you find in later versions. >>>> The patch fixing this particular problem hasn't been applied this >>>> far back, probably because it didn't apply cleanly. >>>> >>>>> The section of code in question is entirely different in 4.4.50 >>>>> than what we currently have: >>>>> >>>>> if (likely(tsk)) { >>>>> sk = &tsk->sk; >>>>> if (likely(spin_trylock_bh(&sk->sk_lock.slock))) { >>>>> tipc_sk_enqueue(inputq, sk, dport); >>>>> spin_unlock_bh(&sk->sk_lock.slock); >>>>> } >>>>> sock_put(sk); >>>>> continue; >>>>> } >>>>> >>>>> Does this mean that the 4.4.50 version (as shown above) is still >>>>> susceptible to the original bug? (Our original O/S maintainer >>>>> patched this section because of the original bug that was causing >>>>> an oops there - but obviously the patch he implemented was also >>>>> buggy, as previously discussed.) >>>>> >>>>> Ultimately we would rather upgrade our entire kernel (say, to >>>>> 4.9.11 >>>>> - the latest and greatest longterm release) but I see the TIPC >>>>> design has changed significantly and I'm not sure if it would >>>>> backport into our 4.4.0 kernel without significant effort; i.e. >>>>> perhaps this change in design also depends on other API changes >>>>> within other layers of the kernel. If I am wrong in this and you >>>>> think that the 4.9.11 TIPC code should be able to be backported >>>>> to our 4.4.0 base then I will do so, >>>> >>>> It is absolutely doable. As a matter of fact, this is what Partha >>>> has been doing in one of our own product lines. >>>> AFAIK, the only build issue you will encounter is a change to the >>>> iov handling in msg_build(), and that is easily fixed by reverting >>>> to the old >>> method. >>>> (Correct me Partha, if I am wrong here). But, with new >>>> functionality (e.g., new flow control) there are new issues which >>>> still haven't been ironed out completely. I think Partha is the one >>>> to give a better update >>> here. >>>> >>>> ///jon >>>> >>>>> as there are far more fixes in 4.9.11 than in 4.4.50. The reason >>>>> we can't upgrade the entire kernel to 4.4.50 or 4.9.11 in the >>>>> short term is a bit of a long story (which I will spare you), but >>>>> suffice it to say that that is only an option for a long-term fix >>>>> for our customers and not for this short term emergency fix which >>>>> we need >>> released asap. >>>>> >>>>> All this to say, the goal here is to move to the latest possible >>>>> TIPC code which will (relatively) seamlessly integrate with our >>>>> 4.4.0 kernel, and also be free of the aforementioned bug. Let me >>>>> know what >>>> you think. >>>>> >>>>> Thanks, >>>>> >>>>> Peter >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: Jon Maloy [mailto:jon.ma...@ericsson.com] >>>>> Sent: February-23-17 8:22 AM >>>>> To: Butler, Peter <pbut...@sonusnet.com >>>>> <mailto:pbut...@sonusnet.com>>; tipc- >>>>> discuss...@lists.sourceforge.net >>>>> <mailto:discuss...@lists.sourceforge.net> >>>>> Subject: RE: TIPC Oops in tipc_sk_recv >>>>> >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: Butler, Peter [mailto:pbut...@sonusnet.com] >>>>>> Sent: Wednesday, February 22, 2017 04:31 PM >>>>>> To: Jon Maloy <jon.ma...@ericsson.com >>>>>> <mailto:jon.ma...@ericsson.com>>; tipc- >>>>>> discuss...@lists.sourceforge.net >>>>>> <mailto:discuss...@lists.sourceforge.net> >>>>>> Cc: Butler, Peter <pbut...@sonusnet.com >>>>>> <mailto:pbut...@sonusnet.com>> >>>>>> Subject: RE: TIPC Oops in tipc_sk_recv >>>>>> >>>>>> Hi Jon, >>>>>> >>>>>> I think I found the problem, which ultimately may only exist on >>>>>> our end (see below for an explanation, and let me know if you agree). >>>>>> >>>>>> The fellow that was maintaining our O/S previously (no longer >>>>>> with the >>>>>> company) had made some patches to the 4.4.0 kernel TIPC code, >>>>>> and indeed one of them is in the offending tipc_sk_rcv() function. >>>>>> >>>>>> Specifically, note this segment of code from our kernel source tree: >>>>>> >>>>>> /* Send pending response/rejected messages, if >>>>>> any */ >>>>>> while (!skb_queue_empty(&sk->sk_write_queue)) { >>>>>> skb = skb_dequeue(&sk->sk_write_queue); >>>>>> dnode = msg_destnode(buf_msg(skb)); >>>>>> tipc_node_xmit_skb(net, skb, dnode, >>>>>> dport); >>>>>> } >>>>> >>>>> Yes, this is wrong. The socket write queue is only used for >>>>> outgoing regular messages (Partha has later changed that), and >>>>> should only be emptied by the sending thread. Running this code >>>>> in interrupt context will give exactly the symptom you see, >>>>> because the writing thread might already have freed or sent the buffer in >>>>> question. >>>>>> >>>>>> Whereas the latest and greatest official longterm 4.9.11 kernel has: >>>>>> >>>>>> /* Send pending response/rejected messages, if any */ >>>>>> while ((skb = __skb_dequeue(&xmitq))) { >>>>>> dnode = msg_destnode(buf_msg(skb)); >>>>>> tipc_node_xmit_skb(net, skb, dnode, dport); >>>>>> } >>>>>> >>>>>> The code path that triggers the oops (in our source code) is from: >>>>>> >>>>>> dnode = msg_destnode(buf_msg(skb)); >>>>>> >>>>>> where msg_destnode() calls msg_word() which calls: >>>>>> >>>>>> ntohl(m->hdr[pos]); >>>>>> >>>>>> which is precisely where the oops occurred. >>>>>> >>>>>> I'm not exactly sure where he got that code change - my guess >>>>>> is he posted a question on the tipc-discussion list and got a >>>>>> suggestion to try a code snippet, but in the end the actual >>>>>> changes (that were officially released at kernel.org) differed, >>>>>> as per >>> above. >>>>> >>>>> I rather suspect he might have looked at the more recent code and >>>>> tried to do the same, while misunderstanding the role of the >>>>> write >>> queue. >>>>> >>>>>> Indeed, on Google I can see some threads discussing a 'deadly >>> embrace' >>>>>> deadlock (for example >>>>>> http://www.spinics.net/lists/netdev/msg382379.html) between >>>>>> yourself and him. Another possibility is that the offending >>>>>> source code in question was indeed released sometime after >>>>>> 4.4.0, but has since modified/fixed, thus explaining the discrepancy. >>>>> >>>>> The loop was introduced in conjunction with that discussion, but >>>>> it should not be done in the way it is done above. Indeed, I >>>>> cannot see that this can have solved the "deadly embrace" problem >>>>> at all, unless he made other changes and added the >>>>> rejected/returned messages to the write queue. That might work >>>>> most of the time, but will still sooner or later interfere with a sending >>>>> thread. >>>>> >>>>> There are two ways you can solve this: >>>>> 1: Introduce a stack based queue for reject/return messages, as >>>>> we do, and pass it along in the calls. >>>>> 2: Put send messages on a stack based queue, as Partha has done >>>>> in the later versions. This assuming that the rejected messages >>>>> are added to the write queue, as I am speculating above. >>>>> >>>>> BR >>>>> ///jon >>>>> >>>>>> >>>>>> If either of possibilities is what actually happened, then this >>>>>> may not a bug you need to worry about. Granted, the same >>>>>> msg_destnode() call still exists in the current (4.9.11 and >>>>>> 4.10) code, but the semantics of the encapsulating while loop >>>>>> are different, and maybe as such >>>>> that eliminates the issue. >>>>>> Thoughts? >>>>>> >>>>>> Peter >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Jon Maloy [mailto:jon.ma...@ericsson.com] >>>>>> Sent: February-22-17 3:01 PM >>>>>> To: Butler, Peter <pbut...@sonusnet.com >>>>>> <mailto:pbut...@sonusnet.com>>; tipc- >>>>>> discuss...@lists.sourceforge.net >>>>>> <mailto:discuss...@lists.sourceforge.net> >>>>>> Subject: RE: TIPC Oops in tipc_sk_recv >>>>>> >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Butler, Peter [mailto:pbut...@sonusnet.com] >>>>>>> Sent: Wednesday, February 22, 2017 02:15 PM >>>>>>> To: Jon Maloy <jon.ma...@ericsson.com >>>>>>> <mailto:jon.ma...@ericsson.com>>; tipc- >>>>>>> discuss...@lists.sourceforge.net >>>>>>> <mailto:discuss...@lists.sourceforge.net> >>>>>>> Cc: Butler, Peter <pbut...@sonusnet.com >>>>>>> <mailto:pbut...@sonusnet.com>> >>>>>>> Subject: RE: TIPC Oops in tipc_sk_recv >>>>>>> >>>>>>> For the " Source file is more recent than executable" >>>>>>> message, could this simply be due to the fact that I copied >>>>>>> the kernel source to the lab and then ran the gdb commands as >>>>>>> shown? As such, the newly copied files would have a newer >>>>>>> timestamp than the >>>> kernel/tipc.ko files. >>>>>>> (The kernel is actual built on a separate compiler than the >>>>>>> test lab >>>>>>> machine.) >>>>>> >>>>>> If you are certain that the build was made from the same source >>>>>> this is false alarm, caused by the timestamp as you suggest. >>>>>> >>>>>> ///jon >>>>>> >>>>>>> >>>>>>> Or could I get that message for another reason? >>>>>>> >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Jon Maloy [mailto:jon.ma...@ericsson.com] >>>>>>> Sent: February-22-17 2:11 PM >>>>>>> To: Butler, Peter <pbut...@sonusnet.com >>>>>>> <mailto:pbut...@sonusnet.com>>; tipc- >>>>>>> discuss...@lists.sourceforge.net >>>>>>> <mailto:discuss...@lists.sourceforge.net> >>>>>>> Subject: RE: TIPC Oops in tipc_sk_recv >>>>>>> >>>>>>> >>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Butler, Peter [mailto:pbut...@sonusnet.com] >>>>>>>> Sent: Wednesday, February 22, 2017 01:04 PM >>>>>>>> To: Jon Maloy <jon.ma...@ericsson.com >>>>>>>> <mailto:jon.ma...@ericsson.com>>; tipc- >>>>>>>> discuss...@lists.sourceforge.net >>>>>>>> <mailto:discuss...@lists.sourceforge.net> >>>>>>>> Cc: Butler, Peter <pbut...@sonusnet.com >>>>>>>> <mailto:pbut...@sonusnet.com>> >>>>>>>> Subject: RE: TIPC Oops in tipc_sk_recv >>>>>>>> >>>>>>>> I took a stab at it this way - not sure if I am doing this >>>>>>>> correctly or >>> not. >>>>>>>> >>>>>>>> [root@myVMslot12 ~]# gdb /boot/vmlinuz-4.4.0 /proc/kcore >>>>>>>> GNU >>>> gdb >>>>>>>> (GDB) Fedora (7.3.50.20110722-13.fc16) Copyright (C) 2011 >>>>>>>> Free Software Foundation, Inc. >>>>>>>> License GPLv3+: GNU GPL version 3 or later >>>>>>>> <http://gnu.org/licenses/gpl.html> >>>>>>>> This is free software: you are free to change and redistribute it. >>>>>>>> There is NO WARRANTY, to the extent permitted by law. Type >>>>>>>> "show copying" >>>>>>>> and "show warranty" for details. >>>>>>>> This GDB was configured as "x86_64-redhat-linux-gnu". >>>>>>>> For bug reporting instructions, please see: >>>>>>>> <http://www.gnu.org/software/gdb/bugs/>... >>>>>>>> BFD: /boot/vmlinuz-4.4.0: Warning: Ignoring section flag >>>>>>>> IMAGE_SCN_MEM_NOT_PAGED in section .bss >>>>>>>> BFD: /boot/vmlinuz-4.4.0: Warning: Ignoring section flag >>>>>>>> IMAGE_SCN_MEM_NOT_PAGED in section .bss Reading symbols >>>> from >>>>>>>> /boot/vmlinuz-4.4.0...(no debugging symbols found)...done. >>>>>>>> >>>>>>>> warning: core file may not match specified executable file. >>>>>>>> [New process 1] >>>>>>>> Core was generated by `BOOT_IMAGE=/vmlinuz-4.4.0 >>>>>>> root=UUID=b419f9ff- >>>>>>>> 80ce-459e-855c-614d86a48105 ro rd.'. >>>>>>>> #0 0x0000000000000000 in ?? () >>>>>>>> (gdb) file /lib/modules/4.4.0/kernel/net/tipc/tipc.ko >>>>>>>> warning: core file may not match specified executable file. >>>>>>>> Reading symbols from >>>>> /lib/modules/4.4.0/kernel/net/tipc/tipc.ko...done. >>>>>>>> (gdb) list *(tipc_sk_rcv+0x238) >>>>>>>> 0x14898 is in tipc_sk_rcv (net/tipc/msg.h:131). >>>>>>>> warning: Source file is more recent than executable. >>>>>>> >>>>>>> Seems like you didn't rebuild after you updated the source file? >>>>>>> Try again just to make sure. >>>>>>> >>>>>>>> 126 return (struct tipc_msg *)skb->data; >>>>>>>> 127 } >>>>>>>> 128 >>>>>>>> 129 static inline u32 msg_word(struct tipc_msg *m, u32 pos) >>>>>>>> 130 { >>>>>>>> 131 return ntohl(m->hdr[pos]); >>>>>>> >>>>>>> If this is correct, you are receiving a corrupt buffer where >>>>>>> the data pointer is invalid. This is typical if the buffer >>>>>>> already has been >>>>> released. >>>>>>> >>>>>>> ///jon >>>>>>> >>>>>>>> 132 } >>>>>>>> 133 >>>>>>>> 134 static inline void msg_set_word(struct tipc_msg *m, u32 w, >>> u32 >>>>> val) >>>>>>>> 135 { >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Butler, Peter >>>>>>>> Sent: February-22-17 12:45 PM >>>>>>>> To: Jon Maloy <jon.ma...@ericsson.com >>>>>>>> <mailto:jon.ma...@ericsson.com>>; tipc- >>>>>>>> discuss...@lists.sourceforge.net >>>>>>>> <mailto:discuss...@lists.sourceforge.net> >>>>>>>> Cc: Butler, Peter <pbut...@sonusnet.com >>>>>>>> <mailto:pbut...@sonusnet.com>> >>>>>>>> Subject: RE: TIPC Oops in tipc_sk_recv >>>>>>>> >>>>>>>> Hi Jon >>>>>>>> >>>>>>>> Thanks for the info. >>>>>>>> >>>>>>>> One thing I should clarify. Although we are running the >>>>>>>> 4.4.0 kernel, we had backported a number of post-4.4.0 TIPC >>>>>>>> patches into our 4.4.0 kernel. As such, the offset in >>>>>>>> question >>>>>>>> (tipc_sk_rcv+0x238) will not match that in the vanilla 4.4.0 source. >>>>>>>> >>>>>>>> Should I post the entire socket.c file to this list for your review? >>>>>>>> Or is there an easy way for me to do a similar listing >>>>>>>> using our actual tipc.ko file here in the lab? >>>>>>>> >>>>>>>> Peter >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Jon Maloy [mailto:jon.ma...@ericsson.com] >>>>>>>> Sent: February-22-17 12:29 PM >>>>>>>> To: Butler, Peter <pbut...@sonusnet.com >>>>>>>> <mailto:pbut...@sonusnet.com>>; tipc- >>>>>>>> discuss...@lists.sourceforge.net >>>>>>>> <mailto:discuss...@lists.sourceforge.net> >>>>>>>> Subject: RE: TIPC Oops in tipc_sk_recv >>>>>>>> >>>>>>>> Hi Peter, >>>>>>>> Very hard to make any suggestions on how to reproduce this. >>>>>>>> What I can see is that it is a STREAM message being sent >>>>>>>> from a node local socket, i.e., it doesn't go via any interface. >>>>>>>> The crash seems to happen when the receiving socket is >>>>>>>> owned by the user, and while we are instead adding the >>>>>>>> message to the >>> backlog queue: >>>>>>>> >>>>>>>> Reading symbols from net/tipc/tipc.ko...done. >>>>>>>> (gdb) list *(tipc_sk_rcv+0x238) >>>>>>>> 0x13d78 is in tipc_sk_rcv (./arch/x86/include/asm/atomic.h:214). >>>>>>>> 209 static __always_inline int __atomic_add_unless(atomic_t *v, >>> int >>>>> a, >>>>>> int >>>>>>>> u) >>>>>>>> 210 { >>>>>>>> 211 int c, old; >>>>>>>> 212 c = atomic_read(v); >>>>>>>> 213 for (;;) { >>>>>>>> 214 if (unlikely(c == (u))) >>>>>>>> 215 break; >>>>>>>> 216 old = atomic_cmpxchg((v), c, c + (a)); >>>>>>>> 217 if (likely(old == c)) >>>>>>>> 218 break; >>>>>>>> >>>>>>>> This is about what I can get out of it at the moment. Maybe >>>>>>>> you should try a high-load test between two local sockets >>>>>>>> (try the benchmark demo from >>>>>>>> tipcutils) and see what you can achieve. >>>>>>>> >>>>>>>> BR >>>>>>>> ///jon >>>>>>>> >>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Butler, Peter [mailto:pbut...@sonusnet.com] >>>>>>>>> Sent: Wednesday, February 22, 2017 10:40 AM >>>>>>>>> To: Jon Maloy <jon.ma...@ericsson.com >>>>>>>>> <mailto:jon.ma...@ericsson.com>>; tipc- >>>>>>>>> discuss...@lists.sourceforge.net >>>>>>>>> <mailto:discuss...@lists.sourceforge.net> >>>>>>>>> Cc: Butler, Peter <pbut...@sonusnet.com >>>>>>>>> <mailto:pbut...@sonusnet.com>> >>>>>>>>> Subject: RE: TIPC Oops in tipc_sk_recv >>>>>>>>> >>>>>>>>> If you have any suggestions as to procedures/tricks you >>>>>>>>> think might trigger this bug I can certainly attempt to >>>>>>>>> do so in the >>> lab. >>>>>>>>> Obviously we can't attempt to reproduce it on the >>>>>>>>> customer's >>>>>>>>> (live) >>>>>>> system. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Butler, Peter >>>>>>>>> Sent: February-21-17 3:39 PM >>>>>>>>> To: Jon Maloy <jon.ma...@ericsson.com >>>>>>>>> <mailto:jon.ma...@ericsson.com>>; tipc- >>>>>>>>> discuss...@lists.sourceforge.net >>>>>>>>> <mailto:discuss...@lists.sourceforge.net> >>>>>>>>> Cc: Butler, Peter <pbut...@sonusnet.com >>>>>>>>> <mailto:pbut...@sonusnet.com>> >>>>>>>>> Subject: RE: TIPC Oops in tipc_sk_recv >>>>>>>>> >>>>>>>>> Unfortunately this occurred on a customer system so it is >>>>>>>>> not readily reproducible. We have not seen this occur in our lab. >>>>>>>>> >>>>>>>>> For what it's worth, it occurred while the process was in >>>>>>>>> TASK_UNINTERRUPTIBLE. As such, the kernel could not >>>>>>>>> actually kill off the associated process despite the >>>>>>>>> Oops, and the process remained forever frozen in the 'D' >>>>>>>>> state and the card had to be >>>>>> rebooted. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Jon Maloy [mailto:jon.ma...@ericsson.com] >>>>>>>>> Sent: February-21-17 3:36 PM >>>>>>>>> To: Butler, Peter <pbut...@sonusnet.com >>>>>>>>> <mailto:pbut...@sonusnet.com>>; tipc- >>>>>>>>> discuss...@lists.sourceforge.net >>>>>>>>> <mailto:discuss...@lists.sourceforge.net> >>>>>>>>> Subject: RE: TIPC Oops in tipc_sk_recv >>>>>>>>> >>>>>>>>> Hi Peter, >>>>>>>>> I don't think this is any known bug. Is it repeatable? >>>>>>>>> >>>>>>>>> ///jon >>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Butler, Peter [mailto:pbut...@sonusnet.com] >>>>>>>>>> Sent: Tuesday, February 21, 2017 12:14 PM >>>>>>>>>> To: tipc-discussion@lists.sourceforge.net >> <mailto:tipc-discussion@lists.sourceforge.net> >>>>>>>>>> Cc: Butler, Peter <pbut...@sonusnet.com >>>>>>>>>> <mailto:pbut...@sonusnet.com>> >>>>>>>>>> Subject: [tipc-discussion] TIPC Oops in tipc_sk_recv >>>>>>>>>> >>>>>>>>>> This was with kernel 4.4.0, however I don't see any fix >>>>>>>>>> specifically related to this in any subsequent 4.4.x kernel... >>>>>>>>>> >>>>>>>>>> BUG: unable to handle kernel NULL pointer dereference >>>>>>>>>> at >>>>>>>>>> 00000000000000d8 >>>>>>>>>> IP: [<ffffffffa0148868>] tipc_sk_rcv+0x238/0x4d0 [tipc] >>>>>>>>>> PGD >>>>>>>>>> 34f4c0067 PUD >>>>>>>>>> 34ed95067 PMD 0 >>>>>>>>>> Oops: 0000 [#1] SMP >>>>>>>>>> Modules linked in: nf_log_ipv4 nf_log_common xt_LOG >>>>>>>>>> sctp libcrc32c e1000e tipc udp_tunnel ip6_udp_tunnel >>>>>>>>>> iTCO_wdt 8021q garp >>>>>>>> xt_physdev >>>>>>>>>> br_netfilter bridge stp llc nf_conntrack_ipv4 >>>>>>>>>> ipmiq_drv(O) >>>>>>>>>> nf_defrag_ipv4 >>>>>>>>>> sio_mmc(O) ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 >>>>>>>>>> nf_defrag_ipv6 xt_state nf_conntrack event_drv(O) >>>>>>>>>> ip6table_filter lockd ip6_tables >>>>>>>>>> pt_timer_info(O) ddi(O) grace usb_storage ixgbe igb >>>>>>>>>> iTCO_vendor_support i2c_algo_bit ptp i2c_i801 pps_core >>>>>>>>>> lpc_ich i2c_core intel_ips mfd_core pcspkr ioatdma >>>>>>>>>> sunrpc dca tpm_tis mdio tpm >>>>>>>>> [last unloaded: iTCO_wdt] >>>>>>>>>> CPU: 2 PID: 12144 Comm: dinamo Tainted: G O 4.4.0 #23 >>>>>>>>>> Hardware name: PT AMC124/Base Board Product Name, BIOS >>>>>>>>>> LGNAJFIP.PTI.0012.P15 01/15/2014 >>>>>>>>>> task: ffff880036ad8000 ti: ffff880036900000 task.ti: >>>>>>>>>> ffff880036900000 >>>>>>>>>> RIP: 0010:[<ffffffffa0148868>] [<ffffffffa0148868>] >>>>>>>>>> tipc_sk_rcv+0x238/0x4d0 [tipc] >>>>>>>>>> RSP: 0018:ffff880036903bb8 EFLAGS: 00010292 >>>>>>>>>> RAX: 0000000000000000 RBX: ffff88034def3970 RCX: >>>>>>>>>> 0000000000000001 >>>>>>>>>> RDX: 0000000000000101 RSI: 0000000000000292 RDI: >>>>>>>>>> ffff88034def3984 >>>>>>>>>> RBP: ffff880036903c28 R08: 0000000000000101 R09: >>>>>>>>>> 0000000000000004 >>>>>>>>>> R10: 0000000000000001 R11: 0000000000000000 R12: >>>>>>>>>> ffff880036903d28 >>>>>>>>>> R13: 00000000bd1fd8b2 R14: ffff88034def3840 R15: >>>>>>>>>> ffff880036903d3c >>>>>>>>>> FS: 00007f1e86299740(0000) GS:ffff88035fc40000(0000) >>>>>>>>>> knlGS:0000000000000000 >>>>>>>>>> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>>>>>>>>> CR2: 00000000000000d8 CR3: 0000000036835000 CR4: >>>>>>>>>> 00000000000006e0 >>>>>>>>>> Stack: >>>>>>>>>> 000000000000009b ffff880036903d28 0000000000000018 >>>>>>>>>> ffff88034def38c8 >>>>>>>>>> ffffffff81ce6240 ffff8802b9bdba00 ffff880036903ca8 >>>>>>>>>> ffffffffa013bd7e >>>>>>>>>> ffff8802b99d5ee8 ffff880036903c60 0000000000000000 >>>>>>>>>> ffff88003693cb00 Call >>>>>>>>>> Trace: >>>>>>>>>> [<ffffffffa013bd7e>] ? tipc_msg_build+0xde/0x4f0 >>>>>>>>>> [tipc] [<ffffffffa014358f>] tipc_node_xmit+0x11f/0x150 >>>>>>>>>> [tipc] [<ffffffffa01470ba>] >>>>>>>>>> __tipc_send_stream+0x16a/0x300 [tipc] [<ffffffff81625eb5>] ? >>>>>>>>>> tcp_sendmsg+0x4d5/0xb00 [<ffffffffa0147292>] >>>>>>>>>> tipc_send_stream+0x42/0x70 [tipc] [<ffffffff815bcf77>] >>>>>>>>>> sock_sendmsg+0x47/0x50 [<ffffffff815bd03f>] >>>>>>>>>> sock_write_iter+0x7f/0xd0 [<ffffffff811d799a>] >>>>>>>>>> __vfs_write+0xaa/0xe0 [<ffffffff811d8b16>] >>>>>>>>>> vfs_write+0xb6/0x1a0 [<ffffffff811d8e3f>] >>>>>>>>>> SyS_write+0x4f/0xb0 [<ffffffff816de6d7>] >>>>>>>>>> entry_SYSCALL_64_fastpath+0x12/0x6a >>>>>>>>>> Code: 89 de 4c 89 f7 e8 29 d3 ff ff 48 8b 7d a8 e8 60 >>>>>>>>>> 59 >>>>>>>>>> 59 >>>>>>>>>> e1 >>>>>>>>>> 49 8d 9e 30 01 00 >>>>>>>>>> 00 49 3b 9e 30 01 00 00 74 30 48 89 df e8 b8 b6 47 e1 >>>>>>>>>> <48> 8b >>>>>>>>>> 90 >>>>>>>>>> d8 >>>>>>>>>> 00 >>>>>>>>>> 00 00 48 8b 7d b0 44 89 e9 48 89 c6 48 89 45 c0 RIP >>>>>>>>>> [<ffffffffa0148868>] >>>>>>>>>> tipc_sk_rcv+0x238/0x4d0 [tipc] RSP <ffff880036903bb8> >>>>>>>>>> CR2: 00000000000000d8 >>>>>>>>>> ---[ end trace 1c2d69738941d565 ]--- >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ------------------------------------------------------- >>>>>>>>>> - >>>>>>>>>> -- >>>>>>>>>> -- >>>>>>>>>> -- >>>>>>>>>> -- >>>>>>>>>> -- >>>>>>>>>> -- >>>>>>>>>> -- >>>>>>>>>> -------- Check out the vibrant tech community on one of >>>>>>>>>> the world's most engaging tech sites, SlashDot.org! >>>>>>>>>> http://sdm.link/slashdot >>>>>>>>>> _______________________________________________ >>>>>>>>>> tipc-discussion mailing list >>>>>>>>>> tipc-discussion@lists.sourceforge.net >> <mailto:tipc-discussion@lists.sourceforge.net> >>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/tipc-discu >>>>>>>>>> s >>>>>>>>>> si >>>>>>>>>> on >> >
------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot _______________________________________________ tipc-discussion mailing list tipc-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/tipc-discussion
