date:20160428

Re: [tipc-discussion] [PATCH] tipc: Only process unicast on intended node

2016-04-28 Thread Jon Maloy

(Removed netdev from list, added some others).

This is interesting, and must it have been hard to track.  But I would really 
like to know the real reason why this happens, so we can catch the root 
problem. Broadcast ACK messages are just ordinary STATE messages, and should 
have a correct destination address. Did you find out where these messages 
really came from, and why they have wrong destination addresses?

///jon

> -Original Message-
> From: Hamish Martin [mailto:hamish.mar...@alliedtelesis.co.nz]
> Sent: Thursday, 28 April, 2016 21:35
> To: Jon Maloy; net...@vger.kernel.org
> Cc: Hamish Martin
> Subject: [PATCH] tipc: Only process unicast on intended node
> 
> We have observed complete lock up of broadcast-link transmission due to
> unacknowledged packets never being removed from the 'transmq' queue. This
> is traced to nodes having their ack field set beyond the sequence number
> of packets that have actually been transmitted to them.
> Consider an example where node 1 has sent 10 packets to node 2 on a
> link and node 3 has sent 20 packets to node 2 on another link. We
> see examples of an ack from node 2 destined for node 3 being treated as
> an ack from node 2 at node 1. This leads to the ack on the node 1 to node
> 2 link being increased to 20 even though we have only sent 10 packets.
> When node 1 does get around to sending further packets, none of the
> packets with sequence numbers less than 21 are actually removed from the
> transmq.
> To resolve this we reinstate some code lost in commit d999297c3dbb ("tipc:
> reduce locking scope during packet reception") which ensures that only
> messages destined for the receiving node are processed by that node. This
> prevents the sequence numbers from getting out of sync and resolves the
> packet leakage, thereby resolving the broadcast-link transmission
> lock-ups we observed.
> 
> Signed-off-by: Hamish Martin 
> Reviewed-by: Chris Packham 
> Reviewed-by: John Thompson 
> ---
>  net/tipc/node.c | 5 +
>  1 file changed, 5 insertions(+)
> 
> diff --git a/net/tipc/node.c b/net/tipc/node.c
> index ace178fd3850..e5dda495d4b6 100644
> --- a/net/tipc/node.c
> +++ b/net/tipc/node.c
> @@ -1460,6 +1460,11 @@ void tipc_rcv(struct net *net, struct sk_buff *skb,
> struct tipc_bearer *b)
>   return tipc_node_bc_rcv(net, skb, bearer_id);
>   }
> 
> + /* Discard unicast link messages destined for another node */
> + if (unlikely(!msg_short(hdr) &&
> +  (msg_destnode(hdr) != tipc_own_addr(net
> + goto discard;
> +
>   /* Locate neighboring node that sent packet */
>   n = tipc_node_find(net, msg_prevnode(hdr));
>   if (unlikely(!n))
> --
> 2.8.1


--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
___
tipc-discussion mailing list
tipc-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tipc-discussion

Re: [tipc-discussion] Tipc: name table mismatch between different cards in a system

2016-04-28 Thread GUNA

Thanks Jon. I already applied this patch on currently running module.

On Thursday, April 28, 2016, Jon Maloy  wrote:

> Here it is. (This is just pasted into Outlook, so don’t try to apply it)
>
> If you manually add the two skb_linearize() calls and the update of ‘hdr’
> you should be safe.
>
>
>
> Good luck!
>
> ///jon
>
>
>
>
>
> commit c7cad0d6f70cd4ce8644ffe528a4df1cdc2e77f5
>
> Author: Jon Paul Maloy  >
>
> Date:   Thu Nov 19 14:30:40 2015 -0500
>
>
>
> tipc: move linearization of buffers to generic code
>
>
>
> In commit 5cbb28a4bf65c7e4 ("tipc: linearize arriving NAME_DISTR
>
> and LINK_PROTO buffers") we added linearization of NAME_DISTRIBUTOR,
>
> LINK_PROTOCOL/RESET and LINK_PROTOCOL/ACTIVATE to the function
>
> tipc_udp_recv(). The location of the change was selected in order
>
> to make the commit easily appliable to 'net' and 'stable'.
>
>
>
> We now move this linearization to where it should be done, in the
>
> functions tipc_named_rcv() and tipc_link_proto_rcv() respectively.
>
>
>
> Reviewed-by: Ying Xue  >
>
> Signed-off-by: Jon Maloy  >
>
> Signed-off-by: David S. Miller  >
>
>
>
> diff --git a/net/tipc/link.c b/net/tipc/link.c
>
> index 9efbdbd..fa452fb 100644
>
> --- a/net/tipc/link.c
>
> +++ b/net/tipc/link.c
>
> @@ -1260,6 +1260,8 @@ static int tipc_link_proto_rcv(struct tipc_link *l,
> struct sk_buff *skb,
>
> /* fall thru' */
>
> case ACTIVATE_MSG:
>
> +   skb_linearize(skb);
>
> +   hdr = buf_msg(skb);
>
> /* Complete own link name with peer's interface name */
>
> if_name =  strrchr(l->name, ':') + 1;
>
> diff --git a/net/tipc/name_distr.c b/net/tipc/name_distr.c
>
> index c07612b..f51c8bd 100644
>
> --- a/net/tipc/name_distr.c
>
> +++ b/net/tipc/name_distr.c
>
> @@ -397,6 +397,7 @@ void tipc_named_rcv(struct net *net, struct
> sk_buff_head *inputq)
>
> spin_lock_bh(>nametbl_lock);
>
> for (skb = skb_dequeue(inputq); skb; skb = skb_dequeue(inputq)) {
>
> +   skb_linearize(skb);
>
> msg = buf_msg(skb);
>
> mtype = msg_type(msg);
>
> item = (struct distr_item *)msg_data(msg);
>
> diff --git a/net/tipc/udp_media.c b/net/tipc/udp_media.c
>
> index ad2719a..816914e 100644
>
> --- a/net/tipc/udp_media.c
>
> +++ b/net/tipc/udp_media.c
>
> @@ -48,7 +48,6 @@
>
> #include 
>
> #include "core.h"
>
> #include "bearer.h"
>
> -#include "msg.h"
>
>  /* IANA assigned UDP port */
>
> #define UDP_PORT_DEFAULT   6118
>
> @@ -221,10 +220,6 @@ static int tipc_udp_recv(struct sock *sk, struct
> sk_buff *skb)
>
> {
>
> struct udp_bearer *ub;
>
> struct tipc_bearer *b;
>
> -   int usr = msg_user(buf_msg(skb));
>
> -
>
> -   if ((usr == LINK_PROTOCOL) || (usr == NAME_DISTRIBUTOR))
>
> -   skb_linearize(skb);
>
> ub = rcu_dereference_sk_user_data(sk);
>
> if (!ub) {
>
>
>
> *From:* GUNA [mailto:gbala...@gmail.com
> ]
> *Sent:* Thursday, 28 April, 2016 19:24
> *To:* Jon Maloy
> *Cc:* tipc-discussion@lists.sourceforge.net
> 
> *Subject:* Re: Tipc: name table mismatch between different cards in a
> system
>
>
>
> See inline
>
> On Thursday, April 28, 2016, Jon Maloy  > wrote:
>
>
>
> > -Original Message-
> > From: GUNA [mailto:gbala...@gmail.com]
> > Sent: Thursday, 28 April, 2016 17:43
> > To: tipc-discussion@lists.sourceforge.net
> > Subject: [tipc-discussion] Tipc: name table mismatch between different
> cards in a
> > system
> >
> > After upgraded CPU cards to 4.4.0 Kernel, there is table mismatch between
> > CPU and IO cards. The IO Publication value = CPU Publication + 1 as you
> see
> > example below:
> >
> > In CPU (slot 2)
> >
> > Type   Lower  Upper  Port Identity   Publication
> >
> > 16789314 3201 3201 <1.1.6:1540208445>   1540208445
> >
> > 16789823 4 4 <1.1.6:3035967304>   3035967304
> >
> > 16832168 3201 3201 <1.1.6:723652841> 723652841
> >
> > …
> >
> >
> >
> > In IO (slot10)
> >
> > 16789314 3201 3201 <1.1.6:1540208445>  1540208446
> >
> > 16789823 4 4 <1.1.6:3035967304>  3035967305
> >
> > 16832168 3201 3201 <1.1.6:723652841>723652842
> >
> > …
>
> This looks like another artefact of the problem we have in 4.4 with
> corrupted entries, where old entries are not withdrawn, so the new ones
> cannot

Re: [tipc-discussion] Kernel 4.4.0 TIPC: links were bouncing and not stable enough

2016-04-28 Thread GUNA

Thanks Jon.

Please see inline


On Thursday, April 28, 2016, Jon Maloy  wrote:

> Hi Guna,
> see below.
>
>
> > -Original Message-
> > From: GUNA [mailto:gbala...@gmail.com ]
> > Sent: Thursday, 28 April, 2016 10:27
> > To: tipc-discussion@lists.sourceforge.net 
> > Subject: [tipc-discussion] Kernel 4.4.0 TIPC: links were bouncing and
> not stable
> > enough
> >
> > Hi Jon,
> >
> > Back to debugging the table mismatch and standby links issues ...
> >
> > I need to clarify two items first as described below. The both issues are
> > reported by our audit script and works fine for kernel 3.4.2 but not for
> > new kernel 4.4.0
> >
> > 1. Table mismatch
> > This is due to bunch of entries with type 2, "node" scope that differs
> from
> > each other.
> > Since the type "2"  is internal and "node" scope, do we expect this to be
> > matched with other node's table? Any change on latest TIPC?
> >
> > // slot3
> >
> > 2  16781314   16781314   <1.1.3:0>  0
>  node
> > 2  16781314   16781314   <1.1.3:1>  1
>  node
> > 2  16781324   16781324   <1.1.3:1>  1
>  node
> > 2  16781324   16781324   <1.1.3:0>  0
>  node
> > 2  16781325   16781325   <1.1.3:0>  0
>  node
> > 2  16781325   16781325   <1.1.3:1>  1
>  node
> >
>
> This is a new feature in 4.0+. It actually shows the working links on this
> node towards other nodes, not only the connectivity to a node, as type "0"
> does.
> I can read form this that you typed the command on node <1.1.3>, and that
> that node has two links towards each of <1.1.2>,< 1.1.12> and <1.1.13>
> respectively, (16781314 in hex is 1001002, i.e. 1.1.2).
> You will find corresponding entries for the other endpoints of the links
> on the respective nodes.
> The fact that they are present in the table means they are up and working.
>
>
> >
> > // slot2
> > Type   Lower  Upper  Port Identity  Publication
> > Scope
> >
> > 2  16781315   16781315   <1.1.2:0>  0
>  node
> > 2  16781315   16781315   <1.1.2:1>  1
>  node
> > 2  16781324   16781324   <1.1.2:0>  0
>  node
> > 2  16781324   16781324   <1.1.2:1>  1
>  node
> > 2  16781325   16781325   <1.1.2:0>  0
>  node
> > 2  16781325   16781325   <1.1.2:1>  1
>  node
> >
> >
> See above.
> <1.1.2> has links towards <1.1.3> as expected, and also towards <1.1.12>
> and <1.1.13>.
> So, everything is correct here.


Entries are correct on each node, however I can not use them when I compare
the tables between the nodes. Since this is a "node" scope, I will filter
out the type 2 prior to comparison.


> > 2. Active and standby links.
> > Our system has 2 bearer p19p1 and p19p2. Both links are ACTIVE in 3.4.2
> > kernel, but on new kernel the one comes as STANDBY.  Both have same
> > priority.
> > Is it expected behavior on latest TIPC?
>
> No. If they have the same priority they should both be active.
> I just checked this in the current code, and there is actually a bug, but
> only in the flag used to present the node, state, not the state as such.
> So, even if the statistics say STANDBY, it is still ACTIVE, and will take
> its full part of the load sharing. You can safely us it as before.
> I will post a patch to fix this upstream, but it probably won't go into
> kernel 4.4, since this is just a "cosmetic" bug.
> Thank you for reporting this.


If you let me know the fix, I will integrate with my kernel. Even though it
is safer to use, fIx is needed to avoid errors reported by our audit script.


> >
> > If both are expected behavior then I would change our audit script
> > accordingly. Otherwise, need to debug the issue.
> >
> > Link <1.1.2:p19p1-1.1.3:p19p1>
> >   ACTIVE  MTU:1500  Priority:10  Tolerance:1200 ms  Window:50 packets
> >   RX packets:4294901760 fragments:0/0 bundles:0/0
> >   TX packets:54487 fragments:0/0 bundles:0/0
> >   TX profile sample:164252 packets  average:30 octets
> >   0-64:97% -256:3% -1024:0% -4096:0% -16384:0% -32768:0% -66000:0%
> >   RX states:473939 probes:513 naks:1407 defs:0 dups:0
> >   TX states:999870 probes:472019 naks:0 acks:0 dups:1407
> >   Congestion link:0  Send queue max:0 avg:0
> >
> > Link <1.1.2:p19p2-1.1.3:p19p2>
> >   STANDBY  MTU:1500  Priority:10  Tolerance:1200 ms  Window:50 packets
> >   RX packets:4294508544 fragments:0/0 bundles:0/0
> >   TX packets:43737 fragments:0/0 bundles:0/0
> >   TX profile sample:255979 packets  average:29 octets
> >   0-64:98% -256:2% -1024:0% -4096:0% -16384:0% -32768:0% -66000:0%
> >   RX states:419534 probes:509 naks:14 defs:242 dups:247
> >   TX states:182 probes:419011 naks:248 acks:0 dups:14
> >   Congestion link:0  Send queue max:0 avg:0
> >
> > [root@Slot2 ~]# tipc-config -b
> > Bearers:
> >

Re: [tipc-discussion] Tipc: name table mismatch between different cards in a system

2016-04-28 Thread Jon Maloy

Here it is. (This is just pasted into Outlook, so don’t try to apply it)
If you manually add the two skb_linearize() calls and the update of ‘hdr’ you 
should be safe.

Good luck!
///jon

commit c7cad0d6f70cd4ce8644ffe528a4df1cdc2e77f5
Author: Jon Paul Maloy 
Date:   Thu Nov 19 14:30:40 2015 -0500

tipc: move linearization of buffers to generic code

In commit 5cbb28a4bf65c7e4 ("tipc: linearize arriving NAME_DISTR
and LINK_PROTO buffers") we added linearization of NAME_DISTRIBUTOR,
LINK_PROTOCOL/RESET and LINK_PROTOCOL/ACTIVATE to the function
tipc_udp_recv(). The location of the change was selected in order
to make the commit easily appliable to 'net' and 'stable'.

We now move this linearization to where it should be done, in the
functions tipc_named_rcv() and tipc_link_proto_rcv() respectively.

Reviewed-by: Ying Xue 
Signed-off-by: Jon Maloy 
Signed-off-by: David S. Miller 

diff --git a/net/tipc/link.c b/net/tipc/link.c
index 9efbdbd..fa452fb 100644
--- a/net/tipc/link.c
+++ b/net/tipc/link.c
@@ -1260,6 +1260,8 @@ static int tipc_link_proto_rcv(struct tipc_link *l, 
struct sk_buff *skb,
/* fall thru' */
case ACTIVATE_MSG:
+   skb_linearize(skb);
+   hdr = buf_msg(skb);
/* Complete own link name with peer's interface name */
if_name =  strrchr(l->name, ':') + 1;
diff --git a/net/tipc/name_distr.c b/net/tipc/name_distr.c
index c07612b..f51c8bd 100644
--- a/net/tipc/name_distr.c
+++ b/net/tipc/name_distr.c
@@ -397,6 +397,7 @@ void tipc_named_rcv(struct net *net, struct sk_buff_head 
*inputq)
spin_lock_bh(>nametbl_lock);
for (skb = skb_dequeue(inputq); skb; skb = skb_dequeue(inputq)) {
+   skb_linearize(skb);
msg = buf_msg(skb);
mtype = msg_type(msg);
item = (struct distr_item *)msg_data(msg);
diff --git a/net/tipc/udp_media.c b/net/tipc/udp_media.c
index ad2719a..816914e 100644
--- a/net/tipc/udp_media.c
+++ b/net/tipc/udp_media.c
@@ -48,7 +48,6 @@
#include 
#include "core.h"
#include "bearer.h"
-#include "msg.h"
 /* IANA assigned UDP port */
#define UDP_PORT_DEFAULT   6118
@@ -221,10 +220,6 @@ static int tipc_udp_recv(struct sock *sk, struct sk_buff 
*skb)
{
struct udp_bearer *ub;
struct tipc_bearer *b;
-   int usr = msg_user(buf_msg(skb));
-
-   if ((usr == LINK_PROTOCOL) || (usr == NAME_DISTRIBUTOR))
-   skb_linearize(skb);
ub = rcu_dereference_sk_user_data(sk);
if (!ub) {

From: GUNA [mailto:gbala...@gmail.com]
Sent: Thursday, 28 April, 2016 19:24
To: Jon Maloy
Cc: tipc-discussion@lists.sourceforge.net
Subject: Re: Tipc: name table mismatch between different cards in a system

See inline

On Thursday, April 28, 2016, Jon Maloy 
> wrote:

> -Original Message-
> From: GUNA [mailto:gbala...@gmail.com]
> Sent: Thursday, 28 April, 2016 17:43
> To: tipc-discussion@lists.sourceforge.net
> Subject: [tipc-discussion] Tipc: name table mismatch between different cards 
> in a
> system
>
> After upgraded CPU cards to 4.4.0 Kernel, there is table mismatch between
> CPU and IO cards. The IO Publication value = CPU Publication + 1 as you see
> example below:
>
> In CPU (slot 2)
>
> Type   Lower  Upper  Port Identity   Publication
>
> 16789314 3201 3201 <1.1.6:1540208445>   1540208445
>
> 16789823 4 4 <1.1.6:3035967304>   3035967304
>
> 16832168 3201 3201 <1.1.6:723652841> 723652841
>
> …
>
>
>
> In IO (slot10)
>
> 16789314 3201 3201 <1.1.6:1540208445>  1540208446
>
> 16789823 4 4 <1.1.6:3035967304>  3035967305
>
> 16832168 3201 3201 <1.1.6:723652841>723652842
>
> …

This looks like another artefact of the problem we have in 4.4 with corrupted 
entries, where old entries are not withdrawn, so the new ones cannot replace 
them in "CPU", while it happens to work in "IO".   If  1.1.6 is "IO" or a third 
node this could make sense.

1.1.6 is node6 but I see similar for all other nodes as well. Ie: same for 
1.1.2,1.1.3,1.1.13.

This problem has been fixed in Ubuntu 16.04, but has not been released yet. I 
would suggest you try 4.5 or 4.6rc4 to see if it disappears. According to Rune 
Torgersen it is definitely working in 4.6.

///jon

I could not move to new kernel now, but I could rebuild the 4.4.0 if I get the 
fix.
If you are aware of the fix please let me know detail of the fix.

>
>
> All the entries are similar pattern in both IO & CPU. The CPU match with
> rest of the CPUs.
>
>
> The system runs on previous kernel did not see this issue.
>
>
> There are two links on IO and CPU cards.  Both links are ACTIVE in IO and
> in CPU one ACTIVE and other STANDBY.
>
> Is that cause the

Re: [tipc-discussion] Tipc: name table mismatch between different cards in a system

2016-04-28 Thread GUNA

See inline

On Thursday, April 28, 2016, Jon Maloy  wrote:

>
>
> > -Original Message-
> > From: GUNA [mailto:gbala...@gmail.com ]
> > Sent: Thursday, 28 April, 2016 17:43
> > To: tipc-discussion@lists.sourceforge.net 
> > Subject: [tipc-discussion] Tipc: name table mismatch between different
> cards in a
> > system
> >
> > After upgraded CPU cards to 4.4.0 Kernel, there is table mismatch between
> > CPU and IO cards. The IO Publication value = CPU Publication + 1 as you
> see
> > example below:
> >
> > In CPU (slot 2)
> >
> > Type   Lower  Upper  Port Identity   Publication
> >
> > 16789314 3201 3201 <1.1.6:1540208445>   1540208445
> >
> > 16789823 4 4 <1.1.6:3035967304>   3035967304
> >
> > 16832168 3201 3201 <1.1.6:723652841> 723652841
> >
> > …
> >
> >
> >
> > In IO (slot10)
> >
> > 16789314 3201 3201 <1.1.6:1540208445>  1540208446
> >
> > 16789823 4 4 <1.1.6:3035967304>  3035967305
> >
> > 16832168 3201 3201 <1.1.6:723652841>723652842
> >
> > …
>
> This looks like another artefact of the problem we have in 4.4 with
> corrupted entries, where old entries are not withdrawn, so the new ones
> cannot replace them in "CPU", while it happens to work in "IO".   If  1.1.6
> is "IO" or a third node this could make sense.


1.1.6 is node6 but I see similar for all other nodes as well. Ie: same for
1.1.2,1.1.3,1.1.13.



> This problem has been fixed in Ubuntu 16.04, but has not been released
> yet. I would suggest you try 4.5 or 4.6rc4 to see if it disappears.
> According to Rune Torgersen it is definitely working in 4.6.
>
> ///jon
>
>
I could not move to new kernel now, but I could rebuild the 4.4.0 if I get
the fix.
If you are aware of the fix please let me know detail of the fix.


> >
> >
> > All the entries are similar pattern in both IO & CPU. The CPU match with
> > rest of the CPUs.
> >
> >
> > The system runs on previous kernel did not see this issue.
> >
> >
> > There are two links on IO and CPU cards.  Both links are ACTIVE in IO and
> > in CPU one ACTIVE and other STANDBY.
> >
> > Is that cause the issue?  If so, is it possible to make both links to be
> > ACTIVE ?
> >
> >
> > // IO
> >
> > Link <1.1.10:eth0-1.1.2:p19p1>
> >
> >   ACTIVE  MTU:1500  Priority:10  Tolerance:1200 ms  Window:50 packets
> >
> > Link <1.1.10:eth1-1.1.2:p19p2>
> >
> >   ACTIVE  MTU:1500  Priority:10  Tolerance:1200 ms  Window:50 packets
> >
> > // CPU
> > Link <1.1.2:p19p1-1.1.10:eth0>
> >   ACTIVE  MTU:1500  Priority:10  Tolerance:1200 ms  Window:50 packets
> >
> > Link <1.1.2:p19p2-1.1.10:eth1>
> >   STANDBY  MTU:1500  Priority:10  Tolerance:1200 ms  Window:50 packets
> >
> > thanks,
> >
> > Guna
> >
> --
> > Find and fix application performance issues faster with Applications
> Manager
> > Applications Manager provides deep performance insights into multiple
> tiers of
> > your business applications. It resolves application problems quickly and
> > reduces your MTTR. Get your free trial!
> > https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
> > ___
> > tipc-discussion mailing list
> > tipc-discussion@lists.sourceforge.net 
> > https://lists.sourceforge.net/lists/listinfo/tipc-discussion
>
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
___
tipc-discussion mailing list
tipc-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tipc-discussion

Re: [tipc-discussion] Tipc: name table mismatch between different cards in a system

2016-04-28 Thread Jon Maloy



> -Original Message-
> From: GUNA [mailto:gbala...@gmail.com]
> Sent: Thursday, 28 April, 2016 17:43
> To: tipc-discussion@lists.sourceforge.net
> Subject: [tipc-discussion] Tipc: name table mismatch between different cards 
> in a
> system
> 
> After upgraded CPU cards to 4.4.0 Kernel, there is table mismatch between
> CPU and IO cards. The IO Publication value = CPU Publication + 1 as you see
> example below:
> 
> In CPU (slot 2)
> 
> Type   Lower  Upper  Port Identity   Publication
> 
> 16789314 3201 3201 <1.1.6:1540208445>   1540208445
> 
> 16789823 4 4 <1.1.6:3035967304>   3035967304
> 
> 16832168 3201 3201 <1.1.6:723652841> 723652841
> 
> …
> 
> 
> 
> In IO (slot10)
> 
> 16789314 3201 3201 <1.1.6:1540208445>  1540208446
> 
> 16789823 4 4 <1.1.6:3035967304>  3035967305
> 
> 16832168 3201 3201 <1.1.6:723652841>723652842
> 
> …

This looks like another artefact of the problem we have in 4.4 with corrupted 
entries, where old entries are not withdrawn, so the new ones cannot replace 
them in "CPU", while it happens to work in "IO".   If  1.1.6 is "IO" or a third 
node this could make sense.
This problem has been fixed in Ubuntu 16.04, but has not been released yet. I 
would suggest you try 4.5 or 4.6rc4 to see if it disappears. According to Rune 
Torgersen it is definitely working in 4.6.

///jon

> 
> 
> All the entries are similar pattern in both IO & CPU. The CPU match with
> rest of the CPUs.
> 
> 
> The system runs on previous kernel did not see this issue.
> 
> 
> There are two links on IO and CPU cards.  Both links are ACTIVE in IO and
> in CPU one ACTIVE and other STANDBY.
> 
> Is that cause the issue?  If so, is it possible to make both links to be
> ACTIVE ?
> 
> 
> // IO
> 
> Link <1.1.10:eth0-1.1.2:p19p1>
> 
>   ACTIVE  MTU:1500  Priority:10  Tolerance:1200 ms  Window:50 packets
> 
> Link <1.1.10:eth1-1.1.2:p19p2>
> 
>   ACTIVE  MTU:1500  Priority:10  Tolerance:1200 ms  Window:50 packets
> 
> // CPU
> Link <1.1.2:p19p1-1.1.10:eth0>
>   ACTIVE  MTU:1500  Priority:10  Tolerance:1200 ms  Window:50 packets
> 
> Link <1.1.2:p19p2-1.1.10:eth1>
>   STANDBY  MTU:1500  Priority:10  Tolerance:1200 ms  Window:50 packets
> 
> thanks,
> 
> Guna
> --
> Find and fix application performance issues faster with Applications Manager
> Applications Manager provides deep performance insights into multiple tiers of
> your business applications. It resolves application problems quickly and
> reduces your MTTR. Get your free trial!
> https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
> ___
> tipc-discussion mailing list
> tipc-discussion@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/tipc-discussion
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
___
tipc-discussion mailing list
tipc-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tipc-discussion

Re: [tipc-discussion] Kernel 4.4.0 TIPC: links were bouncing and not stable enough

2016-04-28 Thread Jon Maloy

Hi Guna,
see below.


> -Original Message-
> From: GUNA [mailto:gbala...@gmail.com]
> Sent: Thursday, 28 April, 2016 10:27
> To: tipc-discussion@lists.sourceforge.net
> Subject: [tipc-discussion] Kernel 4.4.0 TIPC: links were bouncing and not 
> stable
> enough
> 
> Hi Jon,
> 
> Back to debugging the table mismatch and standby links issues ...
> 
> I need to clarify two items first as described below. The both issues are
> reported by our audit script and works fine for kernel 3.4.2 but not for
> new kernel 4.4.0
> 
> 1. Table mismatch
> This is due to bunch of entries with type 2, "node" scope that differs from
> each other.
> Since the type "2"  is internal and "node" scope, do we expect this to be
> matched with other node's table? Any change on latest TIPC?
> 
> // slot3
> 
> 2  16781314   16781314   <1.1.3:0>  0   node
> 2  16781314   16781314   <1.1.3:1>  1   node
> 2  16781324   16781324   <1.1.3:1>  1   node
> 2  16781324   16781324   <1.1.3:0>  0   node
> 2  16781325   16781325   <1.1.3:0>  0   node
> 2  16781325   16781325   <1.1.3:1>  1   node
> 

This is a new feature in 4.0+. It actually shows the working links on this node 
towards other nodes, not only the connectivity to a node, as type "0" does.
I can read form this that you typed the command on node <1.1.3>, and that that 
node has two links towards each of <1.1.2>,< 1.1.12> and <1.1.13> respectively, 
(16781314 in hex is 1001002, i.e. 1.1.2).
You will find corresponding entries for the other endpoints of the links on the 
respective nodes.
The fact that they are present in the table means they are up and working.


> 
> // slot2
> Type   Lower  Upper  Port Identity  Publication
> Scope
> 
> 2  16781315   16781315   <1.1.2:0>  0   node
> 2  16781315   16781315   <1.1.2:1>  1   node
> 2  16781324   16781324   <1.1.2:0>  0   node
> 2  16781324   16781324   <1.1.2:1>  1   node
> 2  16781325   16781325   <1.1.2:0>  0   node
> 2  16781325   16781325   <1.1.2:1>  1   node
> 
> 
See above.
<1.1.2> has links towards <1.1.3> as expected, and also towards <1.1.12> and 
<1.1.13>.
So, everything is correct here.

> 2. Active and standby links.
> Our system has 2 bearer p19p1 and p19p2. Both links are ACTIVE in 3.4.2
> kernel, but on new kernel the one comes as STANDBY.  Both have same
> priority.
> Is it expected behavior on latest TIPC?

No. If they have the same priority they should both be active.
I just checked this in the current code, and there is actually a bug, but only 
in the flag used to present the node, state, not the state as such.
So, even if the statistics say STANDBY, it is still ACTIVE, and will take its 
full part of the load sharing. You can safely us it as before.
I will post a patch to fix this upstream, but it probably won't go into kernel 
4.4, since this is just a "cosmetic" bug.
Thank you for reporting this.

> 
> If both are expected behavior then I would change our audit script
> accordingly. Otherwise, need to debug the issue.
> 
> Link <1.1.2:p19p1-1.1.3:p19p1>
>   ACTIVE  MTU:1500  Priority:10  Tolerance:1200 ms  Window:50 packets
>   RX packets:4294901760 fragments:0/0 bundles:0/0
>   TX packets:54487 fragments:0/0 bundles:0/0
>   TX profile sample:164252 packets  average:30 octets
>   0-64:97% -256:3% -1024:0% -4096:0% -16384:0% -32768:0% -66000:0%
>   RX states:473939 probes:513 naks:1407 defs:0 dups:0
>   TX states:999870 probes:472019 naks:0 acks:0 dups:1407
>   Congestion link:0  Send queue max:0 avg:0
> 
> Link <1.1.2:p19p2-1.1.3:p19p2>
>   STANDBY  MTU:1500  Priority:10  Tolerance:1200 ms  Window:50 packets
>   RX packets:4294508544 fragments:0/0 bundles:0/0
>   TX packets:43737 fragments:0/0 bundles:0/0
>   TX profile sample:255979 packets  average:29 octets
>   0-64:98% -256:2% -1024:0% -4096:0% -16384:0% -32768:0% -66000:0%
>   RX states:419534 probes:509 naks:14 defs:242 dups:247
>   TX states:182 probes:419011 naks:248 acks:0 dups:14
>   Congestion link:0  Send queue max:0 avg:0
> 
> [root@Slot2 ~]# tipc-config -b
> Bearers:
> eth:p19p1
> eth:p19p2
> 
> On last discussion, you asked me to turn on debug from the node.c. Could
> you let me know how I could turned on? Do I need to add printf or just
> enable any micro?

You have to recompile the kernel with DEBUG option to see all pr_debug() 
printouts.
I don't think you need to do that now according to the above.

BR
///jon

> 
> I tried to use the latest tipcutils, but having issue on compiling it.
> 
> Thanks,
> Guna
> --
> Find and fix application performance issues

[tipc-discussion] Tipc: name table mismatch between different cards in a system

2016-04-28 Thread GUNA

After upgraded CPU cards to 4.4.0 Kernel, there is table mismatch between
CPU and IO cards. The IO Publication value = CPU Publication + 1 as you see
example below:

In CPU (slot 2)

Type   Lower  Upper  Port Identity   Publication

16789314 3201 3201 <1.1.6:1540208445>   1540208445

16789823 4 4 <1.1.6:3035967304>   3035967304

16832168 3201 3201 <1.1.6:723652841> 723652841

…



In IO (slot10)

16789314 3201 3201 <1.1.6:1540208445>  1540208446

16789823 4 4 <1.1.6:3035967304>  3035967305

16832168 3201 3201 <1.1.6:723652841>723652842

…


All the entries are similar pattern in both IO & CPU. The CPU match with
rest of the CPUs.


The system runs on previous kernel did not see this issue.


There are two links on IO and CPU cards.  Both links are ACTIVE in IO and
in CPU one ACTIVE and other STANDBY.

Is that cause the issue?  If so, is it possible to make both links to be
ACTIVE ?


// IO

Link <1.1.10:eth0-1.1.2:p19p1>

  ACTIVE  MTU:1500  Priority:10  Tolerance:1200 ms  Window:50 packets

Link <1.1.10:eth1-1.1.2:p19p2>

  ACTIVE  MTU:1500  Priority:10  Tolerance:1200 ms  Window:50 packets

// CPU
Link <1.1.2:p19p1-1.1.10:eth0>
  ACTIVE  MTU:1500  Priority:10  Tolerance:1200 ms  Window:50 packets

Link <1.1.2:p19p2-1.1.10:eth1>
  STANDBY  MTU:1500  Priority:10  Tolerance:1200 ms  Window:50 packets

thanks,

Guna
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
___
tipc-discussion mailing list
tipc-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tipc-discussion

Re: [tipc-discussion] [PATCH net-next v3 1/1] tipc: add neighbor monitoring framework

2016-04-28 Thread Jon Maloy

Hi Partha,
See below.


> -Original Message-
> From: Parthasarathy Bhuvaragan
> Sent: Thursday, 28 April, 2016 10:16
> To: Jon Maloy; tipc-discussion@lists.sourceforge.net; Ying Xue; Richard Alpe
> Cc: ma...@donjonn.com
> Subject: Re: [PATCH net-next v3 1/1] tipc: add neighbor monitoring framework
> 
> Hi Jon,
> 
> I added my interpretation of members corresponding to your description
> along with some comments. Please confirm if they are correct.
> 
>  Description of the printout:
>  
> 
>  Indented line: This peer is not actively monitored from the local node.
> [partha] tipc_peer->is_head || tipc_peer->is_local
Yes.

> 
>  Node: the identity of a peer node
> [partha] tipc_peer->addr
Yes.

> 
>  Reported members: The number of members reported by the peer in its
>  domain record
> [partha] tipc_mon_domain->member_cnt
Yes.

> 
>  Applied members: The number of peers in the monitor list that matches
>  the contents of this peer's domain record. The peer node may see fewer
>  or more nodes than this node, so the lists may not match exactly. A
>  received domain record is matched member by member against the own list
>  until the end, until there is a mismatch, or until it encounters the
>  list's own node. In the latter two cases this number will differ from
>  "Reported Domain Members".
> [partha] tipc_peer->monitoring
Yes.

> 
>  Domain Generation: An integer which is stepped every time something
>  changes (peer add/remove/up/down) in a node's local domain. This helps
>  the receiver of a domain record to know if it can ignore the record, or
>  if it must process it.
> [partha] tipc_mon_domain->gen
Yes.

> 
>  Domain State Map: A bit map showing a peer's view of its domain
>  members' up/down state. Bit 0 corresponds to its view of the first
>  domain member, bit 1 to the next one etc.
> [partha] tipc_mon_domain->up_map
Yes.

> 
>  Head State Map: A bit map showing how the preceding peers in the list
>  sees the up/down state of this peer node. Position 0 corresponds to
>  the view of immediate preceding node in the list, position 1 the
>  the previous one etc.
> [partha] tipc_peer->head_map
Yes.

> If the Head State Map differs from Domain State Map (i.e preceding peers
> view differs from peers view for a given peer), there is an error OR are
> there cases when they can differ?

They will of course differ, but since head_map is generated bit-by-bit from the 
preceding up-map, they must match for the applied bits. Also, if "applied 
members" is smaller than "reported members" according to the previous, the 
up_map of that peer will contain bits which are not reflected in the 
corresponding head_maps. You will be able to see this by using "full mode", 
where you will see that the last members of the domain record are not present 
in the list, or that, in the case where the own node is a member of a domain, 
the applied domain is truncated at there.

> From a tipc user's perspective, how does he ensure that after enabling the
> monitor everything is "OK"?
> It will be of great benefit to the user to present these attributes in a
> format which performs the above.

The closest I can imagine is if we present the bitmaps in some sort of binary 
view, e.g.: uuuddduuuu etc. for both maps, but that takes space (in a 
1000-node cluster we will have 32 bits set in each map), and some "hands-on" 
calculation will still be needed. An easy comparison of correctness would be 
provided if we could present them in a matrix, but that would take even more 
space. 
But your suggestion is good and valid. I will try to spend some time pondering 
on this.


> 
>  Domain Member List: The actual contents of a peer's domain record, i.e.,
>  its reported member list. The suffix ":u" means it considers the member
>  to be "up", the suffux ":d" that it considers it being down.
> [partha] For each of the peer listed in tipc_mon_domain->members, the
> peer states are derived from the tipc_mon_domain->up_map.
> But in full listing, we seem to miss the head state map. Or is this
> implicitly covered?

Since it is generated from the former, yes. But if we don't see it we can't 
verify if it is correct or not. That's why I added it in the "compact", view.
Also, remember that the head_map is an implementation detail, -we could do 
without it at the cost of some more computing.
But as already said, some combined view would be desirable.

> 
>  List Generation: An integer which is stepped every time something changes
>  in the local monitor list, including changes in the local domain. When a
>  link timer asks the monitor whether its should send a probe message or
>  not, it also hands it the value of this field as it was at the previous
>  request. By comparing this cached value with the current list generation,
>  the monitor can know if it can respond with a cached copy of the the
>  pervious response, or if it has to consult the monitor list to obtain
>  an updated response.
> [partha]

Re: [tipc-discussion] [patch] tipc: remove an unnecessary NULL check

2016-04-28 Thread David Miller

From: Dan Carpenter 
Date: Wed, 27 Apr 2016 11:05:28 +0300

> This is never called with a NULL "buf" and anyway, we dereference 's' on
> the lines before so it would Oops before we reach the check.
> 
> Signed-off-by: Dan Carpenter 

Applied to net-next, thanks.

--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
___
tipc-discussion mailing list
tipc-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tipc-discussion

[tipc-discussion] Kernel 4.4.0 TIPC: links were bouncing and not stable enough

2016-04-28 Thread GUNA

Hi Jon,

Back to debugging the table mismatch and standby links issues ...

I need to clarify two items first as described below. The both issues are
reported by our audit script and works fine for kernel 3.4.2 but not for
new kernel 4.4.0

1. Table mismatch
This is due to bunch of entries with type 2, "node" scope that differs from
each other.
Since the type "2"  is internal and "node" scope, do we expect this to be
matched with other node's table? Any change on latest TIPC?

// slot3

2  16781314   16781314   <1.1.3:0>  0   node
2  16781314   16781314   <1.1.3:1>  1   node
2  16781324   16781324   <1.1.3:1>  1   node
2  16781324   16781324   <1.1.3:0>  0   node
2  16781325   16781325   <1.1.3:0>  0   node
2  16781325   16781325   <1.1.3:1>  1   node


// slot2
Type   Lower  Upper  Port Identity  Publication
Scope

2  16781315   16781315   <1.1.2:0>  0   node
2  16781315   16781315   <1.1.2:1>  1   node
2  16781324   16781324   <1.1.2:0>  0   node
2  16781324   16781324   <1.1.2:1>  1   node
2  16781325   16781325   <1.1.2:0>  0   node
2  16781325   16781325   <1.1.2:1>  1   node


2. Active and standby links.
Our system has 2 bearer p19p1 and p19p2. Both links are ACTIVE in 3.4.2
kernel, but on new kernel the one comes as STANDBY.  Both have same
priority.
Is it expected behavior on latest TIPC?

If both are expected behavior then I would change our audit script
accordingly. Otherwise, need to debug the issue.

Link <1.1.2:p19p1-1.1.3:p19p1>
  ACTIVE  MTU:1500  Priority:10  Tolerance:1200 ms  Window:50 packets
  RX packets:4294901760 fragments:0/0 bundles:0/0
  TX packets:54487 fragments:0/0 bundles:0/0
  TX profile sample:164252 packets  average:30 octets
  0-64:97% -256:3% -1024:0% -4096:0% -16384:0% -32768:0% -66000:0%
  RX states:473939 probes:513 naks:1407 defs:0 dups:0
  TX states:999870 probes:472019 naks:0 acks:0 dups:1407
  Congestion link:0  Send queue max:0 avg:0

Link <1.1.2:p19p2-1.1.3:p19p2>
  STANDBY  MTU:1500  Priority:10  Tolerance:1200 ms  Window:50 packets
  RX packets:4294508544 fragments:0/0 bundles:0/0
  TX packets:43737 fragments:0/0 bundles:0/0
  TX profile sample:255979 packets  average:29 octets
  0-64:98% -256:2% -1024:0% -4096:0% -16384:0% -32768:0% -66000:0%
  RX states:419534 probes:509 naks:14 defs:242 dups:247
  TX states:182 probes:419011 naks:248 acks:0 dups:14
  Congestion link:0  Send queue max:0 avg:0

[root@Slot2 ~]# tipc-config -b
Bearers:
eth:p19p1
eth:p19p2

On last discussion, you asked me to turn on debug from the node.c. Could
you let me know how I could turned on? Do I need to add printf or just
enable any micro?

I tried to use the latest tipcutils, but having issue on compiling it.

Thanks,
Guna
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial!
https://ad.doubleclick.net/ddm/clk/302982198;130105516;z
___
tipc-discussion mailing list
tipc-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tipc-discussion

Re: [tipc-discussion] [PATCH net-next v3 1/1] tipc: add neighbor monitoring framework

2016-04-28 Thread Parthasarathy Bhuvaragan

Hi Jon,

I added my interpretation of members corresponding to your description
along with some comments. Please confirm if they are correct.

 Description of the printout:

 Indented line: This peer is not actively monitored from the local node.
[partha] tipc_peer->is_head || tipc_peer->is_local

 Node: the identity of a peer node
[partha] tipc_peer->addr

 Reported members: The number of members reported by the peer in its
 domain record
[partha] tipc_mon_domain->member_cnt

 Applied members: The number of peers in the monitor list that matches
 the contents of this peer's domain record. The peer node may see fewer
 or more nodes than this node, so the lists may not match exactly. A
 received domain record is matched member by member against the own list
 until the end, until there is a mismatch, or until it encounters the
 list's own node. In the latter two cases this number will differ from
 "Reported Domain Members".
[partha] tipc_peer->monitoring

 Domain Generation: An integer which is stepped every time something
 changes (peer add/remove/up/down) in a node's local domain. This helps
 the receiver of a domain record to know if it can ignore the record, or
 if it must process it.
[partha] tipc_mon_domain->gen

 Domain State Map: A bit map showing a peer's view of its domain
 members' up/down state. Bit 0 corresponds to its view of the first
 domain member, bit 1 to the next one etc.
[partha] tipc_mon_domain->up_map

 Head State Map: A bit map showing how the preceding peers in the list
 sees the up/down state of this peer node. Position 0 corresponds to
 the view of immediate preceding node in the list, position 1 the
 the previous one etc.
[partha] tipc_peer->head_map
If the Head State Map differs from Domain State Map (i.e preceding peers
view differs from peers view for a given peer), there is an error OR are
there cases when they can differ?
>From a tipc user's perspective, how does he ensure that after enabling the
monitor everything is "OK"?
It will be of great benefit to the user to present these attributes in a
format which performs the above.

 Domain Member List: The actual contents of a peer's domain record, i.e.,
 its reported member list. The suffix ":u" means it considers the member
 to be "up", the suffux ":d" that it considers it being down.
[partha] For each of the peer listed in tipc_mon_domain->members, the
peer states are derived from the tipc_mon_domain->up_map.
But in full listing, we seem to miss the head state map. Or is this
implicitly covered?

 List Generation: An integer which is stepped every time something changes
 in the local monitor list, including changes in the local domain. When a
 link timer asks the monitor whether its should send a probe message or
 not, it also hands it the value of this field as it was at the previous
 request. By comparing this cached value with the current list generation,
 the monitor can know if it can respond with a cached copy of the the
 pervious response, or if it has to consult the monitor list to obtain
 an updated response.
[partha] tipc_monitor->list_gen

regards
Partha

On 04/26/2016 11:02 PM, Jon Maloy wrote:
> Hi Partha,
> See attachment. I spent a little time on both suggesting a readable format 
> and to explain what the different attributes mean. I hope this helps.
>
> BR
> ///jon
>
>> -Original Message-
>> From: Parthasarathy Bhuvaragan
>> Sent: Tuesday, 26 April, 2016 12:12
>> To: Jon Maloy; tipc-discussion@lists.sourceforge.net; Ying Xue; Richard Alpe
>> Cc: ma...@donjonn.com
>> Subject: Re: [PATCH net-next v3 1/1] tipc: add neighbor monitoring framework
>>
>> Hi Jon,
>>
>> I find it hard to express the monitoring attributes in tipc tool.
>> Can you provide a short description for the required parameters?
>>
>> regards
>> Partha
>>
>> On 2016-04-20 18:22, Jon Maloy wrote:
>>> TIPC based clusters are by default set up with full-mesh link
>>> connectivity between all nodes. Those links are expected to provide
>>> a short failure detection time, by default set to 1500 ms. Because
>>> of this, the background load for neighbor monitoring in an N-node
>>> cluster increases with a factor N on each node, while the overall
>>> monitoring traffic through the network infrastructure inceases at
>>> a ~(N * (N - 1)) rate. Experience has shown that such clusters don't
>>> scale well beyond ~100 nodes unless we significantly increase failure
>>> discovery tolerance.
>>>
>>> This commit introduces a framework and an algorithm that drastically
>>> reduces this background load, while basically maintaining the original
>>> failure detection times across the whole cluster. Using this algortithm,
>>> background load will now grow at a rate of ~(2 * sqrt(N)) per node, and
>>> at ~(2 * N * sqrt(N)) in traffic overhead. As an example, each node will
>>> now have to actively monitor 38 neighbors in a 400-node cluster, instead
>>> of as before 399.
>>>
>>> This "Overlapping Ring Supervision

Re: [tipc-discussion] [PATCH] tipc: Only process unicast on intended node

Re: [tipc-discussion] Tipc: name table mismatch between different cards in a system

Re: [tipc-discussion] Kernel 4.4.0 TIPC: links were bouncing and not stable enough

Re: [tipc-discussion] Tipc: name table mismatch between different cards in a system

Re: [tipc-discussion] Tipc: name table mismatch between different cards in a system

Re: [tipc-discussion] Tipc: name table mismatch between different cards in a system

Re: [tipc-discussion] Kernel 4.4.0 TIPC: links were bouncing and not stable enough

[tipc-discussion] Tipc: name table mismatch between different cards in a system

Re: [tipc-discussion] [PATCH net-next v3 1/1] tipc: add neighbor monitoring framework

Re: [tipc-discussion] [patch] tipc: remove an unnecessary NULL check

[tipc-discussion] Kernel 4.4.0 TIPC: links were bouncing and not stable enough

Re: [tipc-discussion] [PATCH net-next v3 1/1] tipc: add neighbor monitoring framework

12 matches

Site Navigation

Mail list logo

Footer information