Re: [tipc-discussion] kernel page allocation problem

[Erik Hugne]

On Apr 10, 2016 15:27, "Leon Pollak"  wrote:
>
> After your answer I again looked through the both manuals and did not find
> anything saying that connectionless messaging allows drops. Vice verse,
the
> programmer's manual explicitly states that "TIPC is designed to be a
reliable
> messaging mechanism, in which an application can send a message and assume
> that the message will be delivered to the specified destination as long as
> that destination is reachable."
>
> Now, is it a bug, corrected in Nov 6th 2013? Or a feature?
>
> i can't move to connection oriented methods, because I need to support
one-
> to-3 and 3-to-one.
>

This sounds a lot like the problem when connectionless messages are
received, acked on the tipc link layer and passed to the socket, but
dropped there because the socket receive buffer is full.
If the receiving application does not drain the queue faster than messages
build up, there will be losses as there us no concept of flow control at
this level.
Try bumping the prio of the server, maybe run it as an rt thread?

//E

> Sorry.
> --
> Leon
--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial! http://pubads.g.doubleclick.net/
gampad/clk?id=1444514301=/ca-pub-7940484522588532
___
tipc-discussion mailing list
tipc-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tipc-discussion

Re: [tipc-discussion] kernel page allocation problem

[Leon Pollak]

On Friday 01 April 2016 15:02:05 Jon Maloy wrote:
> > I saw that the problem is in the lack of memory.
> > I also made a simple stress test (one side sends in a loop) and received 
> > this problem immediately.
> > But User's Manual states that if there is no room for the receiving side 
> > to accept message it prevents the sending side from sending, effectively
> > blocking the sender.
> 
> That is true only for connection oriented messaging (SOC_STREAM, 
> SOCK_SEQPACKET). Is that what you are running?
> If you are just doing a tight loop with 66k messages using TIPC_PORT_NAME 
> you will not have any flow control, and the messages might be dropped in the 
> receiving socket.
> But if you are really tight of memory they might also be dropped at the link 
> layer, which is what you are seeing. 
> I have never seen that happening before, but it is fully possible in TIPC 
> versions before commit 40ba3cdf542a469aaa9  ("tipc: message reassembly using 
> fragment chain") from Nov 6th 2013. (I don't remember which Linux version 
> this corresponds to, but that should be easy to find out). Furthermore, this 
> might even happen with connection oriented messaging, but is less likely.
> 
> So, if you are running connectionless, I recommend you to go to a connection 
oriented mode. If you already are connection oriented, you can try to find out 
which version this fix was in, and re-adapt the code.

Thanks a lot, Jon, again.

After your answer I again looked through the both manuals and did not find 
anything saying that connectionless messaging allows drops. Vice verse, the 
programmer's manual explicitly states that "TIPC is designed to be a reliable 
messaging mechanism, in which an application can send a message and assume 
that the message will be delivered to the specified destination as long as 
that destination is reachable."

Now, is it a bug, corrected in Nov 6th 2013? Or a feature?

i can't move to connection oriented methods, because I need to support one-
to-3 and 3-to-one.

Sorry.
-- 
Leon

--
Find and fix application performance issues faster with Applications Manager
Applications Manager provides deep performance insights into multiple tiers of
your business applications. It resolves application problems quickly and
reduces your MTTR. Get your free trial! http://pubads.g.doubleclick.net/
gampad/clk?id=1444514301=/ca-pub-7940484522588532
___
tipc-discussion mailing list
tipc-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tipc-discussion

Re: [tipc-discussion] kernel page allocation problem

[Leon Pollak]

On Wednesday 30 March 2016 14:40:58 Jon Maloy wrote:
> Hi Leon,
> I took a quick look at your dump, and to me it looks like you have simply 
run out of memory. I think that is the first thing you should look at.
> As for the most recent code version to use, I cannot give you aby definite 
answer, since the kernel API and environment keeps changing continuously, so 
new code will not easily compile on older kernels and vice versa.
> If you want to go down that road, I would recommend a bisectional approach, 
e.g., try if the code from 3.15 applies and runs, and then go forwards or 
backwards from there.
> But, as already said, look at the problem you have at hand first.


Jon, thank you for the answer.

I saw that the problem is in the lack of memory.
I also made a simple stress test (one side sends in a loop) and received this 
problem immediately.

But User's Manual states that if there is no room for the receiving side to 
accept message it prevents the sending side from sending, effectively blocking 
the sender.
This is what I expected to occur. But this doesn't happen in my case.
I logged the sendto() execution time and it always was less then 2ms, except 
the case when this page error occurred on the accepting side - then sendto() 
took about 500-700 ms.

Excuse me, please, may be this is the simple newbie question...:-)
What do I do wrong?

-- 
Leon

--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785471=/4140
___
tipc-discussion mailing list
tipc-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tipc-discussion

Re: [tipc-discussion] kernel page allocation problem

[Jon Maloy]

Hi Leon,
Yes, you are running TIPC 2.0,  but that does not really mean much. The version 
number hasn't changed the last 4-5 years, while huge parts of the code has been 
fundamentally rewritten during that period.
There were several issues like the one you have encountered, and almost all of 
them have been dealt with or just disappeared in the newer versions.
The code you are using is in reality very old, and I don't know  if we could 
fix it even if we tried, since I am not sure there are any patch releases for 
this old kernel versions.

I would strongly recommend you to move to a more recent kernel if possible 
(Preferable 4.2 or later), so you can benefit from the maintenance and upgrades 
we are working on now.

Regards
///jon

PS. We will step code version number to 3.0 later this year, if everything goes 
according to plan, but the protocol version will not be changed.


> -Original Message-
> From: Leon Pollak [mailto:le...@plris.com]
> Sent: Tuesday, 29 March, 2016 07:18
> To: tipc-discussion@lists.sourceforge.net
> Subject: [tipc-discussion] kernel page allocation problem
> 
> Hello, gurus.
> 
> Recently I encountered the kernel "page allocation" error in the TIPC code.
> I attached the backtrace printout.
> I use 2.6.37 Linux version, which seems to use TIPC 2.0 (although I am not
> sure, but the code is prom the Arago distribution).
> All this runs in the embedded ARM system.
> Although it seems to me irrelevant, but anyway - the largest packet
> transferred by me is 15KiB.
> 
> I will be very thankful for any comment about the issue.
> --
> Leon

--
Transform Data into Opportunity.
Accelerate data analysis in your applications with
Intel Data Analytics Acceleration Library.
Click to learn more.
http://pubads.g.doubleclick.net/gampad/clk?id=278785471=/4140
___
tipc-discussion mailing list
tipc-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/tipc-discussion