Yes I'm testing the modifications I made a few weeks ago, so I have not checked 
the changes in.  The race condition appears to be fixed on our Mellanox FDR 
fabric, but I am still seeing issues on our Qlogic QDR fabric.  I'll take a 
look at your patch and see if it makes sense to merge my changes with yours.

-Randy

From: Kyle Schochenmaier <kscho...@gmail.com<mailto:kscho...@gmail.com>>
Date: Friday, January 25, 2013 8:14 AM
To: "Zhang, Jingwang" <jingwang.zh...@emc.com<mailto:jingwang.zh...@emc.com>>
Cc: PVFS2-developers 
<pvfs2-developers@beowulf-underground.org<mailto:pvfs2-developers@beowulf-underground.org>>,
 "faibish, sorin" <faibish_so...@emc.com<mailto:faibish_so...@emc.com>>, 
Randall Martin <w...@clemson.edu<mailto:w...@clemson.edu>>
Subject: Re: [Pvfs2-developers] Potential connection starvation in bmi_ib

Hi Jingwang -

I believe Randy and I stumbled upon the same issue a few weeks ago and we came 
up with a slightly different approach to resolve the race condition.
I'm not sure if that fix has made it back into the main branch yet, but I'll 
double-check and see if we can send you our version first.

If you're able to test out our fix to see if you still see the race condition 
that would be great, as this may have been fixed already.
I'll check today and try to send you our patch.

Regards,
~Kyle

Kyle Schochenmaier


On Fri, Jan 25, 2013 at 1:27 AM, Zhang, Jingwang 
<jingwang.zh...@emc.com<mailto:jingwang.zh...@emc.com>> wrote:
Hi All,

Recently I experienced a performance issue with bmi_ib. The problem is as 
following:
Assuming that it is a client-server architecture, and they exchange lots of 
messages through bmi_ib. If multiple client processes are started at the same 
time, they cannot run concurrently at the same time. Instead, they will be 
serialized to run one after another. It’s a strange behavior and it hurts the 
performance greatly.

After some investigation to the source code, I found the reason for that is as 
following:
The new coming connection are handled in the function 
ib_tcp_server_check_new_connections(); and this is called inside the function 
ib_block_for_activity(). However the ib_block_for_activity() is only called 
when the network is idle in BMI_ib_testcontext() or BMI_ib_testunexpected().
As a result, when the server is busy serving one client process, the other 
processes can’t make a new connections to the server and thus they can’t 
transfer data to the server concurrently.

I made a pretty simple fix for this problem and it worked for me. The idea is 
checking new connections inside the testunexpected() so that new connections 
can be handled in time to avoid starvation of client processes. Here it is:

diff --git a/src/io/bmi/bmi_ib/ib.c b/src/io/bmi/bmi_ib/ib.c
index 0808797..b349938 100644
--- a/src/io/bmi/bmi_ib/ib.c
+++ b/src/io/bmi/bmi_ib/ib.c
@@ -1436,6 +1436,8 @@ restart:
        }
     }

+    ib_tcp_server_check_new_connections();
+
     *outcount = n;
     return activity + n;
}

Please feel free to share your thoughts and comments, thank you very much.

Best Regards,
Jingwang.


_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org<mailto:Pvfs2-developers@beowulf-underground.org>
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers


_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers

Reply via email to