On Jan 11, 2007, at 2:51 PM, Julian Martin Kunkel wrote:
Hi,
These results are quite interesting and the possible throughput you
get with
IB is amazing.
Do you mean Pete's IB 4x numbers or my MX-10G numbers? Or both?
However, I think you get into trouble to find the bottleneck of the
operation and the reason for the observed gaps due to the
complexity of the software
stack.
Maybe you could benefit from using TAS instead of DBPF so that you
could avoid
network activity at all. (Also you might use only one buffer to
already fill
the network).
I am not sure that I follow you here. Ideally, I only want to measure
network activity and PVFS2 overhead. I would prefer to avoid
measuring disk activity but these old nodes do not have enough memory
to use ramfs well.
My MX-10G results are from newer nodes that have enough memory to use
ramfs effectively. I can't keep them tied up for bmi_mx development
as long as I can these older nodes. :-)
On the other hand you could use the pvfs2-hint-branch which
provides you with
better MPE logging on the server side, we have some tools to
convert and
merge client and pvfs2-server logs and show the results. I could
upgrade the
current pvfs2-hint-branch with your patches (which is currently
somewhere at
release 2.5). The reason we need parts of the advanced logging is
that logs
have a problem on the server side if multiple start events occur
before the
end events happen, for example if you use multiple flow streams.
@Scott
I have seen you solved it by using events and wonder which tool you
have used
to create states out of the events.
You said you have problems with the MPE log on the server, maybe we
could help
you if you give details ?
Best regards,
Julian
I am using MPE. Since I am not using MPI, I compile mpich2 with
CLFAGS="-DCLOG_NOMPI". I then add "-lmpe_nompi" to my LIBS and the
path to mpich2 to my LDFLAGS.
In my initialization function, I have:
#if BMX_LOGGING
MPE_Init_log();
send_start = MPE_Log_get_event_number();
send_finish = MPE_Log_get_event_number();
recv_start = MPE_Log_get_event_number();
recv_finish = MPE_Log_get_event_number();
sendunex_start = MPE_Log_get_event_number();
sendunex_finish = MPE_Log_get_event_number();
recvunex_start = MPE_Log_get_event_number();
recvunex_finish = MPE_Log_get_event_number();
MPE_Describe_state(send_start, send_finish, "Send", "red");
MPE_Describe_state(recv_start, recv_finish, "Recv", "blue");
MPE_Describe_state(sendunex_start, sendunex_finish,
"SendUnex", "orange");
MPE_Describe_state(recvunex_start, recvunex_finish,
"RecvUnex", "green");
#endif
I then insert MPE_Log_event() in my functions that send, recv and
complete messages.
I can now get server logs. My SERVER_LDFLAGS were wrong. Also, on the
server, I had to specify an absolute path (I did not on the client).
I would be interested in merging the logs if you can provide some
tools or insight.
Thanks,
Scott
_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
