If it would help, I have time and am willing to add notifier calls to this area of the code base. You'll still get the errors shown here as I always bury the notifier call behind the error check that surrounds these error messages to avoid impacting the critical path, but you would be able to gt syslog messages (or whatever channel you choose) as well.
Ralph On Dec 10, 2009, at 4:19 PM, Jeff Squyres wrote: > On Dec 10, 2009, at 5:06 PM, Brock Palen wrote: > >> I would like to try out the notifier framework, problem is I am having >> trouble finding documentation for it, I am digging around the website and >> not finding much. >> >> Currently we have a problem where hosts are throwing up errors like: >> [nyx0891.engin.umich.edu][[25560,1],45][btl_tcp_endpoint.c: >> 631:mca_btl_tcp_endpoint_complete_connect] connect() failed: >> Connection timed out (110) > > Yoinks. Any idea why this is happening? > >> We would like when this happens to notify us, so we can put time >> stamps on events going on on the network. Is this even possible with >> the frame work? See we don't show any interfaces coming up and down, >> or any errors on interfaces, so we are looking to isolate the problem >> more. Only the MPI library knows when this happens. > > It's not well documented. So let's start here... > > The first issue is that we currently only have notifier calls down in the > openib BTL -- not any of the others. :-( We put it there because there was > specific requests to be notified when IB links went down. We then used those > as a request for comment from the community, asking "do you like this? do you > want more?" We kinda got nothing back, and I'll admit that we kinda forgot > about it -- and therefore never added notifier calls elsewhere in the code. > :-\ > > We designed the notifier in OMPI to be trivially easy to use throughout the > code base -- it's just adding a single function call where the error occurs. > Would you, perchance, be interested in adding any of these in the TCP BTL? > I'd be happy to point you in the right direction... :-) > > After that, it's just a matter of enabling a notifier: > > mpirun --mca notifier syslog ... > > Each notifier has some MCA params that are fairly obvious -- use: > > ompi_info --param notifier all > > to see them. There's 3 notifier plugins: > > - command: execute any arbitrary command. It must run in finite (short) > time. You use MCA params to set the command (we can pass some strings down > to the command; see the ompi_info help string for more details), and set a > timeout such that if the command runs for that many seconds without exiting, > we'll kill it. > > - syslog: because it was simple to do -- we just output a string to the > syslog. > > - twitter: because it was fun to do. ;-) Actually, the rationale was that > you can tweet to a private feed and then slave an RSS reader to it to see if > anything happens. It will need to be able to reach the general internet > (i.e., twitter.com); proxies are not supported. Set your twitter > username/password via MCA params. > > -- > Jeff Squyres > jsquy...@cisco.com > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
