Re: Ummunotify: progress at last!
I wonder, is that a win? I guess you don't even have to pin it, just do copy_to_user() to update the counter, but mmap doesn't seem so bad. Not sure you can call copy_to_user from a mmu_notifier callback? What if it faults? Good point... we could defer it to a context where we could block but... I think the idea would be that by letting user space select the counter location it could place it in a sensible cachline/etc and probably avoid a pointer indirection. OK, I have to look at the perf stuff. That makes sense but OTOH userspace could (unknowingly) pick a highmen page and then we end up having to do kmap_atomic or something from an mmu notifier callback. -- Roland Dreier rola...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: opensm/main.c: foce stdout to be line-buffered
On 23/Mar/10 23:36, Yevgeny Kliteynik wrote: On 23/Mar/10 18:37, Sasha Khapyorsky wrote: On 14:25 Tue 23 Mar , Yevgeny Kliteynik wrote: I'm running opensm somefile, and I don't see SM's stdout (such as SUBNET UP message, or new cached options after SIGHUP), because when stdout is assigned to file and not terminal, it is handled differently. Instead of flushing on printing '\n', it becomes buffered, which means that you don't control when is this buffer flushed. My fix forces stdout to always flush stdout when printing '\n'. It has no effect when stdout is assigned to terminal, and it changes buffering when SM's stdout is redirected. More details about stdout/stderr buffering: http://www.pixelbeat.org/programming/stdio_buffering/ There you can find couple of ways to workaround this issue, for example: stdbuf -o L opensm somefile This is not a usual shell command, nor some common tool that is used in Linux distros. It's just a tool that is provided in some package called coreutils 7.5. Correction: the coreutils package is common, stdbuf isn't - it was added in coreutils 7.5. For instance, RH 5.3 has version 5.97, SLES 11 has version 6.12 -- Yevgeny I would prefer to not change an external settings so the program would work as expected. IMHO, on the contrary - the expected behavior would be achieved with the patch. But in any case, what exactly seems problematic here? I can't see any impact on any aspect whatsoever. It is somehow related to performance (more flushes when the stream is line-buffered instead of just buffered), but we're talking about stdout here, not the log file, so the performance is not affected too. -- Yevgeny Sasha -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Dimension port order file support
Hi Dale, On 18:06 Wed 03 Mar , Dale Purdy wrote: Provide a means to specify on a per switch basis the mapping (order) between switch ports and dimensions for Dimension Order Routing. This allows the DOR routing engine to be used when the cabling is not properly aligned for DOR, either initially, or for an upgrade. Nice stuff. Is this something useful with ! '-R dor'? Signed-off-by: Dale Purdy pu...@sgi.com The patch itself is broken somehow - it has double space at start of non-changed line (it is fixable with sed -e 's/^ / /', so don't resend patch only for this). Some more minor comments are below. --- opensm/include/opensm/osm_subnet.h |1 + opensm/include/opensm/osm_switch.h | 30 + opensm/man/opensm.8.in | 31 +++-- opensm/opensm/main.c | 13 - opensm/opensm/osm_subnet.c |7 ++ opensm/opensm/osm_switch.c |2 +- opensm/opensm/osm_ucast_mgr.c | 120 7 files changed, 195 insertions(+), 9 deletions(-) diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h index 3970e98..e4e298e 100644 --- a/opensm/include/opensm/osm_subnet.h +++ b/opensm/include/opensm/osm_subnet.h @@ -186,6 +186,7 @@ typedef struct osm_subn_opt { uint16_t console_port; char *port_prof_ignore_file; char *hop_weights_file; + char *dimn_ports_file; boolean_t port_profile_switch_nodes; boolean_t sweep_on_trap; char *routing_engine_names; diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h index cb6e5ac..1c6807e 100644 --- a/opensm/include/opensm/osm_switch.h +++ b/opensm/include/opensm/osm_switch.h @@ -100,6 +100,7 @@ typedef struct osm_switch { uint16_t num_hops; uint8_t **hops; osm_port_profile_t *p_prof; + uint8_t *dimn_ports; uint8_t *lft; uint8_t *new_lft; uint16_t lft_size; @@ -871,6 +872,35 @@ static inline uint8_t osm_switch_get_mft_max_position(IN osm_switch_t * p_sw) * RETURN VALUE */ +/f* OpenSM: Switch/osm_switch_get_dimn_port +* NAME +*osm_switch_get_dimn_port +* +* DESCRIPTION +* Get the routing ordered port +* +* SYNOPSIS +*/ +static inline uint8_t osm_switch_get_dimn_port(IN const osm_switch_t * p_sw, +IN uint8_t port_num) +{ + CL_ASSERT(p_sw); + if (p_sw-dimn_ports == NULL) + return port_num; + return p_sw-dimn_ports[port_num]; +} +/* +* PARAMETERS +*p_sw +*[in] Pointer to the switch object. +* +*port_num +*[in] Port number in the switch +* +* RETURN VALUES +*Returns the port number ordered for routing purposes. +*/ + /f* OpenSM: Switch/osm_switch_recommend_path * NAME * osm_switch_recommend_path diff --git a/opensm/man/opensm.8.in b/opensm/man/opensm.8.in index 7aca8f9..9053611 100644 --- a/opensm/man/opensm.8.in +++ b/opensm/man/opensm.8.in @@ -37,6 +37,7 @@ opensm \- InfiniBand subnet manager and administration (SM/SA) [\-console-port port] [\-i(gnore-guids) equalize-ignore-guids-file] [\-w | \-\-hop_weights_file path to file] +[\-O | \-\-dimn_ports_file path to file] [\-f log file path | \-\-log_file log file path ] [\-L | \-\-log_limit size in MB] [\-e(rase_log_file)] [\-P(config) partition config file ] @@ -273,6 +274,16 @@ factor of 1. Lines starting with # are comments. Weights affect only the output route from the port, so many useful configurations will require weights to be specified in pairs. .TP +\fB\-O\fR, \fB\-\-dimn_ports_file\fR path to file +This option provides a mapping between hypercube dimensions and ports +on a per switch basis for the DOR routing engine. The file consists +of lines containing a switch node GUID (specified as a 64 bit hex +number, with leading 0x) followed by a list of non-zero port numbers, +separated by spaces, one switch per line. The order for the port +numbers is in one to one correspondence to the dimensions. Ports not +listed on a line are assigned to the remaining dimensions, in port +order. Anything after a # is a comment. +.TP \fB\-x\fR, \fB\-\-honor_guid2lid\fR This option forces OpenSM to honor the guid2lid file, when it comes out of Standby state, if such file exists @@ -969,17 +980,20 @@ algorithm and so uses shortest paths. Instead of spreading traffic out across different paths with the same shortest distance, it chooses among the available shortest paths based on an ordering of dimensions. Each port must be consistently cabled to represent a hypercube -dimension or a mesh dimension. Paths are grown from a destination -back to a source using the lowest dimension (port) of available paths -at each step. This provides the ordering necessary to avoid deadlock. +dimension or a mesh dimension.
[PATCH] opensm: fixing compilation issues in some header files
All the compilation issues refer to implicit casting from void* to some_struct_t* Signed-off-by: Yevgeny Kliteynik klit...@dev.mellanox.co.il --- opensm/include/opensm/osm_pkey.h |8 +--- opensm/include/opensm/osm_port.h |4 ++-- opensm/include/opensm/osm_subnet.h |2 +- 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/opensm/include/opensm/osm_pkey.h b/opensm/include/opensm/osm_pkey.h index d10479d..53e9657 100644 --- a/opensm/include/opensm/osm_pkey.h +++ b/opensm/include/opensm/osm_pkey.h @@ -252,7 +252,8 @@ static inline ib_pkey_table_t *osm_pkey_tbl_block_get(const osm_pkey_tbl_t * uint16_t block) { return ((block cl_ptr_vector_get_size(p_pkey_tbl-blocks)) ? - cl_ptr_vector_get(p_pkey_tbl-blocks, block) : NULL); + (ib_pkey_table_t *)cl_ptr_vector_get( + p_pkey_tbl-blocks, block) : NULL); }; /* @@ -282,8 +283,9 @@ static inline ib_pkey_table_t *osm_pkey_tbl_new_block_get(const osm_pkey_tbl_t * p_pkey_tbl, uint16_t block) { - return (block cl_ptr_vector_get_size(p_pkey_tbl-new_blocks)) ? - cl_ptr_vector_get(p_pkey_tbl-new_blocks, block) : NULL; + return ((block cl_ptr_vector_get_size(p_pkey_tbl-new_blocks)) ? + (ib_pkey_table_t *)cl_ptr_vector_get( + p_pkey_tbl-new_blocks, block) : NULL); }; /f* OpenSM: osm_pkey_tbl_set_new_entry diff --git a/opensm/include/opensm/osm_port.h b/opensm/include/opensm/osm_port.h index ff0a178..8c68c99 100644 --- a/opensm/include/opensm/osm_port.h +++ b/opensm/include/opensm/osm_port.h @@ -549,7 +549,7 @@ static inline void osm_physp_set_slvl_tbl(IN osm_physp_t * p_physp, CL_ASSERT(p_slvl_tbl); CL_ASSERT(osm_physp_is_valid(p_physp)); - p_tbl = cl_ptr_vector_get(p_physp-slvl_by_port, in_port_num); + p_tbl = (ib_slvl_table_t *)cl_ptr_vector_get(p_physp-slvl_by_port, in_port_num); *p_tbl = *p_slvl_tbl; } @@ -590,7 +590,7 @@ static inline ib_slvl_table_t *osm_physp_get_slvl_tbl(IN const osm_physp_t * ib_slvl_table_t *p_tbl; CL_ASSERT(osm_physp_is_valid(p_physp)); - p_tbl = cl_ptr_vector_get(p_physp-slvl_by_port, in_port_num); + p_tbl = (ib_slvl_table_t *)cl_ptr_vector_get(p_physp-slvl_by_port, in_port_num); return p_tbl; } diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h index 3970e98..2eef9c7 100644 --- a/opensm/include/opensm/osm_subnet.h +++ b/opensm/include/opensm/osm_subnet.h @@ -1015,7 +1015,7 @@ struct osm_mgrp *osm_get_mgrp_by_mgid(IN osm_subn_t * subn, IN ib_gid_t * mgid); */ static inline struct osm_mgrp_box *osm_get_mbox_by_mlid(osm_subn_t const *p_subn, ib_net16_t mlid) { - return p_subn-mboxes[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; + return (struct osm_mgrp_box *)p_subn-mboxes[cl_ntoh16(mlid) - IB_LID_MCAST_START_HO]; } /* * PARAMETERS -- 1.5.1.4 -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
ibstat stuck in state initialized after reboot
I hope this is the correct place to get help with the problem I have. I have an IB fabric running on a Cisco SFS switch with a 7000D as the subnet manager and the whole thing has been running great for well over a year now, but today I noticed that after any node gets rebooted its IB link doesn't initialize. This has happened on 4 hosts now. What I see is as follows: [r...@compute-2-7 ~]# ibstat CA 'mthca0' CA type: MT25204 Number of ports: 1 Firmware version: 1.2.917 Hardware version: 20 Node GUID: 0x0005ad0c0990 System image GUID: 0x0005ad000100d050 Port 1: State: Initializing Physical state: LinkUp Rate: 20 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02510a68 Port GUID: 0x0005ad0c0991 I don't know much about subnet managers, since ours is in hardware and we've never had to configure anything on it, but I can login to the device and it isn't showing any errors. On a node that hasn't been rebooted recently and is still working I can see what appears to be a working subnet manager: [r...@compute-2-10 ~]# sminfo sminfo: sm lid 2 sm guid 0x5ad1df2a0, activity count 2146213408 priority 10 state 3 SMINFO_MASTER The same command on a non-working node shows this: [r...@compute-2-7 ~]# sminfo sminfo: sm lid 0 sm guid 0x0, activity count 0 priority 0 state 2 SMINFO_STANDBY So far I have reseated all the cables involved on both ends and I have moved the cables on the switch end to new ports and none of that has made a difference even after reboots. I am hoping to find a node that I can take offline tomorrow so I can actually test the cables, but since this seems to be happening to any host that reboots it doesn't appear to be a cabling problem. Can anybody suggest where I should go from here? Is there anything I can do from a working or non-working host to diagnose the problem? Should I try rebooting the subnet manager switch? Will that affect the rest of the fabric? Thanks, Mike Robbert -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ibstat stuck in state initialized after reboot
On Wed, 24 Mar 2010 10:26:02 -0600 Michael Robbert mrobb...@mines.edu wrote: I hope this is the correct place to get help with the problem I have. I have an IB fabric running on a Cisco SFS switch with a 7000D as the subnet manager and the whole thing has been running great for well over a year now, but today I noticed that after any node gets rebooted its IB link doesn't initialize. This has happened on 4 hosts now. What I see is as follows: [r...@compute-2-7 ~]# ibstat CA 'mthca0' CA type: MT25204 Number of ports: 1 Firmware version: 1.2.917 Hardware version: 20 Node GUID: 0x0005ad0c0990 System image GUID: 0x0005ad000100d050 Port 1: State: Initializing Physical state: LinkUp Rate: 20 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02510a68 Port GUID: 0x0005ad0c0991 I don't know much about subnet managers, since ours is in hardware and we've never had to configure anything on it, but I can login to the device and it isn't showing any errors. On a node that hasn't been rebooted recently and is still working I can see what appears to be a working subnet manager: [r...@compute-2-10 ~]# sminfo sminfo: sm lid 2 sm guid 0x5ad1df2a0, activity count 2146213408 priority 10 state 3 SMINFO_MASTER The same command on a non-working node shows this: [r...@compute-2-7 ~]# sminfo sminfo: sm lid 0 sm guid 0x0, activity count 0 priority 0 state 2 SMINFO_STANDBY So far I have reseated all the cables involved on both ends and I have moved the cables on the switch end to new ports and none of that has made a difference even after reboots. I am hoping to find a node that I can take offline tomorrow so I can actually test the cables, but since this seems to be happening to any host that reboots it doesn't appear to be a cabling problem. Can anybody suggest where I should go from here? Is there anything I can do from a working or non-working host to diagnose the problem? Should I try rebooting the subnet manager switch? Will that affect the rest of the fabric? Have you spoken to Cisco about the problem? You say you can log into the device (the SM switch?) if so talk to Cisco about how you may be able to restart the SM there. It does sound like the SM on the switch is failing to transition the links. If you can restart the SM on the switch I would try that first. Otherwise yes rebooting the switch is probably your best bet, and yes it will affect the fabric, although I can't say how much without knowing the topology. Ira Thanks, Mike Robbert -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://*vger.kernel.org/majordomo-info.html -- Ira Weiny Math Programmer/Computer Scientist Lawrence Livermore National Lab 925-423-8008 wei...@llnl.gov -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ibstat stuck in state initialized after reboot
Ira, Thanks for the quick response. That is what I was afraid of. I've been looking through the switch documentation, but it doesn't cover starting, stopping, or even checking the status of the SM service. I'll look into opening a TAC case, but since Cisco has gotten out of the IB business I'm not looking forward to seeing what kind of product support they still have. I can tell you a little more about our topology since it is pretty simple. All of our hosts are connected to the single large SFS switch, then the 7000D which is our subnet-manager is only plugged into that larger switch. Thanks for the help and wish me luck with support! Mike On Mar 24, 2010, at 10:38 AM, Ira Weiny wrote: On Wed, 24 Mar 2010 10:26:02 -0600 Michael Robbert mrobb...@mines.edu wrote: I hope this is the correct place to get help with the problem I have. I have an IB fabric running on a Cisco SFS switch with a 7000D as the subnet manager and the whole thing has been running great for well over a year now, but today I noticed that after any node gets rebooted its IB link doesn't initialize. This has happened on 4 hosts now. What I see is as follows: [r...@compute-2-7 ~]# ibstat CA 'mthca0' CA type: MT25204 Number of ports: 1 Firmware version: 1.2.917 Hardware version: 20 Node GUID: 0x0005ad0c0990 System image GUID: 0x0005ad000100d050 Port 1: State: Initializing Physical state: LinkUp Rate: 20 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02510a68 Port GUID: 0x0005ad0c0991 I don't know much about subnet managers, since ours is in hardware and we've never had to configure anything on it, but I can login to the device and it isn't showing any errors. On a node that hasn't been rebooted recently and is still working I can see what appears to be a working subnet manager: [r...@compute-2-10 ~]# sminfo sminfo: sm lid 2 sm guid 0x5ad1df2a0, activity count 2146213408 priority 10 state 3 SMINFO_MASTER The same command on a non-working node shows this: [r...@compute-2-7 ~]# sminfo sminfo: sm lid 0 sm guid 0x0, activity count 0 priority 0 state 2 SMINFO_STANDBY So far I have reseated all the cables involved on both ends and I have moved the cables on the switch end to new ports and none of that has made a difference even after reboots. I am hoping to find a node that I can take offline tomorrow so I can actually test the cables, but since this seems to be happening to any host that reboots it doesn't appear to be a cabling problem. Can anybody suggest where I should go from here? Is there anything I can do from a working or non-working host to diagnose the problem? Should I try rebooting the subnet manager switch? Will that affect the rest of the fabric? Have you spoken to Cisco about the problem? You say you can log into the device (the SM switch?) if so talk to Cisco about how you may be able to restart the SM there. It does sound like the SM on the switch is failing to transition the links. If you can restart the SM on the switch I would try that first. Otherwise yes rebooting the switch is probably your best bet, and yes it will affect the fabric, although I can't say how much without knowing the topology. Ira Thanks, Mike Robbert -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://*vger.kernel.org/majordomo-info.html -- Ira Weiny Math Programmer/Computer Scientist Lawrence Livermore National Lab 925-423-8008 wei...@llnl.gov -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: ibstat stuck in state initialized after reboot
http://www.cisco.com/en/US/docs/server_nw_virtual/7024/release_4.1/hardware/installation/guide/7024hig.pdf smControl Starts and stops the embedded subnet manager. Syntax: smControl start | stop | restart | status Thanks, Don Meyer Senior Network/System Engineer/Programmer US+ (253) 371-9532 iNet 8-371-9532 *Other names and brands may be claimed as the property of others -Original Message- From: linux-rdma-ow...@vger.kernel.org [mailto:linux-rdma-ow...@vger.kernel.org] On Behalf Of Michael Robbert Sent: Wednesday, March 24, 2010 10:00 AM To: Ira Weiny Cc: linux-rdma@vger.kernel.org Subject: Re: ibstat stuck in state initialized after reboot Ira, Thanks for the quick response. That is what I was afraid of. I've been looking through the switch documentation, but it doesn't cover starting, stopping, or even checking the status of the SM service. I'll look into opening a TAC case, but since Cisco has gotten out of the IB business I'm not looking forward to seeing what kind of product support they still have. I can tell you a little more about our topology since it is pretty simple. All of our hosts are connected to the single large SFS switch, then the 7000D which is our subnet-manager is only plugged into that larger switch. Thanks for the help and wish me luck with support! Mike On Mar 24, 2010, at 10:38 AM, Ira Weiny wrote: On Wed, 24 Mar 2010 10:26:02 -0600 Michael Robbert mrobb...@mines.edu wrote: I hope this is the correct place to get help with the problem I have. I have an IB fabric running on a Cisco SFS switch with a 7000D as the subnet manager and the whole thing has been running great for well over a year now, but today I noticed that after any node gets rebooted its IB link doesn't initialize. This has happened on 4 hosts now. What I see is as follows: [r...@compute-2-7 ~]# ibstat CA 'mthca0' CA type: MT25204 Number of ports: 1 Firmware version: 1.2.917 Hardware version: 20 Node GUID: 0x0005ad0c0990 System image GUID: 0x0005ad000100d050 Port 1: State: Initializing Physical state: LinkUp Rate: 20 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02510a68 Port GUID: 0x0005ad0c0991 I don't know much about subnet managers, since ours is in hardware and we've never had to configure anything on it, but I can login to the device and it isn't showing any errors. On a node that hasn't been rebooted recently and is still working I can see what appears to be a working subnet manager: [r...@compute-2-10 ~]# sminfo sminfo: sm lid 2 sm guid 0x5ad1df2a0, activity count 2146213408 priority 10 state 3 SMINFO_MASTER The same command on a non-working node shows this: [r...@compute-2-7 ~]# sminfo sminfo: sm lid 0 sm guid 0x0, activity count 0 priority 0 state 2 SMINFO_STANDBY So far I have reseated all the cables involved on both ends and I have moved the cables on the switch end to new ports and none of that has made a difference even after reboots. I am hoping to find a node that I can take offline tomorrow so I can actually test the cables, but since this seems to be happening to any host that reboots it doesn't appear to be a cabling problem. Can anybody suggest where I should go from here? Is there anything I can do from a working or non-working host to diagnose the problem? Should I try rebooting the subnet manager switch? Will that affect the rest of the fabric? Have you spoken to Cisco about the problem? You say you can log into the device (the SM switch?) if so talk to Cisco about how you may be able to restart the SM there. It does sound like the SM on the switch is failing to transition the links. If you can restart the SM on the switch I would try that first. Otherwise yes rebooting the switch is probably your best bet, and yes it will affect the fabric, although I can't say how much without knowing the topology. Ira Thanks, Mike Robbert -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://*vger.kernel.org/majordomo-info.html -- Ira Weiny Math Programmer/Computer Scientist Lawrence Livermore National Lab 925-423-8008 wei...@llnl.gov -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Ummunotify: progress at last!
On Tue, Mar 23, 2010 at 10:59:42PM -0700, Roland Dreier wrote: That is all definitely doable. I wonder if it's better to get rid of the dedicated fd though. After all, having the fd means a fancy app can do poll() or sigio or whatever internally. Being able to integrate into an fd-driven event loop seems like a pretty big thing to give up. Not sure, adding the fd is only a small amount of work. But on the other hand integrating with a completion channel might bet better suited to verbs's API, and even less work.. Also, having a uverbs syscall that is exactly like read() seems a bit of a stretch, even within the ugliness that is the uverbs interface. (We love all our children, but sometimes we have to be realistic) I agree, but if the FD is eliminated then that seems like the best bet - it wouldn't be exactly like read, it would be a multiplexed uverb command executed over the uverbs fd. Jason -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ibstat stuck in state initialized after reboot
I just discovered another interesting point. I tried to start opensm on one of my hosts and it went into STANDBY state. Here is the log of it trying to start up: Mar 24 12:23:25 117170 [66DAC170] 0x80 - OpenSM 3.3.5 Entering DISCOVERING state Mar 24 12:23:25 117863 [66DAC170] 0x02 - osm_vendor_init: 1000 pending umads specified Mar 24 12:23:25 118022 [66DAC170] 0x80 - Entering DISCOVERING state Mar 24 12:23:25 120961 [66DAC170] 0x02 - osm_vendor_bind: Binding to port 0x5ad0bf1e1 Mar 24 12:23:25 129023 [66DAC170] 0x02 - osm_vendor_bind: Binding to port 0x5ad0bf1e1 Mar 24 12:23:25 129069 [66DAC170] 0x02 - osm_opensm_bind: Setting IS_SM on port 0x0005ad0bf1e1 Mar 24 12:23:26 120384 [42E1E940] 0x01 - umad_receiver: ERR 5411: DR SMP Send completed with error -- dropping Method 0x1, Attr 0x11, TID 0xf1a51, Hop Ptr: 0x0 Mar 24 12:23:26 120444 [42E1E940] 0x01 - Received SMP on a 4 hop path: Initial path = 0,0,0,0,0, Return path = 0,0,0,0,0 Mar 24 12:23:26 120461 [42E1E940] 0x01 - sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT): SubnGet(NodeInfo), attr_mod 0x0, TID 0x1a51 Using default GUID 0x5ad0bf1e1 Entering STANDBY state Mar 24 12:23:26 120538 [42C1D940] 0x80 - Entering STANDBY state Does that change the diagnosis at all? I'm still waiting for a response from t...@cisco.com Thanks, Mike On Mar 24, 2010, at 11:34 AM, Michael Robbert wrote: Interesting note! The 7024 is our large switch where all the hosts are connected, but I was told that we were sold the 7000D because the 7024 didn't have a subnet manager. Unfortunately the 7000D has a different CLI and that command is not available and I don't have the password for our 7024 so I can't log onto it. On another note I just noticed the uptime on the 7000D is just over 1 day so that must have been the start of the problem, but I have no idea why it rebooted nor why it didn't come up working. I'm pretty sure we tested a reboot of the device during acceptance testing. Oh, I just got your second note: == BTW, I highly recommend running the opensm on a server instead of using the sm on the switch. We found running the sm on the switch was much less reliable. I also recommend using a server dedicated to opensm only. == I will take that into consideration, but we bought this as a turn-key solution from Dell. They designed it and we had no experience with IB so we trusted their knowledge. Thanks, Mike On Mar 24, 2010, at 11:12 AM, Meyer, Donald J wrote: http://www.cisco.com/en/US/docs/server_nw_virtual/7024/release_4.1/hardware/installation/guide/7024hig.pdf smControl Starts and stops the embedded subnet manager. Syntax: smControl start | stop | restart | status Thanks, Don Meyer Senior Network/System Engineer/Programmer US+ (253) 371-9532 iNet 8-371-9532 *Other names and brands may be claimed as the property of others -Original Message- From: linux-rdma-ow...@vger.kernel.org [mailto:linux-rdma-ow...@vger.kernel.org] On Behalf Of Michael Robbert Sent: Wednesday, March 24, 2010 10:00 AM To: Ira Weiny Cc: linux-rdma@vger.kernel.org Subject: Re: ibstat stuck in state initialized after reboot Ira, Thanks for the quick response. That is what I was afraid of. I've been looking through the switch documentation, but it doesn't cover starting, stopping, or even checking the status of the SM service. I'll look into opening a TAC case, but since Cisco has gotten out of the IB business I'm not looking forward to seeing what kind of product support they still have. I can tell you a little more about our topology since it is pretty simple. All of our hosts are connected to the single large SFS switch, then the 7000D which is our subnet-manager is only plugged into that larger switch. Thanks for the help and wish me luck with support! Mike On Mar 24, 2010, at 10:38 AM, Ira Weiny wrote: On Wed, 24 Mar 2010 10:26:02 -0600 Michael Robbert mrobb...@mines.edu wrote: I hope this is the correct place to get help with the problem I have. I have an IB fabric running on a Cisco SFS switch with a 7000D as the subnet manager and the whole thing has been running great for well over a year now, but today I noticed that after any node gets rebooted its IB link doesn't initialize. This has happened on 4 hosts now. What I see is as follows: [r...@compute-2-7 ~]# ibstat CA 'mthca0' CA type: MT25204 Number of ports: 1 Firmware version: 1.2.917 Hardware version: 20 Node GUID: 0x0005ad0c0990 System image GUID: 0x0005ad000100d050 Port 1: State: Initializing Physical state: LinkUp Rate: 20 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02510a68 Port GUID: 0x0005ad0c0991 I don't know much
Re: ibstat stuck in state initialized after reboot
On Wed, 24 Mar 2010 11:34:02 -0600 Michael Robbert mrobb...@mines.edu wrote: Interesting note! The 7024 is our large switch where all the hosts are connected, but I was told that we were sold the 7000D because the 7024 didn't have a subnet manager. Unfortunately the 7000D has a different CLI and that command is not available and I don't have the password for our 7024 so I can't log onto it. On another note I just noticed the uptime on the 7000D is just over 1 day so that must have been the start of the problem, but I have no idea why it rebooted nor why it didn't come up working. I'm pretty sure we tested a reboot of the device during acceptance testing. Oh, I just got your second note: == BTW, I highly recommend running the opensm on a server instead of using the sm on the switch. We found running the sm on the switch was much less reliable. I also recommend using a server dedicated to opensm only. == I will second this. OpenSM has come a long way since the time Cisco was selling IB switches. If I understand your situation you don't even need the 7000D you could just remove it and run OpenSM on a management node. If you can afford it adding a node for OpenSM would be nice but I am not sure you _need_ it. OpenSM is now managing many of the largest IB networks out there, on a 288 node system it will have no problems at all out of the box. :D Ira I will take that into consideration, but we bought this as a turn-key solution from Dell. They designed it and we had no experience with IB so we trusted their knowledge. snip -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ibstat stuck in state initialized after reboot
On Wed, Mar 24, 2010 at 2:25 PM, Ira Weiny wei...@llnl.gov wrote: On Wed, 24 Mar 2010 11:34:02 -0600 Michael Robbert mrobb...@mines.edu wrote: I will second this. OpenSM has come a long way since the time Cisco was selling IB switches. If I understand your situation you don't even need the 7000D you could just remove it and run OpenSM on a management node. If you can afford it adding a node for OpenSM would be nice but I am not sure you _need_ it. OpenSM is now managing many of the largest IB networks out there, on a 288 node system it will have no problems at all out of the box. Can you provide any guidelines to determine when a dedicated management node is beneficial? BTW, we also found that OpenSM is superior to to the SM embedded in our switches. Chuck -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ibstat stuck in state initialized after reboot
I've got good news. I was able to get opensm to take control. I gave it a priority of 15 and rebooted the 7000D. Unfortunately I'm not sure I can leave it like this forever. The only host I had with opensm installed is my test front end for an OS upgrade I'm testing. We're moving from Rocks 4.3 to Rocks 5.3 (RHEL 4.5 to RHEL 5.4). I may need to reboot this node from time to time over the next couple of weeks, but at least I'm working right now. So you say that a 288 node system will work out of the box, what happens when you hit 289? Is that a magic number or just an estimate. We have 268 compute nodes plus a few auxiliary nodes so we're pretty close to that number. Thanks, Mike On Mar 24, 2010, at 12:25 PM, Ira Weiny wrote: On Wed, 24 Mar 2010 11:34:02 -0600 Michael Robbert mrobb...@mines.edu wrote: Interesting note! The 7024 is our large switch where all the hosts are connected, but I was told that we were sold the 7000D because the 7024 didn't have a subnet manager. Unfortunately the 7000D has a different CLI and that command is not available and I don't have the password for our 7024 so I can't log onto it. On another note I just noticed the uptime on the 7000D is just over 1 day so that must have been the start of the problem, but I have no idea why it rebooted nor why it didn't come up working. I'm pretty sure we tested a reboot of the device during acceptance testing. Oh, I just got your second note: == BTW, I highly recommend running the opensm on a server instead of using the sm on the switch. We found running the sm on the switch was much less reliable. I also recommend using a server dedicated to opensm only. == I will second this. OpenSM has come a long way since the time Cisco was selling IB switches. If I understand your situation you don't even need the 7000D you could just remove it and run OpenSM on a management node. If you can afford it adding a node for OpenSM would be nice but I am not sure you _need_ it. OpenSM is now managing many of the largest IB networks out there, on a 288 node system it will have no problems at all out of the box. :D Ira I will take that into consideration, but we bought this as a turn-key solution from Dell. They designed it and we had no experience with IB so we trusted their knowledge. snip -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ibstat stuck in state initialized after reboot
On Wed, 24 Mar 2010 13:42:55 -0600 Michael Robbert mrobb...@mines.edu wrote: I've got good news. I was able to get opensm to take control. I gave it a priority of 15 and rebooted the 7000D. Unfortunately I'm not sure I can leave it like this forever. The only host I had with opensm installed is my test front end for an OS upgrade I'm testing. We're moving from Rocks 4.3 to Rocks 5.3 (RHEL 4.5 to RHEL 5.4). I may need to reboot this node from time to time over the next couple of weeks, but at least I'm working right now. So you say that a 288 node system will work out of the box, what happens when you hit 289? Is that a magic number or just an estimate. We have 268 compute nodes plus a few auxiliary nodes so we're pretty close to that number. Nothing will happen when you hit 289. I chose that number because a 7024 has 288 ports which I assumed was the size of your cluster. There are those running large clusters (thousands of nodes) who have made some changes to OpenSM for specialized topologies or better SA scalability. In the future those changes should be in OpenSM so as you grow, OpenSM grows with you! :-D Ira Thanks, Mike On Mar 24, 2010, at 12:25 PM, Ira Weiny wrote: On Wed, 24 Mar 2010 11:34:02 -0600 Michael Robbert mrobb...@mines.edu wrote: Interesting note! The 7024 is our large switch where all the hosts are connected, but I was told that we were sold the 7000D because the 7024 didn't have a subnet manager. Unfortunately the 7000D has a different CLI and that command is not available and I don't have the password for our 7024 so I can't log onto it. On another note I just noticed the uptime on the 7000D is just over 1 day so that must have been the start of the problem, but I have no idea why it rebooted nor why it didn't come up working. I'm pretty sure we tested a reboot of the device during acceptance testing. Oh, I just got your second note: == BTW, I highly recommend running the opensm on a server instead of using the sm on the switch. We found running the sm on the switch was much less reliable. I also recommend using a server dedicated to opensm only. == I will second this. OpenSM has come a long way since the time Cisco was selling IB switches. If I understand your situation you don't even need the 7000D you could just remove it and run OpenSM on a management node. If you can afford it adding a node for OpenSM would be nice but I am not sure you _need_ it. OpenSM is now managing many of the largest IB networks out there, on a 288 node system it will have no problems at all out of the box. :D Ira I will take that into consideration, but we bought this as a turn-key solution from Dell. They designed it and we had no experience with IB so we trusted their knowledge. snip -- Ira Weiny Math Programmer/Computer Scientist Lawrence Livermore National Lab 925-423-8008 wei...@llnl.gov -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html