Re: opensm with multiple IB subnets
Thank you, Yevgeny. I've modified my init script accordingly and
restarted opensm. For anyone else that may find it useful, here is
the "start" portion of my init script:
start () {
echo -n "Starting opensm: "
for GUID in `${IBSTAT_BIN} ${IBSTAT_ARG}`
do
export OSM_TMP_DIR="/tmp/opensm/${GUID}"
export OSM_CACHE_DIR="/var/cache/opensm/${GUID}"
export OSM_LOG_DIR="/var/log/opensm/${GUID}"
[ -d ${OSM_TMP_DIR} ] || mkdir -p ${OSM_TMP_DIR}
[ -d ${OSM_CACHE_DIR} ] || mkdir -p ${OSM_CACHE_DIR}
[ -d ${OSM_LOG_DIR} ] || mkdir ${OSM_LOG_DIR}
${OPENSM_BIN} --log_file ${OSM_LOG_DIR}/opensm.log
${OPENSM_ARG} ${GUID} > /dev/null
done
if [[ $RETVAL -eq 0 ]]; then
touch /var/lock/subsys/opensm
success
else
failure
fi
echo
}
On Wed, Apr 21, 2010 at 7:56 AM, Yevgeny Kliteynik
wrote:
> Ken,
>
> On 4/21/2010 3:07 AM, Ken Teague wrote:
>>
>> On Tue, Apr 20, 2010 at 2:13 PM, Ken Teague wrote:
>>>
>>> I have a 17-node cluster and each node has a single IB card that has
>>> 2x IB ports (ib0 and ib1).
>>
>> After doing a little more research, I confirmed that my understanding
>> of the manual page is correct. To run opensm for each GUID, I
>> modified my init script to run a for loop based on the information
>> returned from "ibstat -p".
>>
>>
>> I added this near the beginning of the script where the other
>> environment variables are located:
>>
>> OFA_HOME="/usr/local/sbin"
>> IBSTAT_BIN="${OFA_HOME}/ibstat"
>> IBSTAT_ARG="-p"
>> OPENSM_BIN="${OFA_HOME}/opensm"
>> OPENSM_ARG="-B -g"
>>
>>
>>
>> I replaced the single line which started opensm with this for loop:
>> for i in `${IBSTAT_BIN} ${IBSTAT_ARG}`
>> do
>> ${OPENSM_BIN} ${OPENSM_ARG} ${i}
>> done
>>
>>
>> If anyone has a more elegant way to handle this, I'm open to
>> suggestions. Many thanks.
>
> OpenSM dumps various files to /var/log and /var/cache/opensm folders.
> When you have more than one OpenSM process, they will all dump the
> same files, which is probably not a good idea.
>
> To change the output directories, set the OSM_TMP_DIR and
> OSM_CACHE_DIR env. variables to some other place.
> In addition, you need to make sure that each SM instance
> prints its log in a different place. You need to do
> something like this:
>
> foreach guid in guid_list
> export OSM_TMP_DIR=/tmp/osm_dump_dir${guid}
> export OSM_CACHE_DIR=/tmp/osm_dump_dir${guid}
> opensm --log_file /tmp/osm_dump_dir${guid}/osm.log -g ${guid} [your
> other options]
>
> -- Yevgeny
>
>> Ken
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: opensm with multiple IB subnets
On 4/22/2010 5:23 PM, Justin Clift wrote:
We should really put this on the wiki. :)
Good idea :)
-- Yevgeny
On 04/22/2010 12:56 AM, Yevgeny Kliteynik wrote:
OpenSM dumps various files to /var/log and /var/cache/opensm folders.
When you have more than one OpenSM process, they will all dump the
same files, which is probably not a good idea.
To change the output directories, set the OSM_TMP_DIR and
OSM_CACHE_DIR env. variables to some other place.
In addition, you need to make sure that each SM instance
prints its log in a different place. You need to do
something like this:
foreach guid in guid_list
export OSM_TMP_DIR=/tmp/osm_dump_dir${guid}
export OSM_CACHE_DIR=/tmp/osm_dump_dir${guid}
opensm --log_file /tmp/osm_dump_dir${guid}/osm.log -g ${guid} [your
other options]
-- Yevgeny
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: opensm with multiple IB subnets
We should really put this on the wiki. :)
On 04/22/2010 12:56 AM, Yevgeny Kliteynik wrote:
OpenSM dumps various files to /var/log and /var/cache/opensm folders.
When you have more than one OpenSM process, they will all dump the
same files, which is probably not a good idea.
To change the output directories, set the OSM_TMP_DIR and
OSM_CACHE_DIR env. variables to some other place.
In addition, you need to make sure that each SM instance
prints its log in a different place. You need to do
something like this:
foreach guid in guid_list
export OSM_TMP_DIR=/tmp/osm_dump_dir${guid}
export OSM_CACHE_DIR=/tmp/osm_dump_dir${guid}
opensm --log_file /tmp/osm_dump_dir${guid}/osm.log -g ${guid} [your
other options]
-- Yevgeny
--
Salasaga - Open Source eLearning IDE
http://www.salasaga.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: opensm with multiple IB subnets
Ken,
On 4/21/2010 3:07 AM, Ken Teague wrote:
On Tue, Apr 20, 2010 at 2:13 PM, Ken Teague wrote:
I have a 17-node cluster and each node has a single IB card that has
2x IB ports (ib0 and ib1).
After doing a little more research, I confirmed that my understanding
of the manual page is correct. To run opensm for each GUID, I
modified my init script to run a for loop based on the information
returned from "ibstat -p".
I added this near the beginning of the script where the other
environment variables are located:
OFA_HOME="/usr/local/sbin"
IBSTAT_BIN="${OFA_HOME}/ibstat"
IBSTAT_ARG="-p"
OPENSM_BIN="${OFA_HOME}/opensm"
OPENSM_ARG="-B -g"
I replaced the single line which started opensm with this for loop:
for i in `${IBSTAT_BIN} ${IBSTAT_ARG}`
do
${OPENSM_BIN} ${OPENSM_ARG} ${i}
done
If anyone has a more elegant way to handle this, I'm open to
suggestions. Many thanks.
OpenSM dumps various files to /var/log and /var/cache/opensm folders.
When you have more than one OpenSM process, they will all dump the
same files, which is probably not a good idea.
To change the output directories, set the OSM_TMP_DIR and
OSM_CACHE_DIR env. variables to some other place.
In addition, you need to make sure that each SM instance
prints its log in a different place. You need to do
something like this:
foreach guid in guid_list
export OSM_TMP_DIR=/tmp/osm_dump_dir${guid}
export OSM_CACHE_DIR=/tmp/osm_dump_dir${guid}
opensm --log_file /tmp/osm_dump_dir${guid}/osm.log -g ${guid} [your
other options]
-- Yevgeny
Ken
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: opensm with multiple IB subnets
On Tue, Apr 20, 2010 at 2:13 PM, Ken Teague wrote:
> I have a 17-node cluster and each node has a single IB card that has
> 2x IB ports (ib0 and ib1).
After doing a little more research, I confirmed that my understanding
of the manual page is correct. To run opensm for each GUID, I
modified my init script to run a for loop based on the information
returned from "ibstat -p".
I added this near the beginning of the script where the other
environment variables are located:
OFA_HOME="/usr/local/sbin"
IBSTAT_BIN="${OFA_HOME}/ibstat"
IBSTAT_ARG="-p"
OPENSM_BIN="${OFA_HOME}/opensm"
OPENSM_ARG="-B -g"
I replaced the single line which started opensm with this for loop:
for i in `${IBSTAT_BIN} ${IBSTAT_ARG}`
do
${OPENSM_BIN} ${OPENSM_ARG} ${i}
done
If anyone has a more elegant way to handle this, I'm open to
suggestions. Many thanks.
Ken
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
