Re: [openib-general] [PATCH 3/6] [RFC] iser initiator
On Mon, Apr 03, 2006 at 02:59:03PM +0300, Or Gerlitz wrote: > Haven't heard from you re the patch you have supplied me which removes > at least this SCSI IOCTL issuing a non SG SCSI command. As i wrote you i > have patched 2.6.16 and tested it, works great. Is it queued for 2.6.17? It's in scsi-misc and will hopefully still go in before 2.6.17 ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] what are the performance figures of SDP over infiniband?
hi i have started working over infiniband network in my work i am using openib stack which is available along with RHEL4 update 3 and i tried my benchmark to test the performance the figures are not convincing (max 260Mbps at 10Mb data rate) can anybody send me the typical performance figures of sdp over infinband Data rate BandWidth 2.00e+04 1.565233e+085.00e+04 2.080646e+088.00e+04 2.273233e+081.00e+05 2.329996e+082.00e+05 2.479734e+085.00e+05 2.597392e+088.00e+05 2.600675e+081.00e+06 2.623670e+082.00e+06 2.639205e+085.00e+06 2.651772e+088.00e+06 2.663875e+081.00e+07 2.656914e+081.00e+01 5.518821e+061.00e+02 3.419270e+071.00e+03 4.338935e+081.50e+03 7.997614e+082.00e+03 8.501968e+085.00e+03 1.160785e+098.00e+03 9.055712e+071.00e+04 1.127501e+082.00e+04 1.557870e+085.00e+04 2.086995e+088.00e+04 2.260573e+081.00e+05 2.351155e+082.00e+05 2.470192e+085.00e+05 2.585480e+088.00e+05 2.607290e+081.00e+06 2.617710e+082.00e+06 2.637822e+085.00e+06 2.651189e+088.00e+06 2.662102e+081.00e+07 2.656525e+081.00e+01 5.156931e+061.00e+02 3.328813e+071.00e+03 3.475943e+081.50e+03 8.426057e+082.00e+03 6.108210e+085.00e+03 1.014751e+098.00e+03 9.096629e+071.00e+04 1.102024e+082.00e+04 1.585548e+085.00e+04 2.098831e+088.00e+04 2.277244e+08These are the figures i got with SDP. thanx n regards K.Mahesh Jiyo cricket on Yahoo! India cricket Yahoo! Messenger Mobile Stay in touch with your buddies all the time.___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] amso1100 testing with OpenIB
Hi, Thanks for pointing me to the latest firmware/boot loader. After I uninstalled the Ammasso Software, I was able to succesfully update and load the firmware and also the module. It worked fine on one machine but on another machine, when I try to bring the eth2 interface up, the machine hangs. However, I dont see any problem with iw2 interface. I was also able to do krping test successfully between these two machines using the iw2 ip address. It is very likely that if I set this up on another node, everything might run smoothly but I was wondering if I was doing something wrong here. Do you have any suggestions? Also, Will there be a userspace support for the Ammasso driver soon? thanks, Karthik On Thu, 6 Apr 2006, Steve Wise wrote: > On Thu, 2006-04-06 at 14:01 -0400, Karthikeyan Vaidyanathan wrote: > > Hi, > > > > I was trying to follow the discussion on getting the iWARP branch to work > > with Ammasso NICs and tried the following steps: > > > > /include/rdma --> > >/gen2/branches/iwarp/src/linux-kernel/infiniband/include/rdma > > > > /drivers/infiniband --> > >/gen2/branches/iwarp/src/linux-kernel/infiniband > > > > and recompiled the kernel. I'm working on linux 2.6.15.4 kernel. > > > > I had initially updated the firmware from AMSO1100 Release 1.2 Update 1 > > kit. Then I used ccflash2 to update to C2L_H23_B58_F61_080507.bit. > > However, when I reboot the machine, the dmesg reports: > > > > c2: AMSO1100 Gigabit Ethernet driver v1.1 loaded > > c2: Downlevel Firmware boot loader [1/7: got 0x42, exp 0x43]. Use the > > cc_flash utility to update your boot loader > > c2: Adapter not claimed > > c2: probe of :03:02.0 failed with error -5 > > > > I also tried updating the firmware boot loader from AMSO1100 Release 1.2 > > Update 1 kit but I still get this message when the machine boots up. > > > > However, the following command shows that I have the latest firmware > > loaded: > > > > $ ccons 0 bitfile > > FPGA Bitfile ID: C2L_H23_B58_080507 Release 23 > > > > You need the amso1100 openib kit from Open Grid Computing. It contains > new firmware, a new bitfile/boot loader, and scripts to load the > firmware correctly before the openib driver loads. The kit is here: > > http://www.opengridcomputing.com/downloads/ogc_amso_kit_20060308.tgz > > You also need to uninstall the Ammasso software from the system. > > Hope this helps. > > Steve. > > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] $B?M:J$H$A$g$C$HHkL)$N$*IU$-9g$$$r"v%(%C%A$G2a7c$J!D(B
皆さん、どんなエッチを楽しんでいますか？ 今、話題で人気なのがやっぱり即アポ即エッチにムンムンなフェロモンを放つ『人妻』。 そんな人妻とちょっと大人の秘密の出会いを… でもどうやって！？と思ったり感じたりしてませんか？ 彼女たちの本気(マジ)にエロイ実態を大公開♪ そしてそんなサポートをしてくれる『２丁目の奥様』のご紹介、そして出逢いパーティーのお知らせです！！ http://okusama.ock-gatu.com/2cco 出逢い┓┏┓┏┓.*・°☆.。.:*・°☆.。.:*・°☆.。.:*・°☆ 確立♪┻┗┛┗％…┓┏━┓┏…┓┏━┓┏…┓┏━┓ ┃直┃┃ア┃┃ド┃┃Ｏ┃┃Ｋ┃┃♪┃ ┗━┛┗…┛┗━┛┗…┛┗━┛┗…┛ 〜完全無料サイトの決定版〜 ┏━┓┏…┓┏━┓ ┏…┓ ┏━┓┏…┓ ┃２┃┃丁┃┃目┃ ┃の┃ ┃奥┃┃様┃ ┗…┛┗━┛┗…┛ ┗━┛ ┗…┛┗━┛ .。.:*・°☆.。.:*・°☆.。.:*・°☆.。.:*・°☆.。.:*・°☆ ☆★☆★☆★２００６！！【完全無料】にてあなたの近所の人妻を検索！！☆★☆★☆★ ※なんと登録料はもちろん！サイト利用料が全部無料♪完全フルサポートで幅広いユーザー層を誇ってます！！ ☆★『2丁目の奥様』なら、お好みの相手をすぐにＧＥＴ出来ちゃいます！☆★ 無料出会い系サイト『2丁目の奥様』⇒ http://okusama.ock-gatu.com/2cco 〓 この春一番！いい思いをしようよ♪ 〓 ┏━━━★彡 ┌☆ 地域密着型！『ご近所サーチ』 ┿━━ 自分の住んでいる地域はもちろん、他地域全ての異性会員を表示可能！ お近くのお相手検索にてさらに出会いのチャンスが広がります。 http://okusama.ock-gatu.com/2cco ┏━━━☆彡 ┌★ メンクイさん必見『写メサーチ』 ━━ 好みの異性を選り好みで検索。メンクイさんは必見です♪ 写メールをつけるとメールの返信率もアップ間違いなし！ http://okusama.ock-gatu.com/2cco ┏━━━☆彡 ┌★ 気に入った人とだけ直電・直メＯＫ！ ┿━━ 仲良くなった人だけにメールアドレスや電話番号を教える事ができます。 気になるあの人と…もっと仲良くなっちゃいましょう♪ http://okusama.ock-gatu.com/2cco ※無料出会いサイト『２丁目の奥様』は、登録料の他にメールの送受信、掲示板の書込みや 閲覧、顔写真の添付・閲覧など、全てのコンテンツが無料でご利用頂けます。 ▼▼┏┯━┯━┯━┯━┯━┯━┯━┯━┯━┯━┯━┯━┯━┯━┯┓▼▼ ┃│最│新│！│！│掲│示│板│情│報│の│お│届│け│♪│┃ ┗┷━┷━┷━┷━┷━┷━┷━┷━┷━┷━┷━┷━┷━┷━┷┛ ▲▲ .。.:*・°☆.。.:*・°☆.。.:*・°☆.。.:*・°☆.。.:*・°☆▲▲ ■まいさん □２５才 ■結婚暦２年 □体の相性が合えば、生でもいいよん http://okusama.ock-gatu.com/2cco -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ■なぎささん □３０才 ■結婚暦５年 □卑猥な言葉で私を犯して下さい。 http://okusama.ock-gatu.com/2cco -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ┳━ ┃ 会員様必見情報！！出逢いパーティのお知らせ♪ ┻━━ 〜 愛と自由とファッション、そして出会いの提供をコンセプト 〜 ※まだまだ間に合う！弊社主催のカップリングパーティー及び秘密出逢いパーティーに関するお知らせです！ ▼４/１４ ３０対３０カップリングパーティー ３０対３０の会員様限定応募企画！ハイグレードなカップリングパーティーを開催予定。 詳細はサポートに「４/１４カップリング希望」とお問い合わせの上、抽選にてご当選された方にメールにてご連絡致します。 …… ▼４/１５ 人妻厳選♪秘密出逢いパーティー 会員様限定応募企画！普段はなかなか出会うことの出来ないセレブな人妻との○秘交際パーティーです。 詳細はサポートに「４/１４カップリング希望」とお問い合わせの上、抽選にてご当選された方にメールにてご連絡致します。 …… ▼４/２２ 巨乳ちゃん集まれ！！男女入り乱れ♪乱交パーティー 弊社女性登録会員の巨乳女性とのスペシャル乱交企画第３弾！人気企画のため応募はお早めに！！ 詳細はサポートに「４/１４カップリング希望」とお問い合わせの上、抽選にてご当選された方にメールにてご連絡致します。 …… 詳細はこちらから⇒ http://okusama.ock-gatu.com/2cco 今後メール広告の配信を希望されない方は、 [EMAIL PROTECTED] ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH] ipoib_flush_paths
OK, I applied the original patch here, along with this on top of it:
IPoIB: Use spin_lock_irq() instead of spin_lock_irqsave()
We know ipoib_flush_paths() is called from plain process context with
interrupts enabled, since it does wait_for_completion(). So there's
no need to use spin_lock_irqsave() -- spin_lock_irq() is fine.
Signed-off-by: Roland Dreier <[EMAIL PROTECTED]>
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c
b/drivers/infiniband/ulp/ipoib/ipoib_main.c
index 996c6e1..cb078a7 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_main.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -336,9 +336,8 @@ void ipoib_flush_paths(struct net_device
struct ipoib_dev_priv *priv = netdev_priv(dev);
struct ipoib_path *path, *tp;
LIST_HEAD(remove_list);
- unsigned long flags;
- spin_lock_irqsave(&priv->lock, flags);
+ spin_lock_irq(&priv->lock);
list_splice(&priv->path_list, &remove_list);
INIT_LIST_HEAD(&priv->path_list);
@@ -349,12 +348,12 @@ void ipoib_flush_paths(struct net_device
list_for_each_entry_safe(path, tp, &remove_list, list) {
if (path->query)
ib_sa_cancel_query(path->query_id, path->query);
- spin_unlock_irqrestore(&priv->lock, flags);
+ spin_unlock_irq(&priv->lock);
wait_for_completion(&path->done);
path_free(dev, path);
- spin_lock_irqsave(&priv->lock, flags);
+ spin_lock_irq(&priv->lock);
}
- spin_unlock_irqrestore(&priv->lock, flags);
+ spin_unlock_irq(&priv->lock);
}
static void path_rec_completion(int status,
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: ib_umad related kernel panic
On Thu, Apr 06, 2006 at 07:16:25PM -0700, Manpreet Singh wrote: > Doug, > > Thanks for the updated rpms. Just curious as to what svn revision of openib > you used for the rpms. > > Do the IB tools (if compiled separately) have to be from the same svn > revision? Usually, they don't have to be identical revs, but if they get too far apart they might break. -- Doug Ledford <[EMAIL PROTECTED]> 919-754-3700 x44233 Red Hat, Inc. 1801 Varsity Dr. Raleigh, NC 27606 ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: ib_umad related kernel panic
On Fri, Apr 07, 2006 at 12:30:04AM +0300, Michael S. Tsirkin wrote: > Quoting r. Doug Ledford <[EMAIL PROTECTED]>: > > Just to make that point > > clear, I've removed the old RPMs from my site and put up a new set of kernel > > rpms based on the 1.0 release branch code (userspace rpms will be a little > > later). > > Doug, is this > https://openib.org/svn/gen2/branches/1.0/src/linux-kernel/ > happens to be the code that you are using? That's the base, yes. -- Doug Ledford <[EMAIL PROTECTED]> 919-754-3700 x44233 Red Hat, Inc. 1801 Varsity Dr. Raleigh, NC 27606 ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: ib_umad related kernel panic
Doug, Thanks for the updated rpms. Just curious as to what svn revision of openib you used for the rpms. Do the IB tools (if compiled separately) have to be from the same svn revision? Thanks, Manpreet. On 4/6/06, Doug Ledford <[EMAIL PROTECTED]> wrote: On Wed, Apr 05, 2006 at 07:04:29PM -0700, Manpreet Singh wrote:> Hi,>> I am observing the following with redhat kernel rpm at:> http://people.redhat.com/dledford/Infiniband , which uses openib version> 3965. This is on an RHEL4 install.>> When the system is rebooted, ib_core, ib_mad and ib_mthca modules get loaded> automatically. When I load ib_umad after that, I get the following panic. > However, if I unload ib_mthca and load again once before loading ib_umad,> then there is no problem subsequently in the system.The rpms posted on my site, especially the 3985 revision, are not going to be "fixed", they will simply be replaced. We are going to a later rev ofthe code base for the next update (the 1.0 release branch), so time spentfixing bugs in that version is basically wasted. Just to make that point clear, I've removed the old RPMs from my site and put up a new set of kernelrpms based on the 1.0 release branch code (userspace rpms will be a littlelater).-- Doug Ledford < [EMAIL PROTECTED]> 919-754-3700 x44233 Red Hat, Inc. 1801 Varsity Dr. Raleigh, NC 27606 ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Problems running MPI jobs with MVAPICH
A quick followup. I have just build/configure and propagated the mvapich-gen2 installation on two EM64T nodes as root. mvapich-gen2 runs fine with MPD_RING option. Here are the commands I had used. Hope they could help. 1) prepare your mpd passwd/conf files: /root/.mpdpasswd and /root/.mpd.conf, they should be the same with mode 600 [EMAIL PROTECTED] mvapich-gen2]# cat /root/.mpd.conf password=56rtG9 2) make.mvapich.gen2 # select /root/installs as $PREFIX and add USE_MPD_RING into the option. [EMAIL PROTECTED] mvapich-gen2]# scp /root/installs e15:/root/. 3) [EMAIL PROTECTED] mvapich-gen2]# /root/installs/bin/mpicc -o /root/cpi examples/basic/cpi.c [EMAIL PROTECTED] mvapich-gen2]# scp /root/cpi e15:/root/. cpi 100% 294KB 293.8KB/s 00:00 4) Some system info [EMAIL PROTECTED] mvapich-gen2]# cat /etc/redhat-release Red Hat Enterprise Linux AS release 4 (Nahant Update 2) [EMAIL PROTECTED] mvapich-gen2]# uname -a Linux e14-oib 2.6.15 #3 SMP Mon Mar 6 20:48:17 PST 2006 x86_64 x86_64 x86_64 GNU/Linux [EMAIL PROTECTED] mvapich-gen2]# LD_LIBRARY_PATH=/usr/local/lib /root/installs/bin/mpdtrace mpdtrace: e14-oib_43520: lhs=e15-oib_60830 rhs=e15-oib_60830 rhs2=e14-oib_43520 gen=1 mpdtrace: e15-oib_60830: lhs=e14-oib_43520 rhs=e14-oib_43520 rhs2=e15-oib_60830 gen=1 5) running two processes on one or two nodes. [EMAIL PROTECTED] mvapich-gen2]# LD_LIBRARY_PATH=/usr/local/lib /root/installs/bin/mpirun_mpd -np 2 /root/cpi -MPDENV- LD_LIBRARY_PATH=/usr/local/lib Process 0 of 2 on e14-oib Process 1 of 2 on e15-oib pi is approximately 3.1415926544231318, Error is 0.0887 wall clock time = 0.000424 [EMAIL PROTECTED] mvapich-gen2]# LD_LIBRARY_PATH=/usr/local/lib /root/installs/bin/mpirun_mpd -g 2 -np 2 /root/cpi -MPDENV- LD_LIBRARY_PATH=/usr/local/lib Process 0 of 2 on e14-oib Process 1 of 2 on e14-oib pi is approximately 3.1415926544231318, Error is 0.0887 wall clock time = 0.000406 Let us know how we can help further, Weikuan On Apr 6, 2006, at 8:53 PM, Weikuan Yu wrote: Hi, Don, Good to know that you are able to run mvapich with mpirun_rsh. We can now focus on MPD problem. We never had attempted to run MPD_RING option as root user. Just curious, were you able to mvapich2-gen2 with MPD_RING? They are more or less similar code. So could you try the following two possibilities and let us know all the log files and etc. a) rpm -e lam. The reason for this is that I noticed earlier LAM showing up in your config.log. It might help the configure if you can remove the other MPI packages which are on your path. b) Try mvapich-gen2 with mpd_ring, either as root or as user. Please do build/configure/install on one node and propagate the installation to see if it runs. We can look into the separate build later on. BTW, make sure you do `make install' at the end of configure/build. c) If possible, could you try mvapich2-gen2 with mpd_ring since the mpd_ring related code is similar there. That may help to locate the problem. Thanks, Weikuan On Apr 6, 2006, at 8:02 PM, [EMAIL PROTECTED] wrote: Weikuan I previously reported that I was having problems running any MPI jobs between a pair of EM64T machines with RHEL4, Update 3 with the OpenIB modules, (kernel versions 2.6.9-34.ELsmp) and the "mvapich-gen2" code from the OpenIB svn tree. I was having two problems: 1. When I tried to run from user mode, I would get segmentation faults 2. When I ran from root, the jobs would fail with the following message: "cpi: pmgr_client_mpd.c:254: mpd_exchange_info: Assertion `len_remote == len_local' failed. ". The first problem turned out to be a memory problem; I had to increase the size of the max locked-in-memory address space (memlock) in the user limits. The second problem seemed to be more related to process management than to MPI itself. I remembered that when I modified the "make.mvapich.gen2" build script, there was a parameter for MPD: # Whether to use an optimized queue pair exchange scheme. This is not # checked for a setting in in the script. It must be set here explicitly. # Supported: "-DUSE_MPD_RING", "-DUSE_MPD_BASIC" and "" (to disable) HAVE_MPD_RING="" Because I wanted to use MPD to launch jobs, I set HAVE_MPD_RING="-DUSE_MPD_RING" in the build script. I went back and set the parameter to HAVE_MPD_RING="" to disable it, and rebuilt, which meant that MPD was not installed. Using "mpirun_rsh" I am now able to run the MPI jobs, including "cpi", "mping" and other benchmark tests. There seems to be a problem with "USE_MPD_RING". Have you seen this before? Should I try with "USE_MPD_BASIC" instead? -Don Albert- ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-ge
[openib-general] Re: your home
Dear Home O w n e r , Your credit doesn't matter to us ! If you O W N real e s t a t e and want l M M E D l A T E c a s h to s p e n d ANY way you like, or simply wish to L O W E R your monthly p a y m e n t s by a third or more, here are the deals we have T O D A Y : $ 488 , 000 at a 3 , 67 % F i x e d - r a t e $ 372 , 000 at a 3 , 90 % V a r i a b I e - r a t e $ 492 , 000 at a 3 , 21 % l n t e r e s t - o n l y $ 248 , 000 at a 3 , 36 % F i x e d - r a t e $ 198 , 000 at a 3 , 55 % V a r i a b I e - r a t e Hurry, when these deals are gone, they are gone! Don't worry about a p p r o v a l , your credit will not d i s q u a I i f y you ! web site Sincerely, Peronelle Fugate A p p r o v a l Manager___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Problems running MPI jobs with MVAPICH
Hi, Don, Good to know that you are able to run mvapich with mpirun_rsh. We can now focus on MPD problem. We never had attempted to run MPD_RING option as root user. Just curious, were you able to mvapich2-gen2 with MPD_RING? They are more or less similar code. So could you try the following two possibilities and let us know all the log files and etc. a) rpm -e lam. The reason for this is that I noticed earlier LAM showing up in your config.log. It might help the configure if you can remove the other MPI packages which are on your path. b) Try mvapich-gen2 with mpd_ring, either as root or as user. Please do build/configure/install on one node and propagate the installation to see if it runs. We can look into the separate build later on. BTW, make sure you do `make install' at the end of configure/build. c) If possible, could you try mvapich2-gen2 with mpd_ring since the mpd_ring related code is similar there. That may help to locate the problem. Thanks, Weikuan On Apr 6, 2006, at 8:02 PM, [EMAIL PROTECTED] wrote: Weikuan I previously reported that I was having problems running any MPI jobs between a pair of EM64T machines with RHEL4, Update 3 with the OpenIB modules, (kernel versions 2.6.9-34.ELsmp) and the "mvapich-gen2" code from the OpenIB svn tree. I was having two problems: 1. When I tried to run from user mode, I would get segmentation faults 2. When I ran from root, the jobs would fail with the following message: "cpi: pmgr_client_mpd.c:254: mpd_exchange_info: Assertion `len_remote == len_local' failed. ". The first problem turned out to be a memory problem; I had to increase the size of the max locked-in-memory address space (memlock) in the user limits. The second problem seemed to be more related to process management than to MPI itself. I remembered that when I modified the "make.mvapich.gen2" build script, there was a parameter for MPD: # Whether to use an optimized queue pair exchange scheme. This is not # checked for a setting in in the script. It must be set here explicitly. # Supported: "-DUSE_MPD_RING", "-DUSE_MPD_BASIC" and "" (to disable) HAVE_MPD_RING="" Because I wanted to use MPD to launch jobs, I set HAVE_MPD_RING="-DUSE_MPD_RING" in the build script. I went back and set the parameter to HAVE_MPD_RING="" to disable it, and rebuilt, which meant that MPD was not installed. Using "mpirun_rsh" I am now able to run the MPI jobs, including "cpi", "mping" and other benchmark tests. There seems to be a problem with "USE_MPD_RING". Have you seen this before? Should I try with "USE_MPD_BASIC" instead? -Don Albert- ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -- Weikuan Yu, Computer Science, OSU http://www.cse.ohio-state.edu/~yuw ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] Re: [RFC] [PATCH 1/2] multicast support for multiple users
>> Maybe we should give up the ghost and stop trying to support IB switches? > >IMHO, the main reason to have the IB stack on a switch is to support >ipoib for in band management with a stable and well tested code base. >My company is looking closely at doing this in our switches and I'd be >surprised if other companies that already use linux as the management >OS are not doing the same. I've already made the changes to the ib_multicast module to support switches. I will send out an updated version of the patch tomorrow. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: [RFC] [PATCH 1/2] multicast support for multiple users
On Thu, Apr 06, 2006 at 12:51:23PM -0700, Roland Dreier wrote: > Maybe we should give up the ghost and stop trying to support IB switches? IMHO, the main reason to have the IB stack on a switch is to support ipoib for in band management with a stable and well tested code base. My company is looking closely at doing this in our switches and I'd be surprised if other companies that already use linux as the management OS are not doing the same. Regards, Jason ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH] ipoib_flush_paths
Sean Hefty wrote:
+static inline int
+rdma_resolve_ip(struct sockaddr *src_addr, struct sockaddr *dst_addr,
+ struct rdma_dev_addr *addr, int timeout_ms,
+ void (*callback)(int status, struct sockaddr *src_addr,
+struct rdma_dev_addr *addr, void *context),
+ void *context)
+{
+ return __rdma_resolve_ip(src_addr, dst_addr, addr, timeout_ms,
+callback, context, THIS_MODULE);
+}
I missed the compile warning before. This needs a forward declaration of
__rdma_resolve_ip(). Assuming that we've decided to go this route, I'll submit
a corrected version.
- Sean
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Problems running MPI jobs with MVAPICH
Weikuan I previously reported that I was having problems running any MPI jobs between a pair of EM64T machines with RHEL4, Update 3 with the OpenIB modules, (kernel versions 2.6.9-34.ELsmp) and the "mvapich-gen2" code from the OpenIB svn tree. I was having two problems: When I tried to run from user mode, I would get segmentation faults When I ran from root, the jobs would fail with the following message: "cpi: pmgr_client_mpd.c:254: mpd_exchange_info: Assertion `len_remote == len_local' failed. ". The first problem turned out to be a memory problem; I had to increase the size of the max locked-in-memory address space (memlock) in the user limits. The second problem seemed to be more related to process management than to MPI itself. I remembered that when I modified the "make.mvapich.gen2" build script, there was a parameter for MPD: # Whether to use an optimized queue pair exchange scheme. This is not # checked for a setting in in the script. It must be set here explicitly. # Supported: "-DUSE_MPD_RING", "-DUSE_MPD_BASIC" and "" (to disable) HAVE_MPD_RING="" Because I wanted to use MPD to launch jobs, I set HAVE_MPD_RING="-DUSE_MPD_RING" in the build script. I went back and set the parameter to HAVE_MPD_RING="" to disable it, and rebuilt, which meant that MPD was not installed. Using "mpirun_rsh" I am now able to run the MPI jobs, including "cpi", "mping" and other benchmark tests. There seems to be a problem with "USE_MPD_RING". Have you seen this before? Should I try with "USE_MPD_BASIC" instead? -Don Albert- ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: news for you
V A L t U M 1, 22 $ C t A L t S 3, 74 $ V t A G R A 3, 36 $ get more information - http://toaep990.paitemidde.com___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Data Structures at UserSpace
Hi Hal and others, I find that, there are no standard header files exists at userspace which can define structure for PORT_INFO, NODE_INFO and other elements like Path Records, Service record etc. So every individual has to define his own header files to define the data structures for these elements and use in their application program or tool. If it is exists then could you please point me out or if it does not then is there any plan or shall I provide such header files to make standard header files like we have mad.h, umad.h etc. Thanks, - Takshak___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH] uDAPL cma; dat_ep_free can return without freeing cm_id
> Steve, can you test this version and see if it works for your iWARP device. > I think the patch is good. I ran dapltest/regress.sh over the chelsio iwarp device using this new patch instead of my original patch, and things seem as stable as they were before (i'm fighting some intermittent connection setup failures that I think are in cxgb3 provider, not dapl). Thanks, Steve. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Xen Infiniband Source Code Hosted
Hello! Just wanted to let everyone know Jiuxing has populated a mercurial tree (very kindly hosted by XenSource) with his code at the following site: http://xenbits.xensource.com/ext/xen-smartio.hg This contains the current source code for a xen infiniband frontend and backend driver. The source code is in very preliminary stages of development, just a proof of concept for now (works). We have a long way to go. We'd like to invite interested folks to take a look and get involved in the continuing design and development as an open-source community. We will be putting up a Wiki page for this shortly on the Xen Wiki. Stay tuned... There is also a mailing list we set up for discussion on virtualization of smart I/O in Xen at: http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-smartio However, at Ian's request we're going to contain most discussion to xen-devel itself. (Please pardon my earlier post without a subject). thanks, Nivedita___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] (no subject)
Hello! Just wanted to let everyone know Jiuxing has populated a mercurial tree (very kindly hosted by XenSource) with his code at the following site: http://xenbits.xensource.com/ext/xen-smartio.hg This contains the current source code for a xen infiniband frontend and backend driver. The source code is in very preliminary stages of development, just a proof of concept for now (works). We have a long way to go. We'd like to invite interested folks to take a look and get involved in the continuing design and development as an open-source community. We will be putting up a Wiki page for this shortly on the Xen Wiki. Stay tuned... There is also a mailing list we set up for discussion on virtualization of smart I/O in Xen at: http://lists.xensource.com/cgi-bin/mailman/listinfo/xen-smartio However, at Ian's request we're going to contain most discussion to xen-devel itself. thanks, Nivedita ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: [RFC] [PATCH 1/2] multicast support for multiple users
>> Only one MAD request per group is active at any time, regardless if the MAD >is >> for a join or leave. > >One request per group or (per group and join state) ? Per group - with a group identified by an MGID. The intent is to ensure that the state of the group is not driven in different directions in case MADs are processed out of order or are lost. So only a single MAD request can be outstanding per group. >Sorry for being dense but I'm wondering about the SA interactions. In >your example, what are the requests made of the SA ? There seems to be >the original send only join. Sometime later there is the full member >join. Is that it ? In this example, the first join request is for send-only. The second join request is for full member and send-only. The leave request is for full member. A final leave request would later be for send-only. (This is for this example only, the actual order of the requests depends on the users and their join states.) >Is it one request per group as you stated above or one request per group >per join state that can be active ? That's what was confusing me. One request per group outstanding at a time. A change in the join state will result in a second request, but will not be issued until the previous request completes. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH] ipoib_flush_paths
Quoting r. Eli Cohen <[EMAIL PROTECTED]>: > Subject: [PATCH] ipoib_flush_paths > > ib_sa_cancel_query must be called with priv->lock held since > a completion might arrive and set path->query to NULL > > Signed-off-by: Eli Cohen <[EMAIL PROTECTED]> > Roland, with all the noise about module unloading, please don't forget about this one - please note its unreleated to module unloading issue. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: librdmacm/ucma
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Subject: Re: librdmacm/ucma > > Michael S. Tsirkin wrote: > >I just propose killing old ABI support in librdmacm - CMA is sufficiently > >new for that. > > I'm fine doing that. The ABI versions can be reset to 1. Yes, let's do that, -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: [RFC] [PATCH 1/2] multicast support for multiple users
On Thu, 2006-04-06 at 17:56, Sean Hefty wrote: > >There's also code in opensm/osm_sa_mcmember_record.c that you can > >peruse. > > Thanks - that's helpful. > > >> The code uses a promotion/demotion mechanism based on a reference count of > >> membership types. The restriction is that only a single request per group > >> is > >> active at a time. > > > >Meaning only one of the membership types is active outside the node ? If > >so, that seems right as long as the order of precedence is correct. > > Only one MAD request per group is active at any time, regardless if the MAD is > for a join or leave. One request per group or (per group and join state) ? > >How does the change in precedence occur ? Is it a leave followed by the > >new join or the new join followed by the old leave ? > > Either through a join or leave, depending on if the group is being promoted or > demoted. For example, if a send-only group is joined by a full member, a new > join is issued with an updated join state. If the full member later leaves, a > leave request is issued for that join state. Sorry for being dense but I'm wondering about the SA interactions. In your example, what are the requests made of the SA ? There seems to be the original send only join. Sometime later there is the full member join. Is that it ? Is it one request per group as you stated above or one request per group per join state that can be active ? That's what was confusing me. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: RE: Re: [PATCH] ipoib_flush_paths
Michael S. Tsirkin wrote: OK, this means we must add struct module * pointer to ib_create_cm_id and rdma_create_cm_id as well. Could one of you guys build and commit such a patch for these modules? I'm not in the lab. I will create the patches for these. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [RFC] [PATCH 1/2] multicast support for multiple users
Hal> Is it more than the MCMemberRecord RID ? I suppose this keeps Hal> things simpler anyhow. Not really. MCMemberRecord RID identifies a single member of a group, so it needs both MGID and port GID. This means that a group is identified by just the MGID. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: librdmacm/ucma
Michael S. Tsirkin wrote: I just propose killing old ABI support in librdmacm - CMA is sufficiently new for that. I'm fine doing that. The ABI versions can be reset to 1. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: [RFC] [PATCH 1/2] multicast support for multiple users
>There's also code in opensm/osm_sa_mcmember_record.c that you can >peruse. Thanks - that's helpful. >> The code uses a promotion/demotion mechanism based on a reference count of >> membership types. The restriction is that only a single request per group is >> active at a time. > >Meaning only one of the membership types is active outside the node ? If >so, that seems right as long as the order of precedence is correct. Only one MAD request per group is active at any time, regardless if the MAD is for a join or leave. >How does the change in precedence occur ? Is it a leave followed by the >new join or the new join followed by the old leave ? Either through a join or leave, depending on if the group is being promoted or demoted. For example, if a send-only group is joined by a full member, a new join is issued with an updated join state. If the full member later leaves, a leave request is issued for that join state. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: RE: Re: [PATCH] ipoib_flush_paths
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Note that ib_cm and rdma_cm technically have the same issue, since cm_id's can > be destroyed by returning non-zero from a callback. I.e. a user of those > interfaces isn't forced to call anything when unloading. OK, this means we must add struct module * pointer to ib_create_cm_id and rdma_create_cm_id as well. Could one of you guys build and commit such a patch for these modules? I'm not in the lab. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: [RFC] [PATCH 1/2] multicast support for multiple users
On Thu, 2006-04-06 at 17:21, Sean Hefty wrote: > >> I don't think it's needed. MGID and PortGID together form the record > >> identifier for multicast groups. > > > >I see your point so perhaps it is not needed. For IPoIB, that clearly > >works as the PKey is embedded in the MGID. I didn't think that was the > >case with generalized IB multicast. > > It doesn't appear that PKey is used in deletion of the group unless proxy_join > is set to 1. But in that case, the PKey that's used comes from the MAD > source, > and not the value specified as part of the MCMemberRecord. In the case of proxy join/leave, the PKey sharing rule would (still) need to be followed (requester port would need to be in the same partition as the requested port (and the group)) but that's (enforced) on the SA side. As Roland pointed out, the Pkey component is not needed on leaves as it's implicit in the group. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [RFC] [PATCH 1/2] multicast support for multiple users
On Thu, 2006-04-06 at 17:17, Roland Dreier wrote: > Hal> I see your point so perhaps it is not needed. For IPoIB, that > Hal> clearly works as the PKey is embedded in the MGID. I didn't > Hal> think that was the case with generalized IB multicast. > > A multicast group is uniquely identified by MGID. There can't be two > different groups with the same MGID but different P_Keys. Is it more than the MCMemberRecord RID ? I suppose this keeps things simpler anyhow. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: Re: librdmacm/ucma
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Subject: RE: Re: librdmacm/ucma > > >Put another way - why do you already have backward compatibility hacks > >in lirdmacm? There wasn't any released version of cma, was there? > > Because the behavior of the ABI changed. The CMA was released to openib, but > has not yet been merged upstream. What is the issue here? The CMA ABI now > behaves similar to the other userspace components. I just propose killing old ABI support in librdmacm - CMA is sufficiently new for that. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: Re: librdmacm/ucma
>Put another way - why do you already have backward compatibility hacks >in lirdmacm? There wasn't any released version of cma, was there? Because the behavior of the ABI changed. The CMA was released to openib, but has not yet been merged upstream. What is the issue here? The CMA ABI now behaves similar to the other userspace components. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: Re: librdmacm/ucma
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Subject: Re: Re: librdmacm/ucma > > Michael S. Tsirkin wrote: > >In normal usage I expect we must not break the ABI, but should rather > >extend it - without breaking userspace. How do you extend the ABI? > >not by bumping the ABI revision? How does userspace find out > >whether kernel supports new features? > > If the ABI didn't break, can't userspace just make the call for the new > feature and check the return code? Put another way - why do you already have backward compatibility hacks in lirdmacm? There wasn't any released version of cma, was there? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH] uDAPL cma; dat_ep_free can return without freeing cm_id
James and Steve,
Here is revised patch that was tested (free and debug build versions) with
dtest, dapltest, and
Intel MPI test suites. The rdma_destroy_id will block until we acknowledge the
event so there was no
need to add our own wait objects for synchronization. This will never be called
from the async event
thread so there is no chance of deadlock conditions. I also made some changes
to build with
configure enable-debug. Some unused variables that were deleted are actually
used in the debug
messages. Please review the changes.
Steve, can you test this version and see if it works for your iWARP device.
Thanks,
-arlin
Signed-off by: Arlin Davis [EMAIL PROTECTED]
Index: dapl/openib_cma/dapl_ib_cm.c
===
--- dapl/openib_cma/dapl_ib_cm.c(revision 6305)
+++ dapl/openib_cma/dapl_ib_cm.c(working copy)
@@ -62,9 +62,9 @@
/* local prototypes */
static struct dapl_cm_id * dapli_req_recv(struct dapl_cm_id *conn,
struct rdma_cm_event *event);
-static int dapli_cm_active_cb(struct dapl_cm_id *conn,
+static void dapli_cm_active_cb(struct dapl_cm_id *conn,
struct rdma_cm_event *event);
-static int dapli_cm_passive_cb(struct dapl_cm_id *conn,
+static void dapli_cm_passive_cb(struct dapl_cm_id *conn,
struct rdma_cm_event *event);
static void dapli_addr_resolve(struct dapl_cm_id *conn);
static void dapli_route_resolve(struct dapl_cm_id *conn);
@@ -87,7 +87,9 @@ static inline uint64_t cpu_to_be64(uint6
static void dapli_addr_resolve(struct dapl_cm_id *conn)
{
int ret;
-
+#ifdef DAPL_DBG
+ struct rdma_addr *ipaddr = &conn->cm_id->route.addr;
+#endif
dapl_dbg_log(DAPL_DBG_TYPE_CM,
" addr_resolve: cm_id %p SRC %x DST %x\n",
conn->cm_id,
@@ -110,8 +112,10 @@ static void dapli_addr_resolve(struct da
static void dapli_route_resolve(struct dapl_cm_id *conn)
{
int ret;
- struct rdma_cm_id *cm_id = conn->cm_id;
-
+#ifdef DAPL_DBG
+ struct rdma_addr *ipaddr = &conn->cm_id->route.addr;
+ struct ib_addr *ibaddr = &conn->cm_id->route.addr.addr.ibaddr;
+#endif
dapl_dbg_log(DAPL_DBG_TYPE_CM,
" route_resolve: cm_id %p SRC %x DST %x PORT %d\n",
conn->cm_id,
@@ -158,37 +162,51 @@ bail:
NULL, conn->ep);
}
+/*
+ * Called from consumer thread via dat_ep_free().
+ * CANNOT be called from the async event processing thread
+ * dapli_cma_event_cb() since a cm_id reference is held and
+ * a deadlock will occur.
+ */
void dapli_destroy_conn(struct dapl_cm_id *conn)
{
- int in_callback;
+ struct rdma_cm_id *cm_id;
dapl_dbg_log(DAPL_DBG_TYPE_CM,
" destroy_conn: conn %p id %d\n",
conn,conn->cm_id);
-
+
dapl_os_lock(&conn->lock);
conn->destroy = 1;
- in_callback = conn->in_callback;
+
+ if (conn->ep)
+ conn->ep->cm_handle = IB_INVALID_HANDLE;
+
+ cm_id = conn->cm_id;
+ conn->cm_id = NULL;
dapl_os_unlock(&conn->lock);
- if (!in_callback) {
- if (conn->ep)
- conn->ep->cm_handle = IB_INVALID_HANDLE;
- if (conn->cm_id) {
- if (conn->cm_id->qp)
- rdma_destroy_qp(conn->cm_id);
- rdma_destroy_id(conn->cm_id);
- }
-
- conn->cm_id = NULL;
- dapl_os_free(conn, sizeof(*conn));
+ /*
+* rdma_destroy_id will force synchronization with async CM event
+* thread since it blocks until the in-process event reference
+* is cleared during our event processing call exit.
+*/
+ if (cm_id) {
+ if (cm_id->qp)
+ rdma_destroy_qp(cm_id);
+
+ rdma_destroy_id(cm_id);
}
+ dapl_os_free(conn, sizeof(*conn));
}
static struct dapl_cm_id * dapli_req_recv(struct dapl_cm_id *conn,
struct rdma_cm_event *event)
{
struct dapl_cm_id *new_conn;
+#ifdef DAPL_DBG
+ struct rdma_addr *ipaddr = &event->id->route.addr;
+#endif
if (conn->sp == NULL) {
dapl_dbg_log(DAPL_DBG_TYPE_ERR,
@@ -239,11 +257,9 @@ static struct dapl_cm_id * dapli_req_rec
return new_conn;
}
-static int dapli_cm_active_cb(struct dapl_cm_id *conn,
+static void dapli_cm_active_cb(struct dapl_cm_id *conn,
struct rdma_cm_event *event)
{
- int destroy;
-
dapl_dbg_log(DAPL_DBG_TYPE_CM,
" active_cb: conn %p id %d event %d\n",
conn, conn->cm_id, event->event );
@@ -251,9 +267,8 @@ static int dapli_cm_active_cb(struct dap
[openib-general] Re: ib_umad related kernel panic
Quoting r. Doug Ledford <[EMAIL PROTECTED]>: > Just to make that point > clear, I've removed the old RPMs from my site and put up a new set of kernel > rpms based on the 1.0 release branch code (userspace rpms will be a little > later). Doug, is this https://openib.org/svn/gen2/branches/1.0/src/linux-kernel/ happens to be the code that you are using? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: [RFC] [PATCH 1/2] multicast support for multiple users
On Thu, 2006-04-06 at 17:01, Sean Hefty wrote: > >1. My main initial comment is that I think that cmp_rec needs to be more > >complicated that the matching which is there. The selectors include > >things like greater than, less than, and largest available in addition > >to equal to which is what is supported there now. I'm not sure whether > >any of this is used right now so may not be an issue for IPoIB. > > I will review the spec to see where the checks need to be enhanced. There's also code in opensm/osm_sa_mcmember_record.c that you can peruse. > This > probably won't be an issue for a while, since most join requests are limited > to > select fields of the multicast member record. Agreed. > >2. The other comment is I didn't yet follow how multiple joins of > >different JoinStates are handled. I can see there are different slots in > >the groups but I didn't see whether all the joins go out on the wire > >(one per JoinState) or whether there is some "promotion"/"demotion" of > >these. > > The code uses a promotion/demotion mechanism based on a reference count of > membership types. The restriction is that only a single request per group is > active at a time. Meaning only one of the membership types is active outside the node ? If so, that seems right as long as the order of precedence is correct. How does the change in precedence occur ? Is it a leave followed by the new join or the new join followed by the old leave ? > All join requests are queued to a pending list. If a request can be met with > the current join state of the group, it is added. If not, then a request is > sent to promote the group. Leave requests are handled differently, but result > in demotion. Sounds right. Thanks. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: librdmacm/ucma
Michael S. Tsirkin wrote: In normal usage I expect we must not break the ABI, but should rather extend it - without breaking userspace. How do you extend the ABI? not by bumping the ABI revision? How does userspace find out whether kernel supports new features? If the ABI didn't break, can't userspace just make the call for the new feature and check the return code? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: [RFC] [PATCH 1/2] multicast support for multiple users
>> I don't think it's needed. MGID and PortGID together form the record >> identifier for multicast groups. > >I see your point so perhaps it is not needed. For IPoIB, that clearly >works as the PKey is embedded in the MGID. I didn't think that was the >case with generalized IB multicast. It doesn't appear that PKey is used in deletion of the group unless proxy_join is set to 1. But in that case, the PKey that's used comes from the MAD source, and not the value specified as part of the MCMemberRecord. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: librdmacm/ucma
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Subject: Re: [openib-general] Re: librdmacm/ucma > > Michael S. Tsirkin wrote: > >The library now has > >min = 1 max = 2 > >this means that any ABI update in kernel will break userspace. > > Isn't that what it means to break the ABI? In normal usage I expect we must not break the ABI, but should rather extend it - without breaking userspace. How do you extend the ABI? not by bumping the ABI revision? How does userspace find out whether kernel supports new features? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [RFC] [PATCH 1/2] multicast support for multiple users
Hal> I see your point so perhaps it is not needed. For IPoIB, that Hal> clearly works as the PKey is embedded in the MGID. I didn't Hal> think that was the case with generalized IB multicast. A multicast group is uniquely identified by MGID. There can't be two different groups with the same MGID but different P_Keys. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH] ipoib_flush_paths
Quoting r. Roland Dreier <[EMAIL PROTECTED]>: > Subject: Re: [PATCH] ipoib_flush_paths > > Michael> Actually, it turned out to be the simplest solution - and > Michael> quite elegant since there's no room for mistakes: if > Michael> query is going to be running this means module is still > Michael> loaded so we can take a reference to it without races. > > Yes, this is suprisingly clean. > > Michael> As a bonus, and assertion inside __module_get increases > Michael> the chance to catch races where user forgets to cancel > Michael> the query - much nicer than crashing randomly. > > Actually I think __module_get() will do the wrong thing if called > during module unloading -- it doesn't test module_is_live(). In other > words, calling __module_get() without already holding a ref has a > race: __try_stop_module() can see the ref count as 0, then > __module_get() can increment it, and then __try_stop_module() sets the > module state to GOING and returns. > > So the right thing to do is BUG_ON(!try_module_get(owner)) A bit more code but better debugging then. OK. > Also, I don't think that a consumer of ib_sa() would ever pass an > owner other than THIS_MODULE. So how about if we keep the API the > same and just do the THIS_MODULE stuff in an inline wrapper? Good idea. > Like the following... it ends up being a pretty big diff, but just > because I moved some comments around and so on. Also I put the > try_module_get() stuff out of line into call_sa_callback(), because > the compiled code ends up smaller that way. > > Does anyone disagree with this patch? Michael, are you happy with > this tweaked version of yours? I'm happy, go ahead. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [RFC] [PATCH 1/2] multicast support for multiple users
On Thu, 2006-04-06 at 16:58, Roland Dreier wrote: > Hal> Why did the PKey component get removed from the leave ? > > I don't think it's needed. MGID and PortGID together form the record > identifier for multicast groups. I see your point so perhaps it is not needed. For IPoIB, that clearly works as the PKey is embedded in the MGID. I didn't think that was the case with generalized IB multicast. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH] [DAPL] [RFC] - remove duplicate disconnect event.
On Wed, 5 Apr 2006, Steve Wise wrote: > James, > > Running a 4 thread, 8 ep/thread dapltest (the last test in regress.sh), > I was intermittently seeing a seg fault in dapltest. This is running > over the chelsio rnic using the iwarp branch. After debugging I found > out that dapltest was freeing an already freed endpoint due to it > receiving duplicate disconnect events during test shutdown. The code > assumes it will get exactly one disconnect event for every endpoint > (rightly so I guess). > > I tracked this down to the code in dapl_ep_disconnect() that generates > its own disconnect event in certain circumstances. I removed this and > ran regress.sh over both mthca and cxgb3 with no problems. > > So my question to the dapl experts is: why is this code here? This is an artifact of some older verbs definitions. This code should have gone in the verbs specific portion of DAPL instead of the common code. I'll play around with this and see if there are any negative effects on IB. > For our iwarp devices, it ends up sometimes generating duplicate > disconnect events. I don't see why its needed. If anyone can > explain the logic, that would be great. > > With this patch and the previous patch the fixes dat_ep_free() to always > free the endpoint, I'm able to run dapltest 1-6 over the chelsio rnic. > As part of pulling in the iwarp support, I'd like the group to consider > pulling in these patches that fix issues with udapl (once we agree on > the final patches). For now, I'll maintain these patches in the iwarp > branch... ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: [RFC] [PATCH 1/2] multicast support for multiple users
>1. My main initial comment is that I think that cmp_rec needs to be more >complicated that the matching which is there. The selectors include >things like greater than, less than, and largest available in addition >to equal to which is what is supported there now. I'm not sure whether >any of this is used right now so may not be an issue for IPoIB. I will review the spec to see where the checks need to be enhanced. This probably won't be an issue for a while, since most join requests are limited to select fields of the multicast member record. >2. The other comment is I didn't yet follow how multiple joins of >different JoinStates are handled. I can see there are different slots in >the groups but I didn't see whether all the joins go out on the wire >(one per JoinState) or whether there is some "promotion"/"demotion" of >these. The code uses a promotion/demotion mechanism based on a reference count of membership types. The restriction is that only a single request per group is active at a time. All join requests are queued to a pending list. If a request can be met with the current join state of the group, it is added. If not, then a request is sent to promote the group. Leave requests are handled differently, but result in demotion. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [RFC] [PATCH 1/2] multicast support for multiple users
Hal> Why did the PKey component get removed from the leave ? I don't think it's needed. MGID and PortGID together form the record identifier for multicast groups. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [RFC] [PATCH 1/2] multicast support for multiple users
On Thu, 2006-04-06 at 14:13, Sean Hefty wrote:
Another question:
> +static int send_leave(struct mcast_group *group, u8 leave_state)
> +{
> + struct mcast_port *port = group->port;
> + struct ib_sa_mcmember_rec rec;
> + int ret;
> +
> + rec = group->rec;
> + rec.join_state = leave_state;
> +
> + ret = ib_sa_mcmember_rec_delete(port->dev->device, port->port_num, &rec,
> + IB_SA_MCMEMBER_REC_MGID |
> + IB_SA_MCMEMBER_REC_PORT_GID |
> + IB_SA_MCMEMBER_REC_JOIN_STATE,
Why did the PKey component get removed from the leave ?
-- Hal
> + retry_timer, retries, GFP_KERNEL,
> + leave_handler, group, &group->query);
> + if (ret > 0) {
> + group->query_id = ret;
> + ret = 0;
> + }
> + return ret;
> +}
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: librdmacm/ucma
Michael S. Tsirkin wrote: The library now has min = 1 max = 2 this means that any ABI update in kernel will break userspace. Isn't that what it means to break the ABI? ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [RFC] [PATCH 1/2] multicast support for multiple users
On Thu, 2006-04-06 at 14:13, Sean Hefty wrote:
> Add kernel support that tracks joining and leaving multicast groups.
> The SA tracks join/leave operations on a per port basis. In order
> to support multiple users of the same multicast group, we need to
> track join / leave requests locally.
On initial read, this looks pretty good. Aside from the switch port 0
comment/issue, I have 2 comments/questions.
1. My main initial comment is that I think that cmp_rec needs to be more
complicated that the matching which is there. The selectors include
things like greater than, less than, and largest available in addition
to equal to which is what is supported there now. I'm not sure whether
any of this is used right now so may not be an issue for IPoIB.
2. The other comment is I didn't yet follow how multiple joins of
different JoinStates are handled. I can see there are different slots in
the groups but I didn't see whether all the joins go out on the wire
(one per JoinState) or whether there is some "promotion"/"demotion" of
these.
I will also look at this some more because I need at least a second read
as I didn't follow things sufficiently yet.
-- Hal
> Signed-off-by: Sean Hefty <[EMAIL PROTECTED]>
>
> ---
>
> This patch depends on the sa query patch that adds retries to that API.
> I spent considerable time (and a couple of rewrites of the code) trying to
> ensure that all race conditions were handled, and in a way that was as
> simple as possible. Some additional review of the code that looked for
> race conditions would be appreciated.
>
> Also note that this code has the bug that Michael pointed out:
> http://openib.org/pipermail/openib-general/2006-April/019643.html
> I believe that this bug should be fixed, once we agree on a solution, but
> I don't think that the bug is likely enough to occur that it should delay
> the check-in.
>
>
> Index: include/rdma/ib_multicast.h
> ===
> --- include/rdma/ib_multicast.h (revision 0)
> +++ include/rdma/ib_multicast.h (revision 0)
> @@ -0,0 +1,85 @@
> +/*
> + * Copyright (c) 2006 Intel Corporation. All rights reserved.
> + *
> + * This software is available to you under a choice of one of two
> + * licenses. You may choose to be licensed under the terms of the GNU
> + * General Public License (GPL) Version 2, available from the file
> + * COPYING in the main directory of this source tree, or the
> + * OpenIB.org BSD license below:
> + *
> + * Redistribution and use in source and binary forms, with or
> + * without modification, are permitted provided that the following
> + * conditions are met:
> + *
> + * - Redistributions of source code must retain the above
> + *copyright notice, this list of conditions and the following
> + *disclaimer.
> + *
> + * - Redistributions in binary form must reproduce the above
> + *copyright notice, this list of conditions and the following
> + *disclaimer in the documentation and/or other materials
> + *provided with the distribution.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
> + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
> + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
> + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
> + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
> + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
> + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
> + * SOFTWARE.
> + */
> +
> +#ifndef IB_MULTICAST_H
> +#define IB_MULTICAST_H
> +
> +#include
> +
> +struct ib_multicast {
> + struct ib_sa_mcmember_rec rec;
> + ib_sa_comp_mask comp_mask;
> + int (*callback)(int status,
> + struct ib_multicast *multicast);
> + void*context;
> +};
> +
> +/**
> + * ib_join_multicast - Initiates a join request to the specified multicast
> + * group.
> + * @device: Device associated with the multicast group.
> + * @port_num: Port on the specified device to associate with the multicast
> + * group.
> + * @rec: SA multicast member record specifying group attributes.
> + * @comp_mask: Component mask indicating which group attributes of %rec are
> + * valid.
> + * @gfp_mask: GFP mask for memory allocations.
> + * @callback: User callback invoked once the join operation completes.
> + * @context: User specified context stored with the ib_multicast structure.
> + *
> + * This call initiates a multicast join request with the SA for the specified
> + * multicast group. If the join operation is started successfully, it
> returns
> + * an ib_multicast structure that is used to track the multicast operation.
> + * Users must free this structure by calling ib_free_multicast, even if the
> + * join operation later fails. (The callback
[openib-general] Re: [RFC] [PATCH 1/2] multicast support for multiple users
Hal> I would prefer not to. SMI changes for switches have been Hal> provided but not integrated as yet. Enhanced switch port 0 Hal> would need this (at a minimum). OK, then this code needs to do the usual "port 0 if it's a switch, otherwise ports 1...num ports" - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH] ipoib_flush_paths
Sean> Here's a similar patch for ib_addr. Looks good to me. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: librdmacm/ucma
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Subject: RE: librdmacm/ucma > > >- dependency on libsysfs > > I propose we don't get into this again. I know we have it in ibverbs > > but let's avoid for new code. > > > >- negative error codes > > I think this is kernel practice, in userspace we > > either set errno and return -1, or simply return > > positive error code: otherwise utilities like strerror > > do not work > > We can convert all negative error codes to positive. That's OK. > >- abi versioning > > The RDMA_USER_CM_MAX_ABI_VERSION idea is broken - it makes it so that you > > are > > required to upgrade userspace to run on older kernels. > > Bailing out in userspace if the kernel is too new is not good - please > > remove this test. > > There is both a minor and major number. A given library should support > anything > between its known versions. Can you be more specific about the problem? The library now has min = 1 max = 2 this means that any ABI update in kernel will break userspace. People expect to be able to update kernel and keep old userspace. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] Re: [RFC] [PATCH 1/2] multicast support formultiple users
Sean/Roland:
I may be the only one using the openib stack for a switch. I don't need the
module in question at this point, but I request the group to consider
putting all changes for a switch (as well) while adding new functionality.
This way, if and when I get to a point of needing the functionality, I would
have something to start with! And I promise to test and send in changes (if
required).
BTW, as far as the ib_mad/smi stuff goes, I have sent my changes to Hal,
since I am running 2.6.12 and it is best if he does the merge (I think he
said so already in his email).
Thanks a lot,
Suri
> -Original Message-
> From: [EMAIL PROTECTED] [mailto:openib-general-
> [EMAIL PROTECTED] On Behalf Of Sean Hefty
> Sent: Thursday, April 06, 2006 4:17 PM
> To: Roland Dreier
> Cc: openib-general@openib.org
> Subject: Re: [openib-general] Re: [RFC] [PATCH 1/2] multicast support
> formultiple users
>
> Roland Dreier wrote:
> > > +dev = kmalloc(sizeof *dev + device->phys_port_cnt * sizeof
> *port,
> > > + GFP_KERNEL);
> > > +if (!dev)
> > > +return;
> > > +
> > > +for (i = 1; i <= device->phys_port_cnt; i++) {
> >
> > Seems like this is implicitly assuming that the IB device is a CA.
> >
> > Maybe we should give up the ghost and stop trying to support IB
> switches?
>
> I can go either way here. Would someone want to use this module on a
> switch?
> If so, I can fix up the code so that it can work on one, or at least
> change the
> checks that the device is a CA.
>
> I thought that someone was running at least ib_mad on a switch. I'm just
> not
> sure which other modules are likely to run on them.
>
> - Sean
> ___
> openib-general mailing list
> openib-general@openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-
> general
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] amso1100 testing with OpenIB
On Thu, 2006-04-06 at 14:01 -0400, Karthikeyan Vaidyanathan wrote: > Hi, > > I was trying to follow the discussion on getting the iWARP branch to work > with Ammasso NICs and tried the following steps: > > /include/rdma --> >/gen2/branches/iwarp/src/linux-kernel/infiniband/include/rdma > > /drivers/infiniband --> >/gen2/branches/iwarp/src/linux-kernel/infiniband > > and recompiled the kernel. I'm working on linux 2.6.15.4 kernel. > > I had initially updated the firmware from AMSO1100 Release 1.2 Update 1 > kit. Then I used ccflash2 to update to C2L_H23_B58_F61_080507.bit. > However, when I reboot the machine, the dmesg reports: > > c2: AMSO1100 Gigabit Ethernet driver v1.1 loaded > c2: Downlevel Firmware boot loader [1/7: got 0x42, exp 0x43]. Use the > cc_flash utility to update your boot loader > c2: Adapter not claimed > c2: probe of :03:02.0 failed with error -5 > > I also tried updating the firmware boot loader from AMSO1100 Release 1.2 > Update 1 kit but I still get this message when the machine boots up. > > However, the following command shows that I have the latest firmware > loaded: > > $ ccons 0 bitfile > FPGA Bitfile ID: C2L_H23_B58_080507 Release 23 > You need the amso1100 openib kit from Open Grid Computing. It contains new firmware, a new bitfile/boot loader, and scripts to load the firmware correctly before the openib driver loads. The kit is here: http://www.opengridcomputing.com/downloads/ogc_amso_kit_20060308.tgz You also need to uninstall the Ammasso software from the system. Hope this helps. Steve. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: [RFC] [PATCH 1/2] multicast support for multiple users
Roland Dreier wrote:
> +static void queue_join(struct mcast_member *member)
> +{
> + struct mcast_group *group = member->group;
> + unsigned long flags;
> +
> + spin_lock_irqsave(&group->lock, flags);
> + list_add(&member->list, &group->pending_list);
> + if (group->state == MCAST_IDLE) {
> + group->state = MCAST_BUSY;
> + spin_unlock_irqrestore(&group->lock, flags);
> + atomic_inc(&group->refcount);
> + queue_work(mcast_wq, &group->work);
> + } else
> + spin_unlock_irqrestore(&group->lock, flags);
> +}
should the atomic_inc() be outside the lock here? It seems that
leaves a window for things to go bad. It might be simpler to just do
something like:
The mcast_member already holds a reference on the group, or we couldn't have
acquired the spin_lock. The second reference is taken because we're queuing a
work item that will access the group from a callback. That said, your changes
look cleaner, so I'll update the code.
> +static void adjust_membership(struct mcast_group *group, u8 join_state, int
inc)
> +{
> + int i;
> +
> + for (i = 0; i < 3; i++, join_state >>= 1)
> + if (join_state & 0x1)
> + group->members[i] += inc;
> +}
> +
> +static u8 get_leave_state(struct mcast_group *group)
> +{
> + u8 leave_state = 0;
> + int i;
> +
> + for (i = 0; i < 3; i++)
> + if (!group->members[i])
> + leave_state |= (0x1 << i);
> +
> + return leave_state & group->rec.join_state;
> +}
These look rather magical -- perhaps a comment to make them understandable?
I'll add some comments. Basically, a multicast group has 3 types of members.
These calls track the number of members of each type.
> + case MCAST_JOINING:
> + list_del_init(&member->list);
> + break;
Why not just list_del()? I don't see anywhere that this list_head is
used after this.
I think you're correct. list_del_init() is needed elsewhere, but I think this
can be just list_del().
Thanks for the feedback.
- Sean
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: [RFC] [PATCH 1/2] multicast support for multiple users
On Thu, 2006-04-06 at 16:16, Sean Hefty wrote:
> Roland Dreier wrote:
> > > +dev = kmalloc(sizeof *dev + device->phys_port_cnt * sizeof
> > *port,
> > > + GFP_KERNEL);
> > > +if (!dev)
> > > +return;
> > > +
> > > +for (i = 1; i <= device->phys_port_cnt; i++) {
> >
> > Seems like this is implicitly assuming that the IB device is a CA.
> >
> > Maybe we should give up the ghost and stop trying to support IB switches?
>
> I can go either way here. Would someone want to use this module on a switch?
Yes. IPoIB can run on enhanced switch port 0.
-- Hal
> If so, I can fix up the code so that it can work on one, or at least change
> the
> checks that the device is a CA.
>
> I thought that someone was running at least ib_mad on a switch. I'm just not
> sure which other modules are likely to run on them.
>
> - Sean
> ___
> openib-general mailing list
> openib-general@openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: [RFC] [PATCH 1/2] multicast support for multiple users
Roland Dreier wrote:
> + dev = kmalloc(sizeof *dev + device->phys_port_cnt * sizeof *port,
> + GFP_KERNEL);
> + if (!dev)
> + return;
> +
> + for (i = 1; i <= device->phys_port_cnt; i++) {
Seems like this is implicitly assuming that the IB device is a CA.
Maybe we should give up the ghost and stop trying to support IB switches?
I can go either way here. Would someone want to use this module on a switch?
If so, I can fix up the code so that it can work on one, or at least change the
checks that the device is a CA.
I thought that someone was running at least ib_mad on a switch. I'm just not
sure which other modules are likely to run on them.
- Sean
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [RFC] [PATCH 1/2] multicast support for multiple users
On Thu, 2006-04-06 at 15:51, Roland Dreier wrote:
> > + dev = kmalloc(sizeof *dev + device->phys_port_cnt * sizeof *port,
> > +GFP_KERNEL);
> > + if (!dev)
> > + return;
> > +
> > + for (i = 1; i <= device->phys_port_cnt; i++) {
>
> Seems like this is implicitly assuming that the IB device is a CA.
>
> Maybe we should give up the ghost and stop trying to support IB switches?
I would prefer not to. SMI changes for switches have been provided but
not integrated as yet. Enhanced switch port 0 would need this (at a
minimum).
-- Hal
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: [RFC] [PATCH 2/2] ipoib: convert to use new multicast interface
Roland Dreier wrote: and so on. Can you separate that into another patch, so that it's easier to review the real changes here? Sure. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: [RFC] [PATCH] SA query: expose retries through API
Roland Dreier wrote: Looks fine but can you redo this on top of the module unload race fix once we agree on that? I expect the race fix to go into 2.6.17 and this API change to go into 2.6.18, so the API change needs to apply on top of the race fix. Yes - I'll redo this on top of the API fix. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: [PATCH] ipoib_flush_paths
Here's a similar patch for ib_addr.
Signed-off-by: Sean Hefty <[EMAIL PROTECTED]>
---
Index: core/addr.c
===
--- core/addr.c (revision 6307)
+++ core/addr.c (working copy)
@@ -73,6 +73,7 @@ struct addr_req {
struct sockaddr src_addr;
struct sockaddr dst_addr;
struct rdma_dev_addr *addr;
+ struct module *owner;
void *context;
void (*callback)(int status, struct sockaddr *src_addr,
struct rdma_dev_addr *addr, void *context);
@@ -252,8 +253,10 @@ static void process_req(void *data)
list_for_each_entry_safe(req, temp_req, &done_list, list) {
list_del(&req->list);
+ BUG_ON(!try_module_get(req->owner));
req->callback(req->status, &req->src_addr, req->addr,
req->context);
+ module_put(req->owner);
kfree(req);
}
}
@@ -289,11 +292,12 @@ static int addr_resolve_local(struct soc
return ret;
}
-int rdma_resolve_ip(struct sockaddr *src_addr, struct sockaddr *dst_addr,
- struct rdma_dev_addr *addr, int timeout_ms,
- void (*callback)(int status, struct sockaddr *src_addr,
-struct rdma_dev_addr *addr, void *context),
- void *context)
+int __rdma_resolve_ip(struct sockaddr *src_addr, struct sockaddr *dst_addr,
+ struct rdma_dev_addr *addr, int timeout_ms,
+ void (*callback)(int status, struct sockaddr *src_addr,
+ struct rdma_dev_addr *addr,
+ void *context),
+ void *context, struct module *owner)
{
struct sockaddr_in *src_in, *dst_in;
struct addr_req *req;
@@ -310,6 +314,7 @@ int rdma_resolve_ip(struct sockaddr *src
req->addr = addr;
req->callback = callback;
req->context = context;
+ req->owner = owner;
src_in = (struct sockaddr_in *) &req->src_addr;
dst_in = (struct sockaddr_in *) &req->dst_addr;
@@ -335,7 +340,7 @@ int rdma_resolve_ip(struct sockaddr *src
}
return ret;
}
-EXPORT_SYMBOL(rdma_resolve_ip);
+EXPORT_SYMBOL(__rdma_resolve_ip);
void rdma_addr_cancel(struct rdma_dev_addr *addr)
{
Index: include/rdma/ib_addr.h
===
--- include/rdma/ib_addr.h (revision 6307)
+++ include/rdma/ib_addr.h (working copy)
@@ -64,11 +64,16 @@ int rdma_translate_ip(struct sockaddr *a
* or been canceled. A status of 0 indicates success.
* @context: User-specified context associated with the call.
*/
-int rdma_resolve_ip(struct sockaddr *src_addr, struct sockaddr *dst_addr,
- struct rdma_dev_addr *addr, int timeout_ms,
- void (*callback)(int status, struct sockaddr *src_addr,
-struct rdma_dev_addr *addr, void *context),
- void *context);
+static inline int
+rdma_resolve_ip(struct sockaddr *src_addr, struct sockaddr *dst_addr,
+ struct rdma_dev_addr *addr, int timeout_ms,
+ void (*callback)(int status, struct sockaddr *src_addr,
+struct rdma_dev_addr *addr, void *context),
+ void *context)
+{
+ return __rdma_resolve_ip(src_addr, dst_addr, addr, timeout_ms,
+callback, context, THIS_MODULE);
+}
void rdma_addr_cancel(struct rdma_dev_addr *addr);
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [RFC] [PATCH] SA query: expose retries through API
Looks fine but can you redo this on top of the module unload race fix once we agree on that? I expect the race fix to go into 2.6.17 and this API change to go into 2.6.18, so the API change needs to apply on top of the race fix. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [RFC] [PATCH 2/2] ipoib: convert to use new multicast interface
There is a fair amount of simple reformatting here like > -comp_mask = > -IB_SA_MCMEMBER_REC_MGID | > -IB_SA_MCMEMBER_REC_PORT_GID | > -IB_SA_MCMEMBER_REC_PKEY | > -IB_SA_MCMEMBER_REC_JOIN_STATE; > +comp_mask = IB_SA_MCMEMBER_REC_MGID | > +IB_SA_MCMEMBER_REC_PORT_GID | > +IB_SA_MCMEMBER_REC_PKEY | > +IB_SA_MCMEMBER_REC_JOIN_STATE; > priv->mcast_mtu = ib_mtu_enum_to_int(priv->broadcast->mcmember.mtu) - > -IPOIB_ENCAP_LEN; > + IPOIB_ENCAP_LEN; and so on. Can you separate that into another patch, so that it's easier to review the real changes here? Thanks, Roland ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [RFC] [PATCH 1/2] multicast support for multiple users
> +static void queue_join(struct mcast_member *member)
> +{
> +struct mcast_group *group = member->group;
> +unsigned long flags;
> +
> +spin_lock_irqsave(&group->lock, flags);
> +list_add(&member->list, &group->pending_list);
> +if (group->state == MCAST_IDLE) {
> +group->state = MCAST_BUSY;
> +spin_unlock_irqrestore(&group->lock, flags);
> +atomic_inc(&group->refcount);
> +queue_work(mcast_wq, &group->work);
> +} else
> +spin_unlock_irqrestore(&group->lock, flags);
> +}
should the atomic_inc() be outside the lock here? It seems that
leaves a window for things to go bad. It might be simpler to just do
something like:
spin_lock_irqsave(&group->lock, flags);
list_add(&member->list, &group->pending_list);
if (group->state == MCAST_IDLE) {
group->state = MCAST_BUSY;
atomic_inc(&group->refcount);
queue_work(mcast_wq, &group->work);
}
spin_unlock_irqrestore(&group->lock, flags);
> +static void adjust_membership(struct mcast_group *group, u8 join_state, int
> inc)
> +{
> +int i;
> +
> +for (i = 0; i < 3; i++, join_state >>= 1)
> +if (join_state & 0x1)
> +group->members[i] += inc;
> +}
> +
> +static u8 get_leave_state(struct mcast_group *group)
> +{
> +u8 leave_state = 0;
> +int i;
> +
> +for (i = 0; i < 3; i++)
> +if (!group->members[i])
> +leave_state |= (0x1 << i);
> +
> +return leave_state & group->rec.join_state;
> +}
These look rather magical -- perhaps a comment to make them understandable?
> +case MCAST_JOINING:
> +list_del_init(&member->list);
> +break;
Why not just list_del()? I don't see anywhere that this list_head is
used after this.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [RFC] [PATCH 1/2] multicast support for multiple users
> +dev = kmalloc(sizeof *dev + device->phys_port_cnt * sizeof *port,
> + GFP_KERNEL);
> +if (!dev)
> +return;
> +
> +for (i = 1; i <= device->phys_port_cnt; i++) {
Seems like this is implicitly assuming that the IB device is a CA.
Maybe we should give up the ghost and stop trying to support IB switches?
- R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH] ipoib_cleanup_module
Eli> ensure reverse order of creation or else might cause errors Eli> if debugfs is used. Not sure I follow this. What error could occur? - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [RFC] [PATCH 2/2] ipoib: convert to use new multicast interface
Convert IPoIB to use the new ib_multicast interfaces in place of direct
SA query calls.
Signed-off-by: Sean Hefty <[EMAIL PROTECTED]>
---
Testing was limited to bringing up ipoib and verifying that it worked.
If I can get some guidance on some additional testing, I can do that.
Code review would also be beneficial, as this patch removes a multicast
fix that was just added, which I no longer believe is required.
Index: ulp/ipoib/ipoib_multicast.c
===
--- ulp/ipoib/ipoib_multicast.c (revision 6307)
+++ ulp/ipoib/ipoib_multicast.c (working copy)
@@ -45,6 +45,8 @@
#include
+#include
+
#include "ipoib.h"
#ifdef CONFIG_INFINIBAND_IPOIB_DEBUG
@@ -60,14 +62,11 @@ static DEFINE_MUTEX(mcast_mutex);
/* Used for all multicast joins (broadcast, IPv4 mcast and IPv6 mcast) */
struct ipoib_mcast {
struct ib_sa_mcmember_rec mcmember;
+ struct ib_multicast *mc;
struct ipoib_ah *ah;
struct rb_noderb_node;
struct list_head list;
- struct completion done;
-
- int query_id;
- struct ib_sa_query *query;
unsigned long created;
unsigned long backoff;
@@ -299,18 +298,18 @@ static int ipoib_mcast_join_finish(struc
return 0;
}
-static void
+static int
ipoib_mcast_sendonly_join_complete(int status,
- struct ib_sa_mcmember_rec *mcmember,
- void *mcast_ptr)
+ struct ib_multicast *multicast)
{
- struct ipoib_mcast *mcast = mcast_ptr;
+ struct ipoib_mcast *mcast = multicast->context;
struct net_device *dev = mcast->dev;
struct ipoib_dev_priv *priv = netdev_priv(dev);
if (!status)
- ipoib_mcast_join_finish(mcast, mcmember);
- else {
+ status = ipoib_mcast_join_finish(mcast, &multicast->rec);
+
+ if (status) {
if (mcast->logcount++ < 20)
ipoib_dbg_mcast(netdev_priv(dev), "multicast join
failed for "
IPOIB_GID_FMT ", status %d\n",
@@ -325,10 +324,10 @@ ipoib_mcast_sendonly_join_complete(int s
spin_unlock_irq(&priv->tx_lock);
/* Clear the busy flag so we try again */
- clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags);
+ status = test_and_clear_bit(IPOIB_MCAST_FLAG_BUSY,
+ &mcast->flags);
}
-
- complete(&mcast->done);
+ return status;
}
static int ipoib_mcast_sendonly_join(struct ipoib_mcast *mcast)
@@ -358,35 +357,31 @@ static int ipoib_mcast_sendonly_join(str
rec.port_gid = priv->local_gid;
rec.pkey = cpu_to_be16(priv->pkey);
- init_completion(&mcast->done);
-
- ret = ib_sa_mcmember_rec_set(priv->ca, priv->port, &rec,
-IB_SA_MCMEMBER_REC_MGID|
-IB_SA_MCMEMBER_REC_PORT_GID|
-IB_SA_MCMEMBER_REC_PKEY|
-IB_SA_MCMEMBER_REC_JOIN_STATE,
-1000, GFP_ATOMIC,
-ipoib_mcast_sendonly_join_complete,
-mcast, &mcast->query);
- if (ret < 0) {
- ipoib_warn(priv, "ib_sa_mcmember_rec_set failed (ret = %d)\n",
+ mcast->mc = ib_join_multicast(priv->ca, priv->port, &rec,
+ IB_SA_MCMEMBER_REC_MGID |
+ IB_SA_MCMEMBER_REC_PORT_GID |
+ IB_SA_MCMEMBER_REC_PKEY |
+ IB_SA_MCMEMBER_REC_JOIN_STATE,
+ GFP_ATOMIC,
+ ipoib_mcast_sendonly_join_complete,
+ mcast);
+ if (IS_ERR(mcast->mc)) {
+ ret = PTR_ERR(mcast->mc);
+ clear_bit(IPOIB_MCAST_FLAG_BUSY, &mcast->flags);
+ ipoib_warn(priv, "ib_join_multicast failed (ret = %d)\n",
ret);
} else {
ipoib_dbg_mcast(priv, "no multicast record for " IPOIB_GID_FMT
", starting join\n",
IPOIB_GID_ARG(mcast->mcmember.mgid));
-
- mcast->query_id = ret;
}
-
return ret;
}
-static void ipoib_mcast_join_complete(int status,
- struct ib_sa_mcmember_rec *mcmember,
- void *mcast_ptr)
+static int ipoib_mcast_join_complete(int status,
+struct ib_multicast *multicast)
{
- struct ipoib_mcast *mcast = mcast_ptr;
+ struct ipoib_mcast *mcast = multic
[openib-general] Re: [PATCH] ipoib_flush_paths
Same comments as the sa_query patch apply here: - __module_get() has a race, use try_module_get() instead - let's hide the THIS_MODULE inside an inline wrapper so consumers of the API don't change and don't have to think about this at all. Other than that it seems good. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH] ipoib_flush_paths
Michael> Actually, it turned out to be the simplest solution - and
Michael> quite elegant since there's no room for mistakes: if
Michael> query is going to be running this means module is still
Michael> loaded so we can take a reference to it without races.
Yes, this is suprisingly clean.
Michael> As a bonus, and assertion inside __module_get increases
Michael> the chance to catch races where user forgets to cancel
Michael> the query - much nicer than crashing randomly.
Actually I think __module_get() will do the wrong thing if called
during module unloading -- it doesn't test module_is_live(). In other
words, calling __module_get() without already holding a ref has a
race: __try_stop_module() can see the ref count as 0, then
__module_get() can increment it, and then __try_stop_module() sets the
module state to GOING and returns.
So the right thing to do is BUG_ON(!try_module_get(owner))
Also, I don't think that a consumer of ib_sa() would ever pass an
owner other than THIS_MODULE. So how about if we keep the API the
same and just do the THIS_MODULE stuff in an inline wrapper?
Like the following... it ends up being a pretty big diff, but just
because I moved some comments around and so on. Also I put the
try_module_get() stuff out of line into call_sa_callback(), because
the compiled code ends up smaller that way.
Does anyone disagree with this patch? Michael, are you happy with
this tweaked version of yours?
diff --git a/drivers/infiniband/core/sa_query.c
b/drivers/infiniband/core/sa_query.c
index 501cc05..c43ed75 100644
--- a/drivers/infiniband/core/sa_query.c
+++ b/drivers/infiniband/core/sa_query.c
@@ -74,6 +74,7 @@ struct ib_sa_device {
struct ib_sa_query {
void (*callback)(struct ib_sa_query *, int, struct ib_sa_mad *);
void (*release)(struct ib_sa_query *);
+ struct module *owner;
struct ib_sa_port *port;
struct ib_mad_send_buf *mad_buf;
struct ib_sa_sm_ah *sm_ah;
@@ -547,15 +548,16 @@ static void ib_sa_path_rec_release(struc
* error code. Otherwise it is a query ID that can be used to cancel
* the query.
*/
-int ib_sa_path_rec_get(struct ib_device *device, u8 port_num,
- struct ib_sa_path_rec *rec,
- ib_sa_comp_mask comp_mask,
- int timeout_ms, gfp_t gfp_mask,
- void (*callback)(int status,
- struct ib_sa_path_rec *resp,
- void *context),
- void *context,
- struct ib_sa_query **sa_query)
+int __ib_sa_path_rec_get(struct ib_device *device, u8 port_num,
+struct ib_sa_path_rec *rec,
+ib_sa_comp_mask comp_mask,
+int timeout_ms, gfp_t gfp_mask,
+void (*callback)(int status,
+ struct ib_sa_path_rec *resp,
+ void *context),
+void *context,
+struct module *owner,
+struct ib_sa_query **sa_query)
{
struct ib_sa_path_query *query;
struct ib_sa_device *sa_dev = ib_get_client_data(device, &sa_client);
@@ -590,6 +592,7 @@ int ib_sa_path_rec_get(struct ib_device
query->sa_query.callback = callback ? ib_sa_path_rec_callback : NULL;
query->sa_query.release = ib_sa_path_rec_release;
+ query->sa_query.owner= owner;
query->sa_query.port = port;
mad->mad_hdr.method = IB_MGMT_METHOD_GET;
mad->mad_hdr.attr_id = cpu_to_be16(IB_SA_ATTR_PATH_REC);
@@ -613,7 +616,7 @@ err1:
kfree(query);
return ret;
}
-EXPORT_SYMBOL(ib_sa_path_rec_get);
+EXPORT_SYMBOL(__ib_sa_path_rec_get);
static void ib_sa_service_rec_callback(struct ib_sa_query *sa_query,
int status,
@@ -663,15 +666,16 @@ static void ib_sa_service_rec_release(st
* error code. Otherwise it is a request ID that can be used to cancel
* the query.
*/
-int ib_sa_service_rec_query(struct ib_device *device, u8 port_num, u8 method,
- struct ib_sa_service_rec *rec,
- ib_sa_comp_mask comp_mask,
- int timeout_ms, gfp_t gfp_mask,
- void (*callback)(int status,
-struct ib_sa_service_rec *resp,
-void *context),
- void *context,
- struct ib_sa_query **sa_query)
+int __ib_sa_service_rec_query(struct ib_device *device, u8 port_num, u8 method,
+ struct ib_sa_service_rec *rec,
+ ib_sa_comp_mask comp_mask,
+ int timeout_ms, gfp_t gfp_mask,
+ voi
Re: [openib-general] ipath module compilation failure on SLES10
On Thu, 2006-04-06 at 11:11 -0700, Roland Dreier wrote: > I don't think doing this: Thanks for spotting that. Fixed. http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [RFC] [PATCH 1/2] multicast support for multiple users
Add kernel support that tracks joining and leaving multicast groups.
The SA tracks join/leave operations on a per port basis. In order
to support multiple users of the same multicast group, we need to
track join / leave requests locally.
Signed-off-by: Sean Hefty <[EMAIL PROTECTED]>
---
This patch depends on the sa query patch that adds retries to that API.
I spent considerable time (and a couple of rewrites of the code) trying to
ensure that all race conditions were handled, and in a way that was as
simple as possible. Some additional review of the code that looked for
race conditions would be appreciated.
Also note that this code has the bug that Michael pointed out:
http://openib.org/pipermail/openib-general/2006-April/019643.html
I believe that this bug should be fixed, once we agree on a solution, but
I don't think that the bug is likely enough to occur that it should delay
the check-in.
Index: include/rdma/ib_multicast.h
===
--- include/rdma/ib_multicast.h (revision 0)
+++ include/rdma/ib_multicast.h (revision 0)
@@ -0,0 +1,85 @@
+/*
+ * Copyright (c) 2006 Intel Corporation. All rights reserved.
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ *copyright notice, this list of conditions and the following
+ *disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ *copyright notice, this list of conditions and the following
+ *disclaimer in the documentation and/or other materials
+ *provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#ifndef IB_MULTICAST_H
+#define IB_MULTICAST_H
+
+#include
+
+struct ib_multicast {
+ struct ib_sa_mcmember_rec rec;
+ ib_sa_comp_mask comp_mask;
+ int (*callback)(int status,
+ struct ib_multicast *multicast);
+ void*context;
+};
+
+/**
+ * ib_join_multicast - Initiates a join request to the specified multicast
+ * group.
+ * @device: Device associated with the multicast group.
+ * @port_num: Port on the specified device to associate with the multicast
+ * group.
+ * @rec: SA multicast member record specifying group attributes.
+ * @comp_mask: Component mask indicating which group attributes of %rec are
+ * valid.
+ * @gfp_mask: GFP mask for memory allocations.
+ * @callback: User callback invoked once the join operation completes.
+ * @context: User specified context stored with the ib_multicast structure.
+ *
+ * This call initiates a multicast join request with the SA for the specified
+ * multicast group. If the join operation is started successfully, it returns
+ * an ib_multicast structure that is used to track the multicast operation.
+ * Users must free this structure by calling ib_free_multicast, even if the
+ * join operation later fails. (The callback status is non-zero.)
+ */
+struct ib_multicast *ib_join_multicast(struct ib_device *device, u8 port_num,
+ struct ib_sa_mcmember_rec *rec,
+ ib_sa_comp_mask comp_mask, gfp_t
gfp_mask,
+ int (*callback)(int status,
+ struct ib_multicast
+ *multicast),
+ void *context);
+
+/**
+ * ib_free_multicast - Frees the multicast tracking structure, and releases
+ *any reference on the multicast group.
+ * @multicast: Multicast tracking structure allocated by ib_join_multicast.
+ *
+ * This call blocks until the connection identifier is destroyed. It may
+ * not be called from within the multicast callback; however, returning a non-
+ * zero value from the callback will result in destroying the multicast
+ * tracking structure.
+ */
+void ib_free_multicast(struct ib_multicast *multicast);
+
+#endif /* IB_MULTICAST_H */
Index: core/multicas
Re: [openib-general] ipath module compilation failure on SLES10
I don't think doing this: #ifdef IB_NODE_CA dev->node_type = IB_NODE_CA; #else dev->node_type = RDMA_NODE_IB_CA; #endif bought you anything. IB_NODE_CA is an enum, not a macro. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] FW: [openfabrics-ewg] Changes in SM for the new SRP daemon
Yes it should. Ishai From: Woodruff, Robert J [mailto:[EMAIL PROTECTED] Sent: Thursday, April 06, 2006 1:31 AMTo: Ishai Rabinovitz; [EMAIL PROTECTED]Subject: RE: [openfabrics-ewg] Changes in SM for the new SRP daemon Shouldn't this discussion be on openib-general ? From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Ishai RabinovitzSent: Wednesday, April 05, 2006 2:14 PMTo: [EMAIL PROTECTED]Subject: [openfabrics-ewg] Changes in SM for the new SRP daemon Summary: In the next release of SRP we want to add a daemon that is executed in each initiator and finds out which targets exist in the fabric. The new daemon can use SA capabilities from IBTA 1.2 errata (details at the bottom) to improve performance. If you are using SM other then openIB's openSM please support this feature in your SM. Details: When one wants to find all SRP targets and their information in the fabric, he/she currently run "ibsrpdm""ibsrpdm" uses the following procedure to discover all SRP targets available in the fabric "ibsrpdm" sends a query get_table node info with node type is CA. It gets quite a big table and then for every node it sends a port_info query. From the response to this query the initiator can check if this port is an SRP target. (dm bit capability is set) The problem with this procedure is that it may create too much traffic on the fabric. Let's assume there is a cluster of 4096 nodes booting together. Each of this 4096 nodes is sending the first query and gets a list of 4096 nodes. This list is divided into a long number of UD messages and may cause retransmit. After getting this list each node sends 4096 queries for the port of each node. This traffic is quite huge. The SA has a new capability to answer the query: "please return a list of the ports that has the dm bit set" (meaning return ports of SRP targets). (This capability of the SA is part of Errata Release Version: 1.2 1/26/2006 Chapter/Subsection: 15.2.5.3 - quoted at the bottom). Using this capability we can use the following procedure: The daemon will send "get table port_info of ports that has dm bit set" query and gets a table of small number of port_info. For each port It queries for the guid of this ports. This will significantly reduce the traffic on the fabric. Actually, in this solution there is so little traffic that the new daemon will run it periodically (every minute) to look for changes (There will be less traffic than registering to Trap 64 and Trap 65.) Quoting the errata: Errata Tracking Number: MGTWG8372 Sub-Case Number: 0 Reference ID: 4291 Title: Enhanced SA PortInfoRecord searches Submitter: Livingston, James ([EMAIL PROTECTED]) Volume: 1 Revision: 1.2 Errata Release Version: 1.2 1/26/2006 Chapter/Subsection: 15 Page: 885 Line: 20 AssignedIntensity: Status/Disposition: WG_Approved Problem Description: Add a new row to Table 186 SA-Specific ClassPortInfo:CapabilityMask Bits: Problem Resolution: Add a new row to Table 186 SA-Specific ClassPortInfo:CapabilityMask Bits Original Text: Corrected Text: IsPortInfoCapMaskMatchSupported 13 If this value is 1, SA shall support matching the PortInfo:CapabilityMask component as described in 15.2.5.3>. Comment History: Dec 14, 2005 08:11:26 PM Old: New:Pending By:Benner, Alan ([EMAIL PROTECTED]) Errata Tracking Number: MGTWG8372 Sub-Case Number: 1 Reference ID: 4292 Title: Enhanced SA PortInfoRecord searches Submitter: Livingston, James ([EMAIL PROTECTED]) Volume: 1 Revision: 1.2 Errata Release Version: 1.2 1/26/2006 Chapter/Subsection: 15.2.5.3 Page: 891 Line: 36 AssignedIntensity: Status/Disposition: WG_Approved Problem Description: Add optional compliance statement for new capability. Problem Resolution: Make the proposed change to the spec text. Original Text: Corrected Text: o15-0.x.y: If SA's ClassPortInfo:CapabilityMask.IsPortInfoCapMaskMatchSupported is 1, then the AttributeModifier of the SubnAdmGet() and SubnAdmGetTable() methods affects the matching behavior on the PortInfo:CapabilityMask component. If the high-order bit (bit 31) of the AttributeModifier is set to 1, matching on the CapabilityMask component will not be an exact bitwise match as described in . Instead, matching will only be performed on those bits which are set to 1 in the PortInfo:CapabilityMask embedded in the query. In , bits in the PortInfo:CapabilityMask embedded in the query that are set to 0 are bitwise wildcards for purposes of matching. This gives a requester
[openib-general] amso1100 testing with OpenIB
Hi, I was trying to follow the discussion on getting the iWARP branch to work with Ammasso NICs and tried the following steps: /include/rdma --> /gen2/branches/iwarp/src/linux-kernel/infiniband/include/rdma /drivers/infiniband --> /gen2/branches/iwarp/src/linux-kernel/infiniband and recompiled the kernel. I'm working on linux 2.6.15.4 kernel. I had initially updated the firmware from AMSO1100 Release 1.2 Update 1 kit. Then I used ccflash2 to update to C2L_H23_B58_F61_080507.bit. However, when I reboot the machine, the dmesg reports: c2: AMSO1100 Gigabit Ethernet driver v1.1 loaded c2: Downlevel Firmware boot loader [1/7: got 0x42, exp 0x43]. Use the cc_flash utility to update your boot loader c2: Adapter not claimed c2: probe of :03:02.0 failed with error -5 I also tried updating the firmware boot loader from AMSO1100 Release 1.2 Update 1 kit but I still get this message when the machine boots up. However, the following command shows that I have the latest firmware loaded: $ ccons 0 bitfile FPGA Bitfile ID: C2L_H23_B58_080507 Release 23 Am I missing something here? thanks, Karthik ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [RFC] [PATCH] SA query: expose retries through API
Currently, the SA query interface does not permit retrying requests automatically. Expose this capability to take advantage of underlying MAD layer API, which provides it basically for free because of RMPP. Without automatic retries pushed down into the SA query module, retries are assigned new TIDs, and appear as separate requests. This means that a delayed response will be dropped, and the remote side will not detect that the request is a duplicate, so will re-calculate the response. Signed-off-by: Sean Hefty <[EMAIL PROTECTED]> --- This will be used by the multicast code. I changed all of the APIs to be consistent, but for the purposes of multicast, only ib_sa_mcmember_rec_set and ib_sa_mcmember_rec_delete need to change. Also, the MAD layer currently uses linear retries. This could be changed to an exponential backoff algorithm instead. This would still allow for a default retry algorithm that a consumer could use, but consumers that wanted to manage their own timeout algorithm could do so by specifying a retry count of 0.. Index: include/rdma/ib_sa.h === --- include/rdma/ib_sa.h(revision 6230) +++ include/rdma/ib_sa.h(working copy) @@ -285,7 +285,7 @@ void ib_sa_cancel_query(int id, struct i int ib_sa_path_rec_get(struct ib_device *device, u8 port_num, struct ib_sa_path_rec *rec, ib_sa_comp_mask comp_mask, - int timeout_ms, gfp_t gfp_mask, + int timeout_ms, int retries, gfp_t gfp_mask, void (*callback)(int status, struct ib_sa_path_rec *resp, void *context), @@ -296,7 +296,7 @@ int ib_sa_mcmember_rec_query(struct ib_d u8 method, struct ib_sa_mcmember_rec *rec, ib_sa_comp_mask comp_mask, -int timeout_ms, gfp_t gfp_mask, +int timeout_ms, int retries, gfp_t gfp_mask, void (*callback)(int status, struct ib_sa_mcmember_rec *resp, void *context), @@ -307,7 +307,7 @@ int ib_sa_service_rec_query(struct ib_de u8 method, struct ib_sa_service_rec *rec, ib_sa_comp_mask comp_mask, -int timeout_ms, gfp_t gfp_mask, +int timeout_ms, int retries, gfp_t gfp_mask, void (*callback)(int status, struct ib_sa_service_rec *resp, void *context), @@ -321,6 +321,7 @@ int ib_sa_service_rec_query(struct ib_de * @rec:MCMember Record to send in query * @comp_mask:component mask to send in query * @timeout_ms:time to wait for response + * @retries:number of times to retry request * @gfp_mask:GFP mask to use for internal allocations * @callback:function called when query completes, times out or is * canceled @@ -342,7 +343,7 @@ static inline int ib_sa_mcmember_rec_set(struct ib_device *device, u8 port_num, struct ib_sa_mcmember_rec *rec, ib_sa_comp_mask comp_mask, - int timeout_ms, gfp_t gfp_mask, + int timeout_ms, int retries, gfp_t gfp_mask, void (*callback)(int status, struct ib_sa_mcmember_rec *resp, void *context), @@ -352,7 +353,7 @@ ib_sa_mcmember_rec_set(struct ib_device return ib_sa_mcmember_rec_query(device, port_num, IB_MGMT_METHOD_SET, rec, comp_mask, - timeout_ms, gfp_mask, callback, + timeout_ms, retries, gfp_mask, callback, context, query); } @@ -363,6 +364,7 @@ ib_sa_mcmember_rec_set(struct ib_device * @rec:MCMember Record to send in query * @comp_mask:component mask to send in query * @timeout_ms:time to wait for response + * @retries:number of times to retry request * @gfp_mask:GFP mask to use for internal allocations * @callback:function called when query completes, times out or is * canceled @@ -384,7 +386,7 @@ static inline int ib_sa_mcmember_rec_delete(struct ib_device *device, u8 port_num, struct ib_sa_mcmember_rec *rec, ib_sa_comp_mask comp_mask, - int timeout_ms, gfp_t gfp_mask, + int timeout_ms, int retries, gfp_t gfp_mask, void (*callback)(int status, struct i
[openib-general] Re: [PATCH] [RFC] - dapl - dat_ep_free() can return without
James Lentini wrote: On Tue, 4 Apr 2006, Steve Wise wrote: What happens if a consumer attempts to free the EP from a callback? There are no direct consumer callbacks in usermode are there? consumers call dat_evd_wait() or whatever and get scheduled. Not like kernel mode... Or am I confused? You're right. The DAT consumer thread calling dat_ep_free() will never be a provider (or verbs) thread. It looks like there needs to be some synchronization around destroying the cm_id with the dapli_thread(), though. Could we only delete the QP in dat_ep_free as Sean suggested and leave the cm_id cleanup for later as is being done now? I think we should destroy all resources before returning, including the cm_id. This will insure that no events will fire with context associated with an EP(qp,cm_id,etc) that was just freed. I will take a look at Steve's patch and get back to you. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: ib_umad related kernel panic
On Wed, Apr 05, 2006 at 07:04:29PM -0700, Manpreet Singh wrote: > Hi, > > I am observing the following with redhat kernel rpm at: > http://people.redhat.com/dledford/Infiniband , which uses openib version > 3965. This is on an RHEL4 install. > > When the system is rebooted, ib_core, ib_mad and ib_mthca modules get loaded > automatically. When I load ib_umad after that, I get the following panic. > However, if I unload ib_mthca and load again once before loading ib_umad, > then there is no problem subsequently in the system. The rpms posted on my site, especially the 3985 revision, are not going to be "fixed", they will simply be replaced. We are going to a later rev of the code base for the next update (the 1.0 release branch), so time spent fixing bugs in that version is basically wasted. Just to make that point clear, I've removed the old RPMs from my site and put up a new set of kernel rpms based on the 1.0 release branch code (userspace rpms will be a little later). -- Doug Ledford <[EMAIL PROTECTED]> 919-754-3700 x44233 Red Hat, Inc. 1801 Varsity Dr. Raleigh, NC 27606 ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] ipath module compilation failure on RHEL 4.0 U3
Bryan, I have also failure on: OS: Red Hat Enterprise Linux AS release 4 (Nahant Update 3) Kernel: 2.6.9-34.ELsmp Architecture: x86_64 gcc -Wp,-MD,/var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/hw/ipath/.ipath_cq.o.d -nostdinc -iwithprefix include -D__KERNEL__ -I/var/tmp/IBED/tmp/openib/openib/include -I/var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/include -Iinclude -Wall -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Os -fomit-frame-pointer -g -Wdeclaration-after-statement -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -I/var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/include -I/var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/ulp/ipoib -I/var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/ulp/kdapl -I/var/tmp/IBED/tmp/openib/openib/drivers/infiniband/debug -D__nocast= -DIPATH_IDSTR=PathScale kernel.org driver -DIPATH_KERN_TYPE=0 -DMODULE -DKBUILD_BASENAME=ipath_cq -DKBUILD_MODNAME=ib_ipath -c -o /var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/hw/ipath/.tmp_ipath_cq.o /var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/hw/ipath/ipath_cq.c In file included from /var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/hw/ipath/ipath_cq.c:36: /var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/hw/ipath/ipath_verbs.h:395: error: `BITS_PER_BYTE' undeclared here (not in a function) make[3]: *** [/var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/hw/ipath/ipath_cq.o] Error 1 make[2]: *** [/var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/hw/ipath] Error 2 make[1]: *** [_module_/var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband] Error 2 make[1]: Leaving directory `/usr/src/kernels/2.6.9-34.EL-smp-x86_64' make: *** [kernel] Error 2 Regards, Vladimir ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH] [RFC] - dapl - dat_ep_free() can return without
On Tue, 4 Apr 2006, Steve Wise wrote: > > What happens if a consumer attempts to free the EP from a callback? > > There are no direct consumer callbacks in usermode are there? consumers > call dat_evd_wait() or whatever and get scheduled. Not like kernel > mode... Or am I confused? You're right. The DAT consumer thread calling dat_ep_free() will never be a provider (or verbs) thread. It looks like there needs to be some synchronization around destroying the cm_id with the dapli_thread(), though. Could we only delete the QP in dat_ep_free as Sean suggested and leave the cm_id cleanup for later as is being done now? ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: RE: Re: [PATCH] ipoib_flush_paths
>> Actually, it turned out to be the simplest solution - and quite >> elegant since there's no room for mistakes: if query is going to be running >> this means module is still loaded so we can take a reference to it >> without races. > >And Here's the patch to ib_addr. Sean, Roland, please comment. I like this approach, especially given the current implementations. Roland, Hal? What are your thoughts? Note that ib_cm and rdma_cm technically have the same issue, since cm_id's can be destroyed by returning non-zero from a callback. I.e. a user of those interfaces isn't forced to call anything when unloading. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] ipath module compilation failure on SLES10
On Thu, 2006-04-06 at 19:48 +0300, Vladimir Sokolovsky wrote: > I have the following compilation failure: This should be fixed now. http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] ipath module compilation failure on SLES10
Hi Bryan, I have the following compilation failure: OS: Novell Linux Desktop 10 (x86_64) VERSION = 10 RELEASE = 9 Kernel: 2.6.16-rc5-git9-2-smp Architecture: x86_64 gcc -Wp,-MD,/var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/hw/ipath/.ipath_verbs.o.d -nostdinc -isystem /usr/lib64/gcc/x86_64 -suse-linux/4.1.0/include -D__KERNEL__ -I/var/tmp/IBED/tmp/openib/openib/include -I/var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniba nd/include -Iinclude -Iinclude2 -I/usr/src/linux-2.6.16-rc5-git9-2/include -I/var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/h w/ipath -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -Werror-implicit-function-declaration -fno-strict-aliasing -fno-common -ffreestandin g -Os -fomit-frame-pointer -mtune=generic -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-un wind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -Wdeclaration-after-statement -Wno-pointer-sign -I/var/tmp/IBED/tmp/open ib/openib/src/linux-kernel/infiniband/include -I/var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/ulp/ipoib -I/var/tmp/IBED/tmp/o penib/openib/src/linux-kernel/infiniband/ulp/kdapl -I/var/tmp/IBED/tmp/openib/openib/drivers/infiniband/debug -DIPATH_IDSTR='"PathScale kern el.org driver"' -DIPATH_KERN_TYPE=0 -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(ipath_verbs)" -D"KBUILD_MODNAME=KBUILD_STR(i b_ipath)" -c -o /var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/hw/ipath/.tmp_ipath_verbs.o /var/tmp/IBED/tmp/openib/openib/src/l inux-kernel/infiniband/hw/ipath/ipath_verbs.c /var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/hw/ipath/ipath_verbs.c: In function âipath_register_ib_deviceâ: /var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/hw/ipath/ipath_verbs.c:1005: error: âIB_NODE_CAâ undeclared (first use in this fu nction) /var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/hw/ipath/ipath_verbs.c:1005: error: (Each undeclared identifier is reported only once /var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/hw/ipath/ipath_verbs.c:1005: error: for each function it appears in.) make[5]: *** [/var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/hw/ipath/ipath_verbs.o] Error 1 make[4]: *** [/var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband/hw/ipath] Error 2 make[3]: *** [_module_/var/tmp/IBED/tmp/openib/openib/src/linux-kernel/infiniband] Error 2 make[2]: *** [modules] Error 2 make[1]: *** [modules] Error 2 make[1]: Leaving directory `/usr/src/linux-2.6.16-rc5-git9-2-obj/x86_64/smp' make: *** [kernel] Error 2 Regards, Vladimir ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RC2 delayed a bit
On Thu, 2006-04-06 at 09:57, Tziporet Koren wrote: > Bryan O'Sullivan wrote: > > Unfortunately, the coordination of this with the 1.0 process has thus > > far not been very effective, so I am spending a lot of time manually > > filtering diffs to see what has changed between the first EWG software > > release (named "IBED") and the 1.0 tree, so that I can reunify the two. > > > > > User space code that is needed by IBED was reviewed and check it in to > the 1.0 branch. > We did it to all modules except management that is synchronized by Hal > and ipath library that I assumed you will take care for. > ibed directory was opened under 1.0 branch for scripts & backport patches. > We plan to publish IBED rc3 by Monday. If OF rc2 is not done by then, does that mean that IBED rc3 is still using OF rc2 user packages ? -- Hal > Tziporet > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH] ipoib_flush_paths
Quoting r. Roland Dreier <[EMAIL PROTECTED]>: > Subject: Re: [PATCH] ipoib_flush_paths > > The best solution might be just to say, hey, module unloading has races. Ugh. Please, consider my patch instead. It solves these races in 35 LOC, including SA, ADDR and updating all ULPs. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH] ipoib_flush_paths
The best solution might be just to say, hey, module unloading has races. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: RE: Re: [PATCH] ipoib_flush_paths
Quoting r. Michael S. Tsirkin <[EMAIL PROTECTED]>:
> Actually, it turned out to be the simplest solution - and quite
> elegant since there's no room for mistakes: if query is going to be running
> this means module is still loaded so we can take a reference to it
> without races.
And Here's the patch to ib_addr. Sean, Roland, please comment.
---
Prevent module from being unloaded while callback from ib_addr has signaled
completion but is still running.
Signed-off-by: Michael S. Tsirkin <[EMAIL PROTECTED]>
Index: linux-2.6.16/include/rdma/ib_addr.h
===
--- linux-2.6.16/include/rdma/ib_addr.h (revision 6281)
+++ linux-2.6.16/include/rdma/ib_addr.h (working copy)
@@ -63,12 +63,13 @@
* @callback: Call invoked once address resolution has completed, timed out,
* or been canceled. A status of 0 indicates success.
* @context: User-specified context associated with the call.
+ * @owner: Module that owns the callback.
*/
int rdma_resolve_ip(struct sockaddr *src_addr, struct sockaddr *dst_addr,
struct rdma_dev_addr *addr, int timeout_ms,
void (*callback)(int status, struct sockaddr *src_addr,
struct rdma_dev_addr *addr, void *context),
- void *context);
+ void *context, struct module *owner);
void rdma_addr_cancel(struct rdma_dev_addr *addr);
Index: linux-2.6.16/drivers/infiniband/core/addr.c
===
--- linux-2.6.16/drivers/infiniband/core/addr.c (revision 6281)
+++ linux-2.6.16/drivers/infiniband/core/addr.c (working copy)
@@ -76,6 +76,7 @@
void *context;
void (*callback)(int status, struct sockaddr *src_addr,
struct rdma_dev_addr *addr, void *context);
+ struct module *owner;
unsigned long timeout;
int status;
};
@@ -252,8 +253,10 @@
list_for_each_entry_safe(req, temp_req, &done_list, list) {
list_del(&req->list);
+ __module_get(req->owner);
req->callback(req->status, &req->src_addr, req->addr,
req->context);
+ module_put(req->owner);
kfree(req);
}
}
@@ -293,7 +296,7 @@
struct rdma_dev_addr *addr, int timeout_ms,
void (*callback)(int status, struct sockaddr *src_addr,
struct rdma_dev_addr *addr, void *context),
- void *context)
+ void *context, struct module *owner)
{
struct sockaddr_in *src_in, *dst_in;
struct addr_req *req;
@@ -310,6 +313,7 @@
req->addr = addr;
req->callback = callback;
req->context = context;
+ req->owner = owner;
src_in = (struct sockaddr_in *) &req->src_addr;
dst_in = (struct sockaddr_in *) &req->dst_addr;
--
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] [DAPL] - dapl doesn't set max read
On Wed, 5 Apr 2006, Steve Wise wrote: > Ignore this patch. max_sge_rd is not the correct attribute... You're right. I've fixed it on the trunk and 1.0 branch in revision 6289 with this: Index: openib_cma/dapl_ib_util.c === --- openib_cma/dapl_ib_util.c (revision 6287) +++ openib_cma/dapl_ib_util.c (working copy) @@ -442,7 +442,7 @@ DAT_RETURN dapls_ib_query_hca(IN DAPL_HC ia_attr->transport_attr = NULL; ia_attr->num_vendor_attr = 0; ia_attr->vendor_attr = NULL; - ia_attr->max_iov_segments_per_rdma_read = dev_attr.max_sge_rd; + ia_attr->max_iov_segments_per_rdma_read = dev_attr.max_sge; dapl_dbg_log(DAPL_DBG_TYPE_UTIL, " query_hca: (ver=%x) ep %d ep_q %d evd %d evd_q %d\n", @@ -465,7 +465,7 @@ DAT_RETURN dapls_ib_query_hca(IN DAPL_HC ep_attr->max_request_iov = dev_attr.max_sge; ep_attr->max_rdma_read_in = dev_attr.max_qp_rd_atom; ep_attr->max_rdma_read_out= dev_attr.max_qp_rd_atom; - ep_attr->max_rdma_read_iov= dev_attr.max_sge_rd; + ep_attr->max_rdma_read_iov= dev_attr.max_sge; dapl_dbg_log(DAPL_DBG_TYPE_UTIL, " query_hca: MAX msg %llu dto %d iov %d rdma i%d,o%d\n", ep_attr->max_mtu_size, ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] [DAPL] - dapl doesn't set max read iov
On Wed, 5 Apr 2006, Steve Wise wrote: > > Set the IA attribute max_iov_segments_per_rdma_read and the EP attribute > max_rdma_read_iov based on the openib max_sge_rd device attribute. Committed in the trunk and 1.0 branch in revision 6287. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: RE: Re: [PATCH] ipoib_flush_paths
Quoting r. Fabian Tillier <[EMAIL PROTECTED]>:
> Subject: Re: [openib-general] Re: RE: Re: [PATCH] ipoib_flush_paths
>
> On 4/6/06, Michael S. Tsirkin <[EMAIL PROTECTED]> wrote:
> > Quoting r. Muli Ben-Yehuda <[EMAIL PROTECTED]>:
> > > Subject: Re: [openib-general] Re: RE: Re: [PATCH] ipoib_flush_paths
> > >
> > > On Thu, Apr 06, 2006 at 04:38:33PM +0300, Michael S. Tsirkin wrote:
> > >
> > > > No, since we are keeping a callback pointer into that module.
> > >
> > > Sorry if I'm being dense but I don't see it in this patch. Point me at
> > > it?
> >
> > You don't see it in the patch because SA already kept a callback pointer -
> > that's the race I'm solving. Look in sa_query.c
> >
> >
> > If I have
> >
> > struct query {
> >void (*callback)();
> >struct module *owner;
> > }
> >
> > Then it is always safe to do
> >
> >__get_module(query->owner);
> >query->callback();
> >put_module(query->owner);
> >
> > since it is the called module's responsibility to invalidate
> > all such query objects before its unloaded.
>
> Wait, why are you doing __get_module just before the callback? This
> leaves the possibility of crashing - sure, you'll detect that things
> went wrong, but you haven't solved the issue. The whole point of the
> reference is to prevent the crash.
No, this problem is solved today in all ULPs by caller polling on flag
(completion) that callback sets: all ULPs do this already, since the need to
track resources irrespective of module being unloaded.
> You need to call __get_module from the context of teh caller making the
> request.
No, this would prevent module from being unloaded for extended periods
of time - we don't want this.
All I must prevent is problem we missed previously: module being unloaded
while callback is in progress.
--
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: RE: Re: [PATCH] ipoib_flush_paths
On 4/6/06, Michael S. Tsirkin <[EMAIL PROTECTED]> wrote:
> Quoting r. Muli Ben-Yehuda <[EMAIL PROTECTED]>:
> > Subject: Re: [openib-general] Re: RE: Re: [PATCH] ipoib_flush_paths
> >
> > On Thu, Apr 06, 2006 at 04:38:33PM +0300, Michael S. Tsirkin wrote:
> >
> > > No, since we are keeping a callback pointer into that module.
> >
> > Sorry if I'm being dense but I don't see it in this patch. Point me at
> > it?
>
> You don't see it in the patch because SA already kept a callback pointer -
> that's the race I'm solving. Look in sa_query.c
>
>
> If I have
>
> struct query {
>void (*callback)();
>struct module *owner;
> }
>
> Then it is always safe to do
>
>__get_module(query->owner);
>query->callback();
>put_module(query->owner);
>
> since it is the called module's responsibility to invalidate
> all such query objects before its unloaded.
Wait, why are you doing __get_module just before the callback? This
leaves the possibility of crashing - sure, you'll detect that things
went wrong, but you haven't solved the issue. The whole point of the
reference is to prevent the crash.
You need to call __get_module from the context of teh caller making the request.
- Fab
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: RE: Re: [PATCH] ipoib_flush_paths
On 4/6/06, Michael S. Tsirkin <[EMAIL PROTECTED]> wrote: > Quoting r. Fabian Tillier <[EMAIL PROTECTED]>: > > Subject: Re: RE: Re: [PATCH] ipoib_flush_paths > > > > On 4/6/06, Michael S. Tsirkin <[EMAIL PROTECTED]> wrote: > > > Quoting r. Fabian Tillier <[EMAIL PROTECTED]>: > > > > Can't you pass in a reference to the client module for registration, > > > > and then take a reference from the context of each request that is > > > > released after the callback unwinds? I thought Linux had module > > > > reference functions... > > You are right, you might notice I've reversed my opinion. > Please review the patch I've posted. Yep, just noticed that. That's what I get for replying before getting the latest mails... - Fab > > -- > MST > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: RE: Re: [PATCH] ipoib_flush_paths
Quoting r. Fabian Tillier <[EMAIL PROTECTED]>: > Subject: Re: RE: Re: [PATCH] ipoib_flush_paths > > On 4/5/06, Sean Hefty <[EMAIL PROTECTED]> wrote: > > >Can't you pass in a reference to the client module for registration, > > >and then take a reference from the context of each request that is > > >released after the callback unwinds? I thought Linux had module > > >reference functions... > > > > Yes - this is what ib_mad does. The problem is that ib_sa, ib_addr, ib_cm, > > and > > soon to be ib_multicast can invoke callbacks without explicit registration / > > deregistration. For example, the following interface has the issue: > > > > ib_do_async_operation(request, my_callback, my_context); Correct, that's the issue. So this means we can't fix it without some kind of API change. > You don't need explicit registration/deregistration - just add a > module parameter to this function: > > ib_do_async_op_safe( my_module, request, my_callback, my_context ); That's what I did - added owner parameter after callback and context. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: Re: RE: Re: [PATCH] ipoib_flush_paths
Quoting r. Muli Ben-Yehuda <[EMAIL PROTECTED]>:
> Subject: Re: Re: RE: Re: [PATCH] ipoib_flush_paths
>
> On Thu, Apr 06, 2006 at 05:04:07PM +0300, Michael S. Tsirkin wrote:
>
> > struct query {
> > void (*callback)();
> > struct module *owner;
> > }
> >
> > Then it is always safe to do
> >
> > __get_module(query->owner);
> > query->callback();
> > put_module(query->owner);
>
> Ok, that makes sense. By the way, shouldn't __get_module be
> try_module_get(), with proper error handling if it fails?
No, the whole point is it can't fail.
If it fails its a bug - there's nothing I can do.
--
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RC2 delayed a bit
On Thu, 2006-04-06 at 09:57, Tziporet Koren wrote: > Bryan O'Sullivan wrote: > > Unfortunately, the coordination of this with the 1.0 process has thus > > far not been very effective, so I am spending a lot of time manually > > filtering diffs to see what has changed between the first EWG software > > release (named "IBED") and the 1.0 tree, so that I can reunify the two. > > > > > User space code that is needed by IBED was reviewed and check it in to > the 1.0 branch. > We did it to all modules except management that is synchronized by Hal > and ipath library that I assumed you will take care for. Does IBED maintain it's own copy of userspace ? -- Hal > ibed directory was opened under 1.0 branch for scripts & backport patches. > We plan to publish IBED rc3 by Monday. > > Tziporet > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: RE: Re: [PATCH] ipoib_flush_paths
On Thu, Apr 06, 2006 at 05:04:07PM +0300, Michael S. Tsirkin wrote:
> struct query {
> void (*callback)();
> struct module *owner;
> }
>
> Then it is always safe to do
>
> __get_module(query->owner);
> query->callback();
> put_module(query->owner);
Ok, that makes sense. By the way, shouldn't __get_module be
try_module_get(), with proper error handling if it fails?
Cheers,
Muli
--
Muli Ben-Yehuda
http://www.mulix.org | http://mulix.livejournal.com/
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [DAPL] tests
On Wed, 5 Apr 2006, Vladimir Sokolovsky wrote: > Can you add dapl tests to EXTRA_DIST list in the dapl/Makefile.am? Can you send me a patch with exactly what you want? ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: RE: Re: [PATCH] ipoib_flush_paths
Quoting r. Fabian Tillier <[EMAIL PROTECTED]>: > Subject: Re: RE: Re: [PATCH] ipoib_flush_paths > > On 4/6/06, Michael S. Tsirkin <[EMAIL PROTECTED]> wrote: > > Quoting r. Fabian Tillier <[EMAIL PROTECTED]>: > > > Can't you pass in a reference to the client module for registration, > > > and then take a reference from the context of each request that is > > > released after the callback unwinds? I thought Linux had module > > > reference functions... You are right, you might notice I've reversed my opinion. Please review the patch I've posted. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RE: Re: [PATCH] ipoib_flush_paths
On 4/6/06, Michael S. Tsirkin <[EMAIL PROTECTED]> wrote: > Quoting r. Fabian Tillier <[EMAIL PROTECTED]>: > > Can't you pass in a reference to the client module for registration, > > and then take a reference from the context of each request that is > > released after the callback unwinds? I thought Linux had module > > reference functions... > > I thought about that , but this would mean: > > 1. changing API instead of extending it by new functions - lots of > churn for ULPs Adding new functions to work around a flaw in an existing API is like addressing the symptom of a problem instead of the cause. If the API is broken without the client making some other call to close a race, then the API is broken - fix it. > 2. adding overhead on data path rather than unload path where > it belongs I can't imagine a module reference/dereference adds significant overhead, so I don't buy this argument. - Fab ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: RE: Re: [PATCH] ipoib_flush_paths
Quoting r. Muli Ben-Yehuda <[EMAIL PROTECTED]>:
> Subject: Re: [openib-general] Re: RE: Re: [PATCH] ipoib_flush_paths
>
> On Thu, Apr 06, 2006 at 04:38:33PM +0300, Michael S. Tsirkin wrote:
>
> > No, since we are keeping a callback pointer into that module.
>
> Sorry if I'm being dense but I don't see it in this patch. Point me at
> it?
You don't see it in the patch because SA already kept a callback pointer -
that's the race I'm solving. Look in sa_query.c
If I have
struct query {
void (*callback)();
struct module *owner;
}
Then it is always safe to do
__get_module(query->owner);
query->callback();
put_module(query->owner);
since it is the called module's responsibility to invalidate
all such query objects before its unloaded.
--
MST
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RE: Re: [PATCH] ipoib_flush_paths
On 4/5/06, Sean Hefty <[EMAIL PROTECTED]> wrote: > >Can't you pass in a reference to the client module for registration, > >and then take a reference from the context of each request that is > >released after the callback unwinds? I thought Linux had module > >reference functions... > > Yes - this is what ib_mad does. The problem is that ib_sa, ib_addr, ib_cm, > and > soon to be ib_multicast can invoke callbacks without explicit registration / > deregistration. For example, the following interface has the issue: > > ib_do_async_operation(request, my_callback, my_context); You don't need explicit registration/deregistration - just add a module parameter to this function: ib_do_async_op_safe( my_module, request, my_callback, my_context ); > I was able to come up with several possible solutions to this problem. The > easiest to implement is doing what Michael suggested, and calling some sort of > wait_until_all_current_callbacks_return routine. What I don't like about this > approach is that the interface becomes easier to misuse (i.e. callers must > remember to call wait_until_all_current_callbacks_return before unloading), > plus > it requires changes to interfaces that do work. I don't like this - like you said, it's error prone. > My preference, and it's not a very strong one at this point, is to push the > responsibility into the module invoking the callback. To me, that's the > direction that the reference goes, so that's where the responsibility lies. > Besides, it's his thread that's executing random memory as code. Right - that module just needs to know which module to hold a reference on. - Fab ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
