On Sun, 1 Feb 2026, Greg wrote:
Hi there,
I successfully set up iSCSI server based on tgtd. It uses iscsi driver, no
authorization, no problems found.
Since I use InfiniBand I decided to switch to iSER (iSCSI over RDMA).
Unfortunately, after changing driver to iser I get the following:
on the client side:
# iscsiadm -m node -l
iscsiadm: Could not login to [iface: default, target:
iqn.2024-09.org.open-iscsi.hera:iscsivolumea, portal: 192.168.0.129,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI
login failure)
iscsiadm: Could not log into all portals
on the server side:
tgtd: iser_device_init(3259) dev 0x55bbffc24b60
tgtd: iser_alloc_pool(571) shmget rdma pool sz:1073741824 failed
tgtd: iser_init_rdma_buf_pool(647) pool buf:0x7fc1cc2b7000
list:0x55bbffc24d50 mr:0x55bbffc40d60 lkey:0x17ffff
tgtd: iser_device_init(3278) max 524288 CQEs
tgtd: iser_device_init(3295) CQ vector: 0
tgtd: iser_device_init(3303) dev->cq:0x55bbffc40db0
tgtd: iser_cm_connect_request(1493) alloc conn:0x55bbffc410b0
cm_id:0x55bbffc24110
tgtd: iser_conn_login_phase_set(874) conn:0x55bbffc410b0 from:INIT to:START
tgtd: iser_cm_connect_request(1552) conn:0x55bbffc410b0 cm_id:0x55bbffc24110,
created qp:0x55bbffc41688
tgtd: iser_alloc_login_resources(895) conn:0x55bbffc410b0 login tasks num:2,
buf_sz:8544 (rx_sz:8524 tx_sz:8524)
tgtd: iser_alloc_login_resources(944) post_recv login rx task:0x55bbffc73330
tgtd: iser_post_recv(527) conn:0x55bbffc410b0 posted:1 1st
task:0x55bbffc73330 wr_id:0x55bbffc73390 sge_sz:8544
tgtd: iser_conn_getn(1378) refcnt:0 + 1 = 1
tgtd: iser_alloc_login_resources(951) saved login tx-task:0x55bbffc73888
tgtd: iser_conn_get(1371) refcnt:2
tgtd: iser_cm_connect_request(1586) conn:0x55bbffc410b0 cm_id:0x55bbffc24110,
192.168.0.130 -> 192.168.0.129, accepted
tgtd: iser_cm_conn_established(1613) conn:0x55bbffc410b0
cm_id:0x55bbffc24110, 192.168.0.130 -> 192.168.0.129, established
tgtd: handle_wc(2956) recv complete, wr_id:0x55bbffc73390 len:560
tgtd: iser_conn_put(1386) refcnt:1
tgtd: iser_rx_handler_non_ff(2784) login rx, conn:0x55bbffc410b0
tgtd: iser_login_exec(681) conn:0x55bbffc410b0 Login request (operational
negotiation): 15
tgtd: iser_login_rx(2536) conn:0x55bbffc410b0, login failed, class:2 detail:3
tgtd: iser_post_recv(527) conn:0x55bbffc410b0 posted:1 1st
task:0x55bbffc73330 wr_id:0x55bbffc73390 sge_sz:8544
tgtd: iser_conn_getn(1378) refcnt:1 + 1 = 2
tgtd: schedule_resp_tx(156) task:0x55bbffc73888 tag:0x0000 cmdsn:0x0
tgtd: iser_prep_resp_send_req(395) task:0x55bbffc73888 wr_id:0x55bbffc73998
tag:0x0000 dtbuf:0x55bbffc7120c dtsz:0 ahs_sz:0 stat:0x0 statsn:0x0
expcmdsn:0x0
tgtd: iser_post_send(495) conn:0x55bbffc410b0 posted:1 1st wr:0x55bbffc73998
wr_id:0x55bbffc73998 sge_sz:76
tgtd: iser_conn_getn(1378) refcnt:2 + 1 = 3
tgtd: handle_wc(2956) send complete, wr_id:0x55bbffc73998 len:21947
tgtd: iser_tx_complete_handler(2893) conn:0x55bbffc410b0 task:0x55bbffc73888
tag:0x0000 opcode:0x23
tgtd: iser_conn_put(1386) refcnt:2
tgtd: iser_complete_task(831) task:0x55bbffc73888, non-cmd
tgtd: iser_cm_disconnected(1633) conn:0x55bbffc410b0 cm_id:0x55bbffc24110
event:10, RDMA_CM_EVENT_DISCONNECTED
tgtd: iser_conn_close(1277) rdma_disconnect conn:0x55bbffc410b0
tgtd: iser_ib_clear_tx_list(1207) start
tgtd: iser_conn_close(1292) conn:0x55bbffc410b0 cm_id:0x0x55bbffc24110 state:
CLOSE, refcnt:2
tgtd: handle_wc_error(2997) conn:0x55bbffc410b0 task:0x55bbffc73330
tag:0x0000 wr_id:0x0x55bbffc73390 op:recv err:Work Request Flushed Error
vendor_err:0xf5
tgtd: iser_conn_put(1386) refcnt:1
tgtd: iser_cm_timewait_exit(1648) conn:0x55bbffc410b0 refcnt:1
cm_id:0x55bbffc24110
tgtd: iser_ib_clear_tx_list(1207) start
tgtd: iser_cm_timewait_exit(1666) conn:0x55bbffc410b0 refcnt:1
cm_id:0x55bbffc24110 (after cleanup)
tgtd: iser_conn_put(1386) refcnt:0
tgtd: iser_conn_free(1315) conn:0x55bbffc410b0 refcnt:0 qp:0x55bbffc41688
cm_id:0x55bbffc24110
tgtd: iser_ib_clear_iosubmit_list(1179) start
tgtd: iser_ib_clear_rdma_rd_list(1193) start
tgtd: iser_ib_clear_tx_list(1207) start
tgtd: iser_ib_clear_sent_list(1221) start
tgtd: iser_ib_clear_post_recv_list(1234) start
tgtd: iser_free_login_resources(967) conn:0x55bbffc410b0, login phase:START
tgtd: iser_conn_free(1356) conn:0x55bbffc410b0 freed
Which is very strange. First, I do not have authorization, and second, I also
use NFS over RDMA and its working fine.
Any ideas?
PS. Server is Trixi and client is Sid
No idea but I'd start here:
tgtd: iser_alloc_pool(571) shmget rdma pool sz:1073741824 failed
And I see this in the README.iser on github for tgt:
2.2. Memory registration
One of the most severe stumbling blocks in moving any application to
take advantage of RDMA features is memory registration. Before using
RDMA, both the sending and receiving buffers must be registered with
the operating system. This operation ensures that the underlying
hardware pages will not be modified during the transfer, and provides
the physical addresses of the buffers to the network card. However,
the process itself is time consuming, and CPU intensive. Previous
investigations have shown that for InfiniBand, the throughput drops
by up to 40% when memory registration and deregistration are included
in the critical path.
This iSER implementation uses pre-registered buffers for RDMA
operations. In general such a scheme is difficult to justify due
to the large per-connection resource requirements. However, in
this application it may be appropriate. Since the target always
initiates RDMA operations and never advertises RDMA buffers, it can
securely use one pool of buffers for multiple clients and can manage
its memory resources explicitly. Also, the architecture of the code
is such that the iSCSI layer dictates incoming and outgoing buffer
locations to the storage device layer, so supplying a registered
buffer is relatively easy.
Perhaps you need to have some other config somewhere setting this up?
tgt is quite old in debian. I think the maintainer is MIA. Upstream is
responsive though, there's a couple of patches of mine upstream (which
aren't relevant to your issue) and no doubt there are many others that
might be relevant - debian is a dozen or so versions behind, at least in
trixie.
Tim.
