Hi there,
I successfully set up iSCSI server based on tgtd. It uses iscsi driver,
no authorization, no problems found.
Since I use InfiniBand I decided to switch to iSER (iSCSI over RDMA).
Unfortunately, after changing driver to iser I get the following:
on the client side:
# iscsiadm -m node -l
iscsiadm: Could not login to [iface: default, target:
iqn.2024-09.org.open-iscsi.hera:iscsivolumea, portal: 192.168.0.129,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI
login failure)
iscsiadm: Could not log into all portals
on the server side:
tgtd: iser_device_init(3259) dev 0x55bbffc24b60
tgtd: iser_alloc_pool(571) shmget rdma pool sz:1073741824 failed
tgtd: iser_init_rdma_buf_pool(647) pool buf:0x7fc1cc2b7000
list:0x55bbffc24d50 mr:0x55bbffc40d60 lkey:0x17ffff
tgtd: iser_device_init(3278) max 524288 CQEs
tgtd: iser_device_init(3295) CQ vector: 0
tgtd: iser_device_init(3303) dev->cq:0x55bbffc40db0
tgtd: iser_cm_connect_request(1493) alloc conn:0x55bbffc410b0
cm_id:0x55bbffc24110
tgtd: iser_conn_login_phase_set(874) conn:0x55bbffc410b0 from:INIT to:START
tgtd: iser_cm_connect_request(1552) conn:0x55bbffc410b0
cm_id:0x55bbffc24110, created qp:0x55bbffc41688
tgtd: iser_alloc_login_resources(895) conn:0x55bbffc410b0 login tasks
num:2, buf_sz:8544 (rx_sz:8524 tx_sz:8524)
tgtd: iser_alloc_login_resources(944) post_recv login rx task:0x55bbffc73330
tgtd: iser_post_recv(527) conn:0x55bbffc410b0 posted:1 1st
task:0x55bbffc73330 wr_id:0x55bbffc73390 sge_sz:8544
tgtd: iser_conn_getn(1378) refcnt:0 + 1 = 1
tgtd: iser_alloc_login_resources(951) saved login tx-task:0x55bbffc73888
tgtd: iser_conn_get(1371) refcnt:2
tgtd: iser_cm_connect_request(1586) conn:0x55bbffc410b0
cm_id:0x55bbffc24110, 192.168.0.130 -> 192.168.0.129, accepted
tgtd: iser_cm_conn_established(1613) conn:0x55bbffc410b0
cm_id:0x55bbffc24110, 192.168.0.130 -> 192.168.0.129, established
tgtd: handle_wc(2956) recv complete, wr_id:0x55bbffc73390 len:560
tgtd: iser_conn_put(1386) refcnt:1
tgtd: iser_rx_handler_non_ff(2784) login rx, conn:0x55bbffc410b0
tgtd: iser_login_exec(681) conn:0x55bbffc410b0 Login request
(operational negotiation): 15
tgtd: iser_login_rx(2536) conn:0x55bbffc410b0, login failed, class:2
detail:3
tgtd: iser_post_recv(527) conn:0x55bbffc410b0 posted:1 1st
task:0x55bbffc73330 wr_id:0x55bbffc73390 sge_sz:8544
tgtd: iser_conn_getn(1378) refcnt:1 + 1 = 2
tgtd: schedule_resp_tx(156) task:0x55bbffc73888 tag:0x0000 cmdsn:0x0
tgtd: iser_prep_resp_send_req(395) task:0x55bbffc73888
wr_id:0x55bbffc73998 tag:0x0000 dtbuf:0x55bbffc7120c dtsz:0 ahs_sz:0
stat:0x0 statsn:0x0 expcmdsn:0x0
tgtd: iser_post_send(495) conn:0x55bbffc410b0 posted:1 1st
wr:0x55bbffc73998 wr_id:0x55bbffc73998 sge_sz:76
tgtd: iser_conn_getn(1378) refcnt:2 + 1 = 3
tgtd: handle_wc(2956) send complete, wr_id:0x55bbffc73998 len:21947
tgtd: iser_tx_complete_handler(2893) conn:0x55bbffc410b0
task:0x55bbffc73888 tag:0x0000 opcode:0x23
tgtd: iser_conn_put(1386) refcnt:2
tgtd: iser_complete_task(831) task:0x55bbffc73888, non-cmd
tgtd: iser_cm_disconnected(1633) conn:0x55bbffc410b0
cm_id:0x55bbffc24110 event:10, RDMA_CM_EVENT_DISCONNECTED
tgtd: iser_conn_close(1277) rdma_disconnect conn:0x55bbffc410b0
tgtd: iser_ib_clear_tx_list(1207) start
tgtd: iser_conn_close(1292) conn:0x55bbffc410b0 cm_id:0x0x55bbffc24110
state: CLOSE, refcnt:2
tgtd: handle_wc_error(2997) conn:0x55bbffc410b0 task:0x55bbffc73330
tag:0x0000 wr_id:0x0x55bbffc73390 op:recv err:Work Request Flushed Error
vendor_err:0xf5
tgtd: iser_conn_put(1386) refcnt:1
tgtd: iser_cm_timewait_exit(1648) conn:0x55bbffc410b0 refcnt:1
cm_id:0x55bbffc24110
tgtd: iser_ib_clear_tx_list(1207) start
tgtd: iser_cm_timewait_exit(1666) conn:0x55bbffc410b0 refcnt:1
cm_id:0x55bbffc24110 (after cleanup)
tgtd: iser_conn_put(1386) refcnt:0
tgtd: iser_conn_free(1315) conn:0x55bbffc410b0 refcnt:0
qp:0x55bbffc41688 cm_id:0x55bbffc24110
tgtd: iser_ib_clear_iosubmit_list(1179) start
tgtd: iser_ib_clear_rdma_rd_list(1193) start
tgtd: iser_ib_clear_tx_list(1207) start
tgtd: iser_ib_clear_sent_list(1221) start
tgtd: iser_ib_clear_post_recv_list(1234) start
tgtd: iser_free_login_resources(967) conn:0x55bbffc410b0, login phase:START
tgtd: iser_conn_free(1356) conn:0x55bbffc410b0 freed
Which is very strange. First, I do not have authorization, and second, I
also use NFS over RDMA and its working fine.
Any ideas?
PS. Server is Trixi and client is Sid
