Bug#980085: open-iscsi: initiator does not connect
On 2/7/21 8:36 PM, Chris Hofstaedtler wrote: Hello Heinrich, I'm going to upload upstream's fix. Maybe you can give it a try. Best, Chris Debian release 2.1.3-2 added +upstream/0001-iscsiadm-Fix-memory-leak-in-iscsiadm.patch +upstream/0002-Fix-iscsiadm-segfault-when-exiting.patch +upstream/0003-Fix-iscsistart-login-issue-when-target-is-delayed.patch I have built the package from https://salsa.debian.org/linux-blocks-team/open-iscsi commit 7334ed475e26b8 This works for me but still shows the warning iscsistart: initiator reported error (15 - session exists) Tested-by: Heinrich Schuchardt
Bug#980085: open-iscsi: initiator does not connect
Hello Heinrich, I'm going to upload upstream's fix. Maybe you can give it a try. Best, Chris
Bug#980085: open-iscsi: initiator does not connect
Hello Chris,
on my system a partial revert solves the problem:
diff --git a/usr/iscsid_req.c b/usr/iscsid_req.c
index a3aba6d..3bbf5b9 100644
--- a/usr/iscsid_req.c
+++ b/usr/iscsid_req.c
@@ -156,7 +156,7 @@ int iscsid_response(int fd, iscsiadm_cmd_e cmd,
iscsiadm_rsp_t *rsp,
if (!err) {
if (poll_wait)
continue;
- return ISCSI_ERR_SESSION_NOT_CONNECTED;
+ return ISCSI_ERR_ISCSID_NOTCONN;
} else if (err < 0) {
if (errno == EINTR)
continue;
[8.110782] SCSI subsystem initialized
[8.121305] Loading iSCSI transport class v2.0-870.
[8.136984] iscsi: registered transport (tcp)
iscsistart: version 2.1.3+
iscsistart: Logging into iqn.2000-01.de.xypron:disk1 192.168.0.1:3260,1
iscsistart: initiator reported error (15 - session exists)
iscsistart: version 2.1.3+
[ 11.116244] dwmac-sun8i 1c3.ethernet eth0: Link is Up -
1Gbps/Full - flow control rx/tx
[ 11.124686] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
iscsistart: Logging into iqn.2000-01.de.xypron:disk1 192.[ 11.231909]
scsi host0: iSCSI Initiator over TCP/IP
168.0.1:3260,1
iscsistart: Connection1:0 to [target: iqn.2000-01.de.xypron:disk1, po[
11.279574] scsi 0:0:0:0: RAID IET Controller
0001 PQ: 0 ANSI: 5
rtal: 192.168.0.1,3260] through [iface: default] is operational now
[ 11.301919] scsi 0:0:0:1: Direct-Access IET VIRTUAL-DISK
0001 PQ: 0 ANSI: 5
[ 11.327734] sd 0:0:0:1: Power-on or device reset occurred
[ 11.336418] sd 0:0:0:1: [sda] 100663296 512-byte logical blocks:
(51.5 GB/48.0 GiB)
[ 11.344182] sd 0:0:0:1: [sda] 4096-byte physical blocks
[ 11.350139] sd 0:0:0:1: [sda] Write Protect is off
[ 11.356083] sd 0:0:0:1: [sda] Write cache: enabled, read cache:
enabled, supports DPO and FUA
[ 11.392053] sda: sda1 sda2 sda3 sda4
[ 11.403161] sd 0:0:0:1: [sda] Attached SCSI disk
Maybe you want to check this with your short-time disabling of network
in QEMU.
Best regards
Heinrich
Bug#980085: open-iscsi: initiator does not connect
On 14.01.21 20:26, Chris Hofstaedtler wrote: > Hi everyone, > > * Chris Hofstaedtler [210114 19:20]: >> * Heinrich Schuchardt [210114 18:52]: > iscsistart: version 2.1.3 > iscsistart: Logging into iqn.2000-01.de.xypron:disk1 > 192.168.0.1:3260,1 > iscsistart: initiator reported error (32 - target likely not > connected) > > I downgraded to open-iscsi and libopeniscsiusr to version 2.1.2-2 and > reinstalled the kernel package. >> >> Could you try rebuilding 2.1.3-1 with this upstream patch reverted? >> https://github.com/open-iscsi/open-iscsi/commit/9258c8eae046d98511d92912983778ca57ba201f >> >> That seems to have changed timeout behaviour, so maybe that "fixed" >> something broken that nevertheless was needed on your system. > > Heinrich: here is a proposed source package: > https://people.debian.org/~zeha/open-iscsi/ > (incl. amd64 binaries, but they are useless to you) > > I gave this some more tries in my test VM, and from what I can tell > there is a race between the iscsi login and the NIC becoming ready. > (In my test VM, the NIC is immediately ready - but I can "unplug" it > and "plug" it back in in just the right moment to simulate a slow > link setup.) > > The 2.1.2 code gave you just a bit more time (and maybe one retry) > to win this race. The 2.1.3 code just flat fails when the NIC is not > ready, at iscsi login time. > > Maybe the old variant is good enough on Heinrich's system. But it > looks like this would break on an "enterprise network", where the > Ethernet link would take many seconds to come online... > > Not sure if we can claim "iscsi root is supposed to work" where it > would usually be deployed. But I don't know about 2.0.x or earlier. > > Chris > Dear Chris, reverting the patch solves my problem: iscsistart: version 2.1.3 iscsistart: Logging into iqn.2000-01.de.xypron:disk1 192.168.0.1:3260,1 iscsistart: initiator reported error (15 - session exists) iscsistart: version 2.1.3 [ 10.636168] dwmac-sun8i 1c3.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx [ 10.644613] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready iscsistart: Logging into iqn.2000-01.de.xypron:disk1 192.[ 10.732030] scsi host0: iSCSI Initiator over TCP/IP 168.0.1:3260,1 iscsistart: Connection1:0 to [target: iqn.2000-01.de.xypron:disk1, po[ 10.779595] scsi 0:0:0:0: RAID IET Controller 0001 PQ: 0 ANSI: 5 rtal: 192.168.0.1,3260] through [iface: default] is operational now [ 10.801637] scsi 0:0:0:1: Direct-Access IET VIRTUAL-DISK 0001 PQ: 0 ANSI: 5 [ 10.826829] sd 0:0:0:1: Power-on or device reset occurred [ 10.835649] sd 0:0:0:1: [sda] 100663296 512-byte logical blocks: (51.5 GB/48.0 GiB) [ 10.843421] sd 0:0:0:1: [sda] 4096-byte physical blocks [ 10.849319] sd 0:0:0:1: [sda] Write Protect is off [ 10.855275] sd 0:0:0:1: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA Best regards Heinrich
Bug#980085: open-iscsi: initiator does not connect
Hi everyone, * Chris Hofstaedtler [210114 19:20]: > * Heinrich Schuchardt [210114 18:52]: > > >> iscsistart: version 2.1.3 > > >> iscsistart: Logging into iqn.2000-01.de.xypron:disk1 > > >> 192.168.0.1:3260,1 > > >> iscsistart: initiator reported error (32 - target likely not > > >> connected) > > >> > > >> I downgraded to open-iscsi and libopeniscsiusr to version 2.1.2-2 and > > >> reinstalled the kernel package. > > Could you try rebuilding 2.1.3-1 with this upstream patch reverted? > https://github.com/open-iscsi/open-iscsi/commit/9258c8eae046d98511d92912983778ca57ba201f > > That seems to have changed timeout behaviour, so maybe that "fixed" > something broken that nevertheless was needed on your system. Heinrich: here is a proposed source package: https://people.debian.org/~zeha/open-iscsi/ (incl. amd64 binaries, but they are useless to you) I gave this some more tries in my test VM, and from what I can tell there is a race between the iscsi login and the NIC becoming ready. (In my test VM, the NIC is immediately ready - but I can "unplug" it and "plug" it back in in just the right moment to simulate a slow link setup.) The 2.1.2 code gave you just a bit more time (and maybe one retry) to win this race. The 2.1.3 code just flat fails when the NIC is not ready, at iscsi login time. Maybe the old variant is good enough on Heinrich's system. But it looks like this would break on an "enterprise network", where the Ethernet link would take many seconds to come online... Not sure if we can claim "iscsi root is supposed to work" where it would usually be deployed. But I don't know about 2.0.x or earlier. Chris
Bug#980085: open-iscsi: initiator does not connect
Hi Heinrich, * Heinrich Schuchardt [210114 18:52]: > >> iscsistart: version 2.1.3 > >> iscsistart: Logging into iqn.2000-01.de.xypron:disk1 > >> 192.168.0.1:3260,1 > >> iscsistart: initiator reported error (32 - target likely not > >> connected) > >> > >> I downgraded to open-iscsi and libopeniscsiusr to version 2.1.2-2 and > >> reinstalled the kernel package. Could you try rebuilding 2.1.3-1 with this upstream patch reverted? https://github.com/open-iscsi/open-iscsi/commit/9258c8eae046d98511d92912983778ca57ba201f That seems to have changed timeout behaviour, so maybe that "fixed" something broken that nevertheless was needed on your system. Chris
Bug#980085: open-iscsi: initiator does not connect
* Heinrich Schuchardt [210114 18:46]: > The same problem is described in: > > https://github.com/open-iscsi/open-iscsi/issues/241 This issue says stuff about IPv6, but from your log output I see nothing about IPv6. Can you explain more about your setup? Hardware, boot command line, network setup, ... Chris
Bug#980085: open-iscsi: initiator does not connect
Hi, open-iscsi 2.1.3-1 and kernel 5.10.0-1-amd64 seem to work fine for me in a test VM with iscsi root. However, no offloading NIC is involved. Given this works, maybe its best if you bug upstream about this? Chris
Bug#980085: open-iscsi: initiator does not connect
On 14.01.21 17:02, Ritesh Raj Sarraf wrote: > Hi Heinrich, > > On Thu, 2021-01-14 at 11:23 +0100, Heinrich Schuchardt wrote: >> On my system open-iscsi and libopeniscsiusr where upgraded to 2.1.3- >> 1. >> >> With a kernel installed after this upgrade my system cannot boot from >> an >> iSCSI LUN: >> >> iscsistart: version 2.1.3 >> iscsistart: Logging into iqn.2000-01.de.xypron:disk1 >> 192.168.0.1:3260,1 >> iscsistart: initiator reported error (32 - target likely not >> connected) >> >> I downgraded to open-iscsi and libopeniscsiusr to version 2.1.2-2 and >> reinstalled the kernel package. >> > > > I don't know how to react, whether I should thank you or not. > > That's the last version I had uploaded, 2.1.2. And it did clear my > tests, that of an iscsi root disk. > > Now that I know 2.1.3 has broken iscsi root setup, I will be cautious > when/if trying to debug that issue, when time permits. > Hello Ritesh, thank you for looking into this. According to https://tracker.debian.org/pkg/open-iscsi Chris migrated 2.1.3-1 to testing. I replaced %s/iscsistart/iscsistart -d 8/g in file /usr/share/initramfs-tools/scripts/local-top/iscsi to get the extended log at the end of the mail. I hope that helps. >> Now the system boots with: >> >> iscsistart: Logging into iqn.2000-01.de.xypron:disk1 >> 192.168.0.1:3260,1 >> iscsistart: initiator reported error (15 - session exists) >> iscsistart: version 2.1.2 >> iscsistart: Logging into iqn.2000-01.de.xypron:disk1 >> 192.168.0.1:3260,1 >> iscsistart: Connection1:0 to [target: iqn.2000-01.de.xypron:disk1, >> portal: 192.168.0.1,3260] through [iface: default] is operational now > > Just for my own sake. Since the new upgrade broke your setup, how did > you revert back to the older iscsi initiator (2.1.2) for the root LUN ? > Did you build this setup on a VM ? > Did you have snapshots of the root LUN ? Only the newest initrd was concerned. So I could still boot with an old kernel after selecting it in GRUB. You can find old packages on https://snapshot.debian.org/. You could also consider chroot'ing into the image from your workstation. Best regards Heinrich Boot log Begin: Running /scripts/init-premount ... done. Begin: Mounting root file system ... Begin: Running /scripts/local-top ... sh: -d: unknown operand IP-Config: eth0 hardware address xx:xx:xx:xx:xx:xx mtu 1500 DHCP[ 7.484587] dwmac-sun8i 1c3.ethernet eth0: PHY [stmmac-0:01] driver [RTL8211E Gigabit Ethernet] (irq=POLL) RARP [7.502403] dwmac-sun8i 1c3.ethernet eth0: No Safety Features support found [7.509764] dwmac-sun8i 1c3.ethernet eth0: No MAC Management Counters available [7.517438] dwmac-sun8i 1c3.ethernet eth0: PTP not supported by HW [7.524608] dwmac-sun8i 1c3.ethernet eth0: configuring for phy/rgmii-id link mode IP-Config: eth0 guessed broadcast address 192.168.0.255 IP-Config: eth0 complete: address: 192.168.0.18 broadcast: 192.168.0.255 netmask: 255.255.255.0 gateway: 192.168.0.254 dns0 : 0.0.0.0 dns1 : 0.0.0.0 rootserver: 192.168.0.254 rootpath: filename : [7.570722] SCSI subsystem initialized [7.581238] Loading iSCSI transport class v2.0-870. [7.597809] iscsi: registered transport (tcp) iscsistart: in ctldev_open iscsistart: created NETLINK_ISCSI socket... iscsistart: InitiatorName=iqn.1993-08.org.debian:01:xxx iscsistart: InitiatorAlias=(none) iscsistart: TargetName=iqn.2000-01.de.xypron:disk1 iscsistart: TPGT=1 iscsistart: IP Address=192.168.0.1 iscsistart: version 2.1.3 iscsistart: current time 7 iscsistart: nothing on pend_list, deactivating alarm iscsistart: Logging into iqn.2000-01.de.xypron:disk1 192.168.0.1:3260,1 iscsistart: poll result 1 iscsistart: in read_transports iscsistart: Adding new transport tcp iscsistart: Matched transport tcp iscsistart: sysfs_attr_get_value: open '/class/iscsi_transport/tcp'/'handle' iscsistart: sysfs_attr_get_value: open '/class/iscsi_transport/tcp'/'caps' iscsistart: Allocted session 0xfb4531d0 iscsistart: authentication setup complete... iscsistart: resolved 192.168.0.1 to 192.168.0.1 iscsistart: setting iface default, dev , set ip , hw , transport tcp. iscsistart: get ev context 0xfb44b180 iscsistart: set TCP recv window size to 524288, actually got 425984 iscsistart: set TCP send window size to 524288, actually got 425984 iscsistart: connecting to 192.168.0.1:3260 iscsistart: sched conn context 0xfb44b180 event 2, tmo 0 iscsistart: thread 0xfb44b180 schedule: delay 0 state 3 iscsistart: Setting login timer 0xfb45b138 timeout 30 iscsistart: thread fb45b138 delete: state 3 iscsistart: thread 0xfb45b138 schedule: delay 30 state 3 iscsistart: new thread 0xfb45b138 due 38 is first item on pend_list iscsistart: new alarm set for 30 seconds, old alarm 0 iscsistart: current time 8 iscsistart: thread fb45b138 due 38, wait 30 more iscsistart: exec thread fb44b180 callback iscsistart: put ev context 0xfb44b180 isc
Bug#980085: open-iscsi: initiator does not connect
Hi Heinrich, On Thu, 2021-01-14 at 11:23 +0100, Heinrich Schuchardt wrote: > On my system open-iscsi and libopeniscsiusr where upgraded to 2.1.3- > 1. > > With a kernel installed after this upgrade my system cannot boot from > an > iSCSI LUN: > > iscsistart: version 2.1.3 > iscsistart: Logging into iqn.2000-01.de.xypron:disk1 > 192.168.0.1:3260,1 > iscsistart: initiator reported error (32 - target likely not > connected) > > I downgraded to open-iscsi and libopeniscsiusr to version 2.1.2-2 and > reinstalled the kernel package. > I don't know how to react, whether I should thank you or not. That's the last version I had uploaded, 2.1.2. And it did clear my tests, that of an iscsi root disk. Now that I know 2.1.3 has broken iscsi root setup, I will be cautious when/if trying to debug that issue, when time permits. > Now the system boots with: > > iscsistart: Logging into iqn.2000-01.de.xypron:disk1 > 192.168.0.1:3260,1 > iscsistart: initiator reported error (15 - session exists) > iscsistart: version 2.1.2 > iscsistart: Logging into iqn.2000-01.de.xypron:disk1 > 192.168.0.1:3260,1 > iscsistart: Connection1:0 to [target: iqn.2000-01.de.xypron:disk1, > portal: 192.168.0.1,3260] through [iface: default] is operational now Just for my own sake. Since the new upgrade broke your setup, how did you revert back to the older iscsi initiator (2.1.2) for the root LUN ? Did you build this setup on a VM ? Did you have snapshots of the root LUN ? Thanks, Ritesh -- Ritesh Raj Sarraf | http://people.debian.org/~rrs Debian - The Universal Operating System signature.asc Description: This is a digitally signed message part
Bug#980085: open-iscsi: initiator does not connect
The same problem is described in: https://github.com/open-iscsi/open-iscsi/issues/241
Bug#980085: open-iscsi: initiator does not connect
Package: open-iscsi Version: 2.1.3-1 Severity: important On my system open-iscsi and libopeniscsiusr where upgraded to 2.1.3-1. With a kernel installed after this upgrade my system cannot boot from an iSCSI LUN: iscsistart: version 2.1.3 iscsistart: Logging into iqn.2000-01.de.xypron:disk1 192.168.0.1:3260,1 iscsistart: initiator reported error (32 - target likely not connected) I downgraded to open-iscsi and libopeniscsiusr to version 2.1.2-2 and reinstalled the kernel package. Now the system boots with: iscsistart: Logging into iqn.2000-01.de.xypron:disk1 192.168.0.1:3260,1 iscsistart: initiator reported error (15 - session exists) iscsistart: version 2.1.2 iscsistart: Logging into iqn.2000-01.de.xypron:disk1 192.168.0.1:3260,1 iscsistart: Connection1:0 to [target: iqn.2000-01.de.xypron:disk1, portal: 192.168.0.1,3260] through [iface: default] is operational now -- System Information: Debian Release: bullseye/sid APT prefers testing APT policy: (500, 'testing') Architecture: arm64 (aarch64) Kernel: Linux 5.10.7-fwts-arm64+ (SMP w/4 CPU threads) Kernel taint flags: TAINT_UNSIGNED_MODULE Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) LSM: AppArmor: enabled Versions of packages open-iscsi depends on: ii debconf [debconf-2.0] 1.5.74 ii libc6 2.31-9 ii libisns0 0.100-3 ii libkmod2 28-1 ii libmount1 2.36.1-4 ii libopeniscsiusr2.1.3-1 ii libssl1.1 1.1.1i-1 ii libsystemd0247.2-4 ii udev 247.2-4
