Re: CentOS7 and systemd ordering when shutting down results in unclean unmount
Yes, I've found your bug reports and thread on the mailing lists about the issue you have been having. Very similar but unfortunately, different as you've noticed. Here is the output of the relevant services: [root@some-server ~]# systemctl status iscsi iscsid remote-fs.target remote-fs-pre.target iscsi.service - Login and scanning of iSCSI devices Loaded: loaded (/usr/lib/systemd/system/iscsi.service; enabled) Active: active (exited) since Tue 2014-12-09 09:37:01 EST; 1 weeks 2 days ago Docs: man:iscsid(8) man:iscsiadm(8) Main PID: 881 (code=exited, status=0/SUCCESS) Dec 09 09:37:00 some-server systemd[1]: Starting Login and scanning of iSCSI devices... Dec 09 09:37:00 some-server iscsi-mark-root-nodes[869]: iscsiadm: No active sessions. Dec 09 09:37:01 some-server iscsiadm[881]: Logging in to [iface: default, target: some-target, portal: 10.174.1.38,3260] (multiple) Dec 09 09:37:01 some-server iscsiadm[881]: Login to [iface: default, target: some-target, portal: 10.174.1.38,3260] successful. Dec 09 09:37:01 some-server systemd[1]: Started Login and scanning of iSCSI devices. iscsid.service - Open-iSCSI Loaded: loaded (/usr/lib/systemd/system/iscsid.service; enabled) Active: active (running) since Tue 2014-12-09 09:37:00 EST; 1 weeks 2 days ago Docs: man:iscsid(8) man:iscsiadm(8) Main PID: 857 (iscsid) CGroup: /system.slice/iscsid.service +-855 /usr/sbin/iscsid +-857 /usr/sbin/iscsid Dec 09 09:37:00 some-server systemd[1]: Starting Open-iSCSI... Dec 09 09:37:00 some-server iscsid[850]: iSCSI logger with pid=855 started! Dec 09 09:37:00 some-server systemd[1]: Failed to read PID from file /var/run/iscsid.pid: Invalid argument Dec 09 09:37:00 some-server systemd[1]: Started Open-iSCSI. Dec 09 09:37:01 some-server iscsid[855]: iSCSI daemon with pid=857 started! Dec 09 09:37:01 some-server iscsid[855]: Connection1:0 to [target: some-target, portal: 10.174.1.38,3260] through [ifac...ational now remote-fs.target - Remote File Systems Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; enabled) Active: active since Tue 2014-12-09 09:37:03 EST; 1 weeks 2 days ago Docs: man:systemd.special(7) Dec 09 09:37:03 some-server systemd[1]: Starting Remote File Systems. Dec 09 09:37:03 some-server systemd[1]: Reached target Remote File Systems. remote-fs-pre.target - Remote File Systems (Pre) Loaded: loaded (/usr/lib/systemd/system/remote-fs-pre.target; static) Active: inactive (dead) Docs: man:systemd.special(7) I also noticed remote-fs-pre.target is not active and guessed that it might be an issue but haven't been able to figure out what needs to be changed to correct it from not being active. local-fs-pre.target is active and I would think these two would be have the same way for the most part. I just updated the iscsi.service unit file to include Wants=remote-fs-pre.target and it seems to have solved the issue! I'm not sure how I didn't find this myself. I thought I tried just about everything. Also, tried this same thing on RHEL7.1 beta and it seems to be fixed there out of the box (no changes necessary). Just looked at the iscsi.service file on RHEL7.1 and it has Wants=remote-fs-pre.target. I guess, according to the changelog, https://bugzilla.redhat.com/show_bug.cgi?id=1161417 may have fixed this issue in 6.2.0.873-22? I looked through the patch you uploaded and didn't see this particular change and was for the fstab issue, which again seems pretty different than this. Nothing else in the changelog for iscsi-initiator-utils on RHEL7.1 seems to indicate where this fix might have come from. Thanks for your help. Will this fix get applied to RHEL7.0 eventually? Will that happen after 7.1? -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
CentOS7 and systemd ordering when shutting down results in unclean unmount
Setup iSCSI on CentOS7. Mounted a iSCSI disk and am running a small MySQL instance on the disk. The iSCSI disk and MySQL instance all come online fine with booting but when shutting down things seem to get very upset and the drive does not get unmounted cleanly. Does not look like I'm the only one having the issue. Another report that is very similar was posted here: https://www.centos.org/forums/viewtopic.php?f=47t=49337 This might be a systemd issue but figured I'd post here first to see if anyone else has had this issue and has any suggestions. Relevant version info: CentOS Linux release 7.0.1406 (Core) systemd-208-11.el7_0.4.x86_64 iscsi-initiator-utils-6.2.0.873-21.el7.x86_64 Systemd unit file for MySQL server: [Unit] Description=MySQL Server After=nss-lookup.target network.target remote-fs.target time-sync.target Wants=nss-lookup.target network.target remote-fs.target time-sync.target Logs from journalctl: Dec 09 09:27:03 example.server.com kernel: EXT4-fs error (device sdb1): __ext4_get_inode_loc:4039: inode #58989720: block 235930089: comm mysqld: unable to read itable block Dec 09 09:27:03 example.server.com kernel: EXT4-fs error (device sdb1) in ext4_reserve_inode_write:4962: IO failure Dec 09 09:27:03 example.server.com iscsiadm[3906]: Logging out of session [sid: 1, target: example.target, portal: 192.168.1.30,3260] Dec 09 09:27:03 example.server.com iscsiadm[3906]: Logout of [sid: 1, target: example.target, portal: 192.168.1.30,3260] successful. Dec 09 09:27:03 example.server.com systemd[1]: Stopped Login and scanning of iSCSI devices. Dec 09 09:27:03 example.server.com systemd[1]: Stopping Open-iSCSI... Dec 09 09:27:03 example.server.com systemd[1]: Stopped Open-iSCSI. Dec 09 09:27:03 example.server.com systemd[1]: Stopped MySQL database. Dec 09 09:27:03 example.server.com systemd[1]: Stopping System Time Synchronized. Dec 09 09:27:03 example.server.com systemd[1]: Stopped target System Time Synchronized. Dec 09 09:27:03 example.server.com systemd[1]: Stopping Remote File Systems. Dec 09 09:27:03 example.server.com systemd[1]: Stopped target Remote File Systems. Dec 09 09:27:03 example.server.com systemd[1]: Unmounting /iscsi-disk... Dec 09 09:27:03 example.server.com systemd[1]: Stopping Host and Network Name Lookups. Dec 09 09:27:03 example.server.com systemd[1]: Stopped target Host and Network Name Lookups. Dec 09 09:27:03 example.server.com kernel: Buffer I/O error on device sdb1, logical block 40960 Dec 09 09:27:03 example.server.com iscsid[853]: Connection1:0 to [target: example.target, portal: 192.168.1.30,3260] through [iface: default] Dec 09 09:27:03 example.server.com iscsid[853]: iscsid shutting down. Dec 09 09:27:03 example.server.com mysqld_safe[1694]: 141209 09:27:03 mysqld_safe mysqld from pid file /backup/mysql-bacula/data/mysqld.pid ended Dec 09 09:27:03 example.server.com kernel: EXT4-fs warning (device sdb1): ext4_end_bio:287: I/O error writing to inode 58989720 (offset 0 size 4096 starting block 40960) Dec 09 09:27:03 example.server.com kernel: EXT4-fs error (device sdb1): ext4_find_entry:1309: inode #58982448: comm mysqld_safe: reading directory lblock 0 Dec 09 09:27:03 example.server.com kernel: Aborting journal on device sdb1-8. Dec 09 09:27:03 example.server.com kernel: Buffer I/O error on device sdb1, logical block 133726208 Dec 09 09:27:03 example.server.com kernel: lost page write due to I/O error on sdb1 Dec 09 09:27:03 example.server.com kernel: JBD2: Error -5 detected when updating journal superblock for sdb1-8. Dec 09 09:27:03 example.server.com kernel: EXT4-fs error (device sdb1): ext4_put_super:789: Couldn't clean up the journal Dec 09 09:27:03 example.server.com kernel: EXT4-fs (sdb1): Remounting filesystem read-only Dec 09 09:27:03 example.server.com systemd[1]: Unmounted /iscsi-disk. Dec 09 09:27:03 example.server.com systemd[1]: Stopping Network is Online. Dec 09 09:27:03 example.server.com systemd[1]: Stopped target Network is Online. Dec 09 09:27:03 example.server.com systemd[1]: Stopping Network. Dec 09 09:27:03 example.server.com systemd[1]: Stopped target Network. Any help would be greatly appreciated. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: CentOS7 and systemd ordering when shutting down results in unclean unmount
Yes, it does. These are the two main unit files provided for open-iscsi. The first is to log in/out of all automatic targets on startup/shutdown: [Unit] Description=Login and scanning of iSCSI devices Documentation=man:iscsid(8) man:iscsiadm(8) DefaultDependencies=no Conflicts=shutdown.target After=systemd-remount-fs.service network.target iscsid.service iscsiuio.service Before=remote-fs-pre.target ConditionDirectoryNotEmpty=|/var/lib/iscsi/nodes ConditionDirectoryNotEmpty=|/sys/class/iscsi_session [Service] Type=oneshot RemainAfterExit=true ExecStart=/usr/libexec/iscsi-mark-root-nodes SuccessExitStatus=21 ExecStart=/sbin/iscsiadm -m node --loginall=automatic ExecStop=/sbin/iscsiadm -m node --logoutall=automatic ExecReload=/sbin/iscsiadm -m node --loginall=automatic [Install] WantedBy=sysinit.target The second is for iscsid: [Unit] Description=Open-iSCSI Documentation=man:iscsid(8) man:iscsiadm(8) DefaultDependencies=no Conflicts=shutdown.target After=network.target iscsiuio.service Before=remote-fs-pre.target [Service] Type=forking PIDFile=/var/run/iscsid.pid ExecStart=/usr/sbin/iscsid ExecStop=/sbin/iscsiadm -k 0 2 [Install] WantedBy=multi-user.target Here is the systemd generated mount until file for the iscsi-disk found in /run/systemd/generator/iscsi-disk.mount: # Automatically generated by systemd-fstab-generator [Unit] SourcePath=/etc/fstab Before=remote-fs.target [Mount] What=/dev/disk/by-label/\x2fiscsi-disk Where=/iscsi-disk Type=ext4 Options=_netdev Here is the fstab entry: LABEL=/iscsi-disk /iscsi-diskext4_netdev 0 0 -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: CentOS7 and systemd ordering when shutting down results in unclean unmount
Sorry, I didn't post the fstab entry in my original post when I should have. The _netdev entry is being applied to the disk and it seems like systemd generator is seeing that option properly and creating the mount point correctly (or so I think). Here is the fstab entry: LABEL=/iscsi-disk /iscsi-diskext4_netdev 0 0 Long story short, no it doesn't seem to be helping unfortunately. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: CentOS7 and systemd ordering when shutting down results in unclean unmount
Thanks for the feedback and suggestion. I'm fairly certain (haven't actually tried though) that adding iscsi-disk.mount or even iscsi.target to the After= of the MySQL service would probably solve this problem I don't think it's a good solution. Just to start, I don't think the MySQL package includes anything about the remote-fs.target at all. This particular MySQL instance/unit file was home grown to include that since it's known it will be running on the iSCSI disk. The reason I don't feel like adding iscsi-disk.mount is a good solution is because that is a dynamically generated file made by systemd when it looks at fstab. That means if I choose to change my mount point from /iscsi-disk to /foo-bar for example having iscsi.mount as an After= will no longer work. As for adding iscsi.service as an After=, I also feel like that is inadequate. What if the underlying storage becomes NFS instead of iSCSI? The whole point of the special systemd remote-fs.target is to handle this and ordering properly yet clearly something isn't right here. While adding these new orderings to the MySQL unit file will help make MySQL less upset it doesn't change the fact that it appears that /iscsi-disk is being unmounted after iscsi.service is stopped which can cause all sorts of file system problems. Again, I appreciate the attempt at a solution but there is something else wrong here that needs a more concrete fix. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Re: CentOS7 and systemd ordering when shutting down results in unclean unmount
I'm not sure if it is the unit files, iscsi or even systemd itself that is the problem. It all seems very strange to me right now. I certainly will post back if it is an open-iscsi issue. As I said, doesn't really seem like the unit files and their requirements are misconfigured. My hunch is something else is in play here, I am just not sure what that something else is. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To unsubscribe from this group and stop receiving emails from it, send an email to open-iscsi+unsubscr...@googlegroups.com. To post to this group, send email to open-iscsi@googlegroups.com. Visit this group at http://groups.google.com/group/open-iscsi. For more options, visit https://groups.google.com/d/optout.
Need help debugging 1011 connection errors
I seem to be getting this over and over again in the logs: Jun 12 09:13:40 example-server kernel: connection0:0: iscsi: detected conn error (1011) Jun 12 09:13:40 example-server iscsid: Kernel reported iSCSI connection 0:0 error (1011) state (3) Jun 12 09:13:43 example-server iscsid: connection0:0 is operational after recovery (1 attempts) Jun 12 09:14:06 example-server iscsid: Got nop in, but kernel supports nop handling. I'm not really sure what the 1011 error indicates. Also the bit about the Got nop in, but kernel supports nop handling. message is confusing as well. It is the first time I have ever seen it and I'm wondering if it is indicating a problem. The kernel on the machine is quite old and could be the culprit either because drivers or whatever else. Currently the kernel is: 2.6.18-8.1.14.el5 Hopefully, going to update it shortly to see if it fixes any of the issues but thought I would post here to see if anyone had any input. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To view this discussion on the web visit https://groups.google.com/d/msg/open-iscsi/-/AyeuqA44Gj0J. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Disconnected iSCSI and umount problems
I am curious to know if you were able to test this and have any of the same issues. On Wednesday, March 21, 2012 2:36:36 PM UTC-4, awidde...@hotmail.com wrote: Here is the output of 'uname -a' Linux test-server 2.6.32-131.0.15.el6.x86_64 #1 SMP Tue May 10 15:42:40 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux Yes, I have two iSCSI disks on this test machine. When I do ' ls /sys/block/ | grep sd' I still see all of the disks: sda sdb sdc -- You received this message because you are subscribed to the Google Groups open-iscsi group. To view this discussion on the web visit https://groups.google.com/d/msg/open-iscsi/-/zo2S5bcZdq8J. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Need help debugging 1011 connection errors
This indicates something went screwy. The kernel has that el5 string, so are you using Centos or RHEL? If so what is the iscsi tools version It is RHEL5 and the tools are as follows: iscsi-initiator-utils-6.2.0.872-13.el5 Is there anything else in the log before or after that? Something about a nop or ping timing out? No, it is just went I sent you repeated over and over again. What type of target is this with? SUN COMSTAR It could happen if the target is not setting something on the iscsi packet correctly. To detect this we could take a wireshark/tcpdump trace and see the packet causing the problem. I doubt this. We have about 300 other hosts connected to this without an issue. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To view this discussion on the web visit https://groups.google.com/d/msg/open-iscsi/-/Zl8RzmwW7uoJ. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
Re: Need help debugging 1011 connection errors
Awesome. I'll do that and hopefully everything is happy afterward including these random connection issues. -- You received this message because you are subscribed to the Google Groups open-iscsi group. To view this discussion on the web visit https://groups.google.com/d/msg/open-iscsi/-/grCIZutEzE4J. To post to this group, send email to open-iscsi@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
