Re: Rejection I/O to dead device
Wanted to update on this issues again ... It just means that userspace tried to set a feature the kernel did not support. It is not serious. If there was nothing before that first line about the kernel reporting a connection error, then the target could have initiated this. Do you see anything in the target logs? What target is this again? Target is: Centos51 box also. We have noticed this to happen on this one client (centos51 also) every 2-3 days ... This client seems to do a lot of FS-related testing, not against the iscsiRoot I/O to another FS-mounted system. No messages in the target logs pertaining to this particular client and its iscsi target. Today the same symptom occurred again, I was able to scribble the messages in the console ... (now, I have a terminal server console hooked-up and I'm capturing its output in file so I will have a more complete logging info): psid 0 of 1 psid 0 of 1 oversize name in psid 0 of 1 psid 0 of 1 | | readdir corruption in 000 psid 0 of 1 readdir corruption in 000 | | iscsi: cmd 0x2a is not queued (6) end_request: I/O error dev sde; sector 50640 | | Buffer I/O error on device sde1, logical block 6323 last page write due to I/O error on sde 1 iscsi: cmd 0x2a is not queued (6) Aborting journal on device sde1 iscsi: cmd 0x2a is not queued (6) sd 6:0:0:0 rejection I/O to device being mounted (this repeats a few times) | | EXT3-fs error (device sde1): ext3_find_entry: reading directory 96001 offset 0 .. | --- I upgraded this server with a new iscsiRoot image that incorporates Mike's suggestion to increase timeout specific for iscsiRoot type connections ... would it be possible inaccessibility of root device is possibly cause by this? thanks in advance, - aspasia. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
iscsiRoot's access lost for a few seconds
Hello all, I have been wondering a given scenario, where a server boots linux over iscsiRoot; how would iscsi initiator handle, if for instance, access to the iscsiRoot device is interrupted for a second or two. A situation where this could possibly occur would be if one would add a new target in the /etc/ietd.conf in the iscsi-target host, and then do a restart, while certain hosts are booted against an iscsiRoot served by that same target. Any thoughts? Have there been any past discussions on this or any papers/techtips that may address this kind of situation? thanks in advance, - aspasia. -- A S P A S I A . . . . . . . . . . .. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: iscsiRoot's access lost for a few seconds
Thanks for the response!! I will thoroughly review the readme and feedback any observations. One question - how do I check whether the config parameter change has been set correctly? Do I just look at the config file in the /etc/iscsi/nodes/ etc/default? does not seem to show in iscsiadm -m session -P 1, 2 or 3 level thanks again - I am running a script overnight to soak and see if any networking hiccups would reproduce the symptoms now that I have changed the timeout value ... - a. On 6/4/08, Mike Christie [EMAIL PROTECTED] wrote: a s p a s i a wrote: Hello all, I have been wondering a given scenario, where a server boots linux over iscsiRoot; how would iscsi initiator handle, if for instance, access to the iscsiRoot device is interrupted for a second or two. A situation where this could possibly occur would be if one would add a new target in the /etc/ietd.conf in the iscsi-target host, and then do a restart, while certain hosts are booted against an iscsiRoot served by that same target. This is what the replacement timeout is for. Section 8 in the README tries to describe how all those timers and scsi retries works. It is a little confusing, because some of the info on timeouts and retries is valuable for root and multipath but is only in the multipath section. Each scsi command gets 5 retries. If you were to disrupt iscsi service by pulling cables or restarting targets then if the scsi command's execution gets disrupted more than 5 times you will get a error to the FS. For each disruption you have replacement_timeout seconds to relogin to the target. If replacement_timeout expires then it does not matter how many scsi command retries you have left, the command will be failed and you will get an error to the FS. You really really want to use dm-multipath with iscsi. With dm-multipath there are lots of fun settings for if all paths should fail or look like they have failed (for example if you reboot the target or pull cables). Any thoughts? Have there been any past discussions on this or any papers/techtips that may address this kind of situation? It has been discussed a lot on the list and that is why I tried to comment on it in the README. If you could read that and give me some feedback so it is more useful for others in the future it would be helpful. -- A S P A S I A . . . . . . . . . . .. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Rejection I/O to dead device
Oh yeah, are you doing anything on the target? Rebooting it? Changing the config? Adding or removing targets or portals? in the target when the /etc/ietd.conf file is updated with a new iscsi root, a brief service iscsi-target restart occurs we're thinking of just keeping a static list and not bother it any longer - a. On 6/4/08, Mike Christie [EMAIL PROTECTED] wrote: Mike Christie wrote: a s p a s i a wrote: Wanted to update on this issues again ... It just means that userspace tried to set a feature the kernel did not support. It is not serious. If there was nothing before that first line about the kernel reporting a connection error, then the target could have initiated this. Do you see anything in the target logs? What target is this again? Target is: Centos51 box also. What target is running on Centos? Is it IET or stgt or something else? Oh yeah, are you doing anything on the target? Rebooting it? Changing the config? Adding or removing targets or portals? -- A S P A S I A . . . . . . . . . . .. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Install XEN on CentOS iscsi Root
Hi Konrad, Thanks for the response! That is weird. Try running 'new-kernel-pkg --mkinitrd --depmod --install 2.6.18-53.1.21.el5xen' as root and seeing if that generates the initrd images. i got this though .. new-kernel-pkg --mkinitrd --depmod --install 2.6.18-53.1.21.el5xen iscsiadm: no records found! Unable to find iscsi record for /sys/devices/platform/host6/session1/target6:0:0/6:0:0:0 mkinitrd failed [EMAIL PROTECTED] /]# note i'm running root from iscsiroot: df -k Filesystem 1K-blocks Used Available Use% Mounted on /dev/sde1 2015032 1861252 51420 98% / [EMAIL PROTECTED] /]# where /dev/sde1 is my iscsiroot ... any thoughts? -- A S P A S I A . . . . . . . . . . .. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Install XEN on CentOS iscsi Root
oh ok .. i figured that I needed to create the record - so I did this: iscsiadm -m node -T iqn.2008-04.com.mycompany.lab:storage.centos51_x64_xen -p 192.168.17.3 -o new New iSCSI node [tcp:[hw=default,ip=,net_if=default,iscsi_if=default] 192.168.17.3,3260,-1 iqn.2008-04.com.maxiscale.lab:storage.centos51_x64_xen] added Trying to generate the initrd, got a new error - a grubby error! .. :) [EMAIL PROTECTED] /]# new-kernel-pkg --mkinitrd --depmod --install 2.6.18-53.1.21.el5xen grubby fatal error: unable to find a suitable template [EMAIL PROTECTED] /]# any thoughts? google'ing too .. - a. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Install XEN on CentOS iscsi Root
ok .. will check wiith the sh -x Just to be clear: I tried to run the command below using my current - non-xen kernel - that is ok to do so? uname -a Linux dataserv2 2.6.18-53.1.14.el5 #1 SMP Wed Mar 5 11:37:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux [EMAIL PROTECTED] ~]# - a. On Fri, May 30, 2008 at 9:49 AM, Konrad Rzeszutek [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] /]# new-kernel-pkg --mkinitrd --depmod --install 2.6.18-53.1.21.el5xen grubby fatal error: unable to find a suitable template [EMAIL PROTECTED] /]# Hmm.. When you installed the kernel-xen did it update your mkinitrd as well? No idea what this means. You can try to prefix the program with 'sh -x' as [EMAIL PROTECTED] /]# sh -x new-kernel-pkg --mkinitrd --depmod --install 2.6.18-53.1.21.el5xen to see what arguments it passes to grubby. Could be that some of them are no good. -- A S P A S I A . . . . . . . . . . .. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Install XEN on CentOS iscsi Root
h interesting! 1. so yeah, it did seem to error with grubby, BUT it exit 0'ed ... + /sbin/grubby --add-kernel=/boot/vmlinuz-2.6.18-53.1.21.el5xen --initrd /boot/initrd-2.6.18-53.1.21.el5xen.img --copy-default --title 'CentOS (2.6.18-53.1.21.el5xen)' '--args=root=LABEL=iscsiRoot ' '--remove-kernel=TITLE=CentOS (2.6.18-53.1.21.el5xen)' grubby fatal error: unable to find a suitable template + '[' -n '' ']' + '[' -n '' ']' + exit 0 2. And ... when I ls'ed into /boot (which I did not do earlier, assuming it failed) ... seems like it created the initrd ... i'll try this then ... -rw--- 1 root root 3039935 May 30 11:29 initrd-2.6.18-53.1.21.el5xen.img i'll update you and rest ... thanks - a. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Rejection I/O to dead device
It could have if the routing changed, but the initiator would have tried to reconnect and we would have started using eth1. However, if this happened more than 5 times to the same IO then we could have seen IO errors. Either way though you should have seen some iscsi and scsi errors in the logs. I noticed these occurred: May 29 16:47:31 r05s23 iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (3) May 29 16:47:34 r05s23 iscsid: received iferror -38 May 29 16:47:34 r05s23 last message repeated 3 times May 29 16:47:34 r05s23 iscsid: connection1:0 is operational after recovery (1 attempts) ... what id the -38 error code? thanks in advance. - a. -- A S P A S I A . . . . . . . . . . .. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: Install XEN on CentOS iscsi Root
On Thu, May 29, 2008 at 2:47 PM, Tomasz Chmielewski [EMAIL PROTECTED] wrote: See http://wpkg.org/Diskless_/_remote_boot_with_Open-iSCSI on how to make a custom initrd for iSCSI booting. Hi Tomasz I used those instructions for Debian based distros ... for centos the mkinitrd was really fast and quick - but this command generated an initrd based on the running kernel ... I was hoping there is a way of generating an initrd for Centos and point to a non-running kernel .. or that is not possible? I'm relatively new to linux altogether .. so that may have been a wishful thinking ... - a. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: cannot execute binary error when running iscsiadm
Hi Konrad ... You probably booted a 32-bit kernel and are trying to execute 64-bit programs. h ... or could it possibly be the reservse? [EMAIL PROTECTED] etc]# uname -a Linux dataserv2 2.6.18-53.1.14.el5 #1 SMP Wed Mar 5 11:37:38 EST 2008 x86_64 x86_64 x86_64 GNU/Linux [EMAIL PROTECTED] etc]# ... strange though bec. iscsistart works and they were generated from same installation -- A S P A S I A . . . . . . . . . . .. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: in /var/log/messages: conn errors and recovery
thanks for this final note/recommendation! .. i will do so .. have deployed the golden image though and so far my engineering users have not complained seems like they are happy with their iscsi root on CentoS! - a. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: in /var/log/messages: conn errors and recovery
I need more to the log. The parts above this would tell me what happened. OK, I will capture complete console log next time, sorry, but I rebooted the couple of test boxes that had this node.session.timeo.replacement_timeout = 172800 node.conn[0].timeo.noop_out_timeout = 0 node.conn[0].timeo.noop_out_interval = 0 OK! will do and observe - I just updated the /etc/iscsi/iscsi.conf file and rebooted (working on my golden-build server) ... I also downloaded the latest stable release from the open-iscsi.org site, so that it is now on the 869.2 version as opposed to 865 ... will observe and see if any strange behaviours occur. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: in /var/log/messages: conn errors and recovery
Are you using 869 and do you also see the nop out timeout messages or do you just see these connection error messages? just the above connections errors... 865 version: [EMAIL PROTECTED] ~]# iscsiadm -V iscsiadm version 2.0-865 [EMAIL PROTECTED] ~]# iscsistart -v iscsistart version 2.0-865 [EMAIL PROTECTED] ~]# modinfo scsi_transport_iscsi filename: /lib/modules/2.6.18-53.1.14.el5/kernel/drivers/scsi/scsi_transport_iscsi.ko version:2.0-865 Yeah, read the README section 8 for how to modify the nop and replacment timeout settings for iscsi root. yeah ... i'll do so .. maybe adjust and see if it reappears ... in searching through current /var/log/messages, seems like the errors only appeared twice yesterday and once this morning .. no big deal, but interesting to check. - a. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: scsi_transport_iscsi versions
is a little scary. It looks like you might have multiple versions of userspace running/installed (open-iscsi-2.0-865.15 and 2.0-869). That we do not support. Make sure you have one version of userspace installed (do whereis for iscsid, iscsiadm iscsistart to see if there are multiple copies of programs). seems like there is only a single instance of each: whereis iscsid iscsid: /sbin/iscsid /usr/share/man/man8/iscsid.8.gz [EMAIL PROTECTED]:/lib/modules/2.6.24-16-generic/kernel/drivers/scsi# whereis iscsiadm iscsiadm: /sbin/iscsiadm /usr/share/man/man8/iscsiadm.8.gz [EMAIL PROTECTED]:/lib/modules/2.6.24-16-generic/kernel/drivers/scsi# whereis iscsistart iscsistart: /sbin/iscsistart [EMAIL PROTECTED]:/lib/modules/2.6.24-16-generic/kernel/drivers/scsi# would you recommend that I should just remove these and reinstall? ... Just to be certain, this is the right binary for for Ubuntu on AMD 64 right? http://packages.ubuntu.com/hardy/amd64/open-iscsi/download .. - a. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: scsi_transport_iscsi versions
This is a known bug in Ubuntu Hardy.. I just hit it today also: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/218215 Install the newer kernel from the hardy proposed-updates repository and it fixes the problem. arrgh! u kidding .. me .. ok .. thanks for the heads up! - a. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: open-iscsi vs. initiator-tools ... in Gentoo
Don't you mean /etc/init.d/iscsid ? YES! ... sorry ... yes, /etc/init.d/iscsid or (in other flavors the script is - /etc/rc.d/open-iscsi) ... does not seem to exist. I also noticed the iscsi modules have not been installed. Do I need to install the initiator-tools also? - a. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
Re: open-iscsi boot with Intel Pro/1000 PT
Nice. Did you diverge much from the document or pretty much followed it step-by-step? pretty much stuck by the logic of Tomas's script ... but obviously had to add my own modules (so I also peeked inside the init script of a normal Ubuntu install on the same HW platform to ensure that I'm loading the necessary modules) an explicit assignment of static IP address is so necessary ... else one would lose connection ... obviously, unlike NFSroot'ing ... the iscsiroot relies on IP it was fun ... thanks Tomas for the technote! - a. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups open-iscsi group. To post to this group, send email to open-iscsi@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/open-iscsi -~--~~~~--~~--~--~---
