Re: [libvirt] Re: [Libvir] ISCSI howto, example, etc?
Chris Lalancette schreef: Stefan de Konink wrote: I think if you can get iscsi_sysfs.c; get_block_dev_from_lun you have the code already to do this all. So basically get openiscsi into a lib with helper functions. OK. I took a look at this further, and I figured out what is going on here. I can confirm this works for me. Stefan -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Re: [Libvir] ISCSI howto, example, etc?
Stefan de Konink wrote: I think if you can get iscsi_sysfs.c; get_block_dev_from_lun you have the code already to do this all. So basically get openiscsi into a lib with helper functions. OK. I took a look at this further, and I figured out what is going on here. The original impetus for me to re-write the code in virStorageBackendISCSIMakeLUN() was because I thought the open-iscsi people had changed the output of iscsiadm --mode session -r $session -P 3. However, that's not what actually happened. What actually happened is that the sysfs files that export this information changed somewhere between 2.6.21 and 2.6.24 (which is what I was using at the time). So, the code you point to above in open-iscsi tools *also* doesn't work. That code is expecting the block devices to look like block:sdd, which is what you have in your 2.6.21 kernel; but in later kernels, it no longer looks like that. All that being said, the less we have to depend on parsing, and the more we can poke around sysfs and get values, the better. So in that sense, the change was for the better. I've attached a patch which seems to do the right thing on my 2.6.25 based machine; can you test it on 2.6.21 and see if it works? Chris Lalancette Index: src/storage_backend_iscsi.c === RCS file: /data/cvs/libvirt/src/storage_backend_iscsi.c,v retrieving revision 1.5 diff -u -r1.5 storage_backend_iscsi.c --- a/src/storage_backend_iscsi.c 10 Apr 2008 16:53:29 - 1.5 +++ b/src/storage_backend_iscsi.c 21 May 2008 14:49:14 - @@ -170,7 +170,7 @@ virStorageBackendISCSIMakeLUN(virConnectPtr conn, virStoragePoolObjPtr pool, char **const groups, - void *data) + void *data ATTRIBUTE_UNUSED) { virStorageVolDefPtr vol; int fd = -1; @@ -178,13 +178,12 @@ char lunid[100]; int opentries = 0; char *devpath = NULL; -char *session = data; char sysfs_path[PATH_MAX]; char *dev = NULL; DIR *sysdir; struct dirent *block_dirent; -struct stat sbuf; int len; +int new_style_sysfs; if ((virStrToLong_ui(groups[0], NULL, 10, target) 0) || (virStrToLong_ui(groups[1], NULL, 10, channel) 0) || @@ -200,26 +199,33 @@ return 0; } -snprintf(sysfs_path, PATH_MAX, - /sys/class/iscsi_session/session%s/device/ - target%d:%d:%d/%d:%d:%d:%d/block, - session, target, channel, id, target, channel, id, lun); +/* There are two styles of sysfs paths that we have to worry about; the + * old style and the new style. The old style is of the form: + * /sys/bus/scsi/devices/target:channel:id:lun/block:sdd + * + * while the new style is of the form: + * /sys/bus/scsi/devices/target:channel:id:lun/block/sdd + */ -if (stat(sysfs_path, sbuf) 0) { -/* block path in subdir didn't exist; this is unexpected, so fail */ -virStorageReportError(conn, VIR_ERR_INTERNAL_ERROR, - _(Failed to find the sysfs path for %d:%d:%d:%d: %s), - target, channel, id, lun, strerror(errno)); -return -1; -} +/* First try the new style */ +new_style_sysfs = 1; +snprintf(sysfs_path, PATH_MAX, /sys/bus/scsi/devices/%d:%d:%d:%d/block, +target, channel, id, lun); sysdir = opendir(sysfs_path); if (sysdir == NULL) { -/* we failed for some reason; return an error */ -virStorageReportError(conn, VIR_ERR_INTERNAL_ERROR, - _(Failed to opendir sysfs path %s: %s), - sysfs_path, strerror(errno)); -return -1; +/* OK, maybe this is the old style */ +snprintf(sysfs_path, PATH_MAX, /sys/bus/scsi/devices/%d:%d:%d:%d, +target, channel, id, lun); +sysdir = opendir(sysfs_path); +if (sysdir == NULL) { +/* OK, wasn't new or old style; something we don't understand */ +virStorageReportError(conn, VIR_ERR_INTERNAL_ERROR, +_(Failed to opendir sysfs path %s: %s), +sysfs_path, strerror(errno)); +return -1; +} +new_style_sysfs = 0; } while ((block_dirent = readdir(sysdir)) != NULL) { @@ -229,16 +235,19 @@ /* the . and .. directories; just skip them */ continue; } - -/* OK, not . or ..; let's see if it is a SCSI device */ -if (len 2 + +if (new_style_sysfs len 2 block_dirent-d_name[0] == 's' block_dirent-d_name[1] == 'd') { -/* looks like a scsi device, smells like scsi device; it must be - a scsi device */ dev = strdup(block_dirent-d_name); break; }
Re: [libvirt] Re: [Libvir] ISCSI howto, example, etc?
On Wed, May 21, 2008 at 04:50:12PM +0200, Chris Lalancette wrote: Stefan de Konink wrote: I think if you can get iscsi_sysfs.c; get_block_dev_from_lun you have the code already to do this all. So basically get openiscsi into a lib with helper functions. OK. I took a look at this further, and I figured out what is going on here. The original impetus for me to re-write the code in virStorageBackendISCSIMakeLUN() was because I thought the open-iscsi people had changed the output of iscsiadm --mode session -r $session -P 3. However, that's not what actually happened. What actually happened is that the sysfs files that export this information changed somewhere between 2.6.21 and 2.6.24 (which is what I was using at the time). So, the code you point to above in open-iscsi tools *also* doesn't work. That code is expecting the block devices to look like block:sdd, which is what you have in your 2.6.21 kernel; but in later kernels, it no longer looks like that. All that being said, the less we have to depend on parsing, and the more we can poke around sysfs and get values, the better. So in that sense, the change was for the better. I've attached a patch which seems to do the right thing on my 2.6.25 based machine; can you test it on 2.6.21 and see if it works? Can you provide a documentation patch describing the sysfs layout for the kernels the patch is intended to support, along with captured output from the isciadm tool running on each, for the various commands libvirt needs to run. The only way we are going to avoid breaking this repeatedly is to get serious around documentating the variants we need to support then then testing them. The example output from iscsidam tool then can be used to build a fake iscsiadm tool invoked by the test suites exercising all the new code. Regards, Daniel. -- |: Red Hat, Engineering, Boston -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Re: [Libvir] ISCSI howto, example, etc?
Daniel P. Berrange wrote: Can you provide a documentation patch describing the sysfs layout for the kernels the patch is intended to support, along with captured output from the isciadm tool running on each, for the various commands libvirt needs to run. The only way we are going to avoid breaking this repeatedly is to get serious around documentating the variants we need to support then then testing them. The example output from iscsidam tool then can be used to build a fake iscsiadm tool invoked by the test suites exercising all the new code. I didn't get all of the information you requested (yet). But I did connect a number of different kernels up to an iSCSI target and look at the listing in /sys/bus/scsi/devices/target:channel:id:lun Kernel iscsi-initiator-utils sysfs 2.6.9 4.0.3.0-6 block/sda1, block/sda2 (note no block/sda!) 2.6.18 6.2.0.868-0.7.el5 block:sda 2.6.21 6.2.0.865-0.2.fc8 block:sda 2.6.25 6.2.0.868-0.7.fc9.i386 block/sda Now, we probably don't care about 2.6.9. So that leaves the other few kernels, which all seem to follow either the block:sda or block/sda pattern. So, at least for now, Stefan can test my patch to see if it helps him. I'll try to collect further information, including the output from iscsiadm. Chris Lalancette -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
[libvirt] Re: [Libvir] ISCSI howto, example, etc?
On Tue, 6 May 2008, Daniel P. Berrange wrote: As a simple example... - Create an XML definition for the iSCSI target you want to access, I have created: pool type=iscsi namenetapp/name source host name=172.16.103.200/ device path=iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f/ /source target path/dev/disk/by-path/path /target /pool - Define the pool # virsh pool-define virt.xml xen01 netapp # virsh pool-define virt.xml Pool netapp defined from virt.xml - Activate the pool (this step logs into the server) # virsh pool-start virtimages xen01 netapp # virsh pool-start netapp Pool netapp started - List the LUNs available # virsh vol-list virtimages xen01 netapp # virsh vol-list netapp Name Path - pretty empty But when I look in /dev/disk/by-path: lrwxrwxrwx 1 root root 9 May 14 12:35 ip-172.16.103.200:3260-iscsi-iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f-lun-0 - ../../sdc lrwxrwxrwx 1 root root 9 May 14 12:35 ip-172.16.103.200:3260-iscsi-iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f-lun-1 - ../../sdd lrwxrwxrwx 1 root root 9 May 14 12:35 ip-172.16.103.200:3260-iscsi-iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f-lun-2 - ../../sde Something is present, thus the login was succesfull. How to get this to work with the libvirt storage pool? I have tried to change the target path. But it doesn't give me any output. I'll now look in the C code to see what you are giving as command for this list. Stefan -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Re: [Libvir] ISCSI howto, example, etc?
On Wed, 14 May 2008, Stefan de Konink wrote: I consider my output interesting... it seems the -P option is ignored. (I'm running open-iscsi-2.0.868_rc1) After upgrading to open-iscsi-2.0.869.2 i have a working output with -P 3... but... xen01 skinkie # virsh pool-start netapp libvir: Storage error : Failed to find the sysfs path for 4:0:0:2: No such file or directory error: Failed to start pool netapp That doesn't make me happy. snprintf(sysfs_path, PATH_MAX, /sys/class/iscsi_session/session%s/device/ target%d:%d:%d/%d:%d:%d:%d/block, session, target, channel, id, target, channel, id, lun); xen01 5:0:0:1 # pwd /sys/class/iscsi_session/session5/device/target5:0:0/5:0:0:1 xen01 5:0:0:1 # ls -l total 0 lrwxrwxrwx 1 root root0 May 14 13:14 block:sdd - ../../../../../../block/sdd I guess this is not what makes people happy. (:realdisk) Is this something we can solve or should we call the open-iscsi guys? Stefan -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Re: [Libvir] ISCSI howto, example, etc?
On Wed, 14 May 2008, Stefan de Konink wrote: That doesn't make me happy. I have currently solved it and made it work with the new version, I'll send in a patch. But there are more things that don't make me happy. 1) lun-0 is ignored 2) Rescanning session [sid: 8, target: iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f, portal: 172.16.103.200,3260] ** sde ** sdd Link '/dev/disk/by-path/ip-172.16.103.200:3260-iscsi-iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f-lun-2' does not point to '/dev/sdd', ignoring xen01 libvirt # virsh vol-list netapp Name Path - lun-1 /dev/disk/by-path/ip-172.16.103.200:3260-iscsi-iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f-lun-1 lun-2 /dev/disk/by-path/ip-172.16.103.200:3260-iscsi-iqn.1992-08.com.netapp:sn.118046347:vf.88fa4694-0ba6-11dd-b8a9-00a09807592f-lun-2 So something is still not how I want it to be. Stefan -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Re: [Libvir] ISCSI howto, example, etc?
Stefan de Konink wrote: snprintf(sysfs_path, PATH_MAX, /sys/class/iscsi_session/session%s/device/ target%d:%d:%d/%d:%d:%d:%d/block, session, target, channel, id, target, channel, id, lun); xen01 5:0:0:1 # pwd /sys/class/iscsi_session/session5/device/target5:0:0/5:0:0:1 xen01 5:0:0:1 # ls -l total 0 lrwxrwxrwx 1 root root0 May 14 13:14 block:sdd - ../../../../../../block/sdd Ug. Yeah, that will do it. What kernel version are you running that your sysfs looks like that? It seems we have been shoved into a corner; we can't rely on the output of the iscsiadm tools, since we have already seen those change between versions. And apparently we can't rely on sysfs paths to stay stable either. Either we need some iscsiadm command with a known, stable output that we can call, or we need some sort of iSCSI API that we can call to get this information. Either way, I think we need help from the iSCSI people, because the current position seem to be untenable. Chris Lalancette -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Re: [Libvir] ISCSI howto, example, etc?
On Wed, 14 May 2008, Chris Lalancette wrote: Ug. Yeah, that will do it. What kernel version are you running that your sysfs looks like that? I already send a message to open-iscsi group. Fix is trivial and even more simple. I'm running 2.6.21-xen (gentoo) with open-iscsi-2.0.869.2. Fix: snprintf(sysfs_path, PATH_MAX, /sys/class/iscsi_session/session%s/device/ target%d:%d:%d/%d:%d:%d:%d, session, target, channel, id, target, channel, id, lun); (so remove /block) /* OK, not . or ..; let's see if it is a SCSI device */ if (len 8 block_dirent-d_name[0] == 'b' block_dirent-d_name[1] == 'l' block_dirent-d_name[2] == 'o' block_dirent-d_name[3] == 'c' block_dirent-d_name[4] == 'k' block_dirent-d_name[5] == ':' block_dirent-d_name[6] == 's' block_dirent-d_name[7] == 'd') { /* looks like a scsi device, smells like scsi device; it must be a scsi device */ dev = (char *) calloc(sizeof(char), len - 5); strncpy(dev, (block_dirent-d_name[6]), (len - 6)); I guess that can be come a strncmp. And for sake of memory management an if (dev != NULL) would be good too. It seems we have been shoved into a corner; we can't rely on the output of the iscsiadm tools, since we have already seen those change between versions. And apparently we can't rely on sysfs paths to stay stable either. Either we need some iscsiadm command with a known, stable output that we can call, or we need some sort of iSCSI API that we can call to get this information. Either way, I think we need help from the iSCSI people, because the current position seem to be untenable. I would love to see something like a iSCSI API. The parsing looks really cute, but as you say it is so unreliable. Next to this: /* the 0'th LUN isn't a real LUN, it's just a control LUN; skip it */ I think this is also a very strange idea of LUNs. Stefan -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Re: [Libvir] ISCSI howto, example, etc?
Stefan de Konink wrote: Fix: snprintf(sysfs_path, PATH_MAX, /sys/class/iscsi_session/session%s/device/ target%d:%d:%d/%d:%d:%d:%d, session, target, channel, id, target, channel, id, lun); (so remove /block) /* OK, not . or ..; let's see if it is a SCSI device */ if (len 8 block_dirent-d_name[0] == 'b' block_dirent-d_name[1] == 'l' block_dirent-d_name[2] == 'o' block_dirent-d_name[3] == 'c' block_dirent-d_name[4] == 'k' block_dirent-d_name[5] == ':' block_dirent-d_name[6] == 's' block_dirent-d_name[7] == 'd') { /* looks like a scsi device, smells like scsi device; it must be a scsi device */ dev = (char *) calloc(sizeof(char), len - 5); strncpy(dev, (block_dirent-d_name[6]), (len - 6)); I guess that can be come a strncmp. And for sake of memory management an if (dev != NULL) would be good too. Yes, except the problem with your patch is that it will probably break the kernel I made the fixes for originally, which was either 2.6.23 or 2.6.24 (and which had the /sys/class/iscsi_session/session9/device/target9:0:0:0/ 9:0:0:0/block/sdd type paths). We may be able to come up with a hybrid solution that will work on both cases, though. Chris Lalancette -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Re: [Libvir] ISCSI howto, example, etc?
On Wed, May 14, 2008 at 01:57:13PM +0200, Chris Lalancette wrote: Stefan de Konink wrote: snprintf(sysfs_path, PATH_MAX, /sys/class/iscsi_session/session%s/device/ target%d:%d:%d/%d:%d:%d:%d/block, session, target, channel, id, target, channel, id, lun); xen01 5:0:0:1 # pwd /sys/class/iscsi_session/session5/device/target5:0:0/5:0:0:1 xen01 5:0:0:1 # ls -l total 0 lrwxrwxrwx 1 root root0 May 14 13:14 block:sdd - ../../../../../../block/sdd Ug. Yeah, that will do it. What kernel version are you running that your sysfs looks like that? It seems we have been shoved into a corner; we can't rely on the output of the iscsiadm tools, since we have already seen those change between versions. And apparently we can't rely on sysfs paths to stay stable either. Either we need some iscsiadm command with a known, stable output that we can call, or we need some sort of iSCSI API that we can call to get this information. Either way, I think we need help from the iSCSI people, because the current position seem to be untenable. We have no choice. - No standard iSCSI library exists for us to use - Any iSCSI library would encounter these same problems - IF a library were written it still won't be in any distro we use for a long time So we have no choice but to make our current code work. If someone wants to pursue development of an iSCSI library as a long term solution that's a worthy goal, but for libvirt we need solutions that work today, for existing distros. This comes down to, documenting the variations we have to deal with, figuring out the code to support them, and writing test cases to ensure it works and we don't cause regressions. Dan. -- |: Red Hat, Engineering, Boston -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] Re: [Libvir] ISCSI howto, example, etc?
I think if you can get iscsi_sysfs.c; get_block_dev_from_lun you have the code already to do this all. So basically get openiscsi into a lib with helper functions. ISCSI_LIB_SRCS = util.o io.o auth.o login.o log.o md5.o sha1.o iface.o idbm.o iscsi_sysfs.o To something like libopen-iscsi.so, and link it against libvirt, you have what is required. Stefan -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
[Libvir] ISCSI howto, example, etc?
The documentation on the new storage feature is either a little short, or I totally missed it. Anyway, I am looking for some help on how to use it. If anyone could maybe give me an example xml file that would be incredibly helpful, because based on the format description I can't exactly figure out what information I am supposed to put where (for ISCSI). Thanks! -- $ ./Mark -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [Libvir] ISCSI howto, example, etc?
On Tue, May 06, 2008 at 12:50:25AM -0600, Mark Dehus wrote: The documentation on the new storage feature is either a little short, or I totally missed it. Anyway, I am looking for some help on how to use it. If anyone could maybe give me an example xml file that would be incredibly helpful, because based on the format description I can't exactly figure out what information I am supposed to put where (for ISCSI). The main docs we have are http://libvirt.org/formatstorage.html http://libvirt.org/storage.html They are basically more reference guides that howto/tutorials. Also look at the 'virsh help' output for 'vol-XXX' and 'pool-XXX' commands. As a simple example... - Create an XML definition for the iSCSI target you want to access, # cat virt.xml EOF pool type=iscsi namevirtimages/name source host name=iscsi.example.com/ device path=demo-target/ /source target path/dev/disk/by-path/path /target /pool EOF - Define the pool # virsh pool-define virt.xml - Activate the pool (this step logs into the server) # virsh pool-start virtimages - List the LUNs available # virsh vol-list virtimages - Query info about a LUN # virsh vol-info --pool virtimages lun-3 # virsh vol-dumpxml --pool virtimages lun-3 There's many more pool-XXX and vol-XX commands available Regards, Dan. -- |: Red Hat, Engineering, Boston -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :| -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list