[Bug 1336568] Re: LVMiSCSI driver can't issue direct I/O through tgtd to iscsi volume
This bug was fixed in the package cinder - 1:2014.1.5-0ubuntu2 --- cinder (1:2014.1.5-0ubuntu2) trusty; urgency=medium * Enable iscsi_write_cache option for tgtadm backends (LP: #1336568): - d/p/tgtadmin-iscsi-write-cache-config.patch - Includes backport of change from the juno release for enabling iscsi write cache policy for tgtadm. -- Billy Olsen Mon, 27 Jul 2015 17:35:57 -0700 ** Changed in: cinder (Ubuntu Trusty) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1336568 Title: LVMiSCSI driver can't issue direct I/O through tgtd to iscsi volume To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1336568/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1336568] Re: LVMiSCSI driver can't issue direct I/O through tgtd to iscsi volume
The verification of the Stable Release Update for cinder has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions. ** Changed in: cinder (Ubuntu) Status: Confirmed => Fix Released ** Changed in: cinder (Ubuntu Trusty) Assignee: (unassigned) => Billy Olsen (billy-olsen) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1336568 Title: LVMiSCSI driver can't issue direct I/O through tgtd to iscsi volume To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1336568/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1336568] Re: LVMiSCSI driver can't issue direct I/O through tgtd to iscsi volume
Was able to verify the fix for this bug today. Installed cinder from the trusty-proposed pocket and ran the following tests to confirm: # Test one, ensure default option remains to write-cache on 1. create volume 2. attach iscsi volume to instance 3. Verify generated xml in /var/lib/cinder/volumes/ is generated with write-cache on. # Change the iscsi-write-cache to off and restart cinder volumes 1. Set iscsi_write_cache = off in /etc/cinder/cinder.conf 2. Create lvm volume 3. Attach via iscsi to instance 4. Verify generated xml in /var/lib/cinder/volumes/ is generated with write-cache off ** Tags removed: verification-needed ** Tags added: verification-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1336568 Title: LVMiSCSI driver can't issue direct I/O through tgtd to iscsi volume To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1336568/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1336568] Re: LVMiSCSI driver can't issue direct I/O through tgtd to iscsi volume
Hello Mitsuhiro, or anyone else affected, Accepted cinder into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cinder/1:2014.1.5-0ubuntu2 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance! ** Changed in: cinder (Ubuntu Trusty) Status: New => Fix Committed ** Tags added: verification-needed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1336568 Title: LVMiSCSI driver can't issue direct I/O through tgtd to iscsi volume To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1336568/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1336568] Re: LVMiSCSI driver can't issue direct I/O through tgtd to iscsi volume
** Also affects: cinder (Ubuntu Trusty) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1336568 Title: LVMiSCSI driver can't issue direct I/O through tgtd to iscsi volume To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1336568/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1336568] Re: LVMiSCSI driver can't issue direct I/O through tgtd to iscsi volume
** Branch linked: lp:~billy-olsen/cinder/icehouse-1336568 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1336568 Title: LVMiSCSI driver can't issue direct I/O through tgtd to iscsi volume To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1336568/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1336568] Re: LVMiSCSI driver can't issue direct I/O through tgtd to iscsi volume
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: cinder (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1336568 Title: LVMiSCSI driver can't issue direct I/O through tgtd to iscsi volume To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1336568/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1336568] Re: LVMiSCSI driver can't issue direct I/O through tgtd to iscsi volume
This patch was included in the juno release of cinder and thus is available already in utopic, vivid, etc. It is not available in icehouse/trusty - which this patch is for. ** Description changed: I found a problem which LVMiSCSI driver can't issue direct I/O through tgtd to iscsi volume. In current implementation, qemu-kvm opens device(storage volume) using cache='node'at nova side, however, tgtd opens device without "--bsoflags direct" at cinder side. Therefore, I/O from guest instances are cached at control node even though the compute node issues O_DIRECT I/O to iSCSI volume. As a result, if control node has a crash, cached data will be lost. This causes data lost problem of guest instance. I will propose a fix of this issue. Here are test environment and confirmation results. (1) Control node - Control node has nova, cinder(c-sch, c-api, c-vol), glance, horizon services. - Use LVMiSCSI driver for cinder backend. (2) Compute node has n-cpu and n-net services. [Confirmation at compute node] On a compute node, qemu opens device file using cache='none'. This means instance can issue direct I/O from guest to the device. [root@compute ~]# cat /etc/libvirt/qemu/instance-000b.xml instance-000b 9e1eb5cc-4c40-4023-bca5-a7d1720c6f51 # Open the device without cache. fdd23217-6e95-4aee-a586-6ef174567ba5 Confirm a file descriptor whether the device is opened with O_DIRECT or not at compute node. => qemu Process ID is "24836" [root@compute ~]# ps uax | grep qemu root 11421 0.0 0.0 112672 912 pts/6S+ 17:13 0:00 grep --color=auto qemu qemu 24836 13.8 16.0 4638484 1312668 ? Sl Jun29 462:12 /usr/bin/qemu-system-x86_64 -machine accel=kvm -name instance-000b . => Device file of iscsi cinder volume is "/dev/sde". [root@compute ~]# ls -la /dev/disk/by-path/ . -rw-r--r-- 1 root root 349525333 Jun 27 10:01 ip-10.16.42.67 lrwxrwxrwx 1 root root 9 Jun 30 00:16 ip-10.16.42.67:3260-iscsi-iqn.2010-10.org.openstack:volume-fdd23217-6e95-4aee-a586-6ef174567ba5-lun-1 -> ../../sde => "fd18" is infomation of /dev/sde [root@compute ~]# ls -la /proc/24836/fd total 0 dr-x-- 2 qemu qemu 0 Jun 29 09:40 . dr-xr-xr-x 9 qemu qemu 0 Jun 29 09:31 .. . lrwx-- 1 qemu qemu 64 Jun 29 09:40 18 -> /dev/sde => The flags is "02140002". O_DIRECT flag is "0x4". This flag is raised at compute node side. [root@compute ~]# cat /proc/24836/fdinfo/18 pos:10737418240 flags: 02140002 [Confirmation at control node] Confirm iscsi target status and exported disk. Backing store path is /dev/stack-volumes/volume-fdd23217-6e95-4aee-a586-6ef174567ba5 [mtanino@control ~]$ sudo tgt-admin -s . Target 3: iqn.2010-10.org.openstack:volume-fdd23217-6e95-4aee-a586-6ef174567ba5 System information: Driver: iscsi ... LUN: 1 ... Backing store path: /dev/stack-volumes/volume-fdd23217-6e95-4aee-a586-6ef174567ba5 Backing store flags: Account information: ACL information: ALL =>Condirm device mapper file of the backing store [mtanino@control ~]$ ls -la /dev/disk/by-id/ | grep stack lrwxrwxrwx 1 root root 10 Jun 30 00:16 dm-name-stack--volumes-volume--fdd23217--6e95--4aee--a586--6ef174567ba5 -> ../../dm-0 => tgtd Process ID is "31010" [mtanino@control ~]$ ps aux | grep tgtd root 31010 2.0 0.0 476584 900 ?Ssl Jun30 52:27 /usr/sbin/tgtd -f => "fd11" is infomation of /dev/dm-0 [mtanino@control ~]$ sudo ls -la /proc/31010/fd ... lrwx-- 1 root root 64 Jul 1 16:11 11 -> /dev/dm-0 lrwx-- 1 root root 64 Jul 1 16:11 12 -> /dev/sdb2 ... => The flags is "012". O_DIRECT flag is "0x4". This flag is not raised at control node side. [mtanino@control ~]$ sudo cat /proc/31010/fdinfo/11 pos: 0 flags:012 Regards, Mitsuhiro Tanino + + + [Impact] + + * May see data loss without the ability to use write-through caching +(write-cache off) option instead of write-back (write-cache on) +option for iscsi targets. + + [Test Case] + + * Configure Cinder to use LVMiSCSIDriver + * Create cinder volume (cinder create --display-name foo 1G) + * Attach volume to nova instance (nova volume-attach my-instance ) + + * Observe the write-cache policy specified per cinder volume (found in) +- /var/lib/cinder/volumes/volume- + + * Observe above information (detailed by Mitsuhiro) + + [Regression Potential] + + * Low risk of regression as the feature is enabled through a +configurable option in which default value takes original behavior. ** Tags added: sts ** Patch added: "Patch for trusty-icehouse versio
[Bug 1336568] Re: LVMiSCSI driver can't issue direct I/O through tgtd to iscsi volume
** Also affects: cinder (Ubuntu) Importance: Undecided Status: New ** Changed in: cinder (Ubuntu) Assignee: (unassigned) => Billy Olsen (billy-olsen) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1336568 Title: LVMiSCSI driver can't issue direct I/O through tgtd to iscsi volume To manage notifications about this bug go to: https://bugs.launchpad.net/cinder/+bug/1336568/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
