Re: [ceph-users] Ceph with VMWare / XenServer
Uwe, could you please help me a bit with configuring multipathing on two different storage servers and connecting it to xenserver. I am looking at the multipathing howto and it tells me that for multipathing to work the iscsi querry from the target server should return two paths. However, if you have two separate servers with tgt installed, each one would only return a single path. I've configured two servers (tgt1 and tgt2) with tgt, each pointing to the same rbd image. the iscsi config files are identical. One server is using 192.168.170.200 ip, the second one uses 192.168.171.200. When doing a query, the tgt1, it returns: 192.168.170.200:3260,1 iqn.2014-04.iscsi-ibstorage.arhont.com:xenserver-iscsi-export-10TB-1 and tgt2 returns: 192.168.171.200:3260,1 iqn.2014-04.iscsi-ibstorage.arhont.com:xenserver-iscsi-export-10TB-1 According to the documentation, each server should return both paths, like this: 192.168.170.200:3260,1 iqn.2014-04.iscsi-ibstorage.arhont.com:xenserver-iscsi-export-10TB-1 192.168.171.200:3260,1 iqn.2014-04.iscsi-ibstorage.arhont.com:xenserver-iscsi-export-10TB-1 Is there a manual way of configuring multipathing? Or have I not created the tgt configs correctly? Cheers Andrei - Original Message - From: "Uwe Grohnwaldt" To: ceph-users@lists.ceph.com Sent: Monday, 12 May, 2014 12:57:48 PM Subject: Re: [ceph-users] Ceph with VMWare / XenServer Hi, at the moment we are using tgt with RBD backend compiled from source on Ubuntu 12.04 and 14.04 LTS. We have two machines within two ip-ranges (e.g. 192.168.1.0/24 and 192.168.2.0/24). One machine in 192.168.1.0/24 and one machine in 192.168.2.0/24. The config for tgt is the same on both machines, they export the same rbd. This works well for XenServer. For VMWare you have to disable VAAI to use it with tgt (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033665) If you don't disable it, ESXi becomes very slow and unresponsive. I think the problem is the iSCSI Write Same Support but I haven't tried which of the settings of VAAI is responsible for this behavior. Mit freundlichen Grüßen / Best Regards, -- Consultant Dipl.-Inf. Uwe Grohnwaldt Gutleutstr. 351 60327 Frankfurt a. M. eMail: u...@grohnwaldt.eu Telefon: +49-69-34878906 Mobil: +49-172-3209285 Fax: +49-69-348789069 - Original Message - > From: "Andrei Mikhailovsky" > To: ceph-users@lists.ceph.com > Sent: Montag, 12. Mai 2014 12:00:48 > Subject: [ceph-users] Ceph with VMWare / XenServer > > > > Hello guys, > > I am currently running a ceph cluster for running vms with qemu + > rbd. It works pretty well and provides a good degree of failover. I > am able to run maintenance tasks on the ceph nodes without > interrupting vms IO. > > I would like to do the same with VMWare / XenServer hypervisors, but > I am not really sure how to achieve this. Initially I thought of > using iscsi multipathing, however, as it turns out, multipathing is > more for load balancing and nic/switch failure. It does not allow me > to perform maintenance on the iscsi target without interrupting > service to vms. > > Has anyone done either a PoC or better a production environment where > they've used ceph as a backend storage with vmware / xenserver? The > important element for me is to have the ability of performing > maintenance tasks and resilience to failovers without interrupting > IO to vms. Are there any recommendations or howtos on how this could > be achieved? > > Many thanks > > Andrei > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph with VMWare / XenServer
Uwe, do you mind sharing your storage and xenserver iscsi config files? Also, what is your performance like? Thanks - Original Message - From: "Uwe Grohnwaldt" To: ceph-users@lists.ceph.com Sent: Monday, 12 May, 2014 2:45:43 PM Subject: Re: [ceph-users] Ceph with VMWare / XenServer Hi, yes, we use it in production. I can stop/kill the tgt on one server and XenServer goes to the second one. We enabled multipathing in xenserver. In our setup we haven't multiple ip-ranges so we scan/login the second target on xenserverstartup with iscsiadm in rc.local. Thats based on history - we used Dell Equallogic before ceph came in and there was no need to use multipathing (only LACP-channels). No we enabled multipathing and use tgt, but without diffent ip-ranges. Mit freundlichen Grüßen / Best Regards, -- Consultant Dipl.-Inf. Uwe Grohnwaldt Gutleutstr. 351 60327 Frankfurt a. M. eMail: u...@grohnwaldt.eu Telefon: +49-69-34878906 Mobil: +49-172-3209285 Fax: +49-69-348789069 - Original Message - > From: "Andrei Mikhailovsky" > To: "Uwe Grohnwaldt" > Cc: ceph-users@lists.ceph.com > Sent: Montag, 12. Mai 2014 14:48:58 > Subject: Re: [ceph-users] Ceph with VMWare / XenServer > > > Uwe, thanks for your quick reply. > > Do you run the Xenserver setup on production env and have you tried > to test some failover scenarios to see if the xenserver guest vms > are working during the failover of storage servers? > > Also, how did you set up the xenserver iscsi? Have you used the > multipath option to set up the LUNs? > > Cheers > > > > > - Original Message - > > From: "Uwe Grohnwaldt" > To: ceph-users@lists.ceph.com > Sent: Monday, 12 May, 2014 12:57:48 PM > Subject: Re: [ceph-users] Ceph with VMWare / XenServer > > Hi, > > at the moment we are using tgt with RBD backend compiled from source > on Ubuntu 12.04 and 14.04 LTS. We have two machines within two > ip-ranges (e.g. 192.168.1.0/24 and 192.168.2.0/24). One machine in > 192.168.1.0/24 and one machine in 192.168.2.0/24. The config for tgt > is the same on both machines, they export the same rbd. This works > well for XenServer. > > For VMWare you have to disable VAAI to use it with tgt > (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033665) > > If you don't disable it, ESXi becomes very slow and unresponsive. > > I think the problem is the iSCSI Write Same Support but I haven't > tried which of the settings of VAAI is responsible for this > behavior. > > Mit freundlichen Grüßen / Best Regards, > -- > Consultant > Dipl.-Inf. Uwe Grohnwaldt > Gutleutstr. 351 > 60327 Frankfurt a. M. > > eMail: u...@grohnwaldt.eu > Telefon: +49-69-34878906 > Mobil: +49-172-3209285 > Fax: +49-69-348789069 > > - Original Message - > > From: "Andrei Mikhailovsky" > > To: ceph-users@lists.ceph.com > > Sent: Montag, 12. Mai 2014 12:00:48 > > Subject: [ceph-users] Ceph with VMWare / XenServer > > > > > > > > Hello guys, > > > > I am currently running a ceph cluster for running vms with qemu + > > rbd. It works pretty well and provides a good degree of failover. I > > am able to run maintenance tasks on the ceph nodes without > > interrupting vms IO. > > > > I would like to do the same with VMWare / XenServer hypervisors, > > but > > I am not really sure how to achieve this. Initially I thought of > > using iscsi multipathing, however, as it turns out, multipathing is > > more for load balancing and nic/switch failure. It does not allow > > me > > to perform maintenance on the iscsi target without interrupting > > service to vms. > > > > Has anyone done either a PoC or better a production environment > > where > > they've used ceph as a backend storage with vmware / xenserver? The > > important element for me is to have the ability of performing > > maintenance tasks and resilience to failovers without interrupting > > IO to vms. Are there any recommendations or howtos on how this > > could > > be achieved? > > > > Many thanks > > > > Andrei > > > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph with VMWare / XenServer
Le 12/05/2014 15:45, Uwe Grohnwaldt a écrit : Hi, yes, we use it in production. I can stop/kill the tgt on one server and XenServer goes to the second one. We enabled multipathing in xenserver. In our setup we haven't multiple ip-ranges so we scan/login the second target on xenserverstartup with iscsiadm in rc.local. Thats based on history - we used Dell Equallogic before ceph came in and there was no need to use multipathing (only LACP-channels). No we enabled multipathing and use tgt, but without diffent ip-ranges. So you use multipathing in failover mode, that's certainly why it works without state sharing between the tgtd servers. Still, I think you need to deactivate all sort of caching server side. IO must be committed to ceph when the iSCSI initiator think it is. What are the multipath parameters in XenServer (timeout, retry...) ? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph with VMWare / XenServer
We are using a switchstack of Juniper EX4200 and EX3200. Cisco should work, too. Anotheroption is failover bonding (but multipathing with different ips is better) Mit freundlichen Grüßen / Best Regards, -- Consultant Dipl.-Inf. Uwe Grohnwaldt Gutleutstr. 351 60327 Frankfurt a. M. eMail: u...@grohnwaldt.eu Telefon: +49-69-34878906 Mobil: +49-172-3209285 Fax: +49-69-348789069 - Original Message - > From: "Leen Besselink" > To: ceph-users@lists.ceph.com > Cc: "Uwe Grohnwaldt" > Sent: Montag, 12. Mai 2014 19:01:46 > Subject: Re: [ceph-users] Ceph with VMWare / XenServer > > On Mon, May 12, 2014 at 03:45:43PM +0200, Uwe Grohnwaldt wrote: > > Hi, > > > > yes, we use it in production. I can stop/kill the tgt on one server > > and XenServer goes to the second one. We enabled multipathing in > > xenserver. In our setup we haven't multiple ip-ranges so we > > scan/login the second target on xenserverstartup with iscsiadm in > > rc.local. > > > > Thats based on history - we used Dell Equallogic before ceph came > > in and there was no need to use multipathing (only LACP-channels). > > No we enabled multipathing and use tgt, but without diffent > > ip-ranges. > > > > I assume you connected the machines to the same switch ? As normal > LACP don't work with multiple switches. > > Is that correct ? > > It wasn't that I needed different ip-ranges in my setup, it just > makes it simpler/predictable. > > > Mit freundlichen Grüßen / Best Regards, > > -- > > Consultant > > Dipl.-Inf. Uwe Grohnwaldt > > Gutleutstr. 351 > > 60327 Frankfurt a. M. > > > > eMail: u...@grohnwaldt.eu > > Telefon: +49-69-34878906 > > Mobil: +49-172-3209285 > > Fax: +49-69-348789069 > > > > - Original Message - > > > From: "Andrei Mikhailovsky" > > > To: "Uwe Grohnwaldt" > > > Cc: ceph-users@lists.ceph.com > > > Sent: Montag, 12. Mai 2014 14:48:58 > > > Subject: Re: [ceph-users] Ceph with VMWare / XenServer > > > > > > > > > Uwe, thanks for your quick reply. > > > > > > Do you run the Xenserver setup on production env and have you > > > tried > > > to test some failover scenarios to see if the xenserver guest vms > > > are working during the failover of storage servers? > > > > > > Also, how did you set up the xenserver iscsi? Have you used the > > > multipath option to set up the LUNs? > > > > > > Cheers > > > > > > > > > > > > > > > - Original Message - > > > > > > From: "Uwe Grohnwaldt" > > > To: ceph-users@lists.ceph.com > > > Sent: Monday, 12 May, 2014 12:57:48 PM > > > Subject: Re: [ceph-users] Ceph with VMWare / XenServer > > > > > > Hi, > > > > > > at the moment we are using tgt with RBD backend compiled from > > > source > > > on Ubuntu 12.04 and 14.04 LTS. We have two machines within two > > > ip-ranges (e.g. 192.168.1.0/24 and 192.168.2.0/24). One machine > > > in > > > 192.168.1.0/24 and one machine in 192.168.2.0/24. The config for > > > tgt > > > is the same on both machines, they export the same rbd. This > > > works > > > well for XenServer. > > > > > > For VMWare you have to disable VAAI to use it with tgt > > > (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033665) > > > If you don't disable it, ESXi becomes very slow and unresponsive. > > > > > > I think the problem is the iSCSI Write Same Support but I haven't > > > tried which of the settings of VAAI is responsible for this > > > behavior. > > > > > > Mit freundlichen Grüßen / Best Regards, > > > -- > > > Consultant > > > Dipl.-Inf. Uwe Grohnwaldt > > > Gutleutstr. 351 > > > 60327 Frankfurt a. M. > > > > > > eMail: u...@grohnwaldt.eu > > > Telefon: +49-69-34878906 > > > Mobil: +49-172-3209285 > > > Fax: +49-69-348789069 > > > > > > - Original Message - > > > > From: "Andrei Mikhailovsky" > > > > To: ceph-users@lists.ceph.com > > > > Sent: Montag, 12. Mai 2014 12:00:48 > > > > Subject: [ceph-users] Ceph with VMWare / XenServer > > > > > > > > > > > > > > > > Hello guys, > > > > > > > > I am
Re: [ceph-users] Ceph with VMWare / XenServer
On Mon, May 12, 2014 at 07:01:46PM +0200, Leen Besselink wrote: > On Mon, May 12, 2014 at 03:45:43PM +0200, Uwe Grohnwaldt wrote: > > Hi, > > > > yes, we use it in production. I can stop/kill the tgt on one server and > > XenServer goes to the second one. We enabled multipathing in xenserver. In > > our setup we haven't multiple ip-ranges so we scan/login the second target > > on xenserverstartup with iscsiadm in rc.local. > > > > Thats based on history - we used Dell Equallogic before ceph came in and > > there was no need to use multipathing (only LACP-channels). No we enabled > > multipathing and use tgt, but without diffent ip-ranges. > > > > I assume you connected the machines to the same switch ? As normal LACP don't > work with multiple switches. > > Is that correct ? > Or maybe you used a stack or you have Cisco switches with vPC ? > It wasn't that I needed different ip-ranges in my setup, it just makes it > simpler/predictable. > > > Mit freundlichen Grüßen / Best Regards, > > -- > > Consultant > > Dipl.-Inf. Uwe Grohnwaldt > > Gutleutstr. 351 > > 60327 Frankfurt a. M. > > > > eMail: u...@grohnwaldt.eu > > Telefon: +49-69-34878906 > > Mobil: +49-172-3209285 > > Fax: +49-69-348789069 > > > > - Original Message - > > > From: "Andrei Mikhailovsky" > > > To: "Uwe Grohnwaldt" > > > Cc: ceph-users@lists.ceph.com > > > Sent: Montag, 12. Mai 2014 14:48:58 > > > Subject: Re: [ceph-users] Ceph with VMWare / XenServer > > > > > > > > > Uwe, thanks for your quick reply. > > > > > > Do you run the Xenserver setup on production env and have you tried > > > to test some failover scenarios to see if the xenserver guest vms > > > are working during the failover of storage servers? > > > > > > Also, how did you set up the xenserver iscsi? Have you used the > > > multipath option to set up the LUNs? > > > > > > Cheers > > > > > > > > > > > > > > > - Original Message - > > > > > > From: "Uwe Grohnwaldt" > > > To: ceph-users@lists.ceph.com > > > Sent: Monday, 12 May, 2014 12:57:48 PM > > > Subject: Re: [ceph-users] Ceph with VMWare / XenServer > > > > > > Hi, > > > > > > at the moment we are using tgt with RBD backend compiled from source > > > on Ubuntu 12.04 and 14.04 LTS. We have two machines within two > > > ip-ranges (e.g. 192.168.1.0/24 and 192.168.2.0/24). One machine in > > > 192.168.1.0/24 and one machine in 192.168.2.0/24. The config for tgt > > > is the same on both machines, they export the same rbd. This works > > > well for XenServer. > > > > > > For VMWare you have to disable VAAI to use it with tgt > > > (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033665) > > > If you don't disable it, ESXi becomes very slow and unresponsive. > > > > > > I think the problem is the iSCSI Write Same Support but I haven't > > > tried which of the settings of VAAI is responsible for this > > > behavior. > > > > > > Mit freundlichen Grüßen / Best Regards, > > > -- > > > Consultant > > > Dipl.-Inf. Uwe Grohnwaldt > > > Gutleutstr. 351 > > > 60327 Frankfurt a. M. > > > > > > eMail: u...@grohnwaldt.eu > > > Telefon: +49-69-34878906 > > > Mobil: +49-172-3209285 > > > Fax: +49-69-348789069 > > > > > > - Original Message - > > > > From: "Andrei Mikhailovsky" > > > > To: ceph-users@lists.ceph.com > > > > Sent: Montag, 12. Mai 2014 12:00:48 > > > > Subject: [ceph-users] Ceph with VMWare / XenServer > > > > > > > > > > > > > > > > Hello guys, > > > > > > > > I am currently running a ceph cluster for running vms with qemu + > > > > rbd. It works pretty well and provides a good degree of failover. I > > > > am able to run maintenance tasks on the ceph nodes without > > > > interrupting vms IO. > > > > > > > > I would like to do the same with VMWare / XenServer hypervisors, > > > > but > > > > I am not really sure how to achieve this. Initially I thought of > > > > using iscsi multipathing, however, as it turns out, multipathing is > >
Re: [ceph-users] Ceph with VMWare / XenServer
On Mon, May 12, 2014 at 03:45:43PM +0200, Uwe Grohnwaldt wrote: > Hi, > > yes, we use it in production. I can stop/kill the tgt on one server and > XenServer goes to the second one. We enabled multipathing in xenserver. In > our setup we haven't multiple ip-ranges so we scan/login the second target on > xenserverstartup with iscsiadm in rc.local. > > Thats based on history - we used Dell Equallogic before ceph came in and > there was no need to use multipathing (only LACP-channels). No we enabled > multipathing and use tgt, but without diffent ip-ranges. > I assume you connected the machines to the same switch ? As normal LACP don't work with multiple switches. Is that correct ? It wasn't that I needed different ip-ranges in my setup, it just makes it simpler/predictable. > Mit freundlichen Grüßen / Best Regards, > -- > Consultant > Dipl.-Inf. Uwe Grohnwaldt > Gutleutstr. 351 > 60327 Frankfurt a. M. > > eMail: u...@grohnwaldt.eu > Telefon: +49-69-34878906 > Mobil: +49-172-3209285 > Fax: +49-69-348789069 > > - Original Message - > > From: "Andrei Mikhailovsky" > > To: "Uwe Grohnwaldt" > > Cc: ceph-users@lists.ceph.com > > Sent: Montag, 12. Mai 2014 14:48:58 > > Subject: Re: [ceph-users] Ceph with VMWare / XenServer > > > > > > Uwe, thanks for your quick reply. > > > > Do you run the Xenserver setup on production env and have you tried > > to test some failover scenarios to see if the xenserver guest vms > > are working during the failover of storage servers? > > > > Also, how did you set up the xenserver iscsi? Have you used the > > multipath option to set up the LUNs? > > > > Cheers > > > > > > > > > > - Original Message - > > > > From: "Uwe Grohnwaldt" > > To: ceph-users@lists.ceph.com > > Sent: Monday, 12 May, 2014 12:57:48 PM > > Subject: Re: [ceph-users] Ceph with VMWare / XenServer > > > > Hi, > > > > at the moment we are using tgt with RBD backend compiled from source > > on Ubuntu 12.04 and 14.04 LTS. We have two machines within two > > ip-ranges (e.g. 192.168.1.0/24 and 192.168.2.0/24). One machine in > > 192.168.1.0/24 and one machine in 192.168.2.0/24. The config for tgt > > is the same on both machines, they export the same rbd. This works > > well for XenServer. > > > > For VMWare you have to disable VAAI to use it with tgt > > (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033665) > > If you don't disable it, ESXi becomes very slow and unresponsive. > > > > I think the problem is the iSCSI Write Same Support but I haven't > > tried which of the settings of VAAI is responsible for this > > behavior. > > > > Mit freundlichen Grüßen / Best Regards, > > -- > > Consultant > > Dipl.-Inf. Uwe Grohnwaldt > > Gutleutstr. 351 > > 60327 Frankfurt a. M. > > > > eMail: u...@grohnwaldt.eu > > Telefon: +49-69-34878906 > > Mobil: +49-172-3209285 > > Fax: +49-69-348789069 > > > > - Original Message - > > > From: "Andrei Mikhailovsky" > > > To: ceph-users@lists.ceph.com > > > Sent: Montag, 12. Mai 2014 12:00:48 > > > Subject: [ceph-users] Ceph with VMWare / XenServer > > > > > > > > > > > > Hello guys, > > > > > > I am currently running a ceph cluster for running vms with qemu + > > > rbd. It works pretty well and provides a good degree of failover. I > > > am able to run maintenance tasks on the ceph nodes without > > > interrupting vms IO. > > > > > > I would like to do the same with VMWare / XenServer hypervisors, > > > but > > > I am not really sure how to achieve this. Initially I thought of > > > using iscsi multipathing, however, as it turns out, multipathing is > > > more for load balancing and nic/switch failure. It does not allow > > > me > > > to perform maintenance on the iscsi target without interrupting > > > service to vms. > > > > > > Has anyone done either a PoC or better a production environment > > > where > > > they've used ceph as a backend storage with vmware / xenserver? The > > > important element for me is to have the ability of performing > > > maintenance tasks and resilience to failovers without interrupting > > > IO to vms. Are there any recommendations or howtos on how this > > > could > > > be achieved? > > > > > > Many thanks > > > > > > Andrei > > > > > > > > > ___ > > > ceph-users mailing list > > > ceph-users@lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph with VMWare / XenServer
Hi, yes, we use it in production. I can stop/kill the tgt on one server and XenServer goes to the second one. We enabled multipathing in xenserver. In our setup we haven't multiple ip-ranges so we scan/login the second target on xenserverstartup with iscsiadm in rc.local. Thats based on history - we used Dell Equallogic before ceph came in and there was no need to use multipathing (only LACP-channels). No we enabled multipathing and use tgt, but without diffent ip-ranges. Mit freundlichen Grüßen / Best Regards, -- Consultant Dipl.-Inf. Uwe Grohnwaldt Gutleutstr. 351 60327 Frankfurt a. M. eMail: u...@grohnwaldt.eu Telefon: +49-69-34878906 Mobil: +49-172-3209285 Fax: +49-69-348789069 - Original Message - > From: "Andrei Mikhailovsky" > To: "Uwe Grohnwaldt" > Cc: ceph-users@lists.ceph.com > Sent: Montag, 12. Mai 2014 14:48:58 > Subject: Re: [ceph-users] Ceph with VMWare / XenServer > > > Uwe, thanks for your quick reply. > > Do you run the Xenserver setup on production env and have you tried > to test some failover scenarios to see if the xenserver guest vms > are working during the failover of storage servers? > > Also, how did you set up the xenserver iscsi? Have you used the > multipath option to set up the LUNs? > > Cheers > > > > > - Original Message - > > From: "Uwe Grohnwaldt" > To: ceph-users@lists.ceph.com > Sent: Monday, 12 May, 2014 12:57:48 PM > Subject: Re: [ceph-users] Ceph with VMWare / XenServer > > Hi, > > at the moment we are using tgt with RBD backend compiled from source > on Ubuntu 12.04 and 14.04 LTS. We have two machines within two > ip-ranges (e.g. 192.168.1.0/24 and 192.168.2.0/24). One machine in > 192.168.1.0/24 and one machine in 192.168.2.0/24. The config for tgt > is the same on both machines, they export the same rbd. This works > well for XenServer. > > For VMWare you have to disable VAAI to use it with tgt > (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033665) > If you don't disable it, ESXi becomes very slow and unresponsive. > > I think the problem is the iSCSI Write Same Support but I haven't > tried which of the settings of VAAI is responsible for this > behavior. > > Mit freundlichen Grüßen / Best Regards, > -- > Consultant > Dipl.-Inf. Uwe Grohnwaldt > Gutleutstr. 351 > 60327 Frankfurt a. M. > > eMail: u...@grohnwaldt.eu > Telefon: +49-69-34878906 > Mobil: +49-172-3209285 > Fax: +49-69-348789069 > > - Original Message - > > From: "Andrei Mikhailovsky" > > To: ceph-users@lists.ceph.com > > Sent: Montag, 12. Mai 2014 12:00:48 > > Subject: [ceph-users] Ceph with VMWare / XenServer > > > > > > > > Hello guys, > > > > I am currently running a ceph cluster for running vms with qemu + > > rbd. It works pretty well and provides a good degree of failover. I > > am able to run maintenance tasks on the ceph nodes without > > interrupting vms IO. > > > > I would like to do the same with VMWare / XenServer hypervisors, > > but > > I am not really sure how to achieve this. Initially I thought of > > using iscsi multipathing, however, as it turns out, multipathing is > > more for load balancing and nic/switch failure. It does not allow > > me > > to perform maintenance on the iscsi target without interrupting > > service to vms. > > > > Has anyone done either a PoC or better a production environment > > where > > they've used ceph as a backend storage with vmware / xenserver? The > > important element for me is to have the ability of performing > > maintenance tasks and resilience to failovers without interrupting > > IO to vms. Are there any recommendations or howtos on how this > > could > > be achieved? > > > > Many thanks > > > > Andrei > > > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph with VMWare / XenServer
Hello Andrei, I'm trying to accomplish the same thing with VMWare. So far I'm still doing lab testing, but we've gotten as far as simulating a production workload. Forgive the lengthy reply, I happen to be sitting on an airplane . My existing solution is using NFS servers running in ESXi VMs. Each VM serves one or two large (2-4 TB) rbd images. These images are for vmdk storage as well as oracle RAC disks. I tested using multiple NFS servers serving a single rbd, but kept on seeing xfs corruption (which was recoverable with xfs_repair). I initially blamed ceph, but eventually realized that the problem is actually with xfs; well in fact, the problem was with my configuration. It is generally a very bad idea to write to the same xfs file system from two separate computers, whether it is to a ceph rbd or to a physical disk in a shared disk array. What would be required would be a way to synchronize writes between the servers mounting the rbd. There are protocols available to do this, but all of them would introduce more latency, which I'm already struggling to control. My environment is all Cisco UCS hardware. C240 rack mount servers for OSDs and B200 blade servers for VMWare ESXi. The entire network is 10Gb or better. After carefully examining my nfs servers (which are VMs running in ESXi on local storage), I found that I had a tremendous amount of kernel IO. This was because of the high volume of TCP packets it had to constantly process for BOTH the NFS traffic and the ceph traffic. One thing that helped was to enable jumbo frames on every device in the path from ESXi to the OSDs. This is not as simple as it sounds. In ESXi, the vmk port and the vSwitch the vmk is on must have the mtu set to 9000. In the switches, the VLANs and the interfaces need to have the mtu set to 9128 (don't forget about vlan tagging overhead). In the UCSM (Cisco GUI for configuring the Blades and networking), all the vnics and the qos policies must be set to 9000. The Linux interfaces in the nfs servers, mons, and osds all needed to be set to 9000 as well. My kernel io was still high, so I just gave the NFS VM more vCPUs (8 vCPUs, 8 GB RAM). This helped as well. With that all in place, my lab environment is doing a sustained 200 iops bursting up to 500 iops (from VMWare's perspective) on one NFS server VM. The IO is mostly small writes. My lab cluster just has 11 osds in a single node. I have 3x replication as well, so the cluster is actually doing more like 600 - 1400 iops. The osds have an LSI 2208 controller (2GB cache) with each disk in separate single disk RAID1 virtual drives (necessary to take advantage of the write back cache). The OSDs have no separate journal; which means the disks are actually writing at 1200 - 2800 iops (journal + data). Not bad for one node with 11x 7k disks. I still have high latency (though it is much better than before enabling jumbo frames). VMWare shows between 10,000 microseconds and 200,000 microseconds of latency. That is acceptable for this application. IO is mostly asynchronous: alarming/logging writes, database updates. I don't notice the latency on the VMs running in the ceph-NFS datastore. I believe the latency is actually from the osd node being pretty much maxed out. I have 4 more osd servers on order to hopefully smooth out the latency spikes. One huge problem with the NFS server gateway approach is that you have many layers of file systems that are introduced in each OS. My current solution's file system stack looks like this: ext4 - VMs file systems VMFS - ESXi NFS - between ESXi and nfs server XFS - NFS server to mounted rbd disk Rados - NFS server ceph kernel client to OSDs XFS - OSDs to local file system Yuck! Four journaling file systems to write through: VMFS, XFS, OSD, XFS. Clearly the best approach would be for the VMs to directly access the ceph cluster: ext4 - VMs file systems Rados - VM ceph kernel client to OSDs XFS - OSDs to local file system Due to the packaging/deployment procedure of my application (and the ancient RHEL 5 kernel), that won't be possible any time soon. The application will be migrated to openstack, off of VMWare, first. Since I'm using UCS hardware, there is native FCoE built in (with FC frame offload and I can even boot off of FCoE); I am going to build a pair of fiber channel gateways to replace the NFS server. The the filesystem stack will look like this: ext4 - VMs file systems VMFS - ESXi FC - between UCS vHBA and FC Target Rados - FC target via LIO, ceph kernel client to OSDs XFS - OSDs to local file system I had some issues with getting a B200 blade to work in FC target mode (it was only designed to be an initiator), so I'll have to use a C240 in independent mode connected to a nexus 5k switch. As an alternative (while I wait for my new osd nodes and nexus switches to arrive), I was interested in trying tgt with fcoe. I've seen some negative performance reports due to using userland ceph client vs kernel client. More imp
Re: [ceph-users] Ceph with VMWare / XenServer
Uwe, thanks for your quick reply. Do you run the Xenserver setup on production env and have you tried to test some failover scenarios to see if the xenserver guest vms are working during the failover of storage servers? Also, how did you set up the xenserver iscsi? Have you used the multipath option to set up the LUNs? Cheers - Original Message - From: "Uwe Grohnwaldt" To: ceph-users@lists.ceph.com Sent: Monday, 12 May, 2014 12:57:48 PM Subject: Re: [ceph-users] Ceph with VMWare / XenServer Hi, at the moment we are using tgt with RBD backend compiled from source on Ubuntu 12.04 and 14.04 LTS. We have two machines within two ip-ranges (e.g. 192.168.1.0/24 and 192.168.2.0/24). One machine in 192.168.1.0/24 and one machine in 192.168.2.0/24. The config for tgt is the same on both machines, they export the same rbd. This works well for XenServer. For VMWare you have to disable VAAI to use it with tgt (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033665) If you don't disable it, ESXi becomes very slow and unresponsive. I think the problem is the iSCSI Write Same Support but I haven't tried which of the settings of VAAI is responsible for this behavior. Mit freundlichen Grüßen / Best Regards, -- Consultant Dipl.-Inf. Uwe Grohnwaldt Gutleutstr. 351 60327 Frankfurt a. M. eMail: u...@grohnwaldt.eu Telefon: +49-69-34878906 Mobil: +49-172-3209285 Fax: +49-69-348789069 - Original Message - > From: "Andrei Mikhailovsky" > To: ceph-users@lists.ceph.com > Sent: Montag, 12. Mai 2014 12:00:48 > Subject: [ceph-users] Ceph with VMWare / XenServer > > > > Hello guys, > > I am currently running a ceph cluster for running vms with qemu + > rbd. It works pretty well and provides a good degree of failover. I > am able to run maintenance tasks on the ceph nodes without > interrupting vms IO. > > I would like to do the same with VMWare / XenServer hypervisors, but > I am not really sure how to achieve this. Initially I thought of > using iscsi multipathing, however, as it turns out, multipathing is > more for load balancing and nic/switch failure. It does not allow me > to perform maintenance on the iscsi target without interrupting > service to vms. > > Has anyone done either a PoC or better a production environment where > they've used ceph as a backend storage with vmware / xenserver? The > important element for me is to have the ability of performing > maintenance tasks and resilience to failovers without interrupting > IO to vms. Are there any recommendations or howtos on how this could > be achieved? > > Many thanks > > Andrei > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph with VMWare / XenServer
Hi, at the moment we are using tgt with RBD backend compiled from source on Ubuntu 12.04 and 14.04 LTS. We have two machines within two ip-ranges (e.g. 192.168.1.0/24 and 192.168.2.0/24). One machine in 192.168.1.0/24 and one machine in 192.168.2.0/24. The config for tgt is the same on both machines, they export the same rbd. This works well for XenServer. For VMWare you have to disable VAAI to use it with tgt (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033665) If you don't disable it, ESXi becomes very slow and unresponsive. I think the problem is the iSCSI Write Same Support but I haven't tried which of the settings of VAAI is responsible for this behavior. Mit freundlichen Grüßen / Best Regards, -- Consultant Dipl.-Inf. Uwe Grohnwaldt Gutleutstr. 351 60327 Frankfurt a. M. eMail: u...@grohnwaldt.eu Telefon: +49-69-34878906 Mobil: +49-172-3209285 Fax: +49-69-348789069 - Original Message - > From: "Andrei Mikhailovsky" > To: ceph-users@lists.ceph.com > Sent: Montag, 12. Mai 2014 12:00:48 > Subject: [ceph-users] Ceph with VMWare / XenServer > > > > Hello guys, > > I am currently running a ceph cluster for running vms with qemu + > rbd. It works pretty well and provides a good degree of failover. I > am able to run maintenance tasks on the ceph nodes without > interrupting vms IO. > > I would like to do the same with VMWare / XenServer hypervisors, but > I am not really sure how to achieve this. Initially I thought of > using iscsi multipathing, however, as it turns out, multipathing is > more for load balancing and nic/switch failure. It does not allow me > to perform maintenance on the iscsi target without interrupting > service to vms. > > Has anyone done either a PoC or better a production environment where > they've used ceph as a backend storage with vmware / xenserver? The > important element for me is to have the ability of performing > maintenance tasks and resilience to failovers without interrupting > IO to vms. Are there any recommendations or howtos on how this could > be achieved? > > Many thanks > > Andrei > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph with VMWare / XenServer
Hello guys, I am currently running a ceph cluster for running vms with qemu + rbd. It works pretty well and provides a good degree of failover. I am able to run maintenance tasks on the ceph nodes without interrupting vms IO. I would like to do the same with VMWare / XenServer hypervisors, but I am not really sure how to achieve this. Initially I thought of using iscsi multipathing, however, as it turns out, multipathing is more for load balancing and nic/switch failure. It does not allow me to perform maintenance on the iscsi target without interrupting service to vms. Has anyone done either a PoC or better a production environment where they've used ceph as a backend storage with vmware / xenserver? The important element for me is to have the ability of performing maintenance tasks and resilience to failovers without interrupting IO to vms. Are there any recommendations or howtos on how this could be achieved? Many thanks Andrei ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
