Re: [Openstack] about block storage
On a side note, I would love to hear some experiences with GlusterFS as well. We will move to Havana soon and I need to plan the new infrastructure... .a. On Fri, Nov 15, 2013 at 8:45 PM, Vinay Venkataraghavan wrote: > Razique, > > Can you share the type of work loads you ran on Ceph and some performance > numbers. I considered Ceph for a large private cloud that we were building > but never got to it. I ended up using iscsi and nfs which were great. > > I would love to hear more about Ceph performance numbers especially in the > face of replication etc. > > Thanks, > - V > > > On Wed, Nov 13, 2013 at 7:09 PM, Razique Mahroua > wrote: >> >> Hi Dnsbed, >> >> #1: I'd say it depends on the network infrastructure. Maybe you have a >> diagram? >> After I've used Ceph in production for clients, I can definitely say it's >> stable and production-ready (not the CephFS though!) >> >> >> #2: What's the implementation design? >> >> Regards, >> Razique >> >> >> On 13 Nov 2013, at 18:54, Dnsbed Ops wrote: >> >>> Hi, >>> >>> We are deploying openstack for our private cloud apps. >>> For the block storage I have two questions that, >>> >>> #1, is ceph stable enough for product environment as the backend storage >>> of cinder? >>> >>> #2, when using block storage, the disk IO is slow, our apps primarily use >>> it for saving logs. How to improve this? >>> >>> Thanks. >>> >>> >>> ___ >>> Mailing list: >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>> Post to : openstack@lists.openstack.org >>> Unsubscribe : >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> >> >> ___ >> Mailing list: >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> Post to : openstack@lists.openstack.org >> Unsubscribe : >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > > > ___ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > -- antonio.s.mess...@gmail.com antonio.mess...@uzh.ch +41 (0)44 635 42 22 GC3: Grid Computing Competence Center http://www.gc3.uzh.ch/ University of Zurich Winterthurerstrasse 190 CH-8057 Zurich Switzerland ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] about block storage
iscsi is sufficiently performant. I have done extensive testing between iscsi and nfs etc for VM storage. iscsi is very very fast. Ofcourse, the caveat is how your network is also configured. You should definitely perform all the tests that have been recommended in this thread to determine where the bottleneck could be. On another note, I would love to hear your experiences using Ceph. I've heard that its great in theory but in practice its really not that performant. Any performance numbers would be great. Thanks, - V On Fri, Nov 15, 2013 at 3:37 AM, Antonio Messina < antonio.s.mess...@gmail.com> wrote: > I would test the effective network bandwith with iperf. > > Then I would test the speed of the cinder backend by mounting the > volume locally and running, for instance, iozone. > > Finally, I would test the performance of the iSCSI backend by mounting > the volume on the compute node (or another node) via iSCSI and running > iozone on top of it. > > But I have to admint I don't have much experience with iSCSI :) > > .a. > > On Fri, Nov 15, 2013 at 3:30 AM, Jitendra Kumar Bhaskar > wrote: > > Please make sure from block storage to switch and from switch to compute > > node connectivity is in GBs. Means all ports should be GBs. > > > > Regards > > Jitendra Bhaskar > > > > > > > > > > > > > > On Fri, Nov 15, 2013 at 6:20 AM, Dnsbed Ops wrote: > >> > >> They are 1000Mb links. > >> > >> > >> On 2013-11-15 1:40, Razique Mahroua wrote: > >>> > >>> Hi, > >>> > >>> What is the network link between both? > >>> > >>> On 13 Nov 2013, at 19:38, Dnsbed Ops wrote: > >>> > Hi, > > The design is pretty simple. > We run nova-compute for VMs management, for example, a server with > 128GB memory, 12 cores CPU, 300GB SAS (RAID1) disks, to create 20 VMs. > And, we run cinder as the separated storage service to provide block > storage for the VMs. For example, each VM gets a block storage with > 100GB. > > More details, > > OS: ubuntu 12.04 > Hypervisor: KVM > Networking: nova-network for FlatDHCP,multi-host > glance backend: file > cinder backend: ceph > live migration: no need > object storage: no need > > Thanks. > > On 2013-11-14 11:09, Razique Mahroua wrote: > > > > #2: What's the implementation design? > >>> > >>> > >> > >> ___ > >> Mailing list: > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > >> Post to : openstack@lists.openstack.org > >> Unsubscribe : > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > > > > > > > ___ > > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > Post to : openstack@lists.openstack.org > > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > > > > > -- > antonio.s.mess...@gmail.com > antonio.mess...@uzh.ch +41 (0)44 635 42 22 > GC3: Grid Computing Competence Center http://www.gc3.uzh.ch/ > University of Zurich > Winterthurerstrasse 190 > CH-8057 Zurich Switzerland > > ___ > Mailing list: > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] about block storage
Razique, Can you share the type of work loads you ran on Ceph and some performance numbers. I considered Ceph for a large private cloud that we were building but never got to it. I ended up using iscsi and nfs which were great. I would love to hear more about Ceph performance numbers especially in the face of replication etc. Thanks, - V On Wed, Nov 13, 2013 at 7:09 PM, Razique Mahroua wrote: > Hi Dnsbed, > > #1: I'd say it depends on the network infrastructure. Maybe you have a > diagram? > After I've used Ceph in production for clients, I can definitely say it's > stable and production-ready (not the CephFS though!) > > #2: What's the implementation design? > > Regards, > Razique > > > On 13 Nov 2013, at 18:54, Dnsbed Ops wrote: > > Hi, >> >> We are deploying openstack for our private cloud apps. >> For the block storage I have two questions that, >> >> #1, is ceph stable enough for product environment as the backend storage >> of cinder? >> >> #2, when using block storage, the disk IO is slow, our apps primarily use >> it for saving logs. How to improve this? >> >> Thanks. >> >> ___ >> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/ >> openstack >> Post to : openstack@lists.openstack.org >> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/ >> openstack >> > > ___ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > Post to : openstack@lists.openstack.org > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] about block storage
I would test the effective network bandwith with iperf. Then I would test the speed of the cinder backend by mounting the volume locally and running, for instance, iozone. Finally, I would test the performance of the iSCSI backend by mounting the volume on the compute node (or another node) via iSCSI and running iozone on top of it. But I have to admint I don't have much experience with iSCSI :) .a. On Fri, Nov 15, 2013 at 3:30 AM, Jitendra Kumar Bhaskar wrote: > Please make sure from block storage to switch and from switch to compute > node connectivity is in GBs. Means all ports should be GBs. > > Regards > Jitendra Bhaskar > > > > > > > On Fri, Nov 15, 2013 at 6:20 AM, Dnsbed Ops wrote: >> >> They are 1000Mb links. >> >> >> On 2013-11-15 1:40, Razique Mahroua wrote: >>> >>> Hi, >>> >>> What is the network link between both? >>> >>> On 13 Nov 2013, at 19:38, Dnsbed Ops wrote: >>> Hi, The design is pretty simple. We run nova-compute for VMs management, for example, a server with 128GB memory, 12 cores CPU, 300GB SAS (RAID1) disks, to create 20 VMs. And, we run cinder as the separated storage service to provide block storage for the VMs. For example, each VM gets a block storage with 100GB. More details, OS: ubuntu 12.04 Hypervisor: KVM Networking: nova-network for FlatDHCP,multi-host glance backend: file cinder backend: ceph live migration: no need object storage: no need Thanks. On 2013-11-14 11:09, Razique Mahroua wrote: > > #2: What's the implementation design? >>> >>> >> >> ___ >> Mailing list: >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >> Post to : openstack@lists.openstack.org >> Unsubscribe : >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > > > > ___ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > Post to : openstack@lists.openstack.org > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack > -- antonio.s.mess...@gmail.com antonio.mess...@uzh.ch +41 (0)44 635 42 22 GC3: Grid Computing Competence Center http://www.gc3.uzh.ch/ University of Zurich Winterthurerstrasse 190 CH-8057 Zurich Switzerland ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] about block storage
ok, I'd try to bench the I/O locally, then through the network, on the block device,and see when the big difference comes from. - Razique On 14 Nov 2013, at 18:30, Jitendra Kumar Bhaskar wrote: > Please make sure from block storage to switch and from switch to compute > node connectivity is in GBs. Means all ports should be GBs. > > Regards > *Jitendra Bhaskar* > > > > > > > On Fri, Nov 15, 2013 at 6:20 AM, Dnsbed Ops wrote: > >> They are 1000Mb links. >> >> >> On 2013-11-15 1:40, Razique Mahroua wrote: >> >>> Hi, >>> >>> What is the network link between both? >>> >>> On 13 Nov 2013, at 19:38, Dnsbed Ops wrote: >>> >>> Hi, The design is pretty simple. We run nova-compute for VMs management, for example, a server with 128GB memory, 12 cores CPU, 300GB SAS (RAID1) disks, to create 20 VMs. And, we run cinder as the separated storage service to provide block storage for the VMs. For example, each VM gets a block storage with 100GB. More details, OS: ubuntu 12.04 Hypervisor: KVM Networking: nova-network for FlatDHCP,multi-host glance backend: file cinder backend: ceph live migration: no need object storage: no need Thanks. On 2013-11-14 11:09, Razique Mahroua wrote: > #2: What's the implementation design? > >>> >> ___ >> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/ >> openstack >> Post to : openstack@lists.openstack.org >> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/ >> openstack >> ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] about block storage
Please make sure from block storage to switch and from switch to compute node connectivity is in GBs. Means all ports should be GBs. Regards *Jitendra Bhaskar* On Fri, Nov 15, 2013 at 6:20 AM, Dnsbed Ops wrote: > They are 1000Mb links. > > > On 2013-11-15 1:40, Razique Mahroua wrote: > >> Hi, >> >> What is the network link between both? >> >> On 13 Nov 2013, at 19:38, Dnsbed Ops wrote: >> >> Hi, >>> >>> The design is pretty simple. >>> We run nova-compute for VMs management, for example, a server with >>> 128GB memory, 12 cores CPU, 300GB SAS (RAID1) disks, to create 20 VMs. >>> And, we run cinder as the separated storage service to provide block >>> storage for the VMs. For example, each VM gets a block storage with >>> 100GB. >>> >>> More details, >>> >>> OS: ubuntu 12.04 >>> Hypervisor: KVM >>> Networking: nova-network for FlatDHCP,multi-host >>> glance backend: file >>> cinder backend: ceph >>> live migration: no need >>> object storage: no need >>> >>> Thanks. >>> >>> On 2013-11-14 11:09, Razique Mahroua wrote: >>> #2: What's the implementation design? >>> >> > ___ > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > Post to : openstack@lists.openstack.org > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/ > openstack > ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] about block storage
They are 1000Mb links. On 2013-11-15 1:40, Razique Mahroua wrote: Hi, What is the network link between both? On 13 Nov 2013, at 19:38, Dnsbed Ops wrote: Hi, The design is pretty simple. We run nova-compute for VMs management, for example, a server with 128GB memory, 12 cores CPU, 300GB SAS (RAID1) disks, to create 20 VMs. And, we run cinder as the separated storage service to provide block storage for the VMs. For example, each VM gets a block storage with 100GB. More details, OS: ubuntu 12.04 Hypervisor: KVM Networking: nova-network for FlatDHCP,multi-host glance backend: file cinder backend: ceph live migration: no need object storage: no need Thanks. On 2013-11-14 11:09, Razique Mahroua wrote: #2: What's the implementation design? ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] about block storage
Hi, What is the network link between both? On 13 Nov 2013, at 19:38, Dnsbed Ops wrote: Hi, The design is pretty simple. We run nova-compute for VMs management, for example, a server with 128GB memory, 12 cores CPU, 300GB SAS (RAID1) disks, to create 20 VMs. And, we run cinder as the separated storage service to provide block storage for the VMs. For example, each VM gets a block storage with 100GB. More details, OS: ubuntu 12.04 Hypervisor: KVM Networking: nova-network for FlatDHCP,multi-host glance backend: file cinder backend: ceph live migration: no need object storage: no need Thanks. On 2013-11-14 11:09, Razique Mahroua wrote: #2: What's the implementation design? ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] about block storage
Hi, The design is pretty simple. We run nova-compute for VMs management, for example, a server with 128GB memory, 12 cores CPU, 300GB SAS (RAID1) disks, to create 20 VMs. And, we run cinder as the separated storage service to provide block storage for the VMs. For example, each VM gets a block storage with 100GB. More details, OS: ubuntu 12.04 Hypervisor: KVM Networking: nova-network for FlatDHCP,multi-host glance backend: file cinder backend: ceph live migration: no need object storage: no need Thanks. On 2013-11-14 11:09, Razique Mahroua wrote: #2: What's the implementation design? ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Re: [Openstack] about block storage
Hi Dnsbed, #1: I'd say it depends on the network infrastructure. Maybe you have a diagram? After I've used Ceph in production for clients, I can definitely say it's stable and production-ready (not the CephFS though!) #2: What's the implementation design? Regards, Razique On 13 Nov 2013, at 18:54, Dnsbed Ops wrote: Hi, We are deploying openstack for our private cloud apps. For the block storage I have two questions that, #1, is ceph stable enough for product environment as the backend storage of cinder? #2, when using block storage, the disk IO is slow, our apps primarily use it for saving logs. How to improve this? Thanks. ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack ___ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
