Re: [ceph-users] Designing a cluster with ceph and benchmark (ceph vs ext4)
Thank you Alexandre, I'm setting up a new test environment with network 10GB and updated version of Ceph to a new benchmark. []'s -- Thiago Henrique www.adminlinux.com.br Em 01-06-2014 05:23, Alexandre DERUMIER escreveu: hi! See design here: http://adminlinux.com.br/cluster_design.txt # dpkg -l |grep ceph ii ceph 0.41-1ubuntu2.1 distributed storage ii ceph-common0.41-1ubuntu2.1 common utilities to mount and interact with a ceph filesystem ii ceph-deploy1.3.1-1precise Ceph-deploy is an easy to use configuration tool ii ceph-fs-common 0.41-1ubuntu2.1 common utilities to mount and interact with a ceph filesystem ii ceph-mds 0.41-1ubuntu2.1 distributed filesystem service This is prehistoric ceph version ;) you should use repository from ceph.com http://ceph.com/docs/master/install/get-packages/#getting-packages - Mail original - De: Listas@Adminlinux lis...@adminlinux.com.br À: ceph-users@lists.ceph.com Envoyé: Vendredi 23 Mai 2014 20:31:35 Objet: [ceph-users] Designing a cluster with ceph and benchmark (ceph vs ext4) Hi ! I have failover clusters for some aplications. Generally with 2 members configured with Ubuntu + Drbd + Ext4. For example, my IMAP cluster works fine with ~ 50k email accounts and my HTTP cluster hosts ~2k sites. See design here: http://adminlinux.com.br/cluster_design.txt I would like to provide load balancing instead of just failover. So, I would like to use a distributed architecture of the filesystem. As we know, Ext4 isn't a distributed filesystem. So wish to use Ceph in my clusters. Any suggestions for design of the cluster with Ubuntu+Ceph? I built a simple cluster of 2 servers to test simultaneous reading and writing with Ceph. My conf: http://adminlinux.com.br/ceph_conf.txt But in my simultaneous benchmarks found errors in reading and writing. I ran iozone -t 5 -r 4k -s 2m simultaneously on both servers in the cluster. The performance was poor and had errors like this: Error in file: Found ?0? Expecting ?6d6d6d6d6d6d6d6d? addr b660 Error in file: Position 1060864 Record # 259 Record size 4 kb where b660 loop 0 Performance graphs of benchmark: http://adminlinux.com.br/ceph_bench.html Can you help me find what I did wrong? Thanks ! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Designing a cluster with ceph and benchmark (ceph vs ext4)
Hi Pieter, At the time our cluster environment was Ubuntu 10.04 + Kernel-2.6.32 + ocfs2-tools-1.4.3. Later we did the upgrade to Ubuntu 10.10 + Kernel-2.6.35 + ocfs2-tools-1.6.4. We tried to use OCFS2 under production in 2010, but were forced to migrate to a failover design cluster with Ext4. This is the bug that affected us: https://oss.oracle.com/bugzilla/show_bug.cgi?id=1297 Even today, its status is NEW. Thanks! -- Thiago Henrique www.adminlinux.com.br Em 26-05-2014 18:39, Smart Weblications GmbH - Florian Wiessner escreveu: Am 26.05.2014 15:52, schrieb Listas@Adminlinux: Thanks Pieter! I tried using OCFS2 over DRBD, but was not satisfied. I was being affected by various bugs in OCFS2. But Oracle was not committed to solving them. When did you try it? We use such a setup with ocfs2 ontop of rbd with 3.10.40 but also hit bugs with earlier kernel versions. Which Bugs did you hit? I noticed some ocfs2 changes in changelog between 3.10.20 and 3.10.40... I also did online resize of rbd image and then online resize of ocfs2 without problems. Em 24-05-2014 09:14, Pieter Koorts escreveu: If looking for a DRBD alternative and not wanting to use CephFS is it not possible to just use something like OCFS2 or GFS on top of a RDB block device and all worker nodes accessing it via GFS or OCFS2 (obviously with write-through mode)? Would this method not present some advantages over DRBD? DRBD has its uses and will never go away but it does have limited scalability in the general sense. Hi ! I have failover clusters for some aplications. Generally with 2 members configured with Ubuntu + Drbd + Ext4. For example, my IMAP cluster works fine with ~ 50k email accounts and my HTTP cluster hosts ~2k sites. My mailbox servers are also multiple DRBD based cluster pairs. For performance in fully redundant storage there is isn't anything better (in the OSS, generic hardware section at least). See design here:http://adminlinux.com.br/cluster_design.txt I would like to provide load balancing instead of just failover. So, I would like to use a distributed architecture of the filesystem. As we know, Ext4 isn't a distributed filesystem. So wish to use Ceph in my clusters. You will find that all cluster/distributed filesystems have severe performance shortcomings when compared to something like Ext4. On top of that, CephFS isn't ready for production as the MDS isn't HA. A potential middle way might be to use Ceph/RBD volumes formatted in Ext4. That doesn't give you shared access, but it will allow you to separate storage and compute nodes, so when one compute node becomes busy, mount that volume from a more powerful compute node instead. That all said, I can't see any way and reason to replace my mailbox DRBD clusters with Ceph in the foreseeable future. To get similar performance/reliability to DRBD I would have to spend 3-4 times the money. Where Ceph/RBD works well is situations where you can't fit the compute needs into a storage node (as required with DRBD) and where you want to access things from multiple compute nodes, primarily for migration purposes. In short, as a shared storage for VMs. Any suggestions for design of the cluster with Ubuntu+Ceph? I built a simple cluster of 2 servers to test simultaneous reading and writing with Ceph. My conf: http://adminlinux.com.br/ceph_conf.txt Again, CephFS isn't ready for production, but other than that I know very little about it as I don't use it. However your version of Ceph is severely outdated, you really should be looking at something more recent to rule out you're experience long fixed bugs. The same goes for your entire setup and kernel. Also Ceph only starts to perform decently with many OSDs (disks) and the journals on SSDs instead of being on the same disk. Think DRBD AL metadata-internal, but with MUCH more impact. Regards, Christian But in my simultaneous benchmarks found errors in reading and writing. I ran iozone -t 5 -r 4k -s 2m simultaneously on both servers in the cluster. The performance was poor and had errors like this: Error in file: Found ?0? Expecting ?6d6d6d6d6d6d6d6d? addr b660 Error in file: Position 1060864 Record # 259 Record size 4 kb where b660 loop 0 Performance graphs of benchmark: http://adminlinux.com.br/ceph_bench.html Can you help me find what I did wrong? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Designing a cluster with ceph and benchmark (ceph vs ext4)
Hi ! I have failover clusters for some aplications. Generally with 2 members configured with Ubuntu + Drbd + Ext4. For example, my IMAP cluster works fine with ~ 50k email accounts and my HTTP cluster hosts ~2k sites. See design here: http://adminlinux.com.br/cluster_design.txt I would like to provide load balancing instead of just failover. So, I would like to use a distributed architecture of the filesystem. As we know, Ext4 isn't a distributed filesystem. So wish to use Ceph in my clusters. Any suggestions for design of the cluster with Ubuntu+Ceph? I built a simple cluster of 2 servers to test simultaneous reading and writing with Ceph. My conf: http://adminlinux.com.br/ceph_conf.txt But in my simultaneous benchmarks found errors in reading and writing. I ran iozone -t 5 -r 4k -s 2m simultaneously on both servers in the cluster. The performance was poor and had errors like this: Error in file: Found ?0? Expecting ?6d6d6d6d6d6d6d6d? addr b660 Error in file: Position 1060864 Record # 259 Record size 4 kb where b660 loop 0 Performance graphs of benchmark: http://adminlinux.com.br/ceph_bench.html Can you help me find what I did wrong? Thanks ! -- Thiago Henrique www.adminlinux.com.br ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Designing a cluster with ceph and benchmark (ceph vs ext4)
Thanks Cristian. I will reflect on what you told me. There is no free lunch, I'll think it's worth paying the price. -- Thiago Henrique Em 24-05-2014 02:43, Christian Balzer escreveu: Hello, On Fri, 23 May 2014 15:41:23 -0300 Listas@Adminlinux wrote: Hi ! I have failover clusters for some aplications. Generally with 2 members configured with Ubuntu + Drbd + Ext4. For example, my IMAP cluster works fine with ~ 50k email accounts and my HTTP cluster hosts ~2k sites. My mailbox servers are also multiple DRBD based cluster pairs. For performance in fully redundant storage there is isn't anything better (in the OSS, generic hardware section at least). See design here: http://adminlinux.com.br/cluster_design.txt I would like to provide load balancing instead of just failover. So, I would like to use a distributed architecture of the filesystem. As we know, Ext4 isn't a distributed filesystem. So wish to use Ceph in my clusters. You will find that all cluster/distributed filesystems have severe performance shortcomings when compared to something like Ext4. On top of that, CephFS isn't ready for production as the MDS isn't HA. A potential middle way might be to use Ceph/RBD volumes formatted in Ext4. That doesn't give you shared access, but it will allow you to separate storage and compute nodes, so when one compute node becomes busy, mount that volume from a more powerful compute node instead. That all said, I can't see any way and reason to replace my mailbox DRBD clusters with Ceph in the foreseeable future. To get similar performance/reliability to DRBD I would have to spend 3-4 times the money. Where Ceph/RBD works well is situations where you can't fit the compute needs into a storage node (as required with DRBD) and where you want to access things from multiple compute nodes, primarily for migration purposes. In short, as a shared storage for VMs. Any suggestions for design of the cluster with Ubuntu+Ceph? I built a simple cluster of 2 servers to test simultaneous reading and writing with Ceph. My conf: http://adminlinux.com.br/ceph_conf.txt Again, CephFS isn't ready for production, but other than that I know very little about it as I don't use it. However your version of Ceph is severely outdated, you really should be looking at something more recent to rule out you're experience long fixed bugs. The same goes for your entire setup and kernel. Also Ceph only starts to perform decently with many OSDs (disks) and the journals on SSDs instead of being on the same disk. Think DRBD AL metadata-internal, but with MUCH more impact. Regards, Christian But in my simultaneous benchmarks found errors in reading and writing. I ran iozone -t 5 -r 4k -s 2m simultaneously on both servers in the cluster. The performance was poor and had errors like this: Error in file: Found ?0? Expecting ?6d6d6d6d6d6d6d6d? addr b660 Error in file: Position 1060864 Record # 259 Record size 4 kb where b660 loop 0 Performance graphs of benchmark: http://adminlinux.com.br/ceph_bench.html Can you help me find what I did wrong? Thanks ! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Designing a cluster with ceph and benchmark (ceph vs ext4)
Thanks Pieter! I tried using OCFS2 over DRBD, but was not satisfied. I was being affected by various bugs in OCFS2. But Oracle was not committed to solving them. -- Thiago Henrique Em 24-05-2014 09:14, Pieter Koorts escreveu: If looking for a DRBD alternative and not wanting to use CephFS is it not possible to just use something like OCFS2 or GFS on top of a RDB block device and all worker nodes accessing it via GFS or OCFS2 (obviously with write-through mode)? Would this method not present some advantages over DRBD? DRBD has its uses and will never go away but it does have limited scalability in the general sense. Pieter On 24 May 2014, at 06:43, Christian Balzer ch...@gol.com mailto:ch...@gol.com wrote: Hello, On Fri, 23 May 2014 15:41:23 -0300 Listas@Adminlinux wrote: Hi ! I have failover clusters for some aplications. Generally with 2 members configured with Ubuntu + Drbd + Ext4. For example, my IMAP cluster works fine with ~ 50k email accounts and my HTTP cluster hosts ~2k sites. My mailbox servers are also multiple DRBD based cluster pairs. For performance in fully redundant storage there is isn't anything better (in the OSS, generic hardware section at least). See design here:http://adminlinux.com.br/cluster_design.txt I would like to provide load balancing instead of just failover. So, I would like to use a distributed architecture of the filesystem. As we know, Ext4 isn't a distributed filesystem. So wish to use Ceph in my clusters. You will find that all cluster/distributed filesystems have severe performance shortcomings when compared to something like Ext4. On top of that, CephFS isn't ready for production as the MDS isn't HA. A potential middle way might be to use Ceph/RBD volumes formatted in Ext4. That doesn't give you shared access, but it will allow you to separate storage and compute nodes, so when one compute node becomes busy, mount that volume from a more powerful compute node instead. That all said, I can't see any way and reason to replace my mailbox DRBD clusters with Ceph in the foreseeable future. To get similar performance/reliability to DRBD I would have to spend 3-4 times the money. Where Ceph/RBD works well is situations where you can't fit the compute needs into a storage node (as required with DRBD) and where you want to access things from multiple compute nodes, primarily for migration purposes. In short, as a shared storage for VMs. Any suggestions for design of the cluster with Ubuntu+Ceph? I built a simple cluster of 2 servers to test simultaneous reading and writing with Ceph. My conf: http://adminlinux.com.br/ceph_conf.txt Again, CephFS isn't ready for production, but other than that I know very little about it as I don't use it. However your version of Ceph is severely outdated, you really should be looking at something more recent to rule out you're experience long fixed bugs. The same goes for your entire setup and kernel. Also Ceph only starts to perform decently with many OSDs (disks) and the journals on SSDs instead of being on the same disk. Think DRBD AL metadata-internal, but with MUCH more impact. Regards, Christian But in my simultaneous benchmarks found errors in reading and writing. I ran iozone -t 5 -r 4k -s 2m simultaneously on both servers in the cluster. The performance was poor and had errors like this: Error in file: Found ?0? Expecting ?6d6d6d6d6d6d6d6d? addr b660 Error in file: Position 1060864 Record # 259 Record size 4 kb where b660 loop 0 Performance graphs of benchmark: http://adminlinux.com.br/ceph_bench.html Can you help me find what I did wrong? Thanks ! -- Christian BalzerNetwork/Systems Engineer ch...@gol.com mailto:ch...@gol.comGlobal OnLine Japan/Fusion Communications http://www.gol.com/ ___ ceph-users mailing list ceph-users@lists.ceph.com mailto:ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Designing a cluster with ceph and benchmark (ceph vs ext4)
Hi ! I have failover clusters for some aplications. Generally with 2 members configured with Ubuntu + Drbd + Ext4. For example, my IMAP cluster works fine with ~ 50k email accounts and my HTTP cluster hosts ~2k sites. See design here: http://adminlinux.com.br/cluster_design.txt I would like to provide load balancing instead of just failover. So, I would like to use a distributed architecture of the filesystem. As we know, Ext4 isn't a distributed filesystem. So wish to use Ceph in my clusters. Any suggestions for design of the cluster with Ubuntu+Ceph? I built a simple cluster of 2 servers to test simultaneous reading and writing with Ceph. My conf: http://adminlinux.com.br/ceph_conf.txt But in my simultaneous benchmarks found errors in reading and writing. I ran iozone -t 5 -r 4k -s 2m simultaneously on both servers in the cluster. The performance was poor and had errors like this: Error in file: Found ?0? Expecting ?6d6d6d6d6d6d6d6d? addr b660 Error in file: Position 1060864 Record # 259 Record size 4 kb where b660 loop 0 Performance graphs of benchmark: http://adminlinux.com.br/ceph_bench.html Can you help me find what I did wrong? Thanks ! -- Thiago Henrique www.adminlinux.com.br ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] CephFS to provide distributed access read/write
Hi ! I have failover clusters (IMAP service) with 2 members configured with Ubuntu + Drbd + Ext4. My IMAP clusters works fine with ~ 50k email accounts. See design here: http://adminlinux.com.br/my_imap_cluster_design.txt I would like to use a distributed architecture of the filesystem to provide more processing power and availability to my clusters. As we know, Ext4 is a good filesystem but it isn't a distributed filesystem. So I would like to use Ceph in my clusters. I intend to change my IMAP cluster for an architecture like this: Ubuntu + Ceph RBD + CephFS Is CephFS already stable to provide simultaneous access to data in a production environment ? Thanks ! -- Thiago Henrique www.adminlinux.com.br ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] CephFS to provide distributed access read/write
Hi ! I have failover clusters (IMAP service) with 2 members configured with Ubuntu + Drbd + Ext4. My IMAP clusters works fine with ~ 50k email accounts. See design here: http://adminlinux.com.br/my_imap_cluster_design.txt I would like to use a distributed architecture of the filesystem to provide more processing power and availability to my clusters. As we know, Ext4 is a good filesystem but it isn't a distributed filesystem. So I would like to use Ceph in my clusters. I intend to change my IMAP cluster for an architecture like this: Ubuntu + Ceph RBD + CephFS Is CephFS already stable to provide simultaneous access to data in a production environment ? Thanks ! -- Thiago Henrique www.adminlinux.com.br ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Is Ceph a provider of block device too ?
Thank you! I am studying and testing CEPH. I think it will be very good for my needs On 18-11-2013 20:31, Timofey root wrote: On 09 нояб. 2013 г., at 1:46, Gregory Farnum g...@inktank.com wrote: On Fri, Nov 8, 2013 at 8:49 AM, Listas lis...@adminlinux.com.br wrote: Hi ! I have clusters (IMAP service) with 2 members configured with Ubuntu + Drbd + Ext4. Intend to migrate to the use of Ceph and begin to allow distributed access to the data. Does Ceph provides the distributed filesystem and block device? Ceph's RBD is a distributed block device and works very well; you could use it to replace DRBD. The CephFS distributed filesystem is in more of a preview mode and is not supported for general use at this time. Does Ceph work fine in clusters of two members? It should work fine in a cluster of that size, but you're not getting as many advantages over other solutions at such small scales as you do from larger ones. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Is Ceph a provider of block device too ?
Hi ! I have clusters (IMAP service) with 2 members configured with Ubuntu + Drbd + Ext4. Intend to migrate to the use of Ceph and begin to allow distributed access to the data. Does Ceph provides the distributed filesystem and block device? Does Ceph work fine in clusters of two members? Thanks! -- www.adminlinux.com.br Thiago Henrique ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
