Re: [ceph-users] How to think a two different disk's technologies architecture

[German Anders]

Like Alex said network is not an issue now. For example I got a 6 node
cluster with mix of sas and ssd disks running inside cassandra clusters
with heavy load and also mysql clusters and Im getting less than 1ms of IO
latency, on the network part i have infiniband with FDR configured and also
cluster and public network running on the same ib network, no separation of
clus network and runs so far so good! The ib network is shared by a total
of 128 host plus the 6 node and 3 mon ceph cluster.

We are Planning now a all-nvme ceph cluster with ib. It would be nice that
ceph works on the near future with the option of rdma integrated anyone
know if this is on the roadmap? I know that there's a "possible" config of
that but it's not production ready and needs a lot of tuning and
configuration.

Best


On Fri, Mar 24, 2017 at 11:04 Alejandro Comisario 
wrote:

thanks for the recommendations so far.
any one with more experiences and thoughts?

best

On Mar 23, 2017 16:36, "Maxime Guyot"  wrote:

Hi Alexandro,

As I understand you are planning NVMe for Journal for SATA HDD and
collocated journal for SATA SSD?

Option 1:
- 24x SATA SSDs per server, will have a bottleneck with the storage
bus/controller.  Also, I would consider the network capacity 24xSSDs will
deliver more performance than 24xHDD with journal, but you have the same
network capacity on both types of nodes.
- This option is a little easier to implement: just move nodes in different
CRUSHmap root
- Failure of a server (assuming size = 3) will impact all PGs
Option 2:
- You may have noisy neighbors effect between HDDs and SSDs, if HDDs are
able to saturate your NICs or storage controller. So be mindful of this
with the hardware design
- To configure the CRUSHmap for this you need to split each server in 2, I
usually use “server1-hdd” and “server1-ssd” and map the right OSD in the
right bucket, so a little extra work here but you can easily fix a “crush
location hook” script for it (see example
http://www.root314.com/2017/01/15/Ceph-storage-tiers/)
- In case of a server failure recovery will be faster than option 1 and
will impact less PGs

Some general notes:
- SSD pools perform better with higher frequency CPUs
- the 1GB of RAM per TB is a little outdated, the current consensus for HDD
OSDs is around 2GB/OSD (see
https://www.redhat.com/cms/managed-files/st-rhcs-config-guide-technology-detail-inc0387897-201604-en.pdf
)
- Network wise, if the SSD OSDs are rated for 500MB/s and use collocated
journal you could generate up to 250MB/s of traffic per SSD OSD (24Gbps for
12x or 48Gbps for 24x) therefore I would consider doing 4x10G and
consolidate both client and cluster network on that

Cheers,
Maxime

On 23/03/17 18:55, "ceph-users on behalf of Alejandro Comisario" <
ceph-users-boun...@lists.ceph.com on behalf of alejan...@nubeliu.com> wrote:

Hi everyone!
I have to install a ceph cluster (6 nodes) with two "flavors" of
disks, 3 servers with SSD and 3 servers with SATA.

Y will purchase 24 disks servers (the ones with sata with NVE SSD for
the SATA journal)
Processors will be 2 x E5-2620v4 with HT, and ram will be 20GB for the
OS, and 1.3GB of ram per storage TB.

The servers will have 2 x 10Gb bonding for public network and 2 x 10Gb
for cluster network.
My doubts resides, ar want to ask the community about experiences and
pains and gains of choosing between.

Option 1
3 x servers just for SSD
3 x servers jsut for SATA

Option 2
6 x servers with 12 SSD and 12 SATA each

Regarding crushmap configuration and rules everything is clear to make
sure that two pools (poolSSD and poolSATA) uses the right disks.

But, what about performance, maintenance, architecture scalability, etc
?

thank you very much !

--
Alejandrito
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 

*German Anders*
Storage Engineer Leader
*Despegar* | IT Team
*office* +54 11 4894 3500 x3408
*mobile* +54 911 3493 7262
*mail* gand...@despegar.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to think a two different disk's technologies architecture

[Alex Gorbachev]

On Fri, Mar 24, 2017 at 10:04 AM Alejandro Comisario 
wrote:

> thanks for the recommendations so far.
> any one with more experiences and thoughts?
>
> best
>

On the network side, 25, 40, 56 and maybe soon 100 Gbps can now be fairly
affordable, and simplify the architecture for the high throughput nodes.


> On Mar 23, 2017 16:36, "Maxime Guyot"  wrote:
>
> Hi Alexandro,
>
> As I understand you are planning NVMe for Journal for SATA HDD and
> collocated journal for SATA SSD?
>
> Option 1:
> - 24x SATA SSDs per server, will have a bottleneck with the storage
> bus/controller.  Also, I would consider the network capacity 24xSSDs will
> deliver more performance than 24xHDD with journal, but you have the same
> network capacity on both types of nodes.
> - This option is a little easier to implement: just move nodes in
> different CRUSHmap root
> - Failure of a server (assuming size = 3) will impact all PGs
> Option 2:
> - You may have noisy neighbors effect between HDDs and SSDs, if HDDs are
> able to saturate your NICs or storage controller. So be mindful of this
> with the hardware design
> - To configure the CRUSHmap for this you need to split each server in 2, I
> usually use “server1-hdd” and “server1-ssd” and map the right OSD in the
> right bucket, so a little extra work here but you can easily fix a “crush
> location hook” script for it (see example
> http://www.root314.com/2017/01/15/Ceph-storage-tiers/)
> - In case of a server failure recovery will be faster than option 1 and
> will impact less PGs
>
> Some general notes:
> - SSD pools perform better with higher frequency CPUs
> - the 1GB of RAM per TB is a little outdated, the current consensus for
> HDD OSDs is around 2GB/OSD (see
> https://www.redhat.com/cms/managed-files/st-rhcs-config-guide-technology-detail-inc0387897-201604-en.pdf
> )
> - Network wise, if the SSD OSDs are rated for 500MB/s and use collocated
> journal you could generate up to 250MB/s of traffic per SSD OSD (24Gbps for
> 12x or 48Gbps for 24x) therefore I would consider doing 4x10G and
> consolidate both client and cluster network on that
>
> Cheers,
> Maxime
>
> On 23/03/17 18:55, "ceph-users on behalf of Alejandro Comisario" <
> ceph-users-boun...@lists.ceph.com on behalf of alejan...@nubeliu.com>
> wrote:
>
> Hi everyone!
> I have to install a ceph cluster (6 nodes) with two "flavors" of
> disks, 3 servers with SSD and 3 servers with SATA.
>
> Y will purchase 24 disks servers (the ones with sata with NVE SSD for
> the SATA journal)
> Processors will be 2 x E5-2620v4 with HT, and ram will be 20GB for the
> OS, and 1.3GB of ram per storage TB.
>
> The servers will have 2 x 10Gb bonding for public network and 2 x 10Gb
> for cluster network.
> My doubts resides, ar want to ask the community about experiences and
> pains and gains of choosing between.
>
> Option 1
> 3 x servers just for SSD
> 3 x servers jsut for SATA
>
> Option 2
> 6 x servers with 12 SSD and 12 SATA each
>
> Regarding crushmap configuration and rules everything is clear to make
> sure that two pools (poolSSD and poolSATA) uses the right disks.
>
> But, what about performance, maintenance, architecture scalability,
> etc ?
>
> thank you very much !
>
> --
> Alejandrito
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
-- 
--
Alex Gorbachev
Storcium
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to think a two different disk's technologies architecture

[Alejandro Comisario]

thanks for the recommendations so far.
any one with more experiences and thoughts?

best

On Mar 23, 2017 16:36, "Maxime Guyot"  wrote:

> Hi Alexandro,
>
> As I understand you are planning NVMe for Journal for SATA HDD and
> collocated journal for SATA SSD?
>
> Option 1:
> - 24x SATA SSDs per server, will have a bottleneck with the storage
> bus/controller.  Also, I would consider the network capacity 24xSSDs will
> deliver more performance than 24xHDD with journal, but you have the same
> network capacity on both types of nodes.
> - This option is a little easier to implement: just move nodes in
> different CRUSHmap root
> - Failure of a server (assuming size = 3) will impact all PGs
> Option 2:
> - You may have noisy neighbors effect between HDDs and SSDs, if HDDs are
> able to saturate your NICs or storage controller. So be mindful of this
> with the hardware design
> - To configure the CRUSHmap for this you need to split each server in 2, I
> usually use “server1-hdd” and “server1-ssd” and map the right OSD in the
> right bucket, so a little extra work here but you can easily fix a “crush
> location hook” script for it (see example http://www.root314.com/2017/
> 01/15/Ceph-storage-tiers/)
> - In case of a server failure recovery will be faster than option 1 and
> will impact less PGs
>
> Some general notes:
> - SSD pools perform better with higher frequency CPUs
> - the 1GB of RAM per TB is a little outdated, the current consensus for
> HDD OSDs is around 2GB/OSD (see https://www.redhat.com/cms/
> managed-files/st-rhcs-config-guide-technology-detail-
> inc0387897-201604-en.pdf)
> - Network wise, if the SSD OSDs are rated for 500MB/s and use collocated
> journal you could generate up to 250MB/s of traffic per SSD OSD (24Gbps for
> 12x or 48Gbps for 24x) therefore I would consider doing 4x10G and
> consolidate both client and cluster network on that
>
> Cheers,
> Maxime
>
> On 23/03/17 18:55, "ceph-users on behalf of Alejandro Comisario" <
> ceph-users-boun...@lists.ceph.com on behalf of alejan...@nubeliu.com>
> wrote:
>
> Hi everyone!
> I have to install a ceph cluster (6 nodes) with two "flavors" of
> disks, 3 servers with SSD and 3 servers with SATA.
>
> Y will purchase 24 disks servers (the ones with sata with NVE SSD for
> the SATA journal)
> Processors will be 2 x E5-2620v4 with HT, and ram will be 20GB for the
> OS, and 1.3GB of ram per storage TB.
>
> The servers will have 2 x 10Gb bonding for public network and 2 x 10Gb
> for cluster network.
> My doubts resides, ar want to ask the community about experiences and
> pains and gains of choosing between.
>
> Option 1
> 3 x servers just for SSD
> 3 x servers jsut for SATA
>
> Option 2
> 6 x servers with 12 SSD and 12 SATA each
>
> Regarding crushmap configuration and rules everything is clear to make
> sure that two pools (poolSSD and poolSATA) uses the right disks.
>
> But, what about performance, maintenance, architecture scalability,
> etc ?
>
> thank you very much !
>
> --
> Alejandrito
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to think a two different disk's technologies architecture

[Maxime Guyot]

Hi Alexandro,

As I understand you are planning NVMe for Journal for SATA HDD and collocated 
journal for SATA SSD?

Option 1:
- 24x SATA SSDs per server, will have a bottleneck with the storage 
bus/controller.  Also, I would consider the network capacity 24xSSDs will 
deliver more performance than 24xHDD with journal, but you have the same 
network capacity on both types of nodes.
- This option is a little easier to implement: just move nodes in different 
CRUSHmap root
- Failure of a server (assuming size = 3) will impact all PGs
Option 2:
- You may have noisy neighbors effect between HDDs and SSDs, if HDDs are able 
to saturate your NICs or storage controller. So be mindful of this with the 
hardware design
- To configure the CRUSHmap for this you need to split each server in 2, I 
usually use “server1-hdd” and “server1-ssd” and map the right OSD in the right 
bucket, so a little extra work here but you can easily fix a “crush location 
hook” script for it (see example 
http://www.root314.com/2017/01/15/Ceph-storage-tiers/)
- In case of a server failure recovery will be faster than option 1 and will 
impact less PGs

Some general notes:
- SSD pools perform better with higher frequency CPUs
- the 1GB of RAM per TB is a little outdated, the current consensus for HDD 
OSDs is around 2GB/OSD (see 
https://www.redhat.com/cms/managed-files/st-rhcs-config-guide-technology-detail-inc0387897-201604-en.pdf)
- Network wise, if the SSD OSDs are rated for 500MB/s and use collocated 
journal you could generate up to 250MB/s of traffic per SSD OSD (24Gbps for 12x 
or 48Gbps for 24x) therefore I would consider doing 4x10G and consolidate both 
client and cluster network on that

Cheers,
Maxime

On 23/03/17 18:55, "ceph-users on behalf of Alejandro Comisario" 
 wrote:

Hi everyone!
I have to install a ceph cluster (6 nodes) with two "flavors" of
disks, 3 servers with SSD and 3 servers with SATA.

Y will purchase 24 disks servers (the ones with sata with NVE SSD for
the SATA journal)
Processors will be 2 x E5-2620v4 with HT, and ram will be 20GB for the
OS, and 1.3GB of ram per storage TB.

The servers will have 2 x 10Gb bonding for public network and 2 x 10Gb
for cluster network.
My doubts resides, ar want to ask the community about experiences and
pains and gains of choosing between.

Option 1
3 x servers just for SSD
3 x servers jsut for SATA

Option 2
6 x servers with 12 SSD and 12 SATA each

Regarding crushmap configuration and rules everything is clear to make
sure that two pools (poolSSD and poolSATA) uses the right disks.

But, what about performance, maintenance, architecture scalability, etc ?

thank you very much !

-- 
Alejandrito
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to think a two different disk's technologies architecture

[Udo Lembke]

Hi,
ceph speeds up with more nodes and more OSDs - so go for 6 nodes with
mixed SSD+SATA.

Udo

On 23.03.2017 18:55, Alejandro Comisario wrote:
> Hi everyone!
> I have to install a ceph cluster (6 nodes) with two "flavors" of
> disks, 3 servers with SSD and 3 servers with SATA.
>
> Y will purchase 24 disks servers (the ones with sata with NVE SSD for
> the SATA journal)
> Processors will be 2 x E5-2620v4 with HT, and ram will be 20GB for the
> OS, and 1.3GB of ram per storage TB.
>
> The servers will have 2 x 10Gb bonding for public network and 2 x 10Gb
> for cluster network.
> My doubts resides, ar want to ask the community about experiences and
> pains and gains of choosing between.
>
> Option 1
> 3 x servers just for SSD
> 3 x servers jsut for SATA
>
> Option 2
> 6 x servers with 12 SSD and 12 SATA each
>
> Regarding crushmap configuration and rules everything is clear to make
> sure that two pools (poolSSD and poolSATA) uses the right disks.
>
> But, what about performance, maintenance, architecture scalability, etc ?
>
> thank you very much !
>

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] How to think a two different disk's technologies architecture

[Alejandro Comisario]

Hi everyone!
I have to install a ceph cluster (6 nodes) with two "flavors" of
disks, 3 servers with SSD and 3 servers with SATA.

Y will purchase 24 disks servers (the ones with sata with NVE SSD for
the SATA journal)
Processors will be 2 x E5-2620v4 with HT, and ram will be 20GB for the
OS, and 1.3GB of ram per storage TB.

The servers will have 2 x 10Gb bonding for public network and 2 x 10Gb
for cluster network.
My doubts resides, ar want to ask the community about experiences and
pains and gains of choosing between.

Option 1
3 x servers just for SSD
3 x servers jsut for SATA

Option 2
6 x servers with 12 SSD and 12 SATA each

Regarding crushmap configuration and rules everything is clear to make
sure that two pools (poolSSD and poolSATA) uses the right disks.

But, what about performance, maintenance, architecture scalability, etc ?

thank you very much !

-- 
Alejandrito
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com