[ceph-users] Re: [REEF][cephadm] new cluster all pg unknown

[wodel youchi]

Hi,

Thanks Stefan.

Yes I do have separate port-channel interfaces, for public and cluster
networks.
I just didn't understand the documentation (which is sometimes not that
clear). For me when you put forward the --cluster_network option, it meant
that --mon-ip was in the public network by default and that you want to
define a separate network for your cluster to use as private net.
I was reading the documentation when I came across : *ceph config set mon
public_network* command, it's then when I saw what could be my mistake.
I redid the deployment and so far so good :-)
I did use *ceph config set mon public_network* to specify my public net,
now I have to test my clients if they can connect to it.

Regards.

Le ven. 15 mars 2024 à 08:20, Stefan Kooman  a écrit :

> On 15-03-2024 08:10, wodel youchi wrote:
> > Hi,
> >
> > I found my error, it was a mismatch between the monitor network ip
> address
> > and the --cluster_network which were in different subnets.
> > I misunderstood the --cluster_network subnet, I thought that when
> creating
> > a cluster, the monitor IP designed the public Network, and if I wanted to
> > separate public and private (cluster) networks, I needed to use the
> > --cluster_network option.
> > Maybe I was in over my head, but sometimes it is not that clear.
>
> Ah, good that you found the error. Do you have separate interfaces /
> port-channels for public and cluster networks? Note that a cluster
> network is not required and makes things more complicated (as you have
> noted) and might not give you any benefits when the infrastructure does
> not match this separation of functions.
>
> Gr. Stefan
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: [REEF][cephadm] new cluster all pg unknown

[wodel youchi]

Hi,

I found my error, it was a mismatch between the monitor network ip address
and the --cluster_network which were in different subnets.
I misunderstood the --cluster_network subnet, I thought that when creating
a cluster, the monitor IP designed the public Network, and if I wanted to
separate public and private (cluster) networks, I needed to use the
--cluster_network option.
Maybe I was in over my head, but sometimes it is not that clear.

Regards.

Le ven. 15 mars 2024 à 07:18, wodel youchi  a
écrit :

> Hi,
>
> Note : Firewall is disabled on all hosts.
>
> Regards.
>
> Le ven. 15 mars 2024 à 06:42, wodel youchi  a
> écrit :
>
>> Hi,
>>
>> I did recreate the cluster again, and it is the result.
>>
>> This is my initial bootstrap
>>
>> cephadm --image 192.168.2.36:4000/ceph/ceph:v18 bootstrap
>> --initial-dashboard-user admin \
>> --initial-dashboard-password adminpass --dashboard-password-noupdate
>>  --registry-url 192.168.2.36:4000  \
>> --registry-username admin --registry-password admin --mon-ip 20.1.0.23
>> --cluster-network 20.2.0.0/16 \
>> --ssh-private-key /root/.ssh/id_rsa --ssh-public-key
>> /root/.ssh/id_rsa.pub \
>> -c initial-ceph.conf
>>
>> This is my initial-ceph.conf
>> [mgr]
>> mgr/cephadm/container_image_prometheus =
>> 192.168.2.36:4000/prometheus/prometheus:v2.43.0
>> mgr/cephadm/container_image_node_exporter =
>> 192.168.2.36:4000/prometheus/node-exporter:v1.5.0
>> mgr/cephadm/container_image_grafana =
>> 192.168.2.36:4000/ceph/ceph-grafana:9.4.7
>> mgr/cephadm/container_image_alertmanager =
>> 192.168.2.36:4000/prometheus/alertmanager:v0.25.0
>>
>>
>> Then I added two managers and monitors
>> # ceph orch host add controllerb 20.1.0.27 _admin
>> # ceph orch host add controllerc 20.1.0.31 _admin
>> # ceph orch apply mon --placement="3 controllera controllerb
>> controllerc"
>> # ceph orch apply mgr --placement="3 controllera controllerb controllerc"
>>
>> Then I added node-exporter, prometheus, grafana and crash
>> Then I added osd hosts
>> # ceph orch host add computehci01 20.1.0.2
>> # ceph orch host add computehci02 20.1.0.3
>> # ceph orch host add computehci03 20.1.0.4
>> ...
>> # ceph orch host add computehci09 20.1.0.10
>> ...
>>
>> And finally I added osd daemons
>> # ceph orch daemon add osd
>> computehci01:/dev/nvme0n1,/dev/nvme1n1,/dev/nvme2n1,/dev/nvme3n1
>> # ceph orch daemon add osd
>> computehci02:/dev/nvme0n1,/dev/nvme1n1,/dev/nvme2n1,/dev/nvme3n1
>> ...
>>
>> I created a pool
>> # ceph osd pool create volumes  replicated
>> # ceph osd pool application enable volumes rbd
>>
>> I even created cephfs pools and added mds service, but still 100% pgs are
>> unknown 
>>
>>
>> [root@controllera ~]# ceph -s
>>  cluster:
>>id: df914aa2-e21a-11ee-b8df-3cecef2872f0
>>health: HEALTH_WARN
>>1 MDSs report slow metadata IOs
>>Reduced data availability: 4 pgs inactive
>>
>>  services:
>>mon: 3 daemons, quorum controllera,controllerc,controllerb (age 13h)
>>mgr: controllera.ajttxz(active, since 13h), standbys:
>> controllerb.qtixeq, controllerc.pqyqqo
>>mds: 1/1 daemons up, 2 standby
>>osd: 36 osds: 36 up (since 7h), 36 in (since 7h)
>>
>>  data:
>>volumes: 1/1 healthy
>>pools:   4 pools, 4 pgs
>>objects: 0 objects, 0 B
>>usage:   1.1 GiB used, 110 TiB / 110 TiB avail
>>pgs: 100.000% pgs unknown
>> 4 unknown
>>
>> [root@controllera ~]# ceph osd pool ls detail
>> pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash
>> rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 46 flags
>> hashpspool,creating stripe_width 0 pg_num_max 32 pg_nu
>> m_min 1 application mgr
>> pool 2 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
>> rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 154 flags
>> hashpspool,creating stripe_width 0 application rbd
>> pool 3 'cephfs' replicated size 3 min_size 2 crush_rule 0 object_hash
>> rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 157 flags
>> hashpspool,creating stripe_width 0 application ceph
>> fs
>> pool 4 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0
>> object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 157
>> flags hashpspool,creating stripe_width 0 pg_auto
>> scale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
>>
>> What am I 

[ceph-users] Re: [REEF][cephadm] new cluster all pg unknown

[wodel youchi]

Hi,

Note : Firewall is disabled on all hosts.

Regards.

Le ven. 15 mars 2024 à 06:42, wodel youchi  a
écrit :

> Hi,
>
> I did recreate the cluster again, and it is the result.
>
> This is my initial bootstrap
>
> cephadm --image 192.168.2.36:4000/ceph/ceph:v18 bootstrap
> --initial-dashboard-user admin \
> --initial-dashboard-password adminpass --dashboard-password-noupdate
>  --registry-url 192.168.2.36:4000  \
> --registry-username admin --registry-password admin --mon-ip 20.1.0.23
> --cluster-network 20.2.0.0/16 \
> --ssh-private-key /root/.ssh/id_rsa --ssh-public-key /root/.ssh/id_rsa.pub
> \
> -c initial-ceph.conf
>
> This is my initial-ceph.conf
> [mgr]
> mgr/cephadm/container_image_prometheus =
> 192.168.2.36:4000/prometheus/prometheus:v2.43.0
> mgr/cephadm/container_image_node_exporter =
> 192.168.2.36:4000/prometheus/node-exporter:v1.5.0
> mgr/cephadm/container_image_grafana =
> 192.168.2.36:4000/ceph/ceph-grafana:9.4.7
> mgr/cephadm/container_image_alertmanager =
> 192.168.2.36:4000/prometheus/alertmanager:v0.25.0
>
>
> Then I added two managers and monitors
> # ceph orch host add controllerb 20.1.0.27 _admin
> # ceph orch host add controllerc 20.1.0.31 _admin
> # ceph orch apply mon --placement="3 controllera controllerb controllerc"
> # ceph orch apply mgr --placement="3 controllera controllerb controllerc"
>
> Then I added node-exporter, prometheus, grafana and crash
> Then I added osd hosts
> # ceph orch host add computehci01 20.1.0.2
> # ceph orch host add computehci02 20.1.0.3
> # ceph orch host add computehci03 20.1.0.4
> ...
> # ceph orch host add computehci09 20.1.0.10
> ...
>
> And finally I added osd daemons
> # ceph orch daemon add osd
> computehci01:/dev/nvme0n1,/dev/nvme1n1,/dev/nvme2n1,/dev/nvme3n1
> # ceph orch daemon add osd
> computehci02:/dev/nvme0n1,/dev/nvme1n1,/dev/nvme2n1,/dev/nvme3n1
> ...
>
> I created a pool
> # ceph osd pool create volumes  replicated
> # ceph osd pool application enable volumes rbd
>
> I even created cephfs pools and added mds service, but still 100% pgs are
> unknown 
>
>
> [root@controllera ~]# ceph -s
>  cluster:
>id: df914aa2-e21a-11ee-b8df-3cecef2872f0
>health: HEALTH_WARN
>1 MDSs report slow metadata IOs
>Reduced data availability: 4 pgs inactive
>
>  services:
>mon: 3 daemons, quorum controllera,controllerc,controllerb (age 13h)
>mgr: controllera.ajttxz(active, since 13h), standbys:
> controllerb.qtixeq, controllerc.pqyqqo
>mds: 1/1 daemons up, 2 standby
>osd: 36 osds: 36 up (since 7h), 36 in (since 7h)
>
>  data:
>volumes: 1/1 healthy
>pools:   4 pools, 4 pgs
>objects: 0 objects, 0 B
>usage:   1.1 GiB used, 110 TiB / 110 TiB avail
>pgs: 100.000% pgs unknown
> 4 unknown
>
> [root@controllera ~]# ceph osd pool ls detail
> pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 46 flags
> hashpspool,creating stripe_width 0 pg_num_max 32 pg_nu
> m_min 1 application mgr
> pool 2 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 154 flags
> hashpspool,creating stripe_width 0 application rbd
> pool 3 'cephfs' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 157 flags
> hashpspool,creating stripe_width 0 application ceph
> fs
> pool 4 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0
> object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 157
> flags hashpspool,creating stripe_width 0 pg_auto
> scale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
>
> What am I missing, why PGs won't pair?
>
>
>
> Regards.
>
> Le jeu. 14 mars 2024 à 15:36, wodel youchi  a
> écrit :
>
>> Hi,
>>
>> I am creating a new ceph cluster using REEF.
>>
>> This is my host_specs file
>> [root@controllera config]# cat hosts-specs2.yml
>> service_type: host
>> hostname: computehci01
>> addr: 20.1.0.2
>> location:
>>  chassis: chassis1
>> ---
>> service_type: host
>> hostname: computehci02
>> addr: 20.1.0.3
>> location:
>>  chassis: chassis1
>> ---
>> service_type: host
>> hostname: computehci03
>> addr: 20.1.0.4
>> location:
>>  chassis: chassis1
>> ---
>> service_type: host
>> hostname: computehci04
>> addr: 20.1.0.5
>> location:
>>  chassis: chassis2
>> ---
>> service_type: host
>> hostname: computehci05
>> a

[ceph-users] Re: [REEF][cephadm] new cluster all pg unknown

[wodel youchi]

Hi,

I did recreate the cluster again, and it is the result.

This is my initial bootstrap

cephadm --image 192.168.2.36:4000/ceph/ceph:v18 bootstrap
--initial-dashboard-user admin \
--initial-dashboard-password adminpass --dashboard-password-noupdate
 --registry-url 192.168.2.36:4000  \
--registry-username admin --registry-password admin --mon-ip 20.1.0.23
--cluster-network 20.2.0.0/16 \
--ssh-private-key /root/.ssh/id_rsa --ssh-public-key /root/.ssh/id_rsa.pub \
-c initial-ceph.conf

This is my initial-ceph.conf
[mgr]
mgr/cephadm/container_image_prometheus =
192.168.2.36:4000/prometheus/prometheus:v2.43.0
mgr/cephadm/container_image_node_exporter =
192.168.2.36:4000/prometheus/node-exporter:v1.5.0
mgr/cephadm/container_image_grafana =
192.168.2.36:4000/ceph/ceph-grafana:9.4.7
mgr/cephadm/container_image_alertmanager =
192.168.2.36:4000/prometheus/alertmanager:v0.25.0


Then I added two managers and monitors
# ceph orch host add controllerb 20.1.0.27 _admin
# ceph orch host add controllerc 20.1.0.31 _admin
# ceph orch apply mon --placement="3 controllera controllerb controllerc"
# ceph orch apply mgr --placement="3 controllera controllerb controllerc"

Then I added node-exporter, prometheus, grafana and crash
Then I added osd hosts
# ceph orch host add computehci01 20.1.0.2
# ceph orch host add computehci02 20.1.0.3
# ceph orch host add computehci03 20.1.0.4
...
# ceph orch host add computehci09 20.1.0.10
...

And finally I added osd daemons
# ceph orch daemon add osd
computehci01:/dev/nvme0n1,/dev/nvme1n1,/dev/nvme2n1,/dev/nvme3n1
# ceph orch daemon add osd
computehci02:/dev/nvme0n1,/dev/nvme1n1,/dev/nvme2n1,/dev/nvme3n1
...

I created a pool
# ceph osd pool create volumes  replicated
# ceph osd pool application enable volumes rbd

I even created cephfs pools and added mds service, but still 100% pgs are
unknown 


[root@controllera ~]# ceph -s
 cluster:
   id: df914aa2-e21a-11ee-b8df-3cecef2872f0
   health: HEALTH_WARN
   1 MDSs report slow metadata IOs
   Reduced data availability: 4 pgs inactive

 services:
   mon: 3 daemons, quorum controllera,controllerc,controllerb (age 13h)
   mgr: controllera.ajttxz(active, since 13h), standbys:
controllerb.qtixeq, controllerc.pqyqqo
   mds: 1/1 daemons up, 2 standby
   osd: 36 osds: 36 up (since 7h), 36 in (since 7h)

 data:
   volumes: 1/1 healthy
   pools:   4 pools, 4 pgs
   objects: 0 objects, 0 B
   usage:   1.1 GiB used, 110 TiB / 110 TiB avail
   pgs: 100.000% pgs unknown
4 unknown

[root@controllera ~]# ceph osd pool ls detail
pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 46 flags
hashpspool,creating stripe_width 0 pg_num_max 32 pg_nu
m_min 1 application mgr
pool 2 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 154 flags
hashpspool,creating stripe_width 0 application rbd
pool 3 'cephfs' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 157 flags
hashpspool,creating stripe_width 0 application ceph
fs
pool 4 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0
object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 157
flags hashpspool,creating stripe_width 0 pg_auto
scale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs

What am I missing, why PGs won't pair?



Regards.

Le jeu. 14 mars 2024 à 15:36, wodel youchi  a
écrit :

> Hi,
>
> I am creating a new ceph cluster using REEF.
>
> This is my host_specs file
> [root@controllera config]# cat hosts-specs2.yml
> service_type: host
> hostname: computehci01
> addr: 20.1.0.2
> location:
>  chassis: chassis1
> ---
> service_type: host
> hostname: computehci02
> addr: 20.1.0.3
> location:
>  chassis: chassis1
> ---
> service_type: host
> hostname: computehci03
> addr: 20.1.0.4
> location:
>  chassis: chassis1
> ---
> service_type: host
> hostname: computehci04
> addr: 20.1.0.5
> location:
>  chassis: chassis2
> ---
> service_type: host
> hostname: computehci05
> addr: 20.1.0.6
> location:
>  chassis: chassis2
> ---
> service_type: host
> hostname: computehci06
> addr: 20.1.0.7
> location:
>  chassis: chassis2
> ---
> service_type: host
> hostname: computehci07
> addr: 20.1.0.8
> location:
>  chassis: chassis3
> ---
> service_type: host
> hostname: computehci08
> addr: 20.1.0.9
> location:
>  chassis: chassis3
> ---
> service_type: host
> hostname: computehci09
> addr: 20.1.0.10
> location:
>  chassis: chassis3
> ---
> service_type: host
> hostname: computehci10
> addr: 20.1.0.11
> location:
>  chassis: chassis3
> ---
> service_type: host
> hostname: computehci11
> addr: 20.1.0.12
> locati

[ceph-users] [REEF][cephadm] new cluster all pg unknown

[wodel youchi]

Hi,

I am creating a new ceph cluster using REEF.

This is my host_specs file
[root@controllera config]# cat hosts-specs2.yml
service_type: host
hostname: computehci01
addr: 20.1.0.2
location:
 chassis: chassis1
---
service_type: host
hostname: computehci02
addr: 20.1.0.3
location:
 chassis: chassis1
---
service_type: host
hostname: computehci03
addr: 20.1.0.4
location:
 chassis: chassis1
---
service_type: host
hostname: computehci04
addr: 20.1.0.5
location:
 chassis: chassis2
---
service_type: host
hostname: computehci05
addr: 20.1.0.6
location:
 chassis: chassis2
---
service_type: host
hostname: computehci06
addr: 20.1.0.7
location:
 chassis: chassis2
---
service_type: host
hostname: computehci07
addr: 20.1.0.8
location:
 chassis: chassis3
---
service_type: host
hostname: computehci08
addr: 20.1.0.9
location:
 chassis: chassis3
---
service_type: host
hostname: computehci09
addr: 20.1.0.10
location:
 chassis: chassis3
---
service_type: host
hostname: computehci10
addr: 20.1.0.11
location:
 chassis: chassis3
---
service_type: host
hostname: computehci11
addr: 20.1.0.12
location:
 chassis: chassis4
---
service_type: host
hostname: computehci12
addr: 20.1.0.13
location:
 chassis: chassis4
---
service_type: host
hostname: computehci13
addr: 20.1.0.14
location:
 chassis: chassis4
---
service_type: host
hostname: computehci14
addr: 20.1.0.15
location:
 chassis: chassis4
---
service_type: host
hostname: computehci15
addr: 20.1.0.16
location:
 chassis: chassis5
---
service_type: host
hostname: computehci16
addr: 20.1.0.17
location:
 chassis: chassis5
---
service_type: host
hostname: computehci17
addr: 20.1.0.18
location:
 chassis: chassis5
---
service_type: host
hostname: computehci18
addr: 20.1.0.19
location:
 chassis: chassis5
---
service_type: host
hostname: computehci19
addr: 20.1.0.20
location:
 chassis: chassis6
---
service_type: host
hostname: computehci20
addr: 20.1.0.21
location:
 chassis: chassis6
---
service_type: host
hostname: computehci21
addr: 20.1.0.22
location:
 chassis: chassis6
---
service_type: host
hostname: computehci22
addr: 20.1.0.24
location:
 chassis: chassis7
---
service_type: host
hostname: computehci23
addr: 20.1.0.25
location:
 chassis: chassis7
---
service_type: host
hostname: computehci24
addr: 20.1.0.26
location:
 chassis: chassis7
---
service_type: host
hostname: computehci25
addr: 20.1.0.28
location:
 chassis: chassis8
---
service_type: host
hostname: computehci26
addr: 20.1.0.29
location:
 chassis: chassis8
---
service_type: host
hostname: computehci27
addr: 20.1.0.30
location:
 chassis: chassis8
---
service_type: host
hostname: controllera
addr: 20.1.0.23
---
service_type: host
hostname: controllerb
addr: 20.1.0.27
---
service_type: host
hostname: controllerc
addr: 20.1.0.31
---
service_type: mon
placement:
 hosts:
  - controllera
  - controllerb
  - controllerc
---
service_type: mgr
placement:
 hosts:
  - controllera
  - controllerb
  - controllerc
---
service_type: osd
service_id: default_drive_group
placement:
 hosts:
  - computehci01
  - computehci02
  - computehci03
  - computehci04
  - computehci05
  - computehci06
  - computehci07
  - computehci08
  - computehci09
  - computehci10
  - computehci11
  - computehci12
  - computehci13
  - computehci14
  - computehci15
  - computehci16
  - computehci17
  - computehci18
  - computehci19
  - computehci20
  - computehci21
  - computehci22
  - computehci23
  - computehci24
  - computehci25
  - computehci26
  - computehci27
spec:
 data_devices:
   rotational: 0


All osds were added but, pg still unknown state

I've created a pool, but it didn't change anything.

[root@controllerb ~]# ceph -s
 cluster:
   id: be250ade-e1f2-11ee-a6ff-3cecef2872f0
   health: HEALTH_WARN
   Reduced data availability: 1 pg inactive

 services:
   mon: 3 daemons, quorum controllera,controllerc,controllerb (age 3h)
   mgr: controllerc.jevbkl(active, since 21s), standbys:
controllera.zwlolp, controllerb.vqkdga
   osd: 108 osds: 108 up (since 2m), 108 in (since 24m)

 data:
   pools:   2 pools, 33 pgs
   objects: 0 objects, 0 B
   usage:   5.1 GiB used, 330 TiB / 330 TiB avail


*  pgs: 100.000% pgs unknown 33 unknown*

Did I miss something?

Regards.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] ceph osd different size to create a cluster for Openstack : asking for advice

[wodel youchi]

Hi,

I need some guidance from you folks...

I am going to deploy a ceph cluster in HCI mode for an openstack platform.
My hardware will be :
- 03 control nodes  :
- 27 osd nodes : each node has 03x3.8To nvme + 01x1.9To nvme disks (those
disks will all be used as OSDs)

In my Openstack I will be creating all sorts of pools : RBD, Cephfs and RGW.

I am planning to create two crush rules using the disk size as a parameter.
Then divide my pools between the two rules.
- RBD to use the 3.8To disks since I need more space here.
- Cephfs and RGW to use 1.9To disks.

Is this a good configuration?

Regards
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] [Quincy] cannot configure dashboard to listen on all ports

[wodel youchi]

Hi,
ceph dashboard fails to listen on all IPs.

log_channel(cluster) log [ERR] : Unhandled exception from module 'dashboard'
while running on mgr.controllera: OSError("No socket could be created --
(('0.0.0.0', 8443): [Errno -2] Name or service not known) -- (('::', 8443,
0, 0):


ceph version 17.2.7  quincy (stable)
Regards.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] [Quincy] NFS ingress mode haproxy-protocol not recognized

[wodel youchi]

Hi;

I tried to create an NFS cluster using this command :
[root@controllera ceph]# ceph nfs cluster create mynfs "3 controllera
controllerb controllerc" --ingress --virtual_ip 20.1.0.201 --ingress-mode
haproxy-protocol
Invalid command: haproxy-protocol not in default|keepalive-only

And I got this error : Invalid command haproxy-protocol
I am using Quincy : ceph version 17.2.7 (...) quincy (stable)

Is it not supported yet?

Regards.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph orch doesn't execute commands and doesn't report correct status of daemons

[wodel youchi]

Hi,

I'll try the 'ceph mgr fail' and report back.

In the meantime, my problem with the images...
I am trying to use my local registry to deploy the different services. I
don't know how to use the 'apply' and force my cluster to use my local
registry.
So basically, what I am doing so far is :
1 - ceph orch apply -i rgw-service.yml   < deploy the rgw,
and this will pull the image from the internet
2 - ceph orch daemon redeploy rgw.opsrgw.controllera.gtrttj --image
192.168.2.36:4000/ceph/ceph:v17  < Redeploy the demons of that
service with my local image.

How May I deploy directly from my local registry?

Regards.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Ceph orch doesn't execute commands and doesn't report correct status of daemons

[wodel youchi]

Hi,

I have finished the conversion from ceph-ansible to cephadm yesterday.
Everything seemed to be working until this morning, I wanted to redeploy
rgw service to specify the network to be used.

So I deleted the rgw services with ceph orch rm, then I prepared a yml file
with the new conf. I applied the file and the new rgw service was started
but it was launched with an external image, so I wanted to redeploy using
my local image so I did a redeploy ... and then nothing happened, I get the
rescheduled message but nothing happened, then I restarted one of the
controllers, the orchestrator doesn't seem to be aware that some service
have restarted???

PS : I don't fully master the cephadm command line and use.

Regards.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Migration from ceph-ansible to Cephadm

[wodel youchi]

Hi,

I am in the middle of migration from ceph-ansible to cephadm (version
quincy), so far so good ;-). And I have some questions :
- I still have the ceph-crash container, what should I do with it?
- The new rgw and mds daemons have some random string in their names (like
rgw.opsrgw.controllera.*pkajqw*), is this correct ?
- How should I proceed with the monitoring stack (grafana, prometheus,
alermanager and node-exporter)? should I stop then delete the old ones,
then deploy the new ones with ceph orch?

Regards.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Some questions about cephadm

[wodel youchi]

Hi;

So it was that, create the initial-ceph.conf and use the --config
Now All images are from the local registry.
Thank you all for your help.

Regards.

Le lun. 26 févr. 2024 à 14:09, wodel youchi  a
écrit :

> I've read that, but I didn't find how to use it?
> should I use the : --config *CONFIG_FILE *options?
>
> Le lun. 26 févr. 2024 à 13:59, Robert Sander 
> a écrit :
>
>> Hi,
>>
>> On 2/26/24 13:22, wodel youchi wrote:
>> >
>> > No didn't work, the bootstrap is still downloading the images from quay.
>>
>> For the image locations of the monitoring stack you have to create an
>> initical ceph.conf like it is mentioned in the chapter you referred
>> earlier:
>>
>> https://docs.ceph.com/en/reef/cephadm/install/#deployment-in-an-isolated-environment
>>
>> Regards
>> --
>> Robert Sander
>> Heinlein Consulting GmbH
>> Schwedter Str. 8/9b, 10119 Berlin
>>
>> https://www.heinlein-support.de
>>
>> Tel: 030 / 405051-43
>> Fax: 030 / 405051-19
>>
>> Amtsgericht Berlin-Charlottenburg - HRB 220009 B
>> Geschäftsführer: Peer Heinlein - Sitz: Berlin
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Some questions about cephadm

[wodel youchi]

I've read that, but I didn't find how to use it?
should I use the : --config *CONFIG_FILE *options?

Le lun. 26 févr. 2024 à 13:59, Robert Sander 
a écrit :

> Hi,
>
> On 2/26/24 13:22, wodel youchi wrote:
> >
> > No didn't work, the bootstrap is still downloading the images from quay.
>
> For the image locations of the monitoring stack you have to create an
> initical ceph.conf like it is mentioned in the chapter you referred
> earlier:
>
> https://docs.ceph.com/en/reef/cephadm/install/#deployment-in-an-isolated-environment
>
> Regards
> --
> Robert Sander
> Heinlein Consulting GmbH
> Schwedter Str. 8/9b, 10119 Berlin
>
> https://www.heinlein-support.de
>
> Tel: 030 / 405051-43
> Fax: 030 / 405051-19
>
> Amtsgericht Berlin-Charlottenburg - HRB 220009 B
> Geschäftsführer: Peer Heinlein - Sitz: Berlin
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Some questions about cephadm

[wodel youchi]

Hi,

No didn't work, the bootstrap is still downloading the images from quay.
PS : My local registry does not require any login/pass authentication, I
used fake ones since it's mandatory to give them.

cephadm --image 192.168.2.36:4000/ceph/ceph:v17 bootstrap --registry-url
192.168.2.36:4000  --registry-username admin --registry-password admin
--mon-ip 10.1.0.23 --cluster-network 10.2.0.0/16

[root@controllera ~]# podman images

REPOSITORYTAG IMAGE ID  CREATED
   SIZE
192.168.2.36:4000/ceph/ceph   v17 56993389bc29  11 days ago
   1.29 GB
quay.io/ceph/ceph-grafana 9.4.7   954c08fa6188  2 months ago
  647 MB
quay.io/prometheus/prometheus v2.43.0 a07b618ecd1d  11 months ago
 235 MB
quay.io/prometheus/alertmanager   v0.25.0 c8568f914cd2  14 months ago
 66.5 MB
quay.io/prometheus/node-exporter  v1.5.0  0da6a335fe13  15 months ago
 23.9 MB



Regards.

Le lun. 26 févr. 2024 à 11:42, Robert Sander 
a écrit :

> Hi,
>
> On 26.02.24 11:08, wodel youchi wrote:
>
> > Then I tried to deploy using this command on the admin node:
> > cephadm --image 192.168.2.36:4000/ceph/ceph:v17 bootstrap --mon-ip
> > 10.1.0.23 --cluster-network 10.2.0.0/16
> >
> > After the boot strap I found that it still downloads the images from the
> > internet, even the ceph image itself, I see two images one from my
> registry
> > the second from quay.
>
> To quote the docs: you can run cephadm bootstrap -h to see all of
> cephadm’s available options.
>
> These options are available:
>
>--registry-url REGISTRY_URL
>  url for custom registry
>--registry-username REGISTRY_USERNAME
>  username for custom registry
>--registry-password REGISTRY_PASSWORD
>  password for custom registry
>--registry-json REGISTRY_JSON
>  json file with custom registry login info (URL,
> Username, Password)
>
> Regards
> --
> Robert Sander
> Heinlein Consulting GmbH
> Schwedter Str. 8/9b, 10119 Berlin
>
> http://www.heinlein-support.de
>
> Tel: 030 / 405051-43
> Fax: 030 / 405051-19
>
> Zwangsangaben lt. §35a GmbHG:
> HRB 220009 B / Amtsgericht Berlin-Charlottenburg,
> Geschäftsführer: Peer Heinlein -- Sitz: Berlin
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Some questions about cephadm

[wodel youchi]

Thank you all for your help.

@Adam
From the reading you gave me I have understood the following :
1 - Set osd_memory_target_autotune to true then set
autotune_memory_target_ratio to 0.2
2 - Or do the math. For my setup I have 384Go per node, each node has 4
nvme disks of 7.6To, 0.2 of memory is 19.5G. So each OSD will have 19G of
memory.

Question : Should I take into account the size of the disk when calculating
the required memory for an OSD?


I have another problem, the local registry. I deployed a local registry
with the required images, then I used cephadm-ansible to prepare my hosts
and inject the local registry url into /etc/container/registry.conf file

Then I tried to deploy using this command on the admin node:
cephadm --image 192.168.2.36:4000/ceph/ceph:v17 bootstrap --mon-ip
10.1.0.23 --cluster-network 10.2.0.0/16

After the boot strap I found that it still downloads the images from the
internet, even the ceph image itself, I see two images one from my registry
the second from quay.

There is a section that talks about using a local registry here
https://docs.ceph.com/en/reef/cephadm/install/#deployment-in-an-isolated-environment,
but it's not clear especially about the other images. It talks about
preparing a temporary file named initial-ceph.conf, then it does not use
it???!!!

Could you help?

Regards.

Le jeu. 22 févr. 2024 à 11:10, Eugen Block  a écrit :

> Hi,
>
> just responding to the last questions:
>
> >- After the bootstrap, the Web interface was accessible :
> >   - How can I access the wizard page again? If I don't use it the
> first
> >   time I could not find another way to get it.
>
> I don't know how to recall the wizard, but you should be able to
> create a new dashboard user with your desired role (e. g.
> administrator) from the CLI:
>
> ceph dashboard ac-user-create  [] -i
> 
>
> >   - I had a problem with telemetry, I did not configure telemetry,
> then
> >   when I clicked the button, the web gui became
> inaccessible.!!!
>
> You can see what happened in the active MGR log.
>
> Zitat von wodel youchi :
>
> > Hi,
> >
> > I have some questions about ceph using cephadm.
> >
> > I used to deploy ceph using ceph-ansible, now I have to move to cephadm,
> I
> > am in my learning journey.
> >
> >
> >- How can I tell my cluster that it's a part of an HCI deployment?
> With
> >ceph-ansible it was easy using is_hci : yes
> >- The documentation of ceph does not indicate what versions of
> grafana,
> >prometheus, ...etc should be used with a certain version.
> >   - I am trying to deploy Quincy, I did a bootstrap to see what
> >   containers were downloaded and their version.
> >   - I am asking because I need to use a local registry to deploy
> those
> >   images.
> >- After the bootstrap, the Web interface was accessible :
> >   - How can I access the wizard page again? If I don't use it the
> first
> >   time I could not find another way to get it.
> >   - I had a problem with telemetry, I did not configure telemetry,
> then
> >   when I clicked the button, the web gui became
> inaccessible.!!!
> >
> >
> >
> > Regards.
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Some questions about cephadm

[wodel youchi]

Hi,

I have some questions about ceph using cephadm.

I used to deploy ceph using ceph-ansible, now I have to move to cephadm, I
am in my learning journey.


   - How can I tell my cluster that it's a part of an HCI deployment? With
   ceph-ansible it was easy using is_hci : yes
   - The documentation of ceph does not indicate what versions of grafana,
   prometheus, ...etc should be used with a certain version.
  - I am trying to deploy Quincy, I did a bootstrap to see what
  containers were downloaded and their version.
  - I am asking because I need to use a local registry to deploy those
  images.
   - After the bootstrap, the Web interface was accessible :
  - How can I access the wizard page again? If I don't use it the first
  time I could not find another way to get it.
  - I had a problem with telemetry, I did not configure telemetry, then
  when I clicked the button, the web gui became inaccessible.!!!



Regards.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Moving from ceph-ansible to cephadm and upgrading from pacific to octopus

[wodel youchi]

Hi,

I have an Openstack platform deployed with Yoga and ceph-ansible pacific on
Rocky 8.

Now I need to do an upgrade to Openstack zed with octopus on Rocky 9.

This is the path of the upgrade I have traced
- upgrade my nodes to Rocky 9 keeping Openstack yoga with ceph-ansible
pacific.
- convert ceph pacific from ceph-ansible to cephadm.
- stop Openstack platform yoga
- upgrade ceph pacific to octopus
- upgrade Openstack yoga to zed.

Any thoughts or guide lines to keep in mind and follow regarding ceph
convertion and upgrade.

Ps : on my ceph I have rbd, rgw and cephfs pools.

Regards.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Does ceph permit the definition of new classes?

[wodel youchi]

Hi,

Can I define new device classes in ceph, I know that there are hdd, ssd and
nvme, but can I define other classes?

Regards.


Virus-free.www.avast.com

<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: What is the best way to use disks with different sizes

[wodel youchi]

Hi and thanks,

Maybe I was not able to express myself correctly.

I have 3 nodes, and I will be using 3 replicas for the data, which will be
VMs disks.

*Each node has** 04 disks* :
- 03 nvme disks of 3.8Tb
- and 01 nvme disk of 7.6Tb

All three nodes are equivalent.

As mentioned above, one pool will suffice me for my VMs, my question is :
- Should I create two pools, the first one over the 3.8Tb disks (it will
use 9 disks with replicas 3) and the second pool over the 7.6Tb disks (it
will use 03 disks with replicas 3).
- Or, should I create one big pool and use all the 12 disks, mixing them,
despite the difference in size?


Regards.

<https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail>
Sans
virus.www.avast.com
<https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Le mar. 4 juil. 2023 à 15:32, Anthony D'Atri  a
écrit :

> There aren’t enough drives to split into multiple pools.
>
> Deploy 1 OSD on each of the 3.8T devices and 2 OSDs on each of the 7.6s.
>
> Or, alternately, 2 and 4.
>
>
> > On Jul 4, 2023, at 3:44 AM, Eneko Lacunza  wrote:
> >
> > Hi,
> >
> > El 3/7/23 a las 17:27, wodel youchi escribió:
> >> I will be deploying a Proxmox HCI cluster with 3 nodes. Each node has 3
> >> nvme disks of 3.8Tb each and a 4th nvme disk of 7.6Tb. Technically I
> need
> >> one pool.
> >>
> >> Is it good practice to use all disks to create the one pool I need, or
> is
> >> it better to create two pools, one on each group of disks?
> >>
> >> If the former is good (use all disks and create one pool), should I take
> >> into account the difference in disk size?
> >>
> >
> > What space use % do you expect? If you mix all disks in the same pool,
> if a 7.6TB disk fails that node's other disks will get full if use is near
> 60%, halting writes.
> >
> > With 2 pools, that would be "near 66%" for the 3.8T pool and no limit
> for 7.6TB (but in that case you'll only have 2 replicas with a disk
> failure).
> >
> > Another option would be 4 pools, in that case if a disk in any pool
> fails your VMs on that pool will continue working with only 2 replicas.
> >
> > For the "near" calculus, you must factor in nearfull and full ratios for
> OSDs, and also that data may be unevenly distributed among OSDs...
> >
> > The choice also will affect how well the aggregated IOPS will be spread
> between VMs<->disks.
> >
> > Cheers
> >
> > Eneko Lacunza
> > Zuzendari teknikoa | Director técnico
> > Binovo IT Human Project
> >
> > Tel. +34 943 569 206 | https://www.binovo.es
> > Astigarragako Bidea, 2 - 2º izda. Oficina 10-11, 20180 Oiartzun
> >
> > https://www.youtube.com/user/CANALBINOVO
> > https://www.linkedin.com/company/37269706/
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] What is the best way to use disks with different sizes

[wodel youchi]

Hi,

I will be deploying a Proxmox HCI cluster with 3 nodes. Each node has 3
nvme disks of 3.8Tb each and a 4th nvme disk of 7.6Tb. Technically I need
one pool.

Is it good practice to use all disks to create the one pool I need, or is
it better to create two pools, one on each group of disks?

If the former is good (use all disks and create one pool), should I take
into account the difference in disk size?

Regards.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] [Pacific] Admin keys no longer works I get access denied URGENT!!!

[wodel youchi]

Hi,

After a wrong manipulation, the admin key no longer works, it seems it has
been modified.

My cluster is built using containers.

When I execute ceph -s I get
[root@controllera ceph]# ceph -s
2023-05-31T11:33:20.940+0100 7ff7b2d13700 -1 monclient(hunting):
handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
2023-05-31T11:33:20.940+0100 7ff7b1d11700 -1 monclient(hunting):
handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
2023-05-31T11:33:20.940+0100 7ff7b2512700 -1 monclient(hunting):
handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
[errno 13] RADOS permission denied (error connecting to the cluster)

>From the log file I am getting :
May 31 11:03:02 controllera docker[214909]: debug
2023-05-31T11:03:02.714+0100 7fcfc0c91700  0 cephx server client.admin:
 unexpected key: req.key=5fea877f2a68548b expected_key=8c2074e03ffa449a

How can I recover the correct key?

Regards.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RBD mirroring, asking for clarification

[wodel youchi]

Hi,

The goal is to sync some VMs from site1 - to - site2 and vice-versa sync
some VMs in the other way.
I am thinking of using rdb mirroring for that. But I have little experience
with Ceph management.

I am searching for the best way to do that.

I could create two pools on each site, and cross sync the pools.
PoolA (site1)  -> PoolA (site2)
PoolB (site1) <-  PoolB (site2)

Or create one pool on each site and cross sync the VMs I need.
PoolA (site1) <-> PoolA (site2)


The first option seems to be the safest and the easiest to manage.

Regards.

<https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail>
Virus-free.www.avast.com
<https://www.avast.com/sig-email?utm_medium=email_source=link_campaign=sig-email_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

Le mer. 3 mai 2023 à 08:21, Eugen Block  a écrit :

> Hi,
>
> just to clarify, you mean in addition to the rbd mirroring you want to
> have another sync of different VMs between those clusters (potentially
> within the same pools) or are you looking for one option only? Please
> clarify. Anyway, I would use dedicated pools for rbd mirroring and
> then add more pools for different use-cases.
>
> Regards,
> Eugen
>
> Zitat von wodel youchi :
>
> > Hi,
> >
> > Thanks
> > I am trying to find out what is the best way to synchronize VMS between
> two
> > HCI Proxmox clusters.
> > Each cluster will contain 3 compute/storage nodes and each node will
> > contain 4 nvme osd disks.
> >
> > There will be a 10gbs link between the two platforms.
> >
> > The idea is to be able to sync VMS between the two platforms in case of
> > disaster bring the synced VMS up.
> >
> > Would you recommend to create a dedicated pool in each platform to
> > synchronization?
> >
> > Regards.
> >
> > On Tue, May 2, 2023, 13:30 Eugen Block  wrote:
> >
> >> Hi,
> >>
> >> while your assumptions are correct (you can use the rest of the pool
> >> for other non-mirrored images), at least I'm not aware of any
> >> limitations, can I ask for the motivation behind this question? Mixing
> >> different use-cases doesn't seem like a good idea to me. There's
> >> always a chance that a client with caps for that pool deletes or
> >> modifies images or even the entire pool. Why not simply create a
> >> different pool and separate those clients?
> >>
> >> Thanks,
> >> Eugen
> >>
> >> Zitat von wodel youchi :
> >>
> >> > Hi,
> >> >
> >> > When using rbd mirroring, the mirroring concerns the images only, not
> the
> >> > whole pool? So, we don't need to have a dedicated pool in the
> destination
> >> > site to be mirrored, the only obligation is that the mirrored pools
> must
> >> > have the same name.
> >> >
> >> > In other words, We create two pools with the same name, one on the
> source
> >> > site the other on the destination site, we create the mirror link (one
> >> way
> >> > or two ways replication), then we choose what images to sync.
> >> >
> >> > Both pools can be used simultaneously on both sites, it's the mirrored
> >> > images that cannot be used simultaneously, only promoted ones.
> >> >
> >> > Is this correct?
> >> >
> >> > Regards.
> >> > ___
> >> > ceph-users mailing list -- ceph-users@ceph.io
> >> > To unsubscribe send an email to ceph-users-le...@ceph.io
> >>
> >>
> >> ___
> >> ceph-users mailing list -- ceph-users@ceph.io
> >> To unsubscribe send an email to ceph-users-le...@ceph.io
> >>
>
>
>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: RBD mirroring, asking for clarification

[wodel youchi]

Hi,

Thanks
I am trying to find out what is the best way to synchronize VMS between two
HCI Proxmox clusters.
Each cluster will contain 3 compute/storage nodes and each node will
contain 4 nvme osd disks.

There will be a 10gbs link between the two platforms.

The idea is to be able to sync VMS between the two platforms in case of
disaster bring the synced VMS up.

Would you recommend to create a dedicated pool in each platform to
synchronization?

Regards.

On Tue, May 2, 2023, 13:30 Eugen Block  wrote:

> Hi,
>
> while your assumptions are correct (you can use the rest of the pool
> for other non-mirrored images), at least I'm not aware of any
> limitations, can I ask for the motivation behind this question? Mixing
> different use-cases doesn't seem like a good idea to me. There's
> always a chance that a client with caps for that pool deletes or
> modifies images or even the entire pool. Why not simply create a
> different pool and separate those clients?
>
> Thanks,
> Eugen
>
> Zitat von wodel youchi :
>
> > Hi,
> >
> > When using rbd mirroring, the mirroring concerns the images only, not the
> > whole pool? So, we don't need to have a dedicated pool in the destination
> > site to be mirrored, the only obligation is that the mirrored pools must
> > have the same name.
> >
> > In other words, We create two pools with the same name, one on the source
> > site the other on the destination site, we create the mirror link (one
> way
> > or two ways replication), then we choose what images to sync.
> >
> > Both pools can be used simultaneously on both sites, it's the mirrored
> > images that cannot be used simultaneously, only promoted ones.
> >
> > Is this correct?
> >
> > Regards.
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Re: Ceph recovery

[wodel youchi]

Thank you for the clarification.

On Mon, May 1, 2023, 20:11 Wesley Dillingham  wrote:

> Assuming size=3 and min_size=2 It will run degraded (read/write capable)
> until a third host becomes available at which point it will backfill the
> third copy on the third host. It will be unable to create the third copy of
> data if no third host exists. If an additional host is lost the data will
> become inactive+degraded (below min_size) and will be unavailable for use.
> Though data will not be lost assuming no further failures beyond the 2 full
> hosts occurs and again if the second and third host comes back the data
> will recover. Always best to have an additional host beyond the size
> setting for this reason.
>
> Respectfully,
>
> *Wes Dillingham*
> w...@wesdillingham.com
> LinkedIn <http://www.linkedin.com/in/wesleydillingham>
>
>
> On Mon, May 1, 2023 at 11:34 AM wodel youchi 
> wrote:
>
>> Hi,
>>
>> When creating a ceph cluster, a failover domain is created, and by default
>> it uses host as a minimal domain, that domain can be modified to chassis,
>> or rack, ...etc.
>>
>> My question is :
>> Suppose I have three osd nodes, my replication is 3 and my failover domain
>> is host, which means that each copy of data is stored on a different node.
>>
>> What happens when one node crashes, does Ceph use the remaining free space
>> on the other two to create the third copy, or the ceph cluster will run in
>> degraded mode, like a RAID5
>>  which lost a disk.
>>
>> Regards.
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
>>
>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Ceph recovery

[wodel youchi]

Hi,

When creating a ceph cluster, a failover domain is created, and by default
it uses host as a minimal domain, that domain can be modified to chassis,
or rack, ...etc.

My question is :
Suppose I have three osd nodes, my replication is 3 and my failover domain
is host, which means that each copy of data is stored on a different node.

What happens when one node crashes, does Ceph use the remaining free space
on the other two to create the third copy, or the ceph cluster will run in
degraded mode, like a RAID5
 which lost a disk.

Regards.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] RBD mirroring, asking for clarification

[wodel youchi]

Hi,

When using rbd mirroring, the mirroring concerns the images only, not the
whole pool? So, we don't need to have a dedicated pool in the destination
site to be mirrored, the only obligation is that the mirrored pools must
have the same name.

In other words, We create two pools with the same name, one on the source
site the other on the destination site, we create the mirror link (one way
or two ways replication), then we choose what images to sync.

Both pools can be used simultaneously on both sites, it's the mirrored
images that cannot be used simultaneously, only promoted ones.

Is this correct?

Regards.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Could you please explain the PG concept

[wodel youchi]

Hi,

I am learning Ceph and I am having a hard time understanding PG and PG
calculus .

I know that a PG is a collection of objects, and that PG are replicated
over the hosts to respect the replication size, but...

In traditional storage, we use size in Gb, Tb and so on, we create a pool
from a bunch of disks or raid arrays of some size then we create volumes of
a certain size and use them. If the storage is full we add disks, then we
extend our pools/volumes.
The idea of size is simple to understand.

Ceph, although it supports the notion of pool size in Gb, Tb ...etc. Pools
are created using PGs, and now there is also the notion of % of data.

When I use pg calc from ceph or from redhat, the generated yml file
contains the % variable, but the commands file contains only the PGs, and
when you are configuring 15% and 18% have the same number of PGs
???

The pg calc encourages you to create a %data multiple of 100, in other
words, it assumes that you know all your pools from the start. What if you
won't consume all your raw disk space.
What happens when you need to add a new pool?

Also when you create several pools, and then execute ceph osd df tree, you
can see that all pools show the raw size as a free space, it is like all
pools share the same raw space regardless of their PG number.

If someone can put some light on this concept and how to manage it wisely,
because the documentation keeps saying that it's an important concept, that
you have to pay attention when choosing the number of PGs for a pool from
the start.

Regards.


Virus-free.www.avast.com

<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

[ceph-users] Error deploying Ceph Qunicy using ceph-ansible 7 on Rocky 9

[wodel youchi]

Hi,

I am trying to deploy Ceph Quincy using ceph-ansible on Rocky9. I am having
some problems and I don't know where to search for the reason.

PS : I did the same deployment on Rocky8 using ceph-ansible for the Pacific
version on the same hardware and it worked perfectly.

I have 03 controllers nodes : mon, mgr, mdss and rgws
and 27 osd nodes : with 04 nvme disks (osd) each
I am using a 10Gb network with jumbo frames.

The deployment starts with no issues, the 03 monitors are created
correctly, then the 03 managers are created, after that the OSD are
prepared and formatted, until here everything is working fine, but when the
"*wait for all osd to be up*" task is launched, which means starting all
OSDs containers in all OSD nodes, things go south, the monitors become out
of quorum, ceph -s takes a lot of time to respond and not all OSDs are
being activated, and the deployment fails at the end.

cluster 2023-03-06T12:00:26.431947+0100 mon.controllera (mon.0) 3864 :
cluster [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum controllera,controllerc
cluster 2023-03-06T12:00:26.431953+0100 mon.controllera (mon.0) 3865 :
cluster [WRN] mon.controllerb (rank 1) addr [v2:
20.1.0.27:3300/0,v1:20.1.0.27:6789/0] is down (out of quorum)

The monitor container in 2 of my controllers nodes stays at 100% of cpu
utilization.

CONTAINER ID   NAME   CPU % MEM USAGE / LIMIT
MEM % NET I/O   BLOCK I/OPIDS
068e4e55f299   ceph-mon-controllera   99.91%58.12MiB / 376.1GiB
0.02% 0B / 0B   122MB / 85.3MB   28  <-
87730f89420d   ceph-mgr-controllera   0.32% 408.2MiB / 376.1GiB
0.11% 0B / 0B   181MB / 0B   35

Could that be a resource problem? the monitor containers do not have enough
resources CPU, RAM, ...etc to handle all the OSDs that are being started?
If yes, how may I find this?

thanks in advance.

Regards.
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io