Some thoughts:
* Do you have any error messages form the MDS daemons?
https://docs.ceph.com/en/latest/cephadm/troubleshooting/#gathering-log-files
* Do you have any error messages form the OSDs?
* What do you mean by "osd podman object"?
* Try downgrading to 3.0.1
Am 25.10.21 um 23:05 schrieb Magnus Harlander:
> Hi,
>
> after converting my 2 node cluster to cephadm I'm in lots of trouble.
>
> - containerized mds are not available in the cluster. I must
> run mds from systemd to make my fs available.
>
> - osd podman objects are not found after a reboot of one node. I
> don't want to test it on the second node, because if it looses
> the podman config as well, my cluster is dead!
>
> Can anybody help me?
>
> Best regards Magnus
>
> Output from cephadm.log:
>
> cephadm ['--image', 'docker.io/ceph/ceph:v15', '--no-container-init', 'ls']
> 2021-10-25 22:47:01,929 DEBUG container_init=False
> 2021-10-25 22:47:01,929 DEBUG Running command: systemctl is-enabled
> ceph-mds@s1
> 2021-10-25 22:47:01,935 DEBUG systemctl: stdout enabled
> 2021-10-25 22:47:01,935 DEBUG Running command: systemctl is-active
> ceph-mds@s1
> 2021-10-25 22:47:01,940 DEBUG systemctl: stdout active
> 2021-10-25 22:47:01,940 DEBUG Running command: ceph -v
> 2021-10-25 22:47:02,009 DEBUG ceph: stdout ceph version 15.2.13
> (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable)
> 2021-10-25 22:47:02,016 DEBUG Running command: systemctl is-enabled
> ceph-osd@1
> 2021-10-25 22:47:02,024 DEBUG systemctl: stdout disabled
> 2021-10-25 22:47:02,024 DEBUG Running command: systemctl is-active
> ceph-osd@1
> 2021-10-25 22:47:02,031 DEBUG systemctl: stdout inactive
> 2021-10-25 22:47:02,031 DEBUG Running command: systemctl is-enabled
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@mon.s1
> 2021-10-25 22:47:02,036 DEBUG systemctl: stdout enabled
> 2021-10-25 22:47:02,036 DEBUG Running command: systemctl is-active
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@mon.s1
> 2021-10-25 22:47:02,041 DEBUG systemctl: stdout active
> 2021-10-25 22:47:02,041 DEBUG Running command: /bin/podman --version
> 2021-10-25 22:47:02,068 DEBUG /bin/podman: stdout podman version 3.2.3
> 2021-10-25 22:47:02,069 DEBUG Running command: /bin/podman inspect
> --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
> .Config.Labels "io.ceph.version"}}
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-mon.s1
> 2021-10-25 22:47:02,111 DEBUG /bin/podman: stdout
> 1fd6debed26923212e3b3b263e88505d2b70fe024b7a1c01105299bb746d7c48,docker.io/ceph/ceph:v15,2cf504fded3980c76b59a354fca8f301941f86e369215a08752874d1ddb69b73,2021-10-25
> 22:45:39.315992691 +0200 CEST,
> 2021-10-25 22:47:02,203 DEBUG Running command: /bin/podman exec
> 1fd6debed26923212e3b3b263e88505d2b70fe024b7a1c01105299bb746d7c48 ceph -v
> 2021-10-25 22:47:02,356 DEBUG /bin/podman: stdout ceph version 15.2.13
> (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable)
> 2021-10-25 22:47:02,434 DEBUG Running command: systemctl is-enabled
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@mgr.s1
> 2021-10-25 22:47:02,440 DEBUG systemctl: stdout enabled
> 2021-10-25 22:47:02,440 DEBUG Running command: systemctl is-active
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@mgr.s1
> 2021-10-25 22:47:02,445 DEBUG systemctl: stdout active
> 2021-10-25 22:47:02,445 DEBUG Running command: /bin/podman --version
> 2021-10-25 22:47:02,468 DEBUG /bin/podman: stdout podman version 3.2.3
> 2021-10-25 22:47:02,470 DEBUG Running command: /bin/podman inspect
> --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
> .Config.Labels "io.ceph.version"}}
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-mgr.s1
> 2021-10-25 22:47:02,513 DEBUG /bin/podman: stdout
> 2b08ddb6182e14985939e50a00e2306a77c3068a65ed122dab8bd5604c91af65,docker.io/ceph/ceph:v15,2cf504fded3980c76b59a354fca8f301941f86e369215a08752874d1ddb69b73,2021-10-25
> 22:45:39.535973449 +0200 CEST,
> 2021-10-25 22:47:02,601 DEBUG Running command: systemctl is-enabled
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.0
> 2021-10-25 22:47:02,607 DEBUG systemctl: stdout enabled
> 2021-10-25 22:47:02,607 DEBUG Running command: systemctl is-active
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.0
> 2021-10-25 22:47:02,612 DEBUG systemctl: stdout activating
> 2021-10-25 22:47:02,612 DEBUG Running command: /bin/podman --version
> 2021-10-25 22:47:02,636 DEBUG /bin/podman: stdout podman version 3.2.3
> 2021-10-25 22:47:02,637 DEBUG Running command: /bin/podman inspect
> --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
> .Config.Labels "io.ceph.version"}}
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.0
> 2021-10-25 22:47:02,709 DEBUG /bin/podman: stderr Error: error
> inspecting object: no such object:
> "ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.0"
> 2021-10-25 22:47:02,712 DEBUG Running command: systemctl is-enabled
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.2
> 2021-10-25 22:47:02,718 DEBUG systemctl: stdout enabled
> 2021-10-25 22:47:02,718 DEBUG Running command: systemctl is-active
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.2
> 2021-10-25 22:47:02,723 DEBUG systemctl: stdout activating
> 2021-10-25 22:47:02,723 DEBUG Running command: /bin/podman --version
> 2021-10-25 22:47:02,747 DEBUG /bin/podman: stdout podman version 3.2.3
> 2021-10-25 22:47:02,748 DEBUG Running command: /bin/podman inspect
> --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
> .Config.Labels "io.ceph.version"}}
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.2
> 2021-10-25 22:47:02,821 DEBUG /bin/podman: stderr Error: error
> inspecting object: no such object:
> "ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.2"
> 2021-10-25 22:47:02,823 DEBUG Running command: systemctl is-enabled
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.6
> 2021-10-25 22:47:02,830 DEBUG systemctl: stdout enabled
> 2021-10-25 22:47:02,830 DEBUG Running command: systemctl is-active
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.6
> 2021-10-25 22:47:02,835 DEBUG systemctl: stdout activating
> 2021-10-25 22:47:02,835 DEBUG Running command: /bin/podman --version
> 2021-10-25 22:47:02,860 DEBUG /bin/podman: stdout podman version 3.2.3
> 2021-10-25 22:47:02,861 DEBUG Running command: /bin/podman inspect
> --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
> .Config.Labels "io.ceph.version"}}
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.6
> 2021-10-25 22:47:02,948 DEBUG /bin/podman: stderr Error: error
> inspecting object: no such object:
> "ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.6"
> 2021-10-25 22:47:02,950 DEBUG Running command: systemctl is-enabled
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.7
> 2021-10-25 22:47:02,956 DEBUG systemctl: stdout enabled
> 2021-10-25 22:47:02,957 DEBUG Running command: systemctl is-active
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.7
> 2021-10-25 22:47:02,961 DEBUG systemctl: stdout activating
> 2021-10-25 22:47:02,962 DEBUG Running command: /bin/podman --version
> 2021-10-25 22:47:02,986 DEBUG /bin/podman: stdout podman version 3.2.3
> 2021-10-25 22:47:02,988 DEBUG Running command: /bin/podman inspect
> --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
> .Config.Labels "io.ceph.version"}}
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.7
> 2021-10-25 22:47:03,063 DEBUG /bin/podman: stderr Error: error
> inspecting object: no such object:
> "ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.7"
> 2021-10-25 22:47:03,066 DEBUG Running command: systemctl is-enabled
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.8
> 2021-10-25 22:47:03,072 DEBUG systemctl: stdout enabled
> 2021-10-25 22:47:03,072 DEBUG Running command: systemctl is-active
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524@osd.8
> 2021-10-25 22:47:03,077 DEBUG systemctl: stdout activating
> 2021-10-25 22:47:03,077 DEBUG Running command: /bin/podman --version
> 2021-10-25 22:47:03,099 DEBUG /bin/podman: stdout podman version 3.2.3
> 2021-10-25 22:47:03,101 DEBUG Running command: /bin/podman inspect
> --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
> .Config.Labels "io.ceph.version"}}
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.8
> 2021-10-25 22:47:03,174 DEBUG /bin/podman: stderr Error: error
> inspecting object: no such object:
> "ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-osd.8"
> 2021-10-25 22:47:03,176 DEBUG Running command: systemctl is-enabled
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae...@mds.fs.s1.ajxyaf
> 2021-10-25 22:47:03,182 DEBUG systemctl: stdout enabled
> 2021-10-25 22:47:03,182 DEBUG Running command: systemctl is-active
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae...@mds.fs.s1.ajxyaf
> 2021-10-25 22:47:03,187 DEBUG systemctl: stdout active
> 2021-10-25 22:47:03,187 DEBUG Running command: /bin/podman --version
> 2021-10-25 22:47:03,210 DEBUG /bin/podman: stdout podman version 3.2.3
> 2021-10-25 22:47:03,212 DEBUG Running command: /bin/podman inspect
> --format {{.Id}},{{.Config.Image}},{{.Image}},{{.Created}},{{index
> .Config.Labels "io.ceph.version"}}
> ceph-86bbd6c5-ae96-4c78-8a5e-50623f0ae524-mds.fs.s1.ajxyaf
> 2021-10-25 22:47:03,256 DEBUG /bin/podman: stdout
> 818588b3702cbed9ff38abb07ae39e08f9700b8c8740f40f81641d0129bd8338,docker.io/ceph/ceph:v15,2cf504fded3980c76b59a354fca8f301941f86e369215a08752874d1ddb69b73,2021-10-25
> 22:45:39.314981699 +0200 CEST,
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
