Re: [ceph-users] Use telegraf/influx to detect problems is very difficult
As I mentioned yesterday here, there is an issue with current scheme of
metrics.
With current scheme you cannot do simple math like
> SELECT num_osd - num_osd_up FROM "ceph_cluster_stats"
Instead you will need query like
> SELECT (SELECT last("value") FROM "ceph_cluster_stats" WHERE
"type_instance" = 'num_osd') - (SELECT last("value") FROM
"ceph_cluster_stats" WHERE "type_instance" = 'num_osd_up')
which is not supported by InfluxDB.
I know this type of queries works perfectly in prometheus or SQL world,
but AFAIK you unfortunately cannot easily combine multiple series in
InfluxDB.
To Mario's issue with alerting - maybe you can try to use kapacitor for
alerting purposes. I have no direct experiences with it, but it should
be easily controlled via chronograf and could solve your issue.
M.
On 12/11/19 4:58 AM, Konstantin Shalygin wrote:
>
>> But it is very difficult/complicated to make simple queries because, for
>> example I have osd up and osd total but not osd down metric.
>>
> To determine how much osds down you don't need special metric, because
> you already
>
> have osd_up and osd_in metrics. Just use math.
>
>
>
>
> k
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
--
Miroslav Kalina
Systems developement specialist
miroslav.kal...@livesport.eu
+420 773 071 848
Livesport s.r.o.
Aspira Business Centre
Bucharova 2928/14a, 158 00 Praha 5
www.livesport.eu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] //?? // ceph-mon is blocked after shutting down and ip address changed
Hi,daemon is running when using admin socket
[root@ceph-node1 ceph]# ceph --admin-daemon
/var/run/ceph/ceph-mon.ceph-node1.asok mon_status
{
"name": "ceph-node1",
"rank": 0,
"state": "leader",
"election_epoch": 63,
"quorum": [
0
],
"quorum_age": 40839,
"features": {
"required_con": "2449958747315912708",
"required_mon": [
"kraken",
"luminous",
"mimic",
"osdmap-prune",
"nautilus"
],
"quorum_con": "4611087854031667199",
"quorum_mon": [
"kraken",
"luminous",
"mimic",
"osdmap-prune",
"nautilus"
]
},
"outside_quorum": [],
"extra_probe_peers": [],
"sync_provider": [],
"monmap": {
"epoch": 20,
"fsid": "e384e8e6-94d5-4812-bfbb-d1b0468bdef5",
"modified": "2019-12-10 23:15:34.203415",
"created": "2019-12-09 08:28:04.181288",
"min_mon_release": 14,
"min_mon_release_name": "nautilus",
"features": {
"persistent": [
"kraken",
"luminous",
"mimic",
"osdmap-prune",
"nautilus"
],
"optional": []
},
"mons": [
{
"rank": 0,
"name": "ceph-node1",
"public_addrs": {
"addrvec": [
{
"type": "v1",
"addr": "192.168.1.6:6789",
"nonce": 0
}
]
},
"addr":
"192.168.1.6:6789/0",
"public_addr":
"192.168.1.6:6789/0"
}
]
},
"feature_map": {
"mon": [
{
"features":
"0x3ffddff8ffac",
"release": "luminous",
"num": 1
}
]
}
}
[root@ceph-node1 ceph]# ceph --admin-daemon
/var/run/ceph/ceph-mon.ceph-node1.asok quorum_status
{
"election_epoch": 63,
"quorum": [
0
],
"quorum_names": [
"ceph-node1"
],
"quorum_leader_name": "ceph-node1",
"quorum_age": 40854,
"monmap": {
"epoch": 20,
"fsid": "e384e8e6-94d5-4812-bfbb-d1b0468bdef5",
"modified": "2019-12-10 23:15:34.203415",
"created": "2019-12-09 08:28:04.181288",
"min_mon_release": 14,
"min_mon_release_name": "nautilus",
"features": {
"persistent": [
"kraken",
"luminous",
"mimic",
"osdmap-prune",
"nautilus"
],
"optional": []
},
"mons": [
{
"rank": 0,
"name": "ceph-node1",
"public_addrs": {
"addrvec": [
{
"type": "v1",
"addr": "192.168.1.6:6789",
"nonce": 0
}
]
},
"addr":
"192.168.1.6:6789/0",
"public_addr":
"192.168.1.6:6789/0"
}
]
}
}
Quoting Cc?? (occj at qq.com): > 4.[root at ceph-node1 ceph]# ceph -s >
just blocked ... > error 110 after a few hours Is the daemon running? You
can check for the process to be alive in /var/run/ceph/ceph-mon.$hostname.asok
If so ... then query the monitor for its status: ceph daemon mon.$hostname
quorum_status If there is no monitor in quorum ... then that's your problem.
See [1] for more info on debugging the monitor. Gr. Stefan [1]:
https://docs.ceph.com/docs/nautilus/rados/troubleshooting/troubleshooting-mon/___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph assimilated configuration - unable to remove item
Hi, We assimilated our Ceph configuration to store attributes within Ceph itself and subsequently have a minimal configuration file. Whilst this works perfectly we are unable to remove configuration entries populated by the assimilate-conf command. Ceph Nautilus 14.2.4.1 upgrade notes: cd /etc/pve; ceph config assimilate-conf -i ceph.conf -o ceph.conf.new; mv ceph.conf.new ceph.conf; pico /etc/ceph/ceph.conf # add back: cluster_network # public_network ceph config rm global cluster_network; ceph config rm global public_network; ceph config set global mon_osd_down_out_subtree_limit host; Resulting minimal Ceph configuration file: [admin@kvm1c ~]# cat /etc/ceph/ceph.conf [global] cluster_network = 10.248.1.0/24 filestore_xattr_use_omap = true fsid = 31f6ea46-12cb-47e8-a6f3-60fb6bbd1782 mon_host = 10.248.1.60 10.248.1.61 10.248.1.62 public_network = 10.248.1.0/24 [client] keyring = /etc/pve/priv/$cluster.$name.keyring Ceph configuration entries: [admin@kvm1c ~]# ceph config dump WHOMASK LEVELOPTION VALUE RO global advanced auth_client_required cephx * global advanced auth_cluster_required cephx * global advanced auth_service_required cephx * global advanced cluster_network10.248.1.0/24 * global advanced debug_filestore0/0 global advanced debug_journal 0/0 global advanced debug_ms 0/0 global advanced debug_osd 0/0 global basicdevice_failure_prediction_mode cloud global advanced mon_allow_pool_delete true global advanced mon_osd_down_out_subtree_limit host global advanced osd_deep_scrub_interval1209600.00 global advanced osd_pool_default_min_size 2 global advanced osd_pool_default_size 3 global advanced osd_scrub_begin_hour 19 global advanced osd_scrub_end_hour 6 global advanced osd_scrub_sleep0.10 global advanced public_network 10.248.1.0/24 * global advanced rbd_default_features 7 global advanced rbd_default_features 31 mgr advanced mgr/balancer/activetrue mgr advanced mgr/balancer/mode upmap mgr advanced mgr/devicehealth/enable_monitoring true Note the duplicate 'rdb_default_features' entry. We've switched to kernel 5.3 which supports object-map and fast-diff and subsequently wanted to change the default features for new RBD images to reflect this. Commands we entered to get here: [admin@kvm1b ~]# ceph config dump | grep -e WHO -e rbd_default_features WHOMASK LEVELOPTION VALUE RO global advanced rbd_default_features 7 [admin@kvm1b ~]# ceph config rm global rbd_default_features [admin@kvm1b ~]# ceph config rm global rbd_default_features [admin@kvm1b ~]# ceph config rm global rbd_default_features [admin@kvm1b ~]# ceph config dump | grep -e WHO -e rbd_default_features WHOMASK LEVELOPTION VALUE RO global advanced rbd_default_features 7 [admin@kvm1b ~]# ceph config set global rbd_default_features 31 [admin@kvm1b ~]# ceph config dump | grep -e WHO -e rbd_default_features WHOMASK LEVELOPTION VALUE RO global advanced rbd_default_features 7 global advanced rbd_default_features 31 Regards David Herselman ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] CephFS "denied reconnect attempt" after updating Ceph
It seems like this has been fixed with client kernel version 4.19.0-0.bpo.5-amd64. -- Groeten, William Edwards Tuxis Internet Engineering - Originele bericht - Van: William Edwards (wedwa...@tuxis.nl) Datum: 08/13/19 16:46 Naar: ceph-users@lists.ceph.com Onderwerp: [ceph-users] CephFS "denied reconnect attempt" after updating Ceph Hello, I've been using CephFS for quite a while now, and am very happy with it. However, I'm experiencing an issue that's quite hard to debug. On almost every server where CephFS is mounted, the CephFS mount becomes unusable after updating Ceph (has happened 3 times now, after Ceph update). When attempting to access the CephFS mount, I'd get a permission denied: root@cephmgr:~# cd /mnt/cephfs -bash: cd: /mnt/cephfs: Permission denied [root@da0 ~]# ls /home ls: cannot access /home: Permission denied root@http-cap01:~# ls /var/www/vhosts ls: cannot access '/var/www/vhosts': Permission denied What strikes me as odd is that on some machines, the mount works fine, and I get above error on other machines while server configurations are identical. Here's a log of a client machine where the CephFS mount 'broke' after updating Ceph: -- Jun 12 00:56:47 http-cap01 kernel: [4163998.163498] ceph: mds0 caps stale Jun 12 01:00:20 http-cap01 kernel: [4164210.581767] ceph: mds0 caps went stale, renewing Jun 12 01:00:20 http-cap01 kernel: [4164210.581771] ceph: mds0 caps stale Jun 12 01:00:45 http-cap01 kernel: [4164236.434456] libceph: mds0 [fdb7:b01e:7b8e:0:10:10:10:3]:6816 socket closed (con state OPEN) Jun 12 01:00:46 http-cap01 kernel: [4164236.980990] libceph: reset on mds0 Jun 12 01:00:46 http-cap01 kernel: [4164236.980996] ceph: mds0 closed our session Jun 12 01:00:46 http-cap01 kernel: [4164236.980997] ceph: mds0 reconnect start Jun 12 01:00:46 http-cap01 kernel: [4164237.035294] ceph: mds0 reconnect denied Jun 12 01:00:46 http-cap01 kernel: [4164237.036990] libceph: mds0 [fdb7:b01e:7b8e:0:10:10:10:3]:6816 socket closed (con state NEGOTIATING) Jun 12 01:00:47 http-cap01 kernel: [4164237.972853] ceph: mds0 rejected session Jun 12 01:05:15 http-cap01 kernel: [4164506.065665] libceph: mon0 [fdb7:b01e:7b8e:0:10:10:10:1]:6789 session lost, hunting for new mon Jun 12 01:05:15 http-cap01 kernel: [4164506.068613] libceph: mon1 [fdb7:b01e:7b8e:0:10:10:10:2]:6789 session established Jun 12 01:06:47 http-cap01 kernel: [4164597.858261] libceph: mon1 [fdb7:b01e:7b8e:0:10:10:10:2]:6789 socket closed (con state OPEN) Jun 12 01:06:47 http-cap01 kernel: [4164597.858323] libceph: mon1 [fdb7:b01e:7b8e:0:10:10:10:2]:6789 session lost, hunting for new mon Jun 12 01:06:47 http-cap01 kernel: [4164597.864745] libceph: mon2 [fdb7:b01e:7b8e:0:10:10:10:3]:6789 session established Jun 12 01:23:02 http-cap01 kernel: [4165572.915743] ceph: mds0 reconnect start Jun 12 01:23:02 http-cap01 kernel: [4165572.918197] ceph: mds0 reconnect denied Jun 12 01:23:15 http-cap01 kernel: [4165586.526195] libceph: mds0 [fdb7:b01e:7b8e:0:10:10:10:2]:6817 socket closed (con state NEGOTIATING) Jun 12 01:23:16 http-cap01 kernel: [4165586.992411] ceph: mds0 rejected session -- Note the "ceph: mds0 reconnect denied" Log on a machine where the CephFS mount kept working fine: -- Jun 12 01:08:26 http-hlp02 kernel: [3850613.358329] libceph: mon0 [fdb7:b01e:7b8e:0:10:10:10:1]:6789 socket closed (con state OPEN) Jun 12 01:08:26 http-hlp02 kernel: [3850613.358418] libceph: mon0 [fdb7:b01e:7b8e:0:10:10:10:1]:6789 session lost, hunting for new mon Jun 12 01:08:26 http-hlp02 kernel: [3850613.369597] libceph: mon1 [fdb7:b01e:7b8e:0:10:10:10:2]:6789 session established Jun 12 01:09:50 http-hlp02 kernel: [3850697.708357] libceph: osd9 [fdb7:b01e:7b8e:0:10:10:10:1]:6806 socket closed (con state OPEN) Jun 12 01:09:50 http-hlp02 kernel: [3850697.709897] libceph: osd0 down Jun 12 01:09:50 http-hlp02 kernel: [3850697.709898] libceph: osd1 down Jun 12 01:09:50 http-hlp02 kernel: [3850697.709899] libceph: osd6 down Jun 12 01:09:50 http-hlp02 kernel: [3850697.709899] libceph: osd9 down Jun 12 01:12:37 http-hlp02 kernel: [3850864.673357] libceph: osd9 up Jun 12 01:12:37 http-hlp02 kernel: [3850864.673378] libceph: osd6 up Jun 12 01:12:37 http-hlp02 kernel: [3850864.673394] libceph: osd0 up Jun 12 01:12:37 http-hlp02 kernel: [3850864.673402] libceph: osd1 up Jun 12 01:14:30 http-hlp02 kernel: [3850977.916749] libceph: wrong peer, want [fdb7:b01e:7b8e:0:10:10:10:2]:6808/434742, got [fdb7:b01e:7b8e:0:10:10:10:2]:6808/19887 Jun 12 01:14:30 http-hlp02 kernel: [3850977.916765] libceph: osd10 [fdb7:b01e:7b8e:0:10:10:10:2]:6808 wrong peer at address Jun 12 01:14:30 http-hlp02 kernel: [3850977.918368] libceph: osd4 down Jun 12 01:14:30 http-hlp02 kernel: [3850977.918369] libceph: osd5 down Jun 12 01:14:30 http-hlp02 kernel: [3850977.918370] libceph: osd7 down Jun 12 01:14:30 http-hlp02 kernel: [3850977.918370] libceph: osd10 down Jun 12 01:14:30 http-hlp02 kernel: [3850977.918401] libceph: osd10 up Jun 12 01:14:30 http-hlp02
Re: [ceph-users] Ceph assimilated configuration - unable to remove item
Quoting David Herselman (d...@syrex.co): > Hi, > > We assimilated our Ceph configuration to store attributes within Ceph > itself and subsequently have a minimal configuration file. Whilst this > works perfectly we are unable to remove configuration entries > populated by the assimilate-conf command. I forgot about this issue, but I encountered this when we upgraded to mimic. I can confirm this bug. It's possible to have the same key present with different values. For our production cluster we decided to stick to ceph.conf for the time being. That's also the workaround for now if you want to override the config store: just put that in your config file and reboot the daemon(s). Gr. Stefan -- | BIT BV https://www.bit.nl/Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / i...@bit.nl ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] //： // ceph-mon is blocked after shutting down and ip address changed
Quoting Cc君 (o...@qq.com):
> Hi,daemon is running when using admin socket
> [root@ceph-node1 ceph]# ceph --admin-daemon
> /var/run/ceph/ceph-mon.ceph-node1.asok mon_status
> {
> "name": "ceph-node1",
> "rank": 0,
> "state": "leader",
> "election_epoch": 63,
> "quorum": [
> 0
> ],
> "quorum_age": 40839,
Your ceph.conf show that messenger should listen on 3300 (v2) and 6789
(v1). If only 6789 is actually listening ... and the client tries to
connect to 3300 ... you might get a timeout as well. Not sure if
messenger falls back to v1.
What happens when you change ceph.conf (first without restarting the
mon) and try a "ceph -s" again with a ceph client on the monitor node?
Gr. Stefan
--
| BIT BV https://www.bit.nl/Kamer van Koophandel 09090351
| GPG: 0xD14839C6 +31 318 648 688 / i...@bit.nl
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] It works !Re?? //?? // ceph-mon is blocked after shutting down and ip address changed
It works , after I remove v2 address in ceph.conf.
Hope I see. Thank you !
[root@ceph-node1 ceph]# ceph -s
cluster:
id: e384e8e6-94d5-4812-bfbb-d1b0468bdef5
health: HEALTH_WARN
1 MDSs report slow metadata IOs
noout,nobackfill,norecover flag(s) set
9 osds down
no active mgr
Reduced data availability: 102 pgs
inactive, 128 pgs down, 8 pgs stale
Degraded data redundancy:
3664/29810955 objects degraded (0.012%), 28 pgs degraded, 29 pgs undersized
1 monitors have not enabled msgr2
services:
mon: 1 daemons, quorum ceph-node1 (age 13h)
mgr: no daemons active (since 3d)
mds: cephfs:1 {0=ceph-node1=up:active(laggy or crashed)}
osd: 12 osds: 3 up (since 3d), 12 in (since 3d)
flags noout,nobackfill,norecover
rgw: 1 daemon active (ceph-node1.rgw0)
data:
pools: 6 pools, 168 pgs
objects: 2.49M objects, 9.3 TiB
usage: 14 TiB used, 74 TiB / 88 TiB avail
pgs: 97.024% pgs not active
3664/29810955 objects degraded
(0.012%)
128 down
13
stale+undersized+degraded+peered
11
undersized+degraded+peered
6
stale+undersized+peered
5 undersized+peered
4
active+undersized+degraded
1 active+undersized
-- --
??: "Stefan Kooman"https://www.bit.nl/ ;
Kamer van Koophandel 09090351
| GPG:
0xD14839C6
+31 318 648 688 / i...@bit.nl___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Use telegraf/influx to detect problems is very difficult
Miroslav replied better for us why "is not so simple" to use math. And osd down was the easiest. How can I calculate: - monitor down - osd near full ? I do not understand why ceph plugin cannot send to influx all the metrics it has, especially the most useful for creating alarms. Il giorno mer 11 dic 2019 alle ore 04:58 Konstantin Shalygin ha scritto: > But it is very difficult/complicated to make simple queries because, for > example I have osd up and osd total but not osd down metric. > > > To determine how much osds down you don't need special metric, because you > already > > have osd_up and osd_in metrics. Just use math. > > > > > k > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] HA and data recovery of CEPH
Thanks to all, now we can make that duration to 25 seconds around, this is the best result as we can. BR On Tue, Dec 3, 2019 at 10:30 PM Wido den Hollander wrote: > > > On 12/3/19 3:07 PM, Aleksey Gutikov wrote: > > > >> That is true. When an OSD goes down it will take a few seconds for it's > >> Placement Groups to re-peer with the other OSDs. During that period > >> writes to those PGs will stall for a couple of seconds. > >> > >> I wouldn't say it's 40s, but it can take ~10s. > > > > Hello, > > > > According to my experience, in case of OSD crashes, killed -9 (any kind > > abnormat termination) OSD failure handling contains next steps: > > 1) Failed OSD's peers detect that it does not respond - it can take up > > to osd_heartbeat_grace + osd_heartbeat_interval seconds > > If a 'Connection Refused' is detected the OSD will be marked as down > right away. > > > 2) Peers send reports to monitor > > 3) Monitor makes a decision according to (options from it's own config) > > mon_osd_adjust_heartbeat_grace, osd_heartbeat_grace, > > mon_osd_laggy_halflife, mon_osd_min_down_reporters, ... And finally mark > > OSD down in osdmap. > > True. > > > 4) Monitor send updated OSDmap to OSDs and clients > > 5) OSDs starting peering > > 5.1) Peering itself is complicated process, for example we had > > experienced PGs stuck in inactive state due to > > osd_max_pg_per_osd_hard_ratio. > > I would say that 5.1 isn't relevant for most cases. Yes, it can happen, > but it's rare. > > > 6) Peering finished (PGs' data continue moving) - clients can normally > > access affected PGs. Clients also have their own timeouts that can > > affect time to recover. > > > Again, according to my experience, 40s with default settings is possible. > > > > 40s is possible in certain scenarios. But I wouldn't say that's the > default for all cases. > > Wido > > > > -- The modern Unified Communications provider https://www.portsip.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Cluster in ERR status when rebalancing
Has finally been addressed in 14.2.5, check changelog of release. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] RBD Object-Map Usuage incorrect
Due to the recent 5.3.x kernel having support for Object-Map and other features required in KRBD I have now enabled object-map,fast-diff on some RBD images with CEPH (14.2.5), I have rebuilt the object map using "rbd object-map rebuild" However for some RBD images, the Provisioned/Total Provisioned then listed in the Ceph MGR for some images is the full RBD size and not the true size reflected in a VM using df -h, I have discard enabled and have run fstrim but I know that for example a 20TB RBD has never gone above the current 9TB shown in df -h but in CEPH MGR shows as 20TB under Provisioned/Total Provisioned. Not sure if I am hitting a bug? Or if this is expected behavior? Thanks___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
