[ceph-users] Ceph community - how to make it even stronger

[jesper]

Hi All.

I was reading up and especially the thread on upgrading to mimic and
stable releases - caused me to reflect a bit on our ceph journey so far.

We started approximately 6 months ago - with CephFS as the dominant
use case in our HPC setup - starting at 400TB useable capacity and
as is matures going towards 1PB - mixed slow and SSD.

Some of the first confusions was.
bluestore vs. filestore - what was the recommendation actually?
Figuring out what kernel clients are useable with CephFS - and what
kernels to use on the other end?
Tuning of the MDS ?
Imbalace of OSD nodes rendering the cluster down - how to balance?
Triggering kernel bugs in the kernel client during OSD_FULL ?

This mailing list has been very responsive to the questions, thanks for
that.

But - compared to other open source projects we're lacking a bit of
infrastructure and guidance here.

I did check:
- http://tracker.ceph.com/projects/ceph/wiki/Wiki => Which does not seem
to be operational.
- http://docs.ceph.com/docs/mimic/start/get-involved/
Gmane is probably not coming back - waiting 2 years now, can we easily get
the mailinglist archives indexed otherwise.

I feel that the wealth of knowledge being build up around operating ceph
is not really captured to make the next users journey - better and easier.

I would love to help out - hey - I end up spending the time anyway, but
some guidance on how to do it may help.

I would suggest:

1) Dump a 1-3 monthly status email on the project to the respective
mailing lists => Major releases, Conferences, etc
2) Get the wiki active - one of the main things I want to know about when
messing with the storage is - What is working for other people - just a
page where people can dump an aggregated output of their ceph cluster and
write 2-5 lines about the use-case for it.
3) Either get community more active on the documentation - advocate for it
- or start up more documentation on the wiki => A FAQ would be a nice
first place to start.

There may be an awful lot of things I've missed on the write up - but
please follow up.

If some of the core ceph people allready have thoughts / ideas / guidance,
please share so we collaboratively can make it better.

Lastly - thanks for the great support on the mailing list - so far - the
intent is only to try to make ceph even better.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help Ceph Cluster Down

[Arun POONIA]

Hi Kevin,

You are right. Increasing number of PGs per OSD resolved the issue. I will
probably add this config in /etc/ceph/ceph.conf file of ceph mon and OSDs
so it applies on host boot.

Thanks
Arun

On Fri, Jan 4, 2019 at 3:46 PM Kevin Olbrich  wrote:

> Hi Arun,
>
> actually deleting was no good idea, thats why I wrote, that the OSDs
> should be "out".
> You have down PGs, that because the data is on OSDs that are
> unavailable but known by the cluster.
> This can be checked by using "ceph pg 0.5 query" (change PG name).
>
> Because your PG count is so much oversized, the overdose limits get
> hit on every recovery on your cluster.
> I had the same problem on a medium cluster when I added to many new
> disks at once.
> You already got this info from Caspar earlier in this thread.
>
>
> https://ceph.com/planet/placement-groups-with-ceph-luminous-stay-in-activating-state/
>
> https://blog.widodh.nl/2018/01/placement-groups-with-ceph-luminous-stay-in-activating-state/
>
> The second link shows one of the config params you need to inject to
> all your OSDs like this:
> ceph tell osd.* injectargs --mon_max_pg_per_osd 1
>
> This might help you getting these PGs some sort of "active"
> (+recovery/+degraded/+inconsistent/etc.).
>
> The down PGs will most likely never come back. It would bet, you will
> find OSD IDs that are invalid in the acting set, meaning that
> non-existent OSDs hold your data.
> I had a similar problem on a test cluster with erasure code pools
> where too many disks failed at the same time, you will then see
> negative values as OSD IDs.
>
> Maybe this helps a little bit.
>
> Kevin
>
> Am Sa., 5. Jan. 2019 um 00:20 Uhr schrieb Arun POONIA
> :
> >
> > Hi Kevin,
> >
> > I tried deleting newly added server from Ceph Cluster and looks like
> Ceph is not recovering. I agree with unfound data but it doesn't say about
> unfound data. It says inactive/down for PGs and I can't bring them up.
> >
> >
> > [root@fre101 ~]# ceph health detail
> > 2019-01-04 15:17:05.711641 7f27b0f31700 -1 asok(0x7f27ac0017a0)
> AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to
> bind the UNIX domain socket to
> '/var/run/ceph-guests/ceph-client.admin.129552.139808366139728.asok': (2)
> No such file or directory
> > HEALTH_ERR 3 pools have many more objects per pg than average;
> 523656/12393978 objects misplaced (4.225%); 6517 PGs pending on creation;
> Reduced data availability: 6585 pgs inactive, 1267 pgs down, 2 pgs peering,
> 2703 pgs stale; Degraded data redundancy: 86858/12393978 objects degraded
> (0.701%), 717 pgs degraded, 21 pgs undersized; 99059 slow requests are
> blocked > 32 sec; 4834 stuck requests are blocked > 4096 sec; too many PGs
> per OSD (3003 > max 200)
> > MANY_OBJECTS_PER_PG 3 pools have many more objects per pg than average
> > pool glance-images objects per pg (10478) is more than 92.7257 times
> cluster average (113)
> > pool vms objects per pg (4722) is more than 41.7876 times cluster
> average (113)
> > pool volumes objects per pg (1220) is more than 10.7965 times
> cluster average (113)
> > OBJECT_MISPLACED 523656/12393978 objects misplaced (4.225%)
> > PENDING_CREATING_PGS 6517 PGs pending on creation
> > osds
> [osd.0,osd.1,osd.10,osd.11,osd.12,osd.13,osd.14,osd.15,osd.16,osd.17,osd.18,osd.19,osd.2,osd.20,osd.21,osd.22,osd.23,osd.24,osd.25,osd.26,osd.27,osd.28,osd.29,osd.3,osd.30,osd.31,osd.32,osd.33,osd.34,osd.35,osd.4,osd.5,osd.6,osd.7,osd.8,osd.9]
> have pending PGs.
> > PG_AVAILABILITY Reduced data availability: 6585 pgs inactive, 1267 pgs
> down, 2 pgs peering, 2703 pgs stale
> > pg 10.90e is stuck inactive for 94928.999109, current state
> activating, last acting [2,6]
> > pg 10.913 is stuck inactive for 95094.175400, current state
> activating, last acting [9,5]
> > pg 10.915 is stuck inactive for 94929.184177, current state
> activating, last acting [30,26]
> > pg 11.907 is stuck stale for 9612.906582, current state
> stale+active+clean, last acting [38,24]
> > pg 11.910 is stuck stale for 11822.359237, current state stale+down,
> last acting [21]
> > pg 11.915 is stuck stale for 9612.906604, current state
> stale+active+clean, last acting [38,31]
> > pg 11.919 is stuck inactive for 95636.716568, current state
> activating, last acting [25,12]
> > pg 12.902 is stuck stale for 10810.497213, current state
> stale+activating, last acting [36,14]
> > pg 13.901 is stuck stale for 94889.512234, current state
> stale+active+clean, last acting [1,31]
> > pg 13.904 is stuck stale for 10745.279158, current state
> stale+active+clean, last acting [37,8]
> > pg 13.908 is stuck stale for 10745.279176, current state
> stale+active+clean, last acting [37,19]
> > pg 13.909 is stuck inactive for 95370.129659, current state
> activating, last acting [34,19]
> > pg 13.90e is stuck inactive for 95370.379694, current state
> activating, last acting [21,20]
> > pg 13.911 is stuck inactive for 

Re: [ceph-users] Usage of devices in SSD pool vary very much

[Konstantin Shalygin]

On 1/5/19 1:51 AM, Kevin Olbrich wrote:

PS: Could behttp://tracker.ceph.com/issues/36361
There is one HDD OSD that is out (which will not be replaced because
the SSD pool will get the images and the hdd pool will be deleted).


Paste your `ceph osd tree`, `ceph osd df tree` please.



k

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help Ceph Cluster Down

[Kevin Olbrich]

Hi Arun,

actually deleting was no good idea, thats why I wrote, that the OSDs
should be "out".
You have down PGs, that because the data is on OSDs that are
unavailable but known by the cluster.
This can be checked by using "ceph pg 0.5 query" (change PG name).

Because your PG count is so much oversized, the overdose limits get
hit on every recovery on your cluster.
I had the same problem on a medium cluster when I added to many new
disks at once.
You already got this info from Caspar earlier in this thread.

https://ceph.com/planet/placement-groups-with-ceph-luminous-stay-in-activating-state/
https://blog.widodh.nl/2018/01/placement-groups-with-ceph-luminous-stay-in-activating-state/

The second link shows one of the config params you need to inject to
all your OSDs like this:
ceph tell osd.* injectargs --mon_max_pg_per_osd 1

This might help you getting these PGs some sort of "active"
(+recovery/+degraded/+inconsistent/etc.).

The down PGs will most likely never come back. It would bet, you will
find OSD IDs that are invalid in the acting set, meaning that
non-existent OSDs hold your data.
I had a similar problem on a test cluster with erasure code pools
where too many disks failed at the same time, you will then see
negative values as OSD IDs.

Maybe this helps a little bit.

Kevin

Am Sa., 5. Jan. 2019 um 00:20 Uhr schrieb Arun POONIA
:
>
> Hi Kevin,
>
> I tried deleting newly added server from Ceph Cluster and looks like Ceph is 
> not recovering. I agree with unfound data but it doesn't say about unfound 
> data. It says inactive/down for PGs and I can't bring them up.
>
>
> [root@fre101 ~]# ceph health detail
> 2019-01-04 15:17:05.711641 7f27b0f31700 -1 asok(0x7f27ac0017a0) 
> AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to 
> bind the UNIX domain socket to 
> '/var/run/ceph-guests/ceph-client.admin.129552.139808366139728.asok': (2) No 
> such file or directory
> HEALTH_ERR 3 pools have many more objects per pg than average; 
> 523656/12393978 objects misplaced (4.225%); 6517 PGs pending on creation; 
> Reduced data availability: 6585 pgs inactive, 1267 pgs down, 2 pgs peering, 
> 2703 pgs stale; Degraded data redundancy: 86858/12393978 objects degraded 
> (0.701%), 717 pgs degraded, 21 pgs undersized; 99059 slow requests are 
> blocked > 32 sec; 4834 stuck requests are blocked > 4096 sec; too many PGs 
> per OSD (3003 > max 200)
> MANY_OBJECTS_PER_PG 3 pools have many more objects per pg than average
> pool glance-images objects per pg (10478) is more than 92.7257 times 
> cluster average (113)
> pool vms objects per pg (4722) is more than 41.7876 times cluster average 
> (113)
> pool volumes objects per pg (1220) is more than 10.7965 times cluster 
> average (113)
> OBJECT_MISPLACED 523656/12393978 objects misplaced (4.225%)
> PENDING_CREATING_PGS 6517 PGs pending on creation
> osds 
> [osd.0,osd.1,osd.10,osd.11,osd.12,osd.13,osd.14,osd.15,osd.16,osd.17,osd.18,osd.19,osd.2,osd.20,osd.21,osd.22,osd.23,osd.24,osd.25,osd.26,osd.27,osd.28,osd.29,osd.3,osd.30,osd.31,osd.32,osd.33,osd.34,osd.35,osd.4,osd.5,osd.6,osd.7,osd.8,osd.9]
>  have pending PGs.
> PG_AVAILABILITY Reduced data availability: 6585 pgs inactive, 1267 pgs down, 
> 2 pgs peering, 2703 pgs stale
> pg 10.90e is stuck inactive for 94928.999109, current state activating, 
> last acting [2,6]
> pg 10.913 is stuck inactive for 95094.175400, current state activating, 
> last acting [9,5]
> pg 10.915 is stuck inactive for 94929.184177, current state activating, 
> last acting [30,26]
> pg 11.907 is stuck stale for 9612.906582, current state 
> stale+active+clean, last acting [38,24]
> pg 11.910 is stuck stale for 11822.359237, current state stale+down, last 
> acting [21]
> pg 11.915 is stuck stale for 9612.906604, current state 
> stale+active+clean, last acting [38,31]
> pg 11.919 is stuck inactive for 95636.716568, current state activating, 
> last acting [25,12]
> pg 12.902 is stuck stale for 10810.497213, current state 
> stale+activating, last acting [36,14]
> pg 13.901 is stuck stale for 94889.512234, current state 
> stale+active+clean, last acting [1,31]
> pg 13.904 is stuck stale for 10745.279158, current state 
> stale+active+clean, last acting [37,8]
> pg 13.908 is stuck stale for 10745.279176, current state 
> stale+active+clean, last acting [37,19]
> pg 13.909 is stuck inactive for 95370.129659, current state activating, 
> last acting [34,19]
> pg 13.90e is stuck inactive for 95370.379694, current state activating, 
> last acting [21,20]
> pg 13.911 is stuck inactive for 98449.317873, current state activating, 
> last acting [25,22]
> pg 13.914 is stuck stale for 11827.503651, current state stale+down, last 
> acting [29]
> pg 13.917 is stuck inactive for 94564.811121, current state activating, 
> last acting [16,12]
> pg 14.901 is stuck inactive for 94929.006707, current state 
> activating+degraded, last acting 

Re: [ceph-users] Help Ceph Cluster Down

[Arun POONIA]

Hi Kevin,

I tried deleting newly added server from Ceph Cluster and looks like Ceph
is not recovering. I agree with unfound data but it doesn't say about
unfound data. It says inactive/down for PGs and I can't bring them up.


[root@fre101 ~]# ceph health detail
2019-01-04 15:17:05.711641 7f27b0f31700 -1 asok(0x7f27ac0017a0)
AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to
bind the UNIX domain socket to
'/var/run/ceph-guests/ceph-client.admin.129552.139808366139728.asok': (2)
No such file or directory
HEALTH_ERR 3 pools have many more objects per pg than average;
523656/12393978 objects misplaced (4.225%); 6517 PGs pending on creation;
Reduced data availability: 6585 pgs inactive, 1267 pgs down, 2 pgs peering,
2703 pgs stale; Degraded data redundancy: 86858/12393978 objects degraded
(0.701%), 717 pgs degraded, 21 pgs undersized; 99059 slow requests are
blocked > 32 sec; 4834 stuck requests are blocked > 4096 sec; too many PGs
per OSD (3003 > max 200)
MANY_OBJECTS_PER_PG 3 pools have many more objects per pg than average
pool glance-images objects per pg (10478) is more than 92.7257 times
cluster average (113)
pool vms objects per pg (4722) is more than 41.7876 times cluster
average (113)
pool volumes objects per pg (1220) is more than 10.7965 times cluster
average (113)
OBJECT_MISPLACED 523656/12393978 objects misplaced (4.225%)
PENDING_CREATING_PGS 6517 PGs pending on creation
osds
[osd.0,osd.1,osd.10,osd.11,osd.12,osd.13,osd.14,osd.15,osd.16,osd.17,osd.18,osd.19,osd.2,osd.20,osd.21,osd.22,osd.23,osd.24,osd.25,osd.26,osd.27,osd.28,osd.29,osd.3,osd.30,osd.31,osd.32,osd.33,osd.34,osd.35,osd.4,osd.5,osd.6,osd.7,osd.8,osd.9]
have pending PGs.
PG_AVAILABILITY Reduced data availability: 6585 pgs inactive, 1267 pgs
down, 2 pgs peering, 2703 pgs stale
pg 10.90e is stuck inactive for 94928.999109, current state activating,
last acting [2,6]
pg 10.913 is stuck inactive for 95094.175400, current state activating,
last acting [9,5]
pg 10.915 is stuck inactive for 94929.184177, current state activating,
last acting [30,26]
pg 11.907 is stuck stale for 9612.906582, current state
stale+active+clean, last acting [38,24]
pg 11.910 is stuck stale for 11822.359237, current state stale+down,
last acting [21]
pg 11.915 is stuck stale for 9612.906604, current state
stale+active+clean, last acting [38,31]
pg 11.919 is stuck inactive for 95636.716568, current state activating,
last acting [25,12]
pg 12.902 is stuck stale for 10810.497213, current state
stale+activating, last acting [36,14]
pg 13.901 is stuck stale for 94889.512234, current state
stale+active+clean, last acting [1,31]
pg 13.904 is stuck stale for 10745.279158, current state
stale+active+clean, last acting [37,8]
pg 13.908 is stuck stale for 10745.279176, current state
stale+active+clean, last acting [37,19]
pg 13.909 is stuck inactive for 95370.129659, current state activating,
last acting [34,19]
pg 13.90e is stuck inactive for 95370.379694, current state activating,
last acting [21,20]
pg 13.911 is stuck inactive for 98449.317873, current state activating,
last acting [25,22]
pg 13.914 is stuck stale for 11827.503651, current state stale+down,
last acting [29]
pg 13.917 is stuck inactive for 94564.811121, current state activating,
last acting [16,12]
pg 14.901 is stuck inactive for 94929.006707, current state
activating+degraded, last acting [22,8]
pg 14.910 is stuck inactive for 94929.046256, current state
activating+degraded, last acting [17,2]
pg 14.912 is stuck inactive for 10831.758524, current state activating,
last acting [18,2]
pg 14.915 is stuck inactive for 94929.001390, current state activating,
last acting [34,23]
pg 15.90c is stuck inactive for 93957.371333, current state activating,
last acting [29,10]
pg 15.90d is stuck inactive for 94929.145438, current state activating,
last acting [5,31]
pg 15.913 is stuck stale for 10745.279197, current state
stale+active+clean, last acting [37,12]
pg 15.915 is stuck stale for 12343.606595, current state stale+down,
last acting [0]
pg 15.91c is stuck stale for 10650.058945, current state stale+down,
last acting [12]
pg 16.90e is stuck inactive for 94929.240626, current state activating,
last acting [14,2]
pg 16.919 is stuck inactive for 94564.771129, current state activating,
last acting [20,4]
pg 16.91e is stuck inactive for 94960.007104, current state activating,
last acting [22,12]
pg 17.908 is stuck inactive for 12250.346380, current state activating,
last acting [27,18]
pg 17.90b is stuck inactive for 11714.951268, current state activating,
last acting [12,25]
pg 17.910 is stuck inactive for 94564.819149, current state activating,
last acting [26,16]
pg 17.913 is stuck inactive for 95370.177309, current state activating,
last acting [13,31]
pg 17.91f is stuck inactive for 95147.032346, current state activating,
last acting [6,18]
 

Re: [ceph-users] HDD spindown problem

[Nieporte, Michael]

Hello,


we likely faced the same issue with spindowns.


We set max spindown timers on all HDDs and disabled the tuned.service, which we 
were told might change back/affect the set timers.

To correctly disable tuned:
tuned-adm stop
tuned-adm off
systemctl stop tuned.service
systemctl disable tuned.service



Our spindown problem is 'fixed'.


Von: ceph-users  im Auftrag von Florian 
Engelmann 
Gesendet: Montag, 3. Dezember 2018 11:02:28
An: ceph-users@lists.ceph.com
Betreff: [ceph-users] HDD spindown problem

Hello,

we are fighting a HDD spin-down problem on our production ceph cluster
since two weeks now. The problem is not ceph related but I guess this
topic is interesting to the list and to be honest I hope to find a
solution here.

We do use 6 OSD Nodes like:
OS: Suse 12 SP3
Ceph: SES 5.5 (12.2.8)
Server: Supermicro 6048R-E1CR36L
Controller: LSI 3008 (LSI3008-IT)
Disk: 12x Seagate ST8000NM0055-1RM112 8TB (SN05 Firmware (some still
SN02 and SN04)
NVMe: 1x Intel DC P3700 800GB (used for 80GB RocksDB and 2GB WAL for
each OSD (only 7 Disks are online right now - up to 9 Disks will have
there RocksDB/WAL on one NVMe SSD)


Problem:
This Ceph cluster is used for objectstorage (RadosGW) only and is mostly
used for backups to S3 (RadosGW). There is not that much activity -
mostly at night time. We do not want any HDD to spin down but they do.
We tried to disable the spindown timers by using sdparm and also with
the Seagate tool SeaChest but "something" does re-enable them:


Disable standby on all HDD:
for i in sd{c..n}; do
/root/SeaChestUtilities/Linux/Lin64/SeaChest_PowerControl_191_1183_64 -d
/dev/$i --onlySeagate --changePower --disableMode --powerMode standby ;
done


Monitor standby timer status:

while true; do for i in sd{c..n}; do echo  "$(date) $i
$(/root/SeaChestUtilities/Linux/Lin64/SeaChest_PowerControl_191_1183_64
-d /dev/$i --onlySeagate --showEPCSettings -v0 | grep Stand)";  done;
sleep 1 ; done

This will show:
Mon Dec  3 10:42:54 CET 2018 sdc Standby Z   0 9000
65535120   Y Y
Mon Dec  3 10:42:54 CET 2018 sdd Standby Z   0 9000
65535120   Y Y
Mon Dec  3 10:42:54 CET 2018 sde Standby Z   0 9000
65535120   Y Y
Mon Dec  3 10:42:54 CET 2018 sdf Standby Z   0 9000
65535120   Y Y
Mon Dec  3 10:42:54 CET 2018 sdg Standby Z   0 9000
65535120   Y Y
Mon Dec  3 10:42:54 CET 2018 sdh Standby Z   0 9000
65535120   Y Y
Mon Dec  3 10:42:54 CET 2018 sdi Standby Z   0 9000
65535120   Y Y
Mon Dec  3 10:42:55 CET 2018 sdj Standby Z   0 9000
65535120   Y Y
Mon Dec  3 10:42:55 CET 2018 sdk Standby Z   0 9000
65535120   Y Y
Mon Dec  3 10:42:55 CET 2018 sdl Standby Z   0 9000
65535120   Y Y
Mon Dec  3 10:42:55 CET 2018 sdm Standby Z   0 9000
65535120   Y Y
Mon Dec  3 10:42:55 CET 2018 sdn Standby Z   0 9000
65535120   Y Y


So everything is fine right now. Standby timer is 0 and disabled (no *
shown) while the default value is 9000 and the saved timer is  (we
saved this value so the disks have a huge time after reboots). But after
a unknown amount of time (in this case ~7 minutes) things start to get
weird:

Mon Dec  3 10:47:52 CET 2018 sdc Standby Z  *3500  9000
65535120   Y Y
[...]
65535120   Y Y
Mon Dec  3 10:48:07 CET 2018 sdc Standby Z  *3500  9000
65535120   Y Y
Mon Dec  3 10:48:09 CET 2018 sdc Standby Z  *3500  9000
65535120   Y Y
Mon Dec  3 10:48:12 CET 2018 sdc Standby Z  *4500  9000
65535120   Y Y
Mon Dec  3 10:48:14 CET 2018 sdc Standby Z  *4500  9000
65535120   Y Y
Mon Dec  3 10:48:16 CET 2018 sdc Standby Z  *4500  9000
65535120   Y Y
Mon Dec  3 10:48:19 CET 2018 sdc Standby Z  *4500  9000
65535120   Y Y
Mon Dec  3 10:48:21 CET 2018 sdc Standby Z  *4500  9000
65535120   Y Y
Mon Dec  3 10:48:23 CET 2018 sdc Standby Z  *5500  9000
65535120   Y Y
Mon Dec  3 10:48:26 CET 2018 sdc Standby Z  *5500  9000
65535120   Y Y
Mon Dec  3 10:48:28 CET 2018 sdc Standby Z  *5500  9000
65535120   Y Y
Mon Dec  3 10:48:30 CET 2018 sdc Standby Z  *5500  9000
65535120   Y Y
Mon Dec  3 10:48:32 CET 2018 sdc Standby Z  *5500  9000
65535120   Y Y
Mon Dec  3 10:48:35 CET 2018 sdc Standby Z  *5500  9000
65535120   Y Y
Mon Dec  3 10:48:37 CET 2018 sdc Standby Z  *5500  9000
65535120   Y Y
Mon Dec  3 10:48:40 CET 2018 sdc Standby Z  *5500  9000
65535120   Y Y
Mon Dec  3 

Re: [ceph-users] ceph-mgr fails to restart after upgrade to mimic

[Randall Smith]

Some more info that may or may not matter. :-) First off, I am running
13.2.3 on Ubuntu Xenial (ceph version 13.2.3
(9bf3c8b1a04b0aa4a3cc78456a508f1c48e70279) mimic (stable)).

Next, when I try running ceph-mgr with --no-mon-config, the app core dumps.

 0> 2019-01-04 14:56:56.416 7fbcc71db380 -1
/build/ceph-13.2.3/src/common/Timer.cc: In function 'virtual
SafeTimer::~SafeTimer()' thread 7fbcc71db380 time 2019-01-04 14:56:56.419012
/build/ceph-13.2.3/src/common/Timer.cc: 50: FAILED assert(thread == __null)

 ceph version 13.2.3 (9bf3c8b1a04b0aa4a3cc78456a508f1c48e70279) mimic
(stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x7fbcbe5093c2]
 2: (()+0x2e5587) [0x7fbcbe509587]
 3: (()+0x2e12de) [0x7fbcbe5052de]
 4: (MgrClient::~MgrClient()+0xc4) [0x5594f4]
 5: (MgrStandby::~MgrStandby()+0x14d) [0x55063d]
 6: (main()+0x24b) [0x49446b]

 7: (__libc_start_main()+0xf0) [0x7fbcbcf51830]
 8: (_start()+0x29) [0x497dc9]
 NOTE: a copy of the executable, or `objdump -rdS ` is needed
to interpret this.

--- logging levels ---

   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer


   1/ 5 mds_locker

   1/ 5 mds_log

   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   1/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
  10/10 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 1 reserver
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/ 5 rgw_sync
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
   1/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
  -2/-2 (syslog threshold)
  99/99 (stderr threshold)
  max_recent 1
  max_new 1000
  log_file
--- end dump of recent events ---
*** Caught signal (Aborted) **
 in thread 7fbcc71db380 thread_name:ceph-mgr
 ceph version 13.2.3 (9bf3c8b1a04b0aa4a3cc78456a508f1c48e70279) mimic
(stable)
 1: /usr/bin/ceph-mgr() [0x63ebd0]
 2: (()+0x11390) [0x7fbcbd819390]
 3: (gsignal()+0x38) [0x7fbcbcf66428]
 4: (abort()+0x16a) [0x7fbcbcf6802a]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x250) [0x7fbcbe509510]
 6: (()+0x2e5587) [0x7fbcbe509587]
 7: (()+0x2e12de) [0x7fbcbe5052de]
 8: (MgrClient::~MgrClient()+0xc4) [0x5594f4]
 9: (MgrStandby::~MgrStandby()+0x14d) [0x55063d]
 10: (main()+0x24b) [0x49446b]
 11: (__libc_start_main()+0xf0) [0x7fbcbcf51830]
 12: (_start()+0x29) [0x497dc9]
2019-01-04 14:56:56.420 7fbcc71db380 -1 *** Caught signal (Aborted) **
 in thread 7fbcc71db380 thread_name:ceph-mgr

 ceph version 13.2.3 (9bf3c8b1a04b0aa4a3cc78456a508f1c48e70279) mimic
(stable)
 1: /usr/bin/ceph-mgr() [0x63ebd0]
 2: (()+0x11390) [0x7fbcbd819390]
 3: (gsignal()+0x38) [0x7fbcbcf66428]
 4: (abort()+0x16a) [0x7fbcbcf6802a]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x250) [0x7fbcbe509510]
 6: (()+0x2e5587) [0x7fbcbe509587]
 7: (()+0x2e12de) [0x7fbcbe5052de]
 8: (MgrClient::~MgrClient()+0xc4) [0x5594f4]
 9: (MgrStandby::~MgrStandby()+0x14d) [0x55063d]
 10: (main()+0x24b) [0x49446b]
 11: (__libc_start_main()+0xf0) [0x7fbcbcf51830]
 12: (_start()+0x29) [0x497dc9]
 NOTE: a copy of the executable, or `objdump -rdS ` is needed
to interpret this.


On Fri, Jan 4, 2019 at 1:53 PM Randall Smith  wrote:

> I think this is the relevant section of the debug log. There's no
> AUTH_NONE error which would make things easy. You can see the same "Invalid
> argument" error that I'm seeing in the mgr debug output. The malformed
> request feels like a compatibility or protocol communication issue.
>
> 2019-01-04 13:41:58.972 7f88950f5700 10 mon.07@1(peon) e27
> ms_verify_authorizer 192.168.253.148:0/3301807723 client protocol 0
>
> 2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon) e27 _ms_dispatch
> new session 0x40a58c0 MonSession(client.? 192.168.253.148:0/3301807723 is
> open , features 0x3ffddff8ffa4fffb (luminous)) fea$ures 0x3ffddff8ffa4fffb
> 2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon).auth v87697
> preprocess_query auth(proto 0 26 bytes epoch 0) v1 from client.?
> 192.168.253.148:0/3301807723
> 2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon).auth v87697
> prep_auth() blob_size=26
> 2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon).auth v87697
> AuthMonitor::assign_global_id m=auth(proto 0 26 bytes epoch 0) v1 mon=1/3
> last_allocated=12307825 max_global_id=12353896
> 2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon).auth v87697
> next_global_id should be 12307828
>
> 2019-01-04 13:41:58.972 7f8890143700  2 mon.07@1(peon) e27 send_reply
> 0x5449180 

Re: [ceph-users] Mimic 13.2.3?

[Daniel Baumann]

On 01/04/2019 07:32 PM, Peter Woodman wrote:
> not to mention that the current released version of mimic (.2) has a
> bug that is potentially catastrophic to cephfs, known about for
> months, yet it's not in the release notes. would have upgraded and
> destroyed data had i not caught a thread on this list.

indeed.

we're a big cephfs user here for HPC. everytime I get asked about it by
my peers, sadly I have to tell them that they should not use it for
production, that it's not stable and has serious stability bugs
(eventhough it was declared "stable" upstream some time ago).

(e.g. doing an rsync on, from or to a cephfs, just like someone wrote a
couple of days again on the list, reliably kills it, everytime - we
reproduce it with every kernel release and every ceph release since
february 2015 on several independent clusters. even more catastropic is
that single inconsistent files stopps the whole cephfs which then cannot
be restored unless the affected cephfs is unmounted on all(!) machines
that have it mounted, etc.

we can use cephfs only in our sort-of-stable setup with 12.2.5 because
we have mostly non-malicious users that usualy behave nicely. but it's
to brittle in the end and apparently no silver lining ahead. because of
that, during our scaling up of our cephfs cluster from 300tb to 1.2pb
this spring, we'll be moving away from cephfs entirely and switch to
mounting RBDs and export them with samba instead.

we have good experiences with RBDs on other clusters. but using RBDs
that way is quite painful when knowing that cephfs exists, it's slower,
and not really HA anymore, but it's overall more reliable than cephfs)

as much as I like ceph, I unfortunatly can't say the same for cephfs :(

Regards,
Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph blog RSS/Atom URL?

[Gregory Farnum]

Yeah I think it’s a “planet”-style feed that incorporates some other blogs.
I don’t think it’s been maintained much since being launched though.

On Fri, Jan 4, 2019 at 1:21 PM Jan Kasprzak  wrote:

> Gregory Farnum wrote:
> : It looks like ceph.com/feed is the RSS url?
>
> Close enough, thanks.
>
> Comparing the above with the blog itself, there are some
> posts in (apparently) Chinese in /feed, which are not present in
> /community/blog. The first one being
>
>
> https://ceph.com/planet/vdbench%e6%b5%8b%e8%af%95%e5%ae%9e%e6%97%b6%e5%8f%af%e8%a7%86%e5%8c%96%e6%98%be%e7%a4%ba/
>
> -Yenya
>
> : On Fri, Jan 4, 2019 at 5:52 AM Jan Kasprzak  wrote:
> : > is there any RSS or Atom source for Ceph blog? I have looked inside
> : > the https://ceph.com/community/blog/ HTML source, but there is no
> : >  or anything mentioning RSS or Atom.
>
> --
> | Jan "Yenya" Kasprzak 
> |
> | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5
> |
>  This is the world we live in: the way to deal with computers is to google
>  the symptoms, and hope that you don't have to watch a video. --P. Zaitcev
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph blog RSS/Atom URL?

[Jan Kasprzak]

Gregory Farnum wrote:
: It looks like ceph.com/feed is the RSS url?

Close enough, thanks.

Comparing the above with the blog itself, there are some
posts in (apparently) Chinese in /feed, which are not present in
/community/blog. The first one being

https://ceph.com/planet/vdbench%e6%b5%8b%e8%af%95%e5%ae%9e%e6%97%b6%e5%8f%af%e8%a7%86%e5%8c%96%e6%98%be%e7%a4%ba/

-Yenya

: On Fri, Jan 4, 2019 at 5:52 AM Jan Kasprzak  wrote:
: > is there any RSS or Atom source for Ceph blog? I have looked inside
: > the https://ceph.com/community/blog/ HTML source, but there is no
: >  or anything mentioning RSS or Atom.

-- 
| Jan "Yenya" Kasprzak  |
| http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 |
 This is the world we live in: the way to deal with computers is to google
 the symptoms, and hope that you don't have to watch a video. --P. Zaitcev
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph health JSON format has changed

[Jan Kasprzak]

Gregory Farnum wrote:
: On Wed, Jan 2, 2019 at 5:12 AM Jan Kasprzak  wrote:
: 
: > Thomas Byrne - UKRI STFC wrote:
: > : I recently spent some time looking at this, I believe the 'summary' and
: > : 'overall_status' sections are now deprecated. The 'status' and 'checks'
: > : fields are the ones to use now.
: >
: > OK, thanks.
: >
: > : The 'status' field gives you the OK/WARN/ERR, but returning the most
: > : severe error condition from the 'checks' section is less trivial. AFAIK
: > : all health_warn states are treated as equally severe, and same for
: > : health_err. We ended up formatting our single line human readable output
: > : as something like:
: > :
: > : "HEALTH_ERR: 1 inconsistent pg, HEALTH_ERR: 1 scrub error, HEALTH_WARN:
: > 20 large omap objects"
: >
: > Speaking of scrub errors:
: >
: > In previous versions of Ceph, I was able to determine which PGs had
: > scrub errors, and then a cron.hourly script ran "ceph pg repair" for them,
: > provided that they were not already being scrubbed. In Luminous, the bad PG
: > is not visible in "ceph --status" anywhere. Should I use something like
: > "ceph health detail -f json-pretty" instead?
: >
: > Also, is it possible to configure Ceph to attempt repairing
: > the bad PGs itself, as soon as the scrub fails? I run most of my OSDs on
: > top
: > of a bunch of old spinning disks, and a scrub error almost always means
: > that there is a bad sector somewhere, which can easily be fixed by
: > rewriting the lost data using "ceph pg repair".
: >
: 
: It is possible. It's a lot safer than it used to be, but is still NOT
: RECOMMENDED for replicated pools.
: 
: But if you are very sure, you can use the options osd_scrub_auto_repair
: (default: false) and osd_scrub_auto_repair_num_errors (default:5, which
: will not auto-repair if scrub detects more errors than that value) to
: configure it.

OK, thanks. I just want to say that I am NOT very sure,
but this is about the only way I am aware of, when I want to
handle the scrub error. I have mail notification set up in smartd.conf,
and so far the scrub errors seem to correlate with new reallocated
or pending sectors.

What are the drawbacks of running "ceph pg repair" as soon
asi the cluster enters the HEALTH_ERR state with scrub error?

Thanks for explanation,

-Yenya

-- 
| Jan "Yenya" Kasprzak  |
| http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 |
 This is the world we live in: the way to deal with computers is to google
 the symptoms, and hope that you don't have to watch a video. --P. Zaitcev
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-mgr fails to restart after upgrade to mimic

[Randall Smith]

I think this is the relevant section of the debug log. There's no AUTH_NONE
error which would make things easy. You can see the same "Invalid argument"
error that I'm seeing in the mgr debug output. The malformed request feels
like a compatibility or protocol communication issue.

2019-01-04 13:41:58.972 7f88950f5700 10 mon.07@1(peon) e27
ms_verify_authorizer 192.168.253.148:0/3301807723 client protocol 0

2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon) e27 _ms_dispatch new
session 0x40a58c0 MonSession(client.? 192.168.253.148:0/3301807723 is open
, features 0x3ffddff8ffa4fffb (luminous)) fea$ures 0x3ffddff8ffa4fffb
2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon).auth v87697
preprocess_query auth(proto 0 26 bytes epoch 0) v1 from client.?
192.168.253.148:0/3301807723
2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon).auth v87697
prep_auth() blob_size=26
2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon).auth v87697
AuthMonitor::assign_global_id m=auth(proto 0 26 bytes epoch 0) v1 mon=1/3
last_allocated=12307825 max_global_id=12353896
2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon).auth v87697
next_global_id should be 12307828

2019-01-04 13:41:58.972 7f8890143700  2 mon.07@1(peon) e27 send_reply
0x5449180 0x4ee1c00 auth_reply(proto 2 0 (0) Success) v1

2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon).auth v87697
preprocess_query auth(proto 2 2 bytes epoch 0) v1 from client.?
192.168.253.148:0/3301807723
2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon).auth v87697
prep_auth() blob_size=2
2019-01-04 13:41:58.972 7f8890143700  0 mon.07@1(peon).auth v87697 caught
error when trying to handle auth request, probably malformed request

2019-01-04 13:41:58.972 7f8890143700  2 mon.07@1(peon) e27 send_reply
0x30dc500 0x5caa280 auth_reply(proto 2 -22 (22) Invalid argument) v1

2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon) e27 ms_handle_reset
0x4102a00 192.168.253.148:0/3301807723

2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon) e27 reset/close on
session client.? 192.168.253.148:0/3301807723

2019-01-04 13:41:58.972 7f8890143700 10 mon.07@1(peon) e27 remove_session
0x40a58c0 client.? 192.168.253.148:0/3301807723 features 0x3ffddff8ffa4fffb

On Fri, Jan 4, 2019 at 12:32 PM Gregory Farnum  wrote:

> You can also get more data by checking what the monitor logs for that
> manager on the connect attempt (if you turn up its debug mon or debug
> ms settings). If one of your managers is behaving, I'd examine its
> configuration file and compare to the others. For instance, that
> "Invalid argument" might mean the manager is trying to use "AUTH_NONE"
> (no CephX) and the monitors aren't allowing that.
> -Greg
>
> On Fri, Jan 4, 2019 at 6:26 AM Randall Smith  wrote:
> >
> > Greetings,
> >
> > I'm upgrading my cluster from luminous to mimic. I've upgraded my
> monitors and am attempting to upgrade the mgrs. Unfortunately, after an
> upgrade the mgr daemon exits immediately with error code 1.
> >
> > I've tried running ceph-mgr in debug mode to try to see what's happening
> but the output (below) is a bit cryptic for me. It looks like
> authentication might be failing but it was working prior to the upgrade.
> >
> > I do have "auth supported = cephx" in the global section of ceph.conf.
> >
> > What do I need to do to fix this?
> >
> > Thanks.
> >
> > /usr/bin/ceph-mgr -f --cluster ceph --id 8 --setuser ceph --setgroup
> ceph -d --debug_ms 5
> > 2019-01-04 07:01:38.457 7f808f83f700  2 Event(0x30c42c0 nevent=5000
> time_id=1).set_owner idx=0 owner=140190140331776
> > 2019-01-04 07:01:38.457 7f808f03e700  2 Event(0x30c4500 nevent=5000
> time_id=1).set_owner idx=1 owner=140190131939072
> > 2019-01-04 07:01:38.457 7f808e83d700  2 Event(0x30c4e00 nevent=5000
> time_id=1).set_owner idx=2 owner=140190123546368
> > 2019-01-04 07:01:38.457 7f809dd5b380  1  Processor -- start
> > 2019-01-04 07:01:38.477 7f809dd5b380  1 -- - start start
> > 2019-01-04 07:01:38.481 7f809dd5b380  1 -- - --> 192.168.253.147:6789/0
> -- auth(proto 0 26 bytes epoch 0) v1 -- 0x32a6780 con 0
> > 2019-01-04 07:01:38.481 7f809dd5b380  1 -- - --> 192.168.253.148:6789/0
> -- auth(proto 0 26 bytes epoch 0) v1 -- 0x32a6a00 con 0
> > 2019-01-04 07:01:38.481 7f808e83d700  1 -- 192.168.253.148:0/1359135487
> learned_addr learned my addr 192.168.253.148:0/1359135487
> > 2019-01-04 07:01:38.481 7f808e83d700  2 -- 192.168.253.148:0/1359135487
> >> 192.168.253.148:6789/0 conn(0x332d500 :-1
> s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got
> newly_a$
> > ked_seq 0 vs out_seq 0
> > 2019-01-04 07:01:38.481 7f808f03e700  2 -- 192.168.253.148:0/1359135487
> >> 192.168.253.147:6789/0 conn(0x332ce00 :-1
> s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got
> newly_a$
> > ked_seq 0 vs out_seq 0
> > 2019-01-04 07:01:38.481 7f808f03e700  5 -- 192.168.253.148:0/1359135487
> >> 192.168.253.147:6789/0 conn(0x332ce00 :-1
> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74172 cs=1 l=1). rx 

Re: [ceph-users] ceph health JSON format has changed

[Gregory Farnum]

On Wed, Jan 2, 2019 at 5:12 AM Jan Kasprzak  wrote:

> Thomas Byrne - UKRI STFC wrote:
> : I recently spent some time looking at this, I believe the 'summary' and
> : 'overall_status' sections are now deprecated. The 'status' and 'checks'
> : fields are the ones to use now.
>
> OK, thanks.
>
> : The 'status' field gives you the OK/WARN/ERR, but returning the most
> : severe error condition from the 'checks' section is less trivial. AFAIK
> : all health_warn states are treated as equally severe, and same for
> : health_err. We ended up formatting our single line human readable output
> : as something like:
> :
> : "HEALTH_ERR: 1 inconsistent pg, HEALTH_ERR: 1 scrub error, HEALTH_WARN:
> 20 large omap objects"
>
> Speaking of scrub errors:
>
> In previous versions of Ceph, I was able to determine which PGs had
> scrub errors, and then a cron.hourly script ran "ceph pg repair" for them,
> provided that they were not already being scrubbed. In Luminous, the bad PG
> is not visible in "ceph --status" anywhere. Should I use something like
> "ceph health detail -f json-pretty" instead?
>
> Also, is it possible to configure Ceph to attempt repairing
> the bad PGs itself, as soon as the scrub fails? I run most of my OSDs on
> top
> of a bunch of old spinning disks, and a scrub error almost always means
> that there is a bad sector somewhere, which can easily be fixed by
> rewriting the lost data using "ceph pg repair".
>

It is possible. It's a lot safer than it used to be, but is still NOT
RECOMMENDED for replicated pools.

But if you are very sure, you can use the options osd_scrub_auto_repair
(default: false) and osd_scrub_auto_repair_num_errors (default:5, which
will not auto-repair if scrub detects more errors than that value) to
configure it.
-Greg


>
> Thanks,
>
> -Yenya
>
> --
> | Jan "Yenya" Kasprzak 
> |
> | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5
> |
>  This is the world we live in: the way to deal with computers is to google
>  the symptoms, and hope that you don't have to watch a video. --P. Zaitcev
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help Ceph Cluster Down

[Kevin Olbrich]

I don't think this will help you. Unfound means, the cluster is unable
to find the data anywhere (it's lost).
It would be sufficient to shut down the new host - the OSDs will then be out.

You can also force-heal the cluster, something like "do your best possible":

ceph pg 2.5 mark_unfound_lost revert|delete

Src: http://docs.ceph.com/docs/mimic/rados/troubleshooting/troubleshooting-pg/

Kevin

Am Fr., 4. Jan. 2019 um 20:47 Uhr schrieb Arun POONIA
:
>
> Hi Kevin,
>
> Can I remove newly added server from Cluster and see if it heals cluster ?
>
> When I check Hard Disk Iops on new server which are very low compared to 
> existing cluster server.
>
> Indeed this is a critical cluster but I don't have expertise to make it 
> flawless.
>
> Thanks
> Arun
>
> On Fri, Jan 4, 2019 at 11:35 AM Kevin Olbrich  wrote:
>>
>> If you realy created and destroyed OSDs before the cluster healed
>> itself, this data will be permanently lost (not found / inactive).
>> Also your PG count is so much oversized, the calculation for peering
>> will most likely break because this was never tested.
>>
>> If this is a critical cluster, I would start a new one and bring back
>> the backups (using a better PG count).
>>
>> Kevin
>>
>> Am Fr., 4. Jan. 2019 um 20:25 Uhr schrieb Arun POONIA
>> :
>> >
>> > Can anyone comment on this issue please, I can't seem to bring my cluster 
>> > healthy.
>> >
>> > On Fri, Jan 4, 2019 at 6:26 AM Arun POONIA  
>> > wrote:
>> >>
>> >> Hi Caspar,
>> >>
>> >> Number of IOPs are also quite low. It used be around 1K Plus on one of 
>> >> Pool (VMs) now its like close to 10-30 .
>> >>
>> >> Thansk
>> >> Arun
>> >>
>> >> On Fri, Jan 4, 2019 at 5:41 AM Arun POONIA 
>> >>  wrote:
>> >>>
>> >>> Hi Caspar,
>> >>>
>> >>> Yes and No, numbers are going up and down. If I run ceph -s command I 
>> >>> can see it decreases one time and later it increases again. I see there 
>> >>> are so many blocked/slow requests. Almost all the OSDs have slow 
>> >>> requests. Around 12% PGs are inactive not sure how to activate them 
>> >>> again.
>> >>>
>> >>>
>> >>> [root@fre101 ~]# ceph health detail
>> >>> 2019-01-04 05:39:23.860142 7fc37a3a0700 -1 asok(0x7fc3740017a0) 
>> >>> AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed 
>> >>> to bind the UNIX domain socket to 
>> >>> '/var/run/ceph-guests/ceph-client.admin.1066526.140477441513808.asok': 
>> >>> (2) No such file or directory
>> >>> HEALTH_ERR 1 osds down; 3 pools have many more objects per pg than 
>> >>> average; 472812/12392654 objects misplaced (3.815%); 3610 PGs pending on 
>> >>> creation; Reduced data availability: 6578 pgs inactive, 1882 pgs down, 
>> >>> 86 pgs peering, 850 pgs stale; Degraded data redundancy: 216694/12392654 
>> >>> objects degraded (1.749%), 866 pgs degraded, 16 pgs undersized; 116082 
>> >>> slow requests are blocked > 32 sec; 551 stuck requests are blocked > 
>> >>> 4096 sec; too many PGs per OSD (2709 > max 200)
>> >>> OSD_DOWN 1 osds down
>> >>> osd.28 (root=default,host=fre119) is down
>> >>> MANY_OBJECTS_PER_PG 3 pools have many more objects per pg than average
>> >>> pool glance-images objects per pg (10478) is more than 92.7257 times 
>> >>> cluster average (113)
>> >>> pool vms objects per pg (4717) is more than 41.7434 times cluster 
>> >>> average (113)
>> >>> pool volumes objects per pg (1220) is more than 10.7965 times 
>> >>> cluster average (113)
>> >>> OBJECT_MISPLACED 472812/12392654 objects misplaced (3.815%)
>> >>> PENDING_CREATING_PGS 3610 PGs pending on creation
>> >>> osds 
>> >>> [osd.0,osd.1,osd.10,osd.11,osd.14,osd.15,osd.17,osd.18,osd.19,osd.20,osd.21,osd.22,osd.23,osd.25,osd.26,osd.27,osd.28,osd.3,osd.30,osd.32,osd.33,osd.35,osd.36,osd.37,osd.38,osd.4,osd.5,osd.6,osd.7,osd.9]
>> >>>  have pending PGs.
>> >>> PG_AVAILABILITY Reduced data availability: 6578 pgs inactive, 1882 pgs 
>> >>> down, 86 pgs peering, 850 pgs stale
>> >>> pg 10.900 is down, acting [18]
>> >>> pg 10.90e is stuck inactive for 60266.030164, current state 
>> >>> activating, last acting [2,38]
>> >>> pg 10.913 is stuck stale for 1887.552862, current state stale+down, 
>> >>> last acting [9]
>> >>> pg 10.915 is stuck inactive for 60266.215231, current state 
>> >>> activating, last acting [30,38]
>> >>> pg 11.903 is stuck inactive for 59294.465961, current state 
>> >>> activating, last acting [11,38]
>> >>> pg 11.910 is down, acting [21]
>> >>> pg 11.919 is down, acting [25]
>> >>> pg 12.902 is stuck inactive for 57118.544590, current state 
>> >>> activating, last acting [36,14]
>> >>> pg 13.8f8 is stuck inactive for 60707.167787, current state 
>> >>> activating, last acting [29,37]
>> >>> pg 13.901 is stuck stale for 60226.543289, current state 
>> >>> stale+active+clean, last acting [1,31]
>> >>> pg 13.905 is stuck inactive for 60266.050940, current state 
>> >>> activating, last acting [2,36]
>> >>> pg 13.909 is stuck inactive for 60707.160714, current 

Re: [ceph-users] Help Ceph Cluster Down

[Arun POONIA]

Hi Kevin,

Can I remove newly added server from Cluster and see if it heals cluster ?

When I check Hard Disk Iops on new server which are very low compared to
existing cluster server.

Indeed this is a critical cluster but I don't have expertise to make it
flawless.

Thanks
Arun

On Fri, Jan 4, 2019 at 11:35 AM Kevin Olbrich  wrote:

> If you realy created and destroyed OSDs before the cluster healed
> itself, this data will be permanently lost (not found / inactive).
> Also your PG count is so much oversized, the calculation for peering
> will most likely break because this was never tested.
>
> If this is a critical cluster, I would start a new one and bring back
> the backups (using a better PG count).
>
> Kevin
>
> Am Fr., 4. Jan. 2019 um 20:25 Uhr schrieb Arun POONIA
> :
> >
> > Can anyone comment on this issue please, I can't seem to bring my
> cluster healthy.
> >
> > On Fri, Jan 4, 2019 at 6:26 AM Arun POONIA <
> arun.poo...@nuagenetworks.net> wrote:
> >>
> >> Hi Caspar,
> >>
> >> Number of IOPs are also quite low. It used be around 1K Plus on one of
> Pool (VMs) now its like close to 10-30 .
> >>
> >> Thansk
> >> Arun
> >>
> >> On Fri, Jan 4, 2019 at 5:41 AM Arun POONIA <
> arun.poo...@nuagenetworks.net> wrote:
> >>>
> >>> Hi Caspar,
> >>>
> >>> Yes and No, numbers are going up and down. If I run ceph -s command I
> can see it decreases one time and later it increases again. I see there are
> so many blocked/slow requests. Almost all the OSDs have slow requests.
> Around 12% PGs are inactive not sure how to activate them again.
> >>>
> >>>
> >>> [root@fre101 ~]# ceph health detail
> >>> 2019-01-04 05:39:23.860142 7fc37a3a0700 -1 asok(0x7fc3740017a0)
> AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to
> bind the UNIX domain socket to
> '/var/run/ceph-guests/ceph-client.admin.1066526.140477441513808.asok': (2)
> No such file or directory
> >>> HEALTH_ERR 1 osds down; 3 pools have many more objects per pg than
> average; 472812/12392654 objects misplaced (3.815%); 3610 PGs pending on
> creation; Reduced data availability: 6578 pgs inactive, 1882 pgs down, 86
> pgs peering, 850 pgs stale; Degraded data redundancy: 216694/12392654
> objects degraded (1.749%), 866 pgs degraded, 16 pgs undersized; 116082 slow
> requests are blocked > 32 sec; 551 stuck requests are blocked > 4096 sec;
> too many PGs per OSD (2709 > max 200)
> >>> OSD_DOWN 1 osds down
> >>> osd.28 (root=default,host=fre119) is down
> >>> MANY_OBJECTS_PER_PG 3 pools have many more objects per pg than average
> >>> pool glance-images objects per pg (10478) is more than 92.7257
> times cluster average (113)
> >>> pool vms objects per pg (4717) is more than 41.7434 times cluster
> average (113)
> >>> pool volumes objects per pg (1220) is more than 10.7965 times
> cluster average (113)
> >>> OBJECT_MISPLACED 472812/12392654 objects misplaced (3.815%)
> >>> PENDING_CREATING_PGS 3610 PGs pending on creation
> >>> osds
> [osd.0,osd.1,osd.10,osd.11,osd.14,osd.15,osd.17,osd.18,osd.19,osd.20,osd.21,osd.22,osd.23,osd.25,osd.26,osd.27,osd.28,osd.3,osd.30,osd.32,osd.33,osd.35,osd.36,osd.37,osd.38,osd.4,osd.5,osd.6,osd.7,osd.9]
> have pending PGs.
> >>> PG_AVAILABILITY Reduced data availability: 6578 pgs inactive, 1882 pgs
> down, 86 pgs peering, 850 pgs stale
> >>> pg 10.900 is down, acting [18]
> >>> pg 10.90e is stuck inactive for 60266.030164, current state
> activating, last acting [2,38]
> >>> pg 10.913 is stuck stale for 1887.552862, current state
> stale+down, last acting [9]
> >>> pg 10.915 is stuck inactive for 60266.215231, current state
> activating, last acting [30,38]
> >>> pg 11.903 is stuck inactive for 59294.465961, current state
> activating, last acting [11,38]
> >>> pg 11.910 is down, acting [21]
> >>> pg 11.919 is down, acting [25]
> >>> pg 12.902 is stuck inactive for 57118.544590, current state
> activating, last acting [36,14]
> >>> pg 13.8f8 is stuck inactive for 60707.167787, current state
> activating, last acting [29,37]
> >>> pg 13.901 is stuck stale for 60226.543289, current state
> stale+active+clean, last acting [1,31]
> >>> pg 13.905 is stuck inactive for 60266.050940, current state
> activating, last acting [2,36]
> >>> pg 13.909 is stuck inactive for 60707.160714, current state
> activating, last acting [34,36]
> >>> pg 13.90e is stuck inactive for 60707.410749, current state
> activating, last acting [21,36]
> >>> pg 13.911 is down, acting [25]
> >>> pg 13.914 is stale+down, acting [29]
> >>> pg 13.917 is stuck stale for 580.224688, current state stale+down,
> last acting [16]
> >>> pg 14.901 is stuck inactive for 60266.037762, current state
> activating+degraded, last acting [22,37]
> >>> pg 14.90f is stuck inactive for 60296.996447, current state
> activating, last acting [30,36]
> >>> pg 14.910 is stuck inactive for 60266.077310, current state
> activating+degraded, last acting [17,37]
> >>>

Re: [ceph-users] Help Ceph Cluster Down

[Kevin Olbrich]

If you realy created and destroyed OSDs before the cluster healed
itself, this data will be permanently lost (not found / inactive).
Also your PG count is so much oversized, the calculation for peering
will most likely break because this was never tested.

If this is a critical cluster, I would start a new one and bring back
the backups (using a better PG count).

Kevin

Am Fr., 4. Jan. 2019 um 20:25 Uhr schrieb Arun POONIA
:
>
> Can anyone comment on this issue please, I can't seem to bring my cluster 
> healthy.
>
> On Fri, Jan 4, 2019 at 6:26 AM Arun POONIA  
> wrote:
>>
>> Hi Caspar,
>>
>> Number of IOPs are also quite low. It used be around 1K Plus on one of Pool 
>> (VMs) now its like close to 10-30 .
>>
>> Thansk
>> Arun
>>
>> On Fri, Jan 4, 2019 at 5:41 AM Arun POONIA  
>> wrote:
>>>
>>> Hi Caspar,
>>>
>>> Yes and No, numbers are going up and down. If I run ceph -s command I can 
>>> see it decreases one time and later it increases again. I see there are so 
>>> many blocked/slow requests. Almost all the OSDs have slow requests. Around 
>>> 12% PGs are inactive not sure how to activate them again.
>>>
>>>
>>> [root@fre101 ~]# ceph health detail
>>> 2019-01-04 05:39:23.860142 7fc37a3a0700 -1 asok(0x7fc3740017a0) 
>>> AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to 
>>> bind the UNIX domain socket to 
>>> '/var/run/ceph-guests/ceph-client.admin.1066526.140477441513808.asok': (2) 
>>> No such file or directory
>>> HEALTH_ERR 1 osds down; 3 pools have many more objects per pg than average; 
>>> 472812/12392654 objects misplaced (3.815%); 3610 PGs pending on creation; 
>>> Reduced data availability: 6578 pgs inactive, 1882 pgs down, 86 pgs 
>>> peering, 850 pgs stale; Degraded data redundancy: 216694/12392654 objects 
>>> degraded (1.749%), 866 pgs degraded, 16 pgs undersized; 116082 slow 
>>> requests are blocked > 32 sec; 551 stuck requests are blocked > 4096 sec; 
>>> too many PGs per OSD (2709 > max 200)
>>> OSD_DOWN 1 osds down
>>> osd.28 (root=default,host=fre119) is down
>>> MANY_OBJECTS_PER_PG 3 pools have many more objects per pg than average
>>> pool glance-images objects per pg (10478) is more than 92.7257 times 
>>> cluster average (113)
>>> pool vms objects per pg (4717) is more than 41.7434 times cluster 
>>> average (113)
>>> pool volumes objects per pg (1220) is more than 10.7965 times cluster 
>>> average (113)
>>> OBJECT_MISPLACED 472812/12392654 objects misplaced (3.815%)
>>> PENDING_CREATING_PGS 3610 PGs pending on creation
>>> osds 
>>> [osd.0,osd.1,osd.10,osd.11,osd.14,osd.15,osd.17,osd.18,osd.19,osd.20,osd.21,osd.22,osd.23,osd.25,osd.26,osd.27,osd.28,osd.3,osd.30,osd.32,osd.33,osd.35,osd.36,osd.37,osd.38,osd.4,osd.5,osd.6,osd.7,osd.9]
>>>  have pending PGs.
>>> PG_AVAILABILITY Reduced data availability: 6578 pgs inactive, 1882 pgs 
>>> down, 86 pgs peering, 850 pgs stale
>>> pg 10.900 is down, acting [18]
>>> pg 10.90e is stuck inactive for 60266.030164, current state activating, 
>>> last acting [2,38]
>>> pg 10.913 is stuck stale for 1887.552862, current state stale+down, 
>>> last acting [9]
>>> pg 10.915 is stuck inactive for 60266.215231, current state activating, 
>>> last acting [30,38]
>>> pg 11.903 is stuck inactive for 59294.465961, current state activating, 
>>> last acting [11,38]
>>> pg 11.910 is down, acting [21]
>>> pg 11.919 is down, acting [25]
>>> pg 12.902 is stuck inactive for 57118.544590, current state activating, 
>>> last acting [36,14]
>>> pg 13.8f8 is stuck inactive for 60707.167787, current state activating, 
>>> last acting [29,37]
>>> pg 13.901 is stuck stale for 60226.543289, current state 
>>> stale+active+clean, last acting [1,31]
>>> pg 13.905 is stuck inactive for 60266.050940, current state activating, 
>>> last acting [2,36]
>>> pg 13.909 is stuck inactive for 60707.160714, current state activating, 
>>> last acting [34,36]
>>> pg 13.90e is stuck inactive for 60707.410749, current state activating, 
>>> last acting [21,36]
>>> pg 13.911 is down, acting [25]
>>> pg 13.914 is stale+down, acting [29]
>>> pg 13.917 is stuck stale for 580.224688, current state stale+down, last 
>>> acting [16]
>>> pg 14.901 is stuck inactive for 60266.037762, current state 
>>> activating+degraded, last acting [22,37]
>>> pg 14.90f is stuck inactive for 60296.996447, current state activating, 
>>> last acting [30,36]
>>> pg 14.910 is stuck inactive for 60266.077310, current state 
>>> activating+degraded, last acting [17,37]
>>> pg 14.915 is stuck inactive for 60266.032445, current state activating, 
>>> last acting [34,36]
>>> pg 15.8fa is stuck stale for 560.223249, current state stale+down, last 
>>> acting [8]
>>> pg 15.90c is stuck inactive for 59294.402388, current state activating, 
>>> last acting [29,38]
>>> pg 15.90d is stuck inactive for 60266.176492, current state activating, 
>>> last acting [5,36]
>>> pg 15.915 

Re: [ceph-users] ceph-mgr fails to restart after upgrade to mimic

[Gregory Farnum]

You can also get more data by checking what the monitor logs for that
manager on the connect attempt (if you turn up its debug mon or debug
ms settings). If one of your managers is behaving, I'd examine its
configuration file and compare to the others. For instance, that
"Invalid argument" might mean the manager is trying to use "AUTH_NONE"
(no CephX) and the monitors aren't allowing that.
-Greg

On Fri, Jan 4, 2019 at 6:26 AM Randall Smith  wrote:
>
> Greetings,
>
> I'm upgrading my cluster from luminous to mimic. I've upgraded my monitors 
> and am attempting to upgrade the mgrs. Unfortunately, after an upgrade the 
> mgr daemon exits immediately with error code 1.
>
> I've tried running ceph-mgr in debug mode to try to see what's happening but 
> the output (below) is a bit cryptic for me. It looks like authentication 
> might be failing but it was working prior to the upgrade.
>
> I do have "auth supported = cephx" in the global section of ceph.conf.
>
> What do I need to do to fix this?
>
> Thanks.
>
> /usr/bin/ceph-mgr -f --cluster ceph --id 8 --setuser ceph --setgroup ceph -d 
> --debug_ms 5
> 2019-01-04 07:01:38.457 7f808f83f700  2 Event(0x30c42c0 nevent=5000 
> time_id=1).set_owner idx=0 owner=140190140331776
> 2019-01-04 07:01:38.457 7f808f03e700  2 Event(0x30c4500 nevent=5000 
> time_id=1).set_owner idx=1 owner=140190131939072
> 2019-01-04 07:01:38.457 7f808e83d700  2 Event(0x30c4e00 nevent=5000 
> time_id=1).set_owner idx=2 owner=140190123546368
> 2019-01-04 07:01:38.457 7f809dd5b380  1  Processor -- start
> 2019-01-04 07:01:38.477 7f809dd5b380  1 -- - start start
> 2019-01-04 07:01:38.481 7f809dd5b380  1 -- - --> 192.168.253.147:6789/0 -- 
> auth(proto 0 26 bytes epoch 0) v1 -- 0x32a6780 con 0
> 2019-01-04 07:01:38.481 7f809dd5b380  1 -- - --> 192.168.253.148:6789/0 -- 
> auth(proto 0 26 bytes epoch 0) v1 -- 0x32a6a00 con 0
> 2019-01-04 07:01:38.481 7f808e83d700  1 -- 192.168.253.148:0/1359135487 
> learned_addr learned my addr 192.168.253.148:0/1359135487
> 2019-01-04 07:01:38.481 7f808e83d700  2 -- 192.168.253.148:0/1359135487 >> 
> 192.168.253.148:6789/0 conn(0x332d500 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ 
> pgs=0 cs=0 l=0)._process_connection got newly_a$
> ked_seq 0 vs out_seq 0
> 2019-01-04 07:01:38.481 7f808f03e700  2 -- 192.168.253.148:0/1359135487 >> 
> 192.168.253.147:6789/0 conn(0x332ce00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ 
> pgs=0 cs=0 l=0)._process_connection got newly_a$
> ked_seq 0 vs out_seq 0
> 2019-01-04 07:01:38.481 7f808f03e700  5 -- 192.168.253.148:0/1359135487 >> 
> 192.168.253.147:6789/0 conn(0x332ce00 :-1 
> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74172 cs=1 l=1). rx mon.1 
> seq
> 1 0x30c5440 mon_map magic: 0 v1
> 2019-01-04 07:01:38.481 7f808e83d700  5 -- 192.168.253.148:0/1359135487 >> 
> 192.168.253.148:6789/0 conn(0x332d500 :-1 
> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74275 cs=1 l=1). rx mon.2 
> seq
> 1 0x30c5680 mon_map magic: 0 v1
> 2019-01-04 07:01:38.481 7f808f03e700  5 -- 192.168.253.148:0/1359135487 >> 
> 192.168.253.147:6789/0 conn(0x332ce00 :-1 
> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74172 cs=1 l=1). rx mon.1 
> seq
> 2 0x32a6780 auth_reply(proto 2 0 (0) Success) v1
> 2019-01-04 07:01:38.481 7f808e83d700  5 -- 192.168.253.148:0/1359135487 >> 
> 192.168.253.148:6789/0 conn(0x332d500 :-1 
> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74275 cs=1 l=1). rx mon.2 
> seq
> 2 0x32a6a00 auth_reply(proto 2 0 (0) Success) v1
> 2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 <== 
> mon.1 192.168.253.147:6789/0 1  mon_map magic: 0 v1  370+0+0 
> (3034216899 0 0) 0x30c5440 con 0x332ce00
> 2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 <== 
> mon.2 192.168.253.148:6789/0 1  mon_map magic: 0 v1  370+0+0 
> (3034216899 0 0) 0x30c5680 con 0x332d500
> 2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 <== 
> mon.1 192.168.253.147:6789/0 2  auth_reply(proto 2 0 (0) Success) v1  
> 33+0+0 (3430158761 0 0) 0x32a6780 con 0x33$
> ce00
> 2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 --> 
> 192.168.253.147:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x32a6f00 con 0
> 2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 <== 
> mon.2 192.168.253.148:6789/0 2  auth_reply(proto 2 0 (0) Success) v1  
> 33+0+0 (3242503871 0 0) 0x32a6a00 con 0x33$
> d500
> 2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 --> 
> 192.168.253.148:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x32a6780 con 0
> 2019-01-04 07:01:38.481 7f808f03e700  5 -- 192.168.253.148:0/1359135487 >> 
> 192.168.253.147:6789/0 conn(0x332ce00 :-1 
> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74172 cs=1 l=1). rx mon.1 
> seq
> 3 0x32a6f00 auth_reply(proto 2 -22 (22) Invalid argument) v1
> 2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 <== 
> mon.1 192.168.253.147:6789/0 3  

Re: [ceph-users] Help Ceph Cluster Down

[Arun POONIA]

Can anyone comment on this issue please, I can't seem to bring my cluster
healthy.

On Fri, Jan 4, 2019 at 6:26 AM Arun POONIA 
wrote:

> Hi Caspar,
>
> Number of IOPs are also quite low. It used be around 1K Plus on one of
> Pool (VMs) now its like close to 10-30 .
>
> Thansk
> Arun
>
> On Fri, Jan 4, 2019 at 5:41 AM Arun POONIA 
> wrote:
>
>> Hi Caspar,
>>
>> Yes and No, numbers are going up and down. If I run ceph -s command I can
>> see it decreases one time and later it increases again. I see there are so
>> many blocked/slow requests. Almost all the OSDs have slow requests. Around
>> 12% PGs are inactive not sure how to activate them again.
>>
>>
>> [root@fre101 ~]# ceph health detail
>> 2019-01-04 05:39:23.860142 7fc37a3a0700 -1 asok(0x7fc3740017a0)
>> AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to
>> bind the UNIX domain socket to
>> '/var/run/ceph-guests/ceph-client.admin.1066526.140477441513808.asok': (2)
>> No such file or directory
>> HEALTH_ERR 1 osds down; 3 pools have many more objects per pg than
>> average; 472812/12392654 objects misplaced (3.815%); 3610 PGs pending on
>> creation; Reduced data availability: 6578 pgs inactive, 1882 pgs down, 86
>> pgs peering, 850 pgs stale; Degraded data redundancy: 216694/12392654
>> objects degraded (1.749%), 866 pgs degraded, 16 pgs undersized; 116082 slow
>> requests are blocked > 32 sec; 551 stuck requests are blocked > 4096 sec;
>> too many PGs per OSD (2709 > max 200)
>> OSD_DOWN 1 osds down
>> osd.28 (root=default,host=fre119) is down
>> MANY_OBJECTS_PER_PG 3 pools have many more objects per pg than average
>> pool glance-images objects per pg (10478) is more than 92.7257 times
>> cluster average (113)
>> pool vms objects per pg (4717) is more than 41.7434 times cluster
>> average (113)
>> pool volumes objects per pg (1220) is more than 10.7965 times cluster
>> average (113)
>> OBJECT_MISPLACED 472812/12392654 objects misplaced (3.815%)
>> PENDING_CREATING_PGS 3610 PGs pending on creation
>> osds
>> [osd.0,osd.1,osd.10,osd.11,osd.14,osd.15,osd.17,osd.18,osd.19,osd.20,osd.21,osd.22,osd.23,osd.25,osd.26,osd.27,osd.28,osd.3,osd.30,osd.32,osd.33,osd.35,osd.36,osd.37,osd.38,osd.4,osd.5,osd.6,osd.7,osd.9]
>> have pending PGs.
>> PG_AVAILABILITY Reduced data availability: 6578 pgs inactive, 1882 pgs
>> down, 86 pgs peering, 850 pgs stale
>> pg 10.900 is down, acting [18]
>> pg 10.90e is stuck inactive for 60266.030164, current state
>> activating, last acting [2,38]
>> pg 10.913 is stuck stale for 1887.552862, current state stale+down,
>> last acting [9]
>> pg 10.915 is stuck inactive for 60266.215231, current state
>> activating, last acting [30,38]
>> pg 11.903 is stuck inactive for 59294.465961, current state
>> activating, last acting [11,38]
>> pg 11.910 is down, acting [21]
>> pg 11.919 is down, acting [25]
>> pg 12.902 is stuck inactive for 57118.544590, current state
>> activating, last acting [36,14]
>> pg 13.8f8 is stuck inactive for 60707.167787, current state
>> activating, last acting [29,37]
>> pg 13.901 is stuck stale for 60226.543289, current state
>> stale+active+clean, last acting [1,31]
>> pg 13.905 is stuck inactive for 60266.050940, current state
>> activating, last acting [2,36]
>> pg 13.909 is stuck inactive for 60707.160714, current state
>> activating, last acting [34,36]
>> pg 13.90e is stuck inactive for 60707.410749, current state
>> activating, last acting [21,36]
>> pg 13.911 is down, acting [25]
>> pg 13.914 is stale+down, acting [29]
>> pg 13.917 is stuck stale for 580.224688, current state stale+down,
>> last acting [16]
>> pg 14.901 is stuck inactive for 60266.037762, current state
>> activating+degraded, last acting [22,37]
>> pg 14.90f is stuck inactive for 60296.996447, current state
>> activating, last acting [30,36]
>> pg 14.910 is stuck inactive for 60266.077310, current state
>> activating+degraded, last acting [17,37]
>> pg 14.915 is stuck inactive for 60266.032445, current state
>> activating, last acting [34,36]
>> pg 15.8fa is stuck stale for 560.223249, current state stale+down,
>> last acting [8]
>> pg 15.90c is stuck inactive for 59294.402388, current state
>> activating, last acting [29,38]
>> pg 15.90d is stuck inactive for 60266.176492, current state
>> activating, last acting [5,36]
>> pg 15.915 is down, acting [0]
>> pg 15.917 is stuck inactive for 56279.658951, current state
>> activating, last acting [13,38]
>> pg 15.91c is stuck stale for 374.590704, current state stale+down,
>> last acting [12]
>> pg 16.903 is stuck inactive for 56580.905961, current state
>> activating, last acting [25,37]
>> pg 16.90e is stuck inactive for 60266.271680, current state
>> activating, last acting [14,37]
>> pg 16.919 is stuck inactive for 59901.802184, current state
>> activating, last acting [20,37]
>> pg 16.91e is stuck inactive for 60297.038159, 

Re: [ceph-users] Mimic 13.2.3?

[Gregory Farnum]

Regarding 13.2.3 specifically:

As Abhishek says, there are no known issues in the release. It went
through our full and proper release validation; nobody has spotted any
last-minute bugs. The release notes are available in the git
repository: 
https://github.com/ceph/ceph/blob/master/doc/releases/mimic.rst#v1323-mimic
It was not announced on the mailing list or in a blog post because the
person cutting it didn't have those steps in their checklist, but ff
you install these packages, you will be fine and in good shape.
The release did leave out some non-regression fixes that were intended
to go out so we're building a 13.2.4 with a few more patches included,
so if doing an upgrade is a big deal for you then you should probably
wait for those until you go through the effort.


Regarding Ceph releases more generally:

As we scale up the number of stable releases being maintained and have
more companies involved in Ceph, we are trying as a community to get
more people and groups involved in building and running releases. In
the long term this makes for a more stable and collaborative project.
In the short term, we're dealing with issues like "nobody told me I
should post to the blog when the release was done", figuring out
processes for releases that don't rely on a few blessed people knowing
the steps to take and sharing them over irc, and dealing with the
security concerns presented by making tools like signing keys
available.

I imagine we will discuss all this in more detail after the release,
but everybody's patience is appreciated as we work through these
challenges.
-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Usage of devices in SSD pool vary very much

[Kevin Olbrich]

PS: Could be http://tracker.ceph.com/issues/36361
There is one HDD OSD that is out (which will not be replaced because
the SSD pool will get the images and the hdd pool will be deleted).

Kevin

Am Fr., 4. Jan. 2019 um 19:46 Uhr schrieb Kevin Olbrich :
>
> Hi!
>
> I did what you wrote but my MGRs started to crash again:
> root@adminnode:~# ceph -s
>   cluster:
> id: 086d9f80-6249-4594-92d0-e31b6a9c
> health: HEALTH_WARN
> no active mgr
> 105498/6277782 objects misplaced (1.680%)
>
>   services:
> mon: 3 daemons, quorum mon01,mon02,mon03
> mgr: no daemons active
> osd: 44 osds: 43 up, 43 in
>
>   data:
> pools:   4 pools, 1616 pgs
> objects: 1.88M objects, 7.07TiB
> usage:   13.2TiB used, 16.7TiB / 29.9TiB avail
> pgs: 105498/6277782 objects misplaced (1.680%)
>  1606 active+clean
>  8active+remapped+backfill_wait
>  2active+remapped+backfilling
>
>   io:
> client:   5.51MiB/s rd, 3.38MiB/s wr, 33op/s rd, 317op/s wr
> recovery: 60.3MiB/s, 15objects/s
>
>
> MON 1 log:
>-13> 2019-01-04 14:05:04.432186 7fec56a93700  4 mgr ms_dispatch
> active mgrdigest v1
>-12> 2019-01-04 14:05:04.432194 7fec56a93700  4 mgr ms_dispatch mgrdigest 
> v1
>-11> 2019-01-04 14:05:04.822041 7fec434e1700  4 mgr[balancer]
> Optimize plan auto_2019-01-04_14:05:04
>-10> 2019-01-04 14:05:04.822170 7fec434e1700  4 mgr get_config
> get_configkey: mgr/balancer/mode
> -9> 2019-01-04 14:05:04.822231 7fec434e1700  4 mgr get_config
> get_configkey: mgr/balancer/max_misplaced
> -8> 2019-01-04 14:05:04.822268 7fec434e1700  4 ceph_config_get
> max_misplaced not found
> -7> 2019-01-04 14:05:04.822444 7fec434e1700  4 mgr[balancer] Mode
> upmap, max misplaced 0.05
> -6> 2019-01-04 14:05:04.822849 7fec434e1700  4 mgr[balancer] do_upmap
> -5> 2019-01-04 14:05:04.822923 7fec434e1700  4 mgr get_config
> get_configkey: mgr/balancer/upmap_max_iterations
> -4> 2019-01-04 14:05:04.822964 7fec434e1700  4 ceph_config_get
> upmap_max_iterations not found
> -3> 2019-01-04 14:05:04.823013 7fec434e1700  4 mgr get_config
> get_configkey: mgr/balancer/upmap_max_deviation
> -2> 2019-01-04 14:05:04.823048 7fec434e1700  4 ceph_config_get
> upmap_max_deviation not found
> -1> 2019-01-04 14:05:04.823265 7fec434e1700  4 mgr[balancer] pools
> ['rbd_vms_hdd', 'rbd_vms_ssd', 'rbd_vms_ssd_01', 'rbd_vms_ssd_01_ec']
>  0> 2019-01-04 14:05:04.836124 7fec434e1700 -1
> /build/ceph-12.2.8/src/osd/OSDMap.cc: In function 'int
> OSDMap::calc_pg_upmaps(CephContext*, float, int, const std::set int>&, OSDMap::Incremental*)' thread 7fec434e1700 time 2019-01-04
> 14:05:04.832885
> /build/ceph-12.2.8/src/osd/OSDMap.cc: 4102: FAILED assert(target > 0)
>
>  ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0)
> luminous (stable)
>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x102) [0x558c3c0bb572]
>  2: (OSDMap::calc_pg_upmaps(CephContext*, float, int, std::set std::less, std::allocator > const&,
> OSDMap::Incremental*)+0x2801) [0x558c3c1c0ee1]
>  3: (()+0x2f3020) [0x558c3bf5d020]
>  4: (PyEval_EvalFrameEx()+0x8a51) [0x7fec5e832971]
>  5: (PyEval_EvalCodeEx()+0x85c) [0x7fec5e96805c]
>  6: (PyEval_EvalFrameEx()+0x6ffd) [0x7fec5e830f1d]
>  7: (PyEval_EvalFrameEx()+0x7124) [0x7fec5e831044]
>  8: (PyEval_EvalFrameEx()+0x7124) [0x7fec5e831044]
>  9: (PyEval_EvalCodeEx()+0x85c) [0x7fec5e96805c]
>  10: (()+0x13e370) [0x7fec5e8be370]
>  11: (PyObject_Call()+0x43) [0x7fec5e891273]
>  12: (()+0x1853ac) [0x7fec5e9053ac]
>  13: (PyObject_Call()+0x43) [0x7fec5e891273]
>  14: (PyObject_CallMethod()+0xf4) [0x7fec5e892444]
>  15: (PyModuleRunner::serve()+0x5c) [0x558c3bf5a18c]
>  16: (PyModuleRunner::PyModuleRunnerThread::entry()+0x1b8) [0x558c3bf5a998]
>  17: (()+0x76ba) [0x7fec5d74c6ba]
>  18: (clone()+0x6d) [0x7fec5c7b841d]
>  NOTE: a copy of the executable, or `objdump -rdS ` is
> needed to interpret this.
>
> --- logging levels ---
>0/ 5 none
>0/ 1 lockdep
>0/ 1 context
>1/ 1 crush
>1/ 5 mds
>1/ 5 mds_balancer
>1/ 5 mds_locker
>1/ 5 mds_log
>1/ 5 mds_log_expire
>1/ 5 mds_migrator
>0/ 1 buffer
>0/ 1 timer
>0/ 1 filer
>0/ 1 striper
>0/ 1 objecter
>0/ 5 rados
>0/ 5 rbd
>0/ 5 rbd_mirror
>0/ 5 rbd_replay
>0/ 5 journaler
>0/ 5 objectcacher
>0/ 5 client
>1/ 5 osd
>0/ 5 optracker
>0/ 5 objclass
>1/ 3 filestore
>1/ 3 journal
>0/ 5 ms
>1/ 5 mon
>0/10 monc
>1/ 5 paxos
>0/ 5 tp
>1/ 5 auth
>1/ 5 crypto
>1/ 1 finisher
>1/ 1 reserver
>1/ 5 heartbeatmap
>1/ 5 perfcounter
>1/ 5 rgw
>1/10 civetweb
>1/ 5 javaclient
>1/ 5 asok
>1/ 1 throttle
>0/ 0 refs
>1/ 5 xio
>1/ 5 compressor
>1/ 5 bluestore
>1/ 5 bluefs
>1/ 3 bdev
>1/ 5 kstore
>4/ 5 rocksdb
>4/ 5 leveldb
>4/ 5 memdb
>1/ 5 kinetic
>

Re: [ceph-users] Usage of devices in SSD pool vary very much

[Kevin Olbrich]

Hi!

I did what you wrote but my MGRs started to crash again:
root@adminnode:~# ceph -s
  cluster:
id: 086d9f80-6249-4594-92d0-e31b6a9c
health: HEALTH_WARN
no active mgr
105498/6277782 objects misplaced (1.680%)

  services:
mon: 3 daemons, quorum mon01,mon02,mon03
mgr: no daemons active
osd: 44 osds: 43 up, 43 in

  data:
pools:   4 pools, 1616 pgs
objects: 1.88M objects, 7.07TiB
usage:   13.2TiB used, 16.7TiB / 29.9TiB avail
pgs: 105498/6277782 objects misplaced (1.680%)
 1606 active+clean
 8active+remapped+backfill_wait
 2active+remapped+backfilling

  io:
client:   5.51MiB/s rd, 3.38MiB/s wr, 33op/s rd, 317op/s wr
recovery: 60.3MiB/s, 15objects/s


MON 1 log:
   -13> 2019-01-04 14:05:04.432186 7fec56a93700  4 mgr ms_dispatch
active mgrdigest v1
   -12> 2019-01-04 14:05:04.432194 7fec56a93700  4 mgr ms_dispatch mgrdigest v1
   -11> 2019-01-04 14:05:04.822041 7fec434e1700  4 mgr[balancer]
Optimize plan auto_2019-01-04_14:05:04
   -10> 2019-01-04 14:05:04.822170 7fec434e1700  4 mgr get_config
get_configkey: mgr/balancer/mode
-9> 2019-01-04 14:05:04.822231 7fec434e1700  4 mgr get_config
get_configkey: mgr/balancer/max_misplaced
-8> 2019-01-04 14:05:04.822268 7fec434e1700  4 ceph_config_get
max_misplaced not found
-7> 2019-01-04 14:05:04.822444 7fec434e1700  4 mgr[balancer] Mode
upmap, max misplaced 0.05
-6> 2019-01-04 14:05:04.822849 7fec434e1700  4 mgr[balancer] do_upmap
-5> 2019-01-04 14:05:04.822923 7fec434e1700  4 mgr get_config
get_configkey: mgr/balancer/upmap_max_iterations
-4> 2019-01-04 14:05:04.822964 7fec434e1700  4 ceph_config_get
upmap_max_iterations not found
-3> 2019-01-04 14:05:04.823013 7fec434e1700  4 mgr get_config
get_configkey: mgr/balancer/upmap_max_deviation
-2> 2019-01-04 14:05:04.823048 7fec434e1700  4 ceph_config_get
upmap_max_deviation not found
-1> 2019-01-04 14:05:04.823265 7fec434e1700  4 mgr[balancer] pools
['rbd_vms_hdd', 'rbd_vms_ssd', 'rbd_vms_ssd_01', 'rbd_vms_ssd_01_ec']
 0> 2019-01-04 14:05:04.836124 7fec434e1700 -1
/build/ceph-12.2.8/src/osd/OSDMap.cc: In function 'int
OSDMap::calc_pg_upmaps(CephContext*, float, int, const std::set&, OSDMap::Incremental*)' thread 7fec434e1700 time 2019-01-04
14:05:04.832885
/build/ceph-12.2.8/src/osd/OSDMap.cc: 4102: FAILED assert(target > 0)

 ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0)
luminous (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x558c3c0bb572]
 2: (OSDMap::calc_pg_upmaps(CephContext*, float, int, std::set, std::allocator > const&,
OSDMap::Incremental*)+0x2801) [0x558c3c1c0ee1]
 3: (()+0x2f3020) [0x558c3bf5d020]
 4: (PyEval_EvalFrameEx()+0x8a51) [0x7fec5e832971]
 5: (PyEval_EvalCodeEx()+0x85c) [0x7fec5e96805c]
 6: (PyEval_EvalFrameEx()+0x6ffd) [0x7fec5e830f1d]
 7: (PyEval_EvalFrameEx()+0x7124) [0x7fec5e831044]
 8: (PyEval_EvalFrameEx()+0x7124) [0x7fec5e831044]
 9: (PyEval_EvalCodeEx()+0x85c) [0x7fec5e96805c]
 10: (()+0x13e370) [0x7fec5e8be370]
 11: (PyObject_Call()+0x43) [0x7fec5e891273]
 12: (()+0x1853ac) [0x7fec5e9053ac]
 13: (PyObject_Call()+0x43) [0x7fec5e891273]
 14: (PyObject_CallMethod()+0xf4) [0x7fec5e892444]
 15: (PyModuleRunner::serve()+0x5c) [0x558c3bf5a18c]
 16: (PyModuleRunner::PyModuleRunnerThread::entry()+0x1b8) [0x558c3bf5a998]
 17: (()+0x76ba) [0x7fec5d74c6ba]
 18: (clone()+0x6d) [0x7fec5c7b841d]
 NOTE: a copy of the executable, or `objdump -rdS ` is
needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   1/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 1 reserver
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
   1/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent 1
  max_new 1000
  log_file /var/log/ceph/ceph-mgr.mon01.ceph01.srvfarm.net.log
--- end dump of recent events ---
2019-01-04 14:05:05.032479 7fec434e1700 -1 *** Caught signal (Aborted) **
 in thread 7fec434e1700 thread_name:balancer

 ceph version 12.2.8 (ae699615bac534ea496ee965ac6192cb7e0e07c0)
luminous (stable)
 1: (()+0x4105b4) [0x558c3c07a5b4]
 2: (()+0x11390) [0x7fec5d756390]
 3: 

Re: [ceph-users] Mimic 13.2.3?

[Peter Woodman]

not to mention that the current released version of mimic (.2) has a
bug that is potentially catastrophic to cephfs, known about for
months, yet it's not in the release notes. would have upgraded and
destroyed data had i not caught a thread on this list.

hopefully crowing like this isn't coming off as too obnoxious, but
yeah, the release process seems quite brittle at the moment.

On Fri, Jan 4, 2019 at 1:25 PM Brady Deetz  wrote:
>
> I agree with the comments above. I don't feel comfortable upgrading because I 
> never know what's been deemed stable. We used to get an announcement at the 
> same times that the packages hit the repo. What's going on? Frankly, the 
> entire release cycle of Mimic has seemed very haphazard.
>
> On Fri, Jan 4, 2019 at 10:22 AM Daniel Baumann  wrote:
>>
>> On 01/04/2019 05:07 PM, Matthew Vernon wrote:
>> > how is it still the case that packages are being pushed onto the official 
>> > ceph.com repos that people
>> > shouldn't install?
>>
>> We're still on 12.2.5 because of this. Basically every 12.2.x after that
>> had notes on the mailinglist like "don't use, wait for ..."
>>
>> I don't dare updating to 13.2.
>>
>> For the 10.2.x and 11.2.x cycles, we upgraded our production cluster
>> within a matter of days after the release of an update. Since the second
>> half of the 12.2.x releases, this seems to be not possible anymore.
>>
>> Ceph is great and all, but this decrease of release quality seriously
>> harms the image and perception of Ceph as a stable software platform in
>> the enterprise environment and makes people do the wrong things (rotting
>> systems update-wise, for the sake of stability).
>>
>> Regards,
>> Daniel
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Mimic 13.2.3?

[Brady Deetz]

I agree with the comments above. I don't feel comfortable upgrading because
I never know what's been deemed stable. We used to get an announcement at
the same times that the packages hit the repo. What's going on? Frankly,
the entire release cycle of Mimic has seemed very haphazard.

On Fri, Jan 4, 2019 at 10:22 AM Daniel Baumann 
wrote:

> On 01/04/2019 05:07 PM, Matthew Vernon wrote:
> > how is it still the case that packages are being pushed onto the
> official ceph.com repos that people
> > shouldn't install?
>
> We're still on 12.2.5 because of this. Basically every 12.2.x after that
> had notes on the mailinglist like "don't use, wait for ..."
>
> I don't dare updating to 13.2.
>
> For the 10.2.x and 11.2.x cycles, we upgraded our production cluster
> within a matter of days after the release of an update. Since the second
> half of the 12.2.x releases, this seems to be not possible anymore.
>
> Ceph is great and all, but this decrease of release quality seriously
> harms the image and perception of Ceph as a stable software platform in
> the enterprise environment and makes people do the wrong things (rotting
> systems update-wise, for the sake of stability).
>
> Regards,
> Daniel
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph blog RSS/Atom URL?

[Gregory Farnum]

It looks like ceph.com/feed is the RSS url?

On Fri, Jan 4, 2019 at 5:52 AM Jan Kasprzak  wrote:

> Hello,
>
> is there any RSS or Atom source for Ceph blog? I have looked inside
> the https://ceph.com/community/blog/ HTML source, but there is no
>  or anything mentioning RSS or Atom.
>
> Thanks,
>
> -Yenya
>
> --
> | Jan "Yenya" Kasprzak 
> |
> | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5
> |
>  This is the world we live in: the way to deal with computers is to google
>  the symptoms, and hope that you don't have to watch a video. --P. Zaitcev
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-mgr fails to restart after upgrade to mimic

[Randall Smith]

To test things, I tried created a new mgr in case there was some weird
corruption with the old key but I'm seeing the same behavior with the new
mgr.

On Fri, Jan 4, 2019 at 11:03 AM Randall Smith  wrote:

> The keys in the keyrings for the broken mgrs match what is shows in ceph
> auth list. The relevant entries are below so that you can see the caps.
>
> I am having problems with both mgr.6 and mgr.8. mgr.7 is the only mgr
> currently functioning.
>
> mgr.6
> key: [redacted]
> caps: [mds] allow *
> caps: [mgr] allow r
> caps: [mon] allow profile mgr
> caps: [osd] allow *
> mgr.7
> key: [redacted]
> caps: [mds] allow *
> caps: [mgr] allow r
> caps: [mon] allow profile mgr
> caps: [osd] allow *
> mgr.8
> key: [redacted]
> caps: [mds] allow *
> caps: [mon] allow profile mgr
> caps: [osd] allow *
>
> I agree that an auth issue seems unlikely to have been triggered but I'm
> not sure what else it can be.
>
>
> On Fri, Jan 4, 2019 at 10:51 AM Steve Taylor <
> steve.tay...@storagecraft.com> wrote:
>
>> I can't think of why the upgrade would have broken your keys, but have
>> you verified that the mons still have the correct mgr keys configured?
>> 'ceph auth ls' should list an mgr. key for each mgr with a key
>> matching the contents of /var/lib/ceph/mgr/-/keyring on the
>> mgr host and some caps that should minimally include '[mon] allow profile
>> mgr' and '[osd] allow *' I would think.
>>
>> Again, it seems unlikely that this would have broken with the upgrade if
>> it had been working previously, but if you're seeing auth errors it might
>> be something to check out.
>>
>> --
>>
>> *Steve Taylor* | Senior Software Engineer | *StorageCraft Technology
>> Corporation* 
>> 380 Data Drive Suite 300 | Draper | Utah | 84020
>> *Office:* 801.871.2799 |
>> --
>> If you are not the intended recipient of this message or received it
>> erroneously, please notify the sender and delete it, together with any
>> attachments, and be advised that any dissemination or copying of this
>> message is prohibited.
>> --
>>
>> On Fri, 2019-01-04 at 07:26 -0700, Randall Smith wrote:
>>
>> Greetings,
>>
>> I'm upgrading my cluster from luminous to mimic. I've upgraded my
>> monitors and am attempting to upgrade the mgrs. Unfortunately, after an
>> upgrade the mgr daemon exits immediately with error code 1.
>>
>> I've tried running ceph-mgr in debug mode to try to see what's happening
>> but the output (below) is a bit cryptic for me. It looks like
>> authentication might be failing but it was working prior to the upgrade.
>>
>> I do have "auth supported = cephx" in the global section of ceph.conf.
>>
>> What do I need to do to fix this?
>>
>> Thanks.
>>
>> /usr/bin/ceph-mgr -f --cluster ceph --id 8 --setuser ceph --setgroup ceph
>> -d --debug_ms 5
>>
>> 2019-01-04 07:01:38.457 7f808f83f700  2 Event(0x30c42c0 nevent=5000
>> time_id=1).set_owner idx=0 owner=140190140331776
>>
>> 2019-01-04 07:01:38.457 7f808f03e700  2 Event(0x30c4500 nevent=5000
>> time_id=1).set_owner idx=1 owner=140190131939072
>>
>> 2019-01-04 07:01:38.457 7f808e83d700  2 Event(0x30c4e00 nevent=5000
>> time_id=1).set_owner idx=2 owner=140190123546368
>>
>> 2019-01-04 07:01:38.457 7f809dd5b380  1  Processor -- start
>>
>>
>> 2019-01-04 07:01:38.477 7f809dd5b380  1 -- - start start
>>
>>
>> 2019-01-04 07:01:38.481 7f809dd5b380  1 -- - --> 192.168.253.147:6789/0
>> -- auth(proto 0 26 bytes epoch 0) v1 -- 0x32a6780 con 0
>>
>> 2019-01-04 07:01:38.481 7f809dd5b380  1 -- - --> 192.168.253.148:6789/0
>> -- auth(proto 0 26 bytes epoch 0) v1 -- 0x32a6a00 con 0
>> 2019-01-04 07:01:38.481 7f808e83d700  1 -- 192.168.253.148:0/1359135487
>> learned_addr learned my addr 192.168.253.148:0/1359135487
>> 2019-01-04 07:01:38.481 7f808e83d700  2 -- 192.168.253.148:0/1359135487
>> >> 192.168.253.148:6789/0 conn(0x332d500 :-1
>> s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got
>> newly_a$
>> ked_seq 0 vs out_seq 0
>> 2019-01-04 07:01:38.481 7f808f03e700  2 -- 192.168.253.148:0/1359135487
>> >> 192.168.253.147:6789/0 conn(0x332ce00 :-1
>> s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got
>> newly_a$
>> ked_seq 0 vs out_seq 0
>> 2019-01-04 07:01:38.481 7f808f03e700  5 -- 192.168.253.148:0/1359135487
>> >> 192.168.253.147:6789/0 conn(0x332ce00 :-1
>> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74172 cs=1 l=1). rx mon.1
>> seq
>> 1 0x30c5440 mon_map magic: 0 v1
>> 2019-01-04 07:01:38.481 7f808e83d700  5 -- 192.168.253.148:0/1359135487
>> >> 192.168.253.148:6789/0 conn(0x332d500 :-1
>> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74275 cs=1 l=1). rx mon.2
>> seq
>> 1 0x30c5680 mon_map magic: 0 v1
>> 2019-01-04 07:01:38.481 7f808f03e700  5 -- 192.168.253.148:0/1359135487
>> >> 192.168.253.147:6789/0 conn(0x332ce00 :-1

Re: [ceph-users] ceph-mgr fails to restart after upgrade to mimic

[Randall Smith]

The keys in the keyrings for the broken mgrs match what is shows in ceph
auth list. The relevant entries are below so that you can see the caps.

I am having problems with both mgr.6 and mgr.8. mgr.7 is the only mgr
currently functioning.

mgr.6
key: [redacted]
caps: [mds] allow *
caps: [mgr] allow r
caps: [mon] allow profile mgr
caps: [osd] allow *
mgr.7
key: [redacted]
caps: [mds] allow *
caps: [mgr] allow r
caps: [mon] allow profile mgr
caps: [osd] allow *
mgr.8
key: [redacted]
caps: [mds] allow *
caps: [mon] allow profile mgr
caps: [osd] allow *

I agree that an auth issue seems unlikely to have been triggered but I'm
not sure what else it can be.


On Fri, Jan 4, 2019 at 10:51 AM Steve Taylor 
wrote:

> I can't think of why the upgrade would have broken your keys, but have you
> verified that the mons still have the correct mgr keys configured? 'ceph
> auth ls' should list an mgr. key for each mgr with a key matching the
> contents of /var/lib/ceph/mgr/-/keyring on the mgr host and
> some caps that should minimally include '[mon] allow profile mgr' and
> '[osd] allow *' I would think.
>
> Again, it seems unlikely that this would have broken with the upgrade if
> it had been working previously, but if you're seeing auth errors it might
> be something to check out.
>
> --
>
> *Steve Taylor* | Senior Software Engineer | *StorageCraft Technology
> Corporation* 
> 380 Data Drive Suite 300 | Draper | Utah | 84020
> *Office:* 801.871.2799 |
> --
> If you are not the intended recipient of this message or received it
> erroneously, please notify the sender and delete it, together with any
> attachments, and be advised that any dissemination or copying of this
> message is prohibited.
> --
>
> On Fri, 2019-01-04 at 07:26 -0700, Randall Smith wrote:
>
> Greetings,
>
> I'm upgrading my cluster from luminous to mimic. I've upgraded my monitors
> and am attempting to upgrade the mgrs. Unfortunately, after an upgrade the
> mgr daemon exits immediately with error code 1.
>
> I've tried running ceph-mgr in debug mode to try to see what's happening
> but the output (below) is a bit cryptic for me. It looks like
> authentication might be failing but it was working prior to the upgrade.
>
> I do have "auth supported = cephx" in the global section of ceph.conf.
>
> What do I need to do to fix this?
>
> Thanks.
>
> /usr/bin/ceph-mgr -f --cluster ceph --id 8 --setuser ceph --setgroup ceph
> -d --debug_ms 5
>
> 2019-01-04 07:01:38.457 7f808f83f700  2 Event(0x30c42c0 nevent=5000
> time_id=1).set_owner idx=0 owner=140190140331776
>
> 2019-01-04 07:01:38.457 7f808f03e700  2 Event(0x30c4500 nevent=5000
> time_id=1).set_owner idx=1 owner=140190131939072
>
> 2019-01-04 07:01:38.457 7f808e83d700  2 Event(0x30c4e00 nevent=5000
> time_id=1).set_owner idx=2 owner=140190123546368
>
> 2019-01-04 07:01:38.457 7f809dd5b380  1  Processor -- start
>
>
> 2019-01-04 07:01:38.477 7f809dd5b380  1 -- - start start
>
>
> 2019-01-04 07:01:38.481 7f809dd5b380  1 -- - --> 192.168.253.147:6789/0
> -- auth(proto 0 26 bytes epoch 0) v1 -- 0x32a6780 con 0
>
> 2019-01-04 07:01:38.481 7f809dd5b380  1 -- - --> 192.168.253.148:6789/0
> -- auth(proto 0 26 bytes epoch 0) v1 -- 0x32a6a00 con 0
> 2019-01-04 07:01:38.481 7f808e83d700  1 -- 192.168.253.148:0/1359135487
> learned_addr learned my addr 192.168.253.148:0/1359135487
> 2019-01-04 07:01:38.481 7f808e83d700  2 -- 192.168.253.148:0/1359135487
> >> 192.168.253.148:6789/0 conn(0x332d500 :-1
> s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got
> newly_a$
> ked_seq 0 vs out_seq 0
> 2019-01-04 07:01:38.481 7f808f03e700  2 -- 192.168.253.148:0/1359135487
> >> 192.168.253.147:6789/0 conn(0x332ce00 :-1
> s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got
> newly_a$
> ked_seq 0 vs out_seq 0
> 2019-01-04 07:01:38.481 7f808f03e700  5 -- 192.168.253.148:0/1359135487
> >> 192.168.253.147:6789/0 conn(0x332ce00 :-1
> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74172 cs=1 l=1). rx mon.1
> seq
> 1 0x30c5440 mon_map magic: 0 v1
> 2019-01-04 07:01:38.481 7f808e83d700  5 -- 192.168.253.148:0/1359135487
> >> 192.168.253.148:6789/0 conn(0x332d500 :-1
> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74275 cs=1 l=1). rx mon.2
> seq
> 1 0x30c5680 mon_map magic: 0 v1
> 2019-01-04 07:01:38.481 7f808f03e700  5 -- 192.168.253.148:0/1359135487
> >> 192.168.253.147:6789/0 conn(0x332ce00 :-1
> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74172 cs=1 l=1). rx mon.1
> seq
> 2 0x32a6780 auth_reply(proto 2 0 (0) Success) v1
> 2019-01-04 07:01:38.481 7f808e83d700  5 -- 192.168.253.148:0/1359135487
> >> 192.168.253.148:6789/0 conn(0x332d500 :-1
> s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74275 cs=1 l=1). rx mon.2
> seq
> 2 0x32a6a00 auth_reply(proto 2 0 (0) 

Re: [ceph-users] ceph-mgr fails to restart after upgrade to mimic

[Steve Taylor]

I can't think of why the upgrade would have broken your keys, but have you 
verified that the mons still have the correct mgr keys configured? 'ceph auth 
ls' should list an mgr. key for each mgr with a key matching the contents 
of /var/lib/ceph/mgr/-/keyring on the mgr host and some caps 
that should minimally include '[mon] allow profile mgr' and '[osd] allow *' I 
would think.

Again, it seems unlikely that this would have broken with the upgrade if it had 
been working previously, but if you're seeing auth errors it might be something 
to check out.




[cid:SC_LOGO_VERT_4C_100x72_f823be1a-ae53-43d3-975c-b054a1b22ec3.jpg]


Steve Taylor | Senior Software Engineer | StorageCraft Technology 
Corporation
380 Data Drive Suite 300 | Draper | Utah | 84020
Office: 801.871.2799 |



If you are not the intended recipient of this message or received it 
erroneously, please notify the sender and delete it, together with any 
attachments, and be advised that any dissemination or copying of this message 
is prohibited.



On Fri, 2019-01-04 at 07:26 -0700, Randall Smith wrote:
Greetings,

I'm upgrading my cluster from luminous to mimic. I've upgraded my monitors and 
am attempting to upgrade the mgrs. Unfortunately, after an upgrade the mgr 
daemon exits immediately with error code 1.

I've tried running ceph-mgr in debug mode to try to see what's happening but 
the output (below) is a bit cryptic for me. It looks like authentication might 
be failing but it was working prior to the upgrade.

I do have "auth supported = cephx" in the global section of ceph.conf.

What do I need to do to fix this?

Thanks.

/usr/bin/ceph-mgr -f --cluster ceph --id 8 --setuser ceph --setgroup ceph -d 
--debug_ms 5
2019-01-04 07:01:38.457 7f808f83f700  2 Event(0x30c42c0 nevent=5000 
time_id=1).set_owner idx=0 owner=140190140331776
2019-01-04 07:01:38.457 7f808f03e700  2 Event(0x30c4500 nevent=5000 
time_id=1).set_owner idx=1 owner=140190131939072
2019-01-04 07:01:38.457 7f808e83d700  2 Event(0x30c4e00 nevent=5000 
time_id=1).set_owner idx=2 owner=140190123546368
2019-01-04 07:01:38.457 7f809dd5b380  1  Processor -- start
2019-01-04 07:01:38.477 7f809dd5b380  1 -- - start start
2019-01-04 07:01:38.481 7f809dd5b380  1 -- - --> 
192.168.253.147:6789/0 -- auth(proto 0 26 bytes 
epoch 0) v1 -- 0x32a6780 con 0
2019-01-04 07:01:38.481 7f809dd5b380  1 -- - --> 
192.168.253.148:6789/0 -- auth(proto 0 26 bytes 
epoch 0) v1 -- 0x32a6a00 con 0
2019-01-04 07:01:38.481 7f808e83d700  1 -- 
192.168.253.148:0/1359135487 learned_addr 
learned my addr 
192.168.253.148:0/1359135487
2019-01-04 07:01:38.481 7f808e83d700  2 -- 
192.168.253.148:0/1359135487 >> 
192.168.253.148:6789/0 conn(0x332d500 :-1 
s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_a$
ked_seq 0 vs out_seq 0
2019-01-04 07:01:38.481 7f808f03e700  2 -- 
192.168.253.148:0/1359135487 >> 
192.168.253.147:6789/0 conn(0x332ce00 :-1 
s=STATE_CONNECTING_WAIT_ACK_SEQ pgs=0 cs=0 l=0)._process_connection got newly_a$
ked_seq 0 vs out_seq 0
2019-01-04 07:01:38.481 7f808f03e700  5 -- 
192.168.253.148:0/1359135487 >> 
192.168.253.147:6789/0 conn(0x332ce00 :-1 
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74172 cs=1 l=1). rx mon.1 seq
1 0x30c5440 mon_map magic: 0 v1
2019-01-04 07:01:38.481 7f808e83d700  5 -- 
192.168.253.148:0/1359135487 >> 
192.168.253.148:6789/0 conn(0x332d500 :-1 
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74275 cs=1 l=1). rx mon.2 seq
1 0x30c5680 mon_map magic: 0 v1
2019-01-04 07:01:38.481 7f808f03e700  5 -- 
192.168.253.148:0/1359135487 >> 
192.168.253.147:6789/0 conn(0x332ce00 :-1 
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74172 cs=1 l=1). rx mon.1 seq
2 0x32a6780 auth_reply(proto 2 0 (0) Success) v1
2019-01-04 07:01:38.481 7f808e83d700  5 -- 
192.168.253.148:0/1359135487 >> 
192.168.253.148:6789/0 conn(0x332d500 :-1 
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74275 cs=1 l=1). rx mon.2 seq
2 0x32a6a00 auth_reply(proto 2 0 (0) Success) v1
2019-01-04 07:01:38.481 7f808e03c700  1 -- 
192.168.253.148:0/1359135487 <== mon.1 
192.168.253.147:6789/0 1  mon_map magic: 0 
v1  370+0+0 (3034216899 0 0) 0x30c5440 con 0x332ce00
2019-01-04 07:01:38.481 7f808e03c700  1 -- 

Re: [ceph-users] Mimic 13.2.3?

[Daniel Baumann]

On 01/04/2019 05:07 PM, Matthew Vernon wrote:
> how is it still the case that packages are being pushed onto the official 
> ceph.com repos that people
> shouldn't install?

We're still on 12.2.5 because of this. Basically every 12.2.x after that
had notes on the mailinglist like "don't use, wait for ..."

I don't dare updating to 13.2.

For the 10.2.x and 11.2.x cycles, we upgraded our production cluster
within a matter of days after the release of an update. Since the second
half of the 12.2.x releases, this seems to be not possible anymore.

Ceph is great and all, but this decrease of release quality seriously
harms the image and perception of Ceph as a stable software platform in
the enterprise environment and makes people do the wrong things (rotting
systems update-wise, for the sake of stability).

Regards,
Daniel
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Mimic 13.2.3?

[Reed Dier]

Piggy backing for a +1 on this.
Really would love if bad packages would be recalled, and also if packages would 
follow release announcement, rather than precede it.

For anyone wondering, this is the likely changelog for 13.2.3 in case people 
want to know what is in it.

https://github.com/ceph/ceph/pull/25637/commits/fe854ac1729a5353fa646298a0b4550101f9c6b2
 


Reed

> On Jan 4, 2019, at 10:07 AM, Matthew Vernon  wrote:
> 
> Hi,
> 
> On 04/01/2019 15:34, Abhishek Lekshmanan wrote:
>> Ashley Merrick  writes:
>> 
>>> If this is another nasty bug like .2? Can’t you remove .3 from being
>>> available till .4 comes around?
>> 
>> This time there isn't a nasty bug, just a a couple of more fixes in .4
>> which would be better to have. We're building 12.2.4 as we speak
>>> Myself will wait for proper confirmation always but others may run an apt
>>> upgrade for any other reason and end up with .3 packages.
> 
> Without wishing to bang on about this, how is it still the case that
> packages are being pushed onto the official ceph.com repos that people
> shouldn't install? This has caused plenty of people problems on several
> occasions now, and a number of people have offered help to fix it...
> 
> Regards,
> 
> Matthew
> 
> 
> 
> -- 
> The Wellcome Sanger Institute is operated by Genome Research 
> Limited, a charity registered in England with number 1021457 and a 
> company registered in England with number 2742969, whose registered 
> office is 215 Euston Road, London, NW1 2BE. 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Mimic 13.2.3?

[Matthew Vernon]

Hi,

On 04/01/2019 15:34, Abhishek Lekshmanan wrote:
> Ashley Merrick  writes:
> 
>> If this is another nasty bug like .2? Can’t you remove .3 from being
>> available till .4 comes around?
> 
> This time there isn't a nasty bug, just a a couple of more fixes in .4
> which would be better to have. We're building 12.2.4 as we speak
>> Myself will wait for proper confirmation always but others may run an apt
>> upgrade for any other reason and end up with .3 packages.

Without wishing to bang on about this, how is it still the case that
packages are being pushed onto the official ceph.com repos that people
shouldn't install? This has caused plenty of people problems on several
occasions now, and a number of people have offered help to fix it...

Regards,

Matthew



-- 
 The Wellcome Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Mimic 13.2.3?

[Abhishek Lekshmanan]

Ashley Merrick  writes:

> If this is another nasty bug like .2? Can’t you remove .3 from being
> available till .4 comes around?

This time there isn't a nasty bug, just a a couple of more fixes in .4
which would be better to have. We're building 12.2.4 as we speak
> Myself will wait for proper confirmation always but others may run an apt
> upgrade for any other reason and end up with .3 packages.

>
> ,Ashley
>
> On Fri, 4 Jan 2019 at 11:21 PM, Abhishek Lekshmanan 
> wrote:
>
>> Ashley Merrick  writes:
>>
>> > Another day and still nothing to say there has been an official
>> release..?!?
>>
>> Sorry just wait for a bit, we're building 13.2.4, don't install
>> 13.2.3 yet.
>> >
>> >
>> > On Fri, 4 Jan 2019 at 2:27 AM, Alex Litvak > >
>> > wrote:
>> >
>> >> It is true for all distros.  It doesn't happen the first time either. I
>> >> think it is a bit dangerous.
>> >>
>> >> On 1/3/19 12:25 AM, Ashley Merrick wrote:
>> >> > Have just run an apt update and have noticed there are some CEPH
>> >> > packages now available for update on my mimic cluster / ubuntu.
>> >> >
>> >> > Have yet to install these yet but it look's like we have the next
>> point
>> >> > release of CEPH Mimic, but not able to see any release note's or
>> >> > official comm's yet?..
>> >> >
>> >> > ___
>> >> > ceph-users mailing list
>> >> > ceph-users@lists.ceph.com
>> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >> >
>> >>
>> >>
>> >> ___
>> >> ceph-users mailing list
>> >> ceph-users@lists.ceph.com
>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >>
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>> --
>> Abhishek Lekshmanan
>> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
>> HRB 21284 (AG Nürnberg)
>>

-- 
Abhishek Lekshmanan
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Mimic 13.2.3?

[Ashley Merrick]

If this is another nasty bug like .2? Can’t you remove .3 from being
available till .4 comes around?

Myself will wait for proper confirmation always but others may run an apt
upgrade for any other reason and end up with .3 packages.

,Ashley

On Fri, 4 Jan 2019 at 11:21 PM, Abhishek Lekshmanan 
wrote:

> Ashley Merrick  writes:
>
> > Another day and still nothing to say there has been an official
> release..?!?
>
> Sorry just wait for a bit, we're building 13.2.4, don't install
> 13.2.3 yet.
> >
> >
> > On Fri, 4 Jan 2019 at 2:27 AM, Alex Litvak  >
> > wrote:
> >
> >> It is true for all distros.  It doesn't happen the first time either. I
> >> think it is a bit dangerous.
> >>
> >> On 1/3/19 12:25 AM, Ashley Merrick wrote:
> >> > Have just run an apt update and have noticed there are some CEPH
> >> > packages now available for update on my mimic cluster / ubuntu.
> >> >
> >> > Have yet to install these yet but it look's like we have the next
> point
> >> > release of CEPH Mimic, but not able to see any release note's or
> >> > official comm's yet?..
> >> >
> >> > ___
> >> > ceph-users mailing list
> >> > ceph-users@lists.ceph.com
> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> >
> >>
> >>
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> --
> Abhishek Lekshmanan
> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
> HRB 21284 (AG Nürnberg)
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Mimic 13.2.3?

[Abhishek Lekshmanan]

Ashley Merrick  writes:

> Another day and still nothing to say there has been an official release..?!?

Sorry just wait for a bit, we're building 13.2.4, don't install
13.2.3 yet.
>
>
> On Fri, 4 Jan 2019 at 2:27 AM, Alex Litvak 
> wrote:
>
>> It is true for all distros.  It doesn't happen the first time either. I
>> think it is a bit dangerous.
>>
>> On 1/3/19 12:25 AM, Ashley Merrick wrote:
>> > Have just run an apt update and have noticed there are some CEPH
>> > packages now available for update on my mimic cluster / ubuntu.
>> >
>> > Have yet to install these yet but it look's like we have the next point
>> > release of CEPH Mimic, but not able to see any release note's or
>> > official comm's yet?..
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

-- 
Abhishek Lekshmanan
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Mimic 13.2.3?

[Ashley Merrick]

Another day and still nothing to say there has been an official release..?!?


On Fri, 4 Jan 2019 at 2:27 AM, Alex Litvak 
wrote:

> It is true for all distros.  It doesn't happen the first time either. I
> think it is a bit dangerous.
>
> On 1/3/19 12:25 AM, Ashley Merrick wrote:
> > Have just run an apt update and have noticed there are some CEPH
> > packages now available for update on my mimic cluster / ubuntu.
> >
> > Have yet to install these yet but it look's like we have the next point
> > release of CEPH Mimic, but not able to see any release note's or
> > official comm's yet?..
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help Ceph Cluster Down

[Arun POONIA]

Hi Caspar,

Number of IOPs are also quite low. It used be around 1K Plus on one of Pool
(VMs) now its like close to 10-30 .

Thansk
Arun

On Fri, Jan 4, 2019 at 5:41 AM Arun POONIA 
wrote:

> Hi Caspar,
>
> Yes and No, numbers are going up and down. If I run ceph -s command I can
> see it decreases one time and later it increases again. I see there are so
> many blocked/slow requests. Almost all the OSDs have slow requests. Around
> 12% PGs are inactive not sure how to activate them again.
>
>
> [root@fre101 ~]# ceph health detail
> 2019-01-04 05:39:23.860142 7fc37a3a0700 -1 asok(0x7fc3740017a0)
> AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to
> bind the UNIX domain socket to
> '/var/run/ceph-guests/ceph-client.admin.1066526.140477441513808.asok': (2)
> No such file or directory
> HEALTH_ERR 1 osds down; 3 pools have many more objects per pg than
> average; 472812/12392654 objects misplaced (3.815%); 3610 PGs pending on
> creation; Reduced data availability: 6578 pgs inactive, 1882 pgs down, 86
> pgs peering, 850 pgs stale; Degraded data redundancy: 216694/12392654
> objects degraded (1.749%), 866 pgs degraded, 16 pgs undersized; 116082 slow
> requests are blocked > 32 sec; 551 stuck requests are blocked > 4096 sec;
> too many PGs per OSD (2709 > max 200)
> OSD_DOWN 1 osds down
> osd.28 (root=default,host=fre119) is down
> MANY_OBJECTS_PER_PG 3 pools have many more objects per pg than average
> pool glance-images objects per pg (10478) is more than 92.7257 times
> cluster average (113)
> pool vms objects per pg (4717) is more than 41.7434 times cluster
> average (113)
> pool volumes objects per pg (1220) is more than 10.7965 times cluster
> average (113)
> OBJECT_MISPLACED 472812/12392654 objects misplaced (3.815%)
> PENDING_CREATING_PGS 3610 PGs pending on creation
> osds
> [osd.0,osd.1,osd.10,osd.11,osd.14,osd.15,osd.17,osd.18,osd.19,osd.20,osd.21,osd.22,osd.23,osd.25,osd.26,osd.27,osd.28,osd.3,osd.30,osd.32,osd.33,osd.35,osd.36,osd.37,osd.38,osd.4,osd.5,osd.6,osd.7,osd.9]
> have pending PGs.
> PG_AVAILABILITY Reduced data availability: 6578 pgs inactive, 1882 pgs
> down, 86 pgs peering, 850 pgs stale
> pg 10.900 is down, acting [18]
> pg 10.90e is stuck inactive for 60266.030164, current state
> activating, last acting [2,38]
> pg 10.913 is stuck stale for 1887.552862, current state stale+down,
> last acting [9]
> pg 10.915 is stuck inactive for 60266.215231, current state
> activating, last acting [30,38]
> pg 11.903 is stuck inactive for 59294.465961, current state
> activating, last acting [11,38]
> pg 11.910 is down, acting [21]
> pg 11.919 is down, acting [25]
> pg 12.902 is stuck inactive for 57118.544590, current state
> activating, last acting [36,14]
> pg 13.8f8 is stuck inactive for 60707.167787, current state
> activating, last acting [29,37]
> pg 13.901 is stuck stale for 60226.543289, current state
> stale+active+clean, last acting [1,31]
> pg 13.905 is stuck inactive for 60266.050940, current state
> activating, last acting [2,36]
> pg 13.909 is stuck inactive for 60707.160714, current state
> activating, last acting [34,36]
> pg 13.90e is stuck inactive for 60707.410749, current state
> activating, last acting [21,36]
> pg 13.911 is down, acting [25]
> pg 13.914 is stale+down, acting [29]
> pg 13.917 is stuck stale for 580.224688, current state stale+down,
> last acting [16]
> pg 14.901 is stuck inactive for 60266.037762, current state
> activating+degraded, last acting [22,37]
> pg 14.90f is stuck inactive for 60296.996447, current state
> activating, last acting [30,36]
> pg 14.910 is stuck inactive for 60266.077310, current state
> activating+degraded, last acting [17,37]
> pg 14.915 is stuck inactive for 60266.032445, current state
> activating, last acting [34,36]
> pg 15.8fa is stuck stale for 560.223249, current state stale+down,
> last acting [8]
> pg 15.90c is stuck inactive for 59294.402388, current state
> activating, last acting [29,38]
> pg 15.90d is stuck inactive for 60266.176492, current state
> activating, last acting [5,36]
> pg 15.915 is down, acting [0]
> pg 15.917 is stuck inactive for 56279.658951, current state
> activating, last acting [13,38]
> pg 15.91c is stuck stale for 374.590704, current state stale+down,
> last acting [12]
> pg 16.903 is stuck inactive for 56580.905961, current state
> activating, last acting [25,37]
> pg 16.90e is stuck inactive for 60266.271680, current state
> activating, last acting [14,37]
> pg 16.919 is stuck inactive for 59901.802184, current state
> activating, last acting [20,37]
> pg 16.91e is stuck inactive for 60297.038159, current state
> activating, last acting [22,37]
> pg 17.8e5 is stuck inactive for 60266.149061, current state
> activating, last acting [25,36]
> pg 17.910 is stuck inactive for 59901.850204, current state
> activating, last acting 

[ceph-users] ceph-mgr fails to restart after upgrade to mimic

[Randall Smith]

Greetings,

I'm upgrading my cluster from luminous to mimic. I've upgraded my monitors
and am attempting to upgrade the mgrs. Unfortunately, after an upgrade the
mgr daemon exits immediately with error code 1.

I've tried running ceph-mgr in debug mode to try to see what's happening
but the output (below) is a bit cryptic for me. It looks like
authentication might be failing but it was working prior to the upgrade.

I do have "auth supported = cephx" in the global section of ceph.conf.

What do I need to do to fix this?

Thanks.

/usr/bin/ceph-mgr -f --cluster ceph --id 8 --setuser ceph --setgroup ceph
-d --debug_ms 5

2019-01-04 07:01:38.457 7f808f83f700  2 Event(0x30c42c0 nevent=5000
time_id=1).set_owner idx=0 owner=140190140331776

2019-01-04 07:01:38.457 7f808f03e700  2 Event(0x30c4500 nevent=5000
time_id=1).set_owner idx=1 owner=140190131939072

2019-01-04 07:01:38.457 7f808e83d700  2 Event(0x30c4e00 nevent=5000
time_id=1).set_owner idx=2 owner=140190123546368

2019-01-04 07:01:38.457 7f809dd5b380  1  Processor -- start


2019-01-04 07:01:38.477 7f809dd5b380  1 -- - start start


2019-01-04 07:01:38.481 7f809dd5b380  1 -- - --> 192.168.253.147:6789/0 --
auth(proto 0 26 bytes epoch 0) v1 -- 0x32a6780 con 0

2019-01-04 07:01:38.481 7f809dd5b380  1 -- - --> 192.168.253.148:6789/0 --
auth(proto 0 26 bytes epoch 0) v1 -- 0x32a6a00 con 0
2019-01-04 07:01:38.481 7f808e83d700  1 -- 192.168.253.148:0/1359135487
learned_addr learned my addr 192.168.253.148:0/1359135487
2019-01-04 07:01:38.481 7f808e83d700  2 -- 192.168.253.148:0/1359135487 >>
192.168.253.148:6789/0 conn(0x332d500 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ
pgs=0 cs=0 l=0)._process_connection got newly_a$
ked_seq 0 vs out_seq 0
2019-01-04 07:01:38.481 7f808f03e700  2 -- 192.168.253.148:0/1359135487 >>
192.168.253.147:6789/0 conn(0x332ce00 :-1 s=STATE_CONNECTING_WAIT_ACK_SEQ
pgs=0 cs=0 l=0)._process_connection got newly_a$
ked_seq 0 vs out_seq 0
2019-01-04 07:01:38.481 7f808f03e700  5 -- 192.168.253.148:0/1359135487 >>
192.168.253.147:6789/0 conn(0x332ce00 :-1
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74172 cs=1 l=1). rx mon.1
seq
1 0x30c5440 mon_map magic: 0 v1
2019-01-04 07:01:38.481 7f808e83d700  5 -- 192.168.253.148:0/1359135487 >>
192.168.253.148:6789/0 conn(0x332d500 :-1
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74275 cs=1 l=1). rx mon.2
seq
1 0x30c5680 mon_map magic: 0 v1
2019-01-04 07:01:38.481 7f808f03e700  5 -- 192.168.253.148:0/1359135487 >>
192.168.253.147:6789/0 conn(0x332ce00 :-1
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74172 cs=1 l=1). rx mon.1
seq
2 0x32a6780 auth_reply(proto 2 0 (0) Success) v1
2019-01-04 07:01:38.481 7f808e83d700  5 -- 192.168.253.148:0/1359135487 >>
192.168.253.148:6789/0 conn(0x332d500 :-1
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74275 cs=1 l=1). rx mon.2
seq
2 0x32a6a00 auth_reply(proto 2 0 (0) Success) v1
2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 <==
mon.1 192.168.253.147:6789/0 1  mon_map magic: 0 v1  370+0+0
(3034216899 0 0) 0x30c5440 con 0x332ce00
2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 <==
mon.2 192.168.253.148:6789/0 1  mon_map magic: 0 v1  370+0+0
(3034216899 0 0) 0x30c5680 con 0x332d500
2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 <==
mon.1 192.168.253.147:6789/0 2  auth_reply(proto 2 0 (0) Success) v1
 33+0+0 (3430158761 0 0) 0x32a6780 con 0x33$
ce00
2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 -->
192.168.253.147:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x32a6f00 con
0
2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 <==
mon.2 192.168.253.148:6789/0 2  auth_reply(proto 2 0 (0) Success) v1
 33+0+0 (3242503871 0 0) 0x32a6a00 con 0x33$
d500
2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 -->
192.168.253.148:6789/0 -- auth(proto 2 2 bytes epoch 0) v1 -- 0x32a6780 con
0
2019-01-04 07:01:38.481 7f808f03e700  5 -- 192.168.253.148:0/1359135487 >>
192.168.253.147:6789/0 conn(0x332ce00 :-1
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74172 cs=1 l=1). rx mon.1
seq
3 0x32a6f00 auth_reply(proto 2 -22 (22) Invalid argument) v1
2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 <==
mon.1 192.168.253.147:6789/0 3  auth_reply(proto 2 -22 (22) Invalid
argument) v1  24+0+0 (882932531 0 0) 0x32a6f$
0 con 0x332ce00
2019-01-04 07:01:38.481 7f808e03c700  1 -- 192.168.253.148:0/1359135487 >>
192.168.253.147:6789/0 conn(0x332ce00 :-1 s=STATE_OPEN pgs=74172 cs=1
l=1).mark_down
2019-01-04 07:01:38.481 7f808e03c700  2 -- 192.168.253.148:0/1359135487 >>
192.168.253.147:6789/0 conn(0x332ce00 :-1 s=STATE_OPEN pgs=74172 cs=1
l=1)._stop
2019-01-04 07:01:38.481 7f808e83d700  5 -- 192.168.253.148:0/1359135487 >>
192.168.253.148:6789/0 conn(0x332d500 :-1
s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=74275 cs=1 l=1). rx mon.2
seq
3 0x32a6780 auth_reply(proto 2 -22 (22) 

Re: [ceph-users] Help Ceph Cluster Down

[Arun POONIA]

Hi Caspar,

Yes and No, numbers are going up and down. If I run ceph -s command I can
see it decreases one time and later it increases again. I see there are so
many blocked/slow requests. Almost all the OSDs have slow requests. Around
12% PGs are inactive not sure how to activate them again.


[root@fre101 ~]# ceph health detail
2019-01-04 05:39:23.860142 7fc37a3a0700 -1 asok(0x7fc3740017a0)
AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to
bind the UNIX domain socket to
'/var/run/ceph-guests/ceph-client.admin.1066526.140477441513808.asok': (2)
No such file or directory
HEALTH_ERR 1 osds down; 3 pools have many more objects per pg than average;
472812/12392654 objects misplaced (3.815%); 3610 PGs pending on creation;
Reduced data availability: 6578 pgs inactive, 1882 pgs down, 86 pgs
peering, 850 pgs stale; Degraded data redundancy: 216694/12392654 objects
degraded (1.749%), 866 pgs degraded, 16 pgs undersized; 116082 slow
requests are blocked > 32 sec; 551 stuck requests are blocked > 4096 sec;
too many PGs per OSD (2709 > max 200)
OSD_DOWN 1 osds down
osd.28 (root=default,host=fre119) is down
MANY_OBJECTS_PER_PG 3 pools have many more objects per pg than average
pool glance-images objects per pg (10478) is more than 92.7257 times
cluster average (113)
pool vms objects per pg (4717) is more than 41.7434 times cluster
average (113)
pool volumes objects per pg (1220) is more than 10.7965 times cluster
average (113)
OBJECT_MISPLACED 472812/12392654 objects misplaced (3.815%)
PENDING_CREATING_PGS 3610 PGs pending on creation
osds
[osd.0,osd.1,osd.10,osd.11,osd.14,osd.15,osd.17,osd.18,osd.19,osd.20,osd.21,osd.22,osd.23,osd.25,osd.26,osd.27,osd.28,osd.3,osd.30,osd.32,osd.33,osd.35,osd.36,osd.37,osd.38,osd.4,osd.5,osd.6,osd.7,osd.9]
have pending PGs.
PG_AVAILABILITY Reduced data availability: 6578 pgs inactive, 1882 pgs
down, 86 pgs peering, 850 pgs stale
pg 10.900 is down, acting [18]
pg 10.90e is stuck inactive for 60266.030164, current state activating,
last acting [2,38]
pg 10.913 is stuck stale for 1887.552862, current state stale+down,
last acting [9]
pg 10.915 is stuck inactive for 60266.215231, current state activating,
last acting [30,38]
pg 11.903 is stuck inactive for 59294.465961, current state activating,
last acting [11,38]
pg 11.910 is down, acting [21]
pg 11.919 is down, acting [25]
pg 12.902 is stuck inactive for 57118.544590, current state activating,
last acting [36,14]
pg 13.8f8 is stuck inactive for 60707.167787, current state activating,
last acting [29,37]
pg 13.901 is stuck stale for 60226.543289, current state
stale+active+clean, last acting [1,31]
pg 13.905 is stuck inactive for 60266.050940, current state activating,
last acting [2,36]
pg 13.909 is stuck inactive for 60707.160714, current state activating,
last acting [34,36]
pg 13.90e is stuck inactive for 60707.410749, current state activating,
last acting [21,36]
pg 13.911 is down, acting [25]
pg 13.914 is stale+down, acting [29]
pg 13.917 is stuck stale for 580.224688, current state stale+down, last
acting [16]
pg 14.901 is stuck inactive for 60266.037762, current state
activating+degraded, last acting [22,37]
pg 14.90f is stuck inactive for 60296.996447, current state activating,
last acting [30,36]
pg 14.910 is stuck inactive for 60266.077310, current state
activating+degraded, last acting [17,37]
pg 14.915 is stuck inactive for 60266.032445, current state activating,
last acting [34,36]
pg 15.8fa is stuck stale for 560.223249, current state stale+down, last
acting [8]
pg 15.90c is stuck inactive for 59294.402388, current state activating,
last acting [29,38]
pg 15.90d is stuck inactive for 60266.176492, current state activating,
last acting [5,36]
pg 15.915 is down, acting [0]
pg 15.917 is stuck inactive for 56279.658951, current state activating,
last acting [13,38]
pg 15.91c is stuck stale for 374.590704, current state stale+down, last
acting [12]
pg 16.903 is stuck inactive for 56580.905961, current state activating,
last acting [25,37]
pg 16.90e is stuck inactive for 60266.271680, current state activating,
last acting [14,37]
pg 16.919 is stuck inactive for 59901.802184, current state activating,
last acting [20,37]
pg 16.91e is stuck inactive for 60297.038159, current state activating,
last acting [22,37]
pg 17.8e5 is stuck inactive for 60266.149061, current state activating,
last acting [25,36]
pg 17.910 is stuck inactive for 59901.850204, current state activating,
last acting [26,37]
pg 17.913 is stuck inactive for 60707.208364, current state activating,
last acting [13,36]
pg 17.91a is stuck inactive for 60266.187509, current state activating,
last acting [4,37]
pg 17.91f is down, acting [6]
pg 18.908 is stuck inactive for 60707.216314, current state activating,
last acting [10,36]
pg 18.911 is stuck stale for 244.570413, 

Re: [ceph-users] Help Ceph Cluster Down

[Caspar Smit]

Are the numbers still decreasing?

This one for instance:

"3883 PGs pending on creation"

Caspar


Op vr 4 jan. 2019 om 14:23 schreef Arun POONIA <
arun.poo...@nuagenetworks.net>:

> Hi Caspar,
>
> Yes, cluster was working fine with number of PGs per OSD warning up until
> now. I am not sure how to recover from stale down/inactive PGs. If you
> happen to know about this can you let me know?
>
> Current State:
>
> [root@fre101 ~]# ceph -s
> 2019-01-04 05:22:05.942349 7f314f613700 -1 asok(0x7f31480017a0)
> AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to
> bind the UNIX domain socket to
> '/var/run/ceph-guests/ceph-client.admin.1053724.139849638091088.asok': (2)
> No such file or directory
>   cluster:
> id: adb9ad8e-f458-4124-bf58-7963a8d1391f
> health: HEALTH_ERR
> 3 pools have many more objects per pg than average
> 505714/12392650 objects misplaced (4.081%)
> 3883 PGs pending on creation
> Reduced data availability: 6519 pgs inactive, 1870 pgs down, 1
> pg peering, 886 pgs stale
> Degraded data redundancy: 42987/12392650 objects degraded
> (0.347%), 634 pgs degraded, 16 pgs undersized
> 125827 slow requests are blocked > 32 sec
> 2 stuck requests are blocked > 4096 sec
> too many PGs per OSD (2758 > max 200)
>
>   services:
> mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03
> mgr: ceph-mon03(active), standbys: ceph-mon01, ceph-mon02
> osd: 39 osds: 39 up, 39 in; 76 remapped pgs
> rgw: 1 daemon active
>
>   data:
> pools:   18 pools, 54656 pgs
> objects: 6051k objects, 10944 GB
> usage:   21933 GB used, 50688 GB / 72622 GB avail
> pgs: 11.927% pgs not active
>  42987/12392650 objects degraded (0.347%)
>  505714/12392650 objects misplaced (4.081%)
>  48080 active+clean
>  3885  activating
>    down
>  759   stale+down
>  614   activating+degraded
>  74activating+remapped
>  46stale+active+clean
>  35stale+activating
>  21stale+activating+remapped
>  9 stale+active+undersized
>  9 stale+activating+degraded
>  5 stale+activating+undersized+degraded+remapped
>  3 activating+degraded+remapped
>  1 stale+activating+degraded+remapped
>  1 stale+active+undersized+degraded
>  1 remapped+peering
>  1 active+clean+remapped
>  1 activating+undersized+degraded+remapped
>
>   io:
> client:   0 B/s rd, 25397 B/s wr, 4 op/s rd, 4 op/s wr
>
> I will update number of PGs per OSD once these inactive or stale PGs come
> online. I am not able to access VMs (VMs, Images) which are using Ceph.
>
> Thanks
> Arun
>
> On Fri, Jan 4, 2019 at 4:53 AM Caspar Smit  wrote:
>
>> Hi Arun,
>>
>> How did you end up with a 'working' cluster with so many pgs per OSD?
>>
>> "too many PGs per OSD (2968 > max 200)"
>>
>> To (temporarily) allow this kind of pgs per osd you could try this:
>>
>> Change these values in the global section in your ceph.conf:
>>
>> mon max pg per osd = 200
>> osd max pg per osd hard ratio = 2
>>
>> It allows 200*2 = 400 Pgs per OSD before disabling the creation of new
>> pgs.
>>
>> Above are the defaults (for Luminous, maybe other versions too)
>> You can check your current settings with:
>>
>> ceph daemon mon.ceph-mon01 config show |grep pg_per_osd
>>
>> Since your current pgs per osd ratio is way higher then the default you
>> could set them to for instance:
>>
>> mon max pg per osd = 1000
>> osd max pg per osd hard ratio = 5
>>
>> Which allow for 5000 pgs per osd before disabling creation of new pgs.
>>
>> You'll need to inject the setting into the mons/osds and restart mgrs to
>> make them active.
>>
>> ceph tell mon.* injectargs ‘--mon_max_pg_per_osd 1000’
>> ceph tell mon.* injectargs ‘--osd_max_pg_per_osd_hard_ratio 5’
>> ceph tell osd.* injectargs ‘--mon_max_pg_per_osd 1000’
>> ceph tell osd.* injectargs ‘--osd_max_pg_per_osd_hard_ratio 5’
>> restart mgrs
>>
>> Kind regards,
>> Caspar
>>
>>
>> Op vr 4 jan. 2019 om 04:28 schreef Arun POONIA <
>> arun.poo...@nuagenetworks.net>:
>>
>>> Hi Chris,
>>>
>>> Indeed that's what happened. I didn't set noout flag either and I did
>>> zapped disk on new server every time. In my cluster status fre201 is only
>>> new server.
>>>
>>> Current Status after enabling 3 OSDs on fre201 host.
>>>
>>> [root@fre201 ~]# ceph osd tree
>>> ID  CLASS WEIGHT   TYPE NAME   STATUS REWEIGHT PRI-AFF
>>>  -1   70.92137 root default
>>>  -25.45549 host fre101
>>>   0   hdd  1.81850 osd.0   up  1.0 1.0
>>>   1   hdd  1.81850 osd.1   up  1.0 1.0
>>>   2   hdd  1.81850 osd.2   up  1.0 1.0
>>>  -95.45549 host fre103
>>>   3   hdd  

Re: [ceph-users] Help Ceph Cluster Down

[Arun POONIA]

Hi Caspar,

Yes, cluster was working fine with number of PGs per OSD warning up until
now. I am not sure how to recover from stale down/inactive PGs. If you
happen to know about this can you let me know?

Current State:

[root@fre101 ~]# ceph -s
2019-01-04 05:22:05.942349 7f314f613700 -1 asok(0x7f31480017a0)
AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to
bind the UNIX domain socket to
'/var/run/ceph-guests/ceph-client.admin.1053724.139849638091088.asok': (2)
No such file or directory
  cluster:
id: adb9ad8e-f458-4124-bf58-7963a8d1391f
health: HEALTH_ERR
3 pools have many more objects per pg than average
505714/12392650 objects misplaced (4.081%)
3883 PGs pending on creation
Reduced data availability: 6519 pgs inactive, 1870 pgs down, 1
pg peering, 886 pgs stale
Degraded data redundancy: 42987/12392650 objects degraded
(0.347%), 634 pgs degraded, 16 pgs undersized
125827 slow requests are blocked > 32 sec
2 stuck requests are blocked > 4096 sec
too many PGs per OSD (2758 > max 200)

  services:
mon: 3 daemons, quorum ceph-mon01,ceph-mon02,ceph-mon03
mgr: ceph-mon03(active), standbys: ceph-mon01, ceph-mon02
osd: 39 osds: 39 up, 39 in; 76 remapped pgs
rgw: 1 daemon active

  data:
pools:   18 pools, 54656 pgs
objects: 6051k objects, 10944 GB
usage:   21933 GB used, 50688 GB / 72622 GB avail
pgs: 11.927% pgs not active
 42987/12392650 objects degraded (0.347%)
 505714/12392650 objects misplaced (4.081%)
 48080 active+clean
 3885  activating
   down
 759   stale+down
 614   activating+degraded
 74activating+remapped
 46stale+active+clean
 35stale+activating
 21stale+activating+remapped
 9 stale+active+undersized
 9 stale+activating+degraded
 5 stale+activating+undersized+degraded+remapped
 3 activating+degraded+remapped
 1 stale+activating+degraded+remapped
 1 stale+active+undersized+degraded
 1 remapped+peering
 1 active+clean+remapped
 1 activating+undersized+degraded+remapped

  io:
client:   0 B/s rd, 25397 B/s wr, 4 op/s rd, 4 op/s wr

I will update number of PGs per OSD once these inactive or stale PGs come
online. I am not able to access VMs (VMs, Images) which are using Ceph.

Thanks
Arun

On Fri, Jan 4, 2019 at 4:53 AM Caspar Smit  wrote:

> Hi Arun,
>
> How did you end up with a 'working' cluster with so many pgs per OSD?
>
> "too many PGs per OSD (2968 > max 200)"
>
> To (temporarily) allow this kind of pgs per osd you could try this:
>
> Change these values in the global section in your ceph.conf:
>
> mon max pg per osd = 200
> osd max pg per osd hard ratio = 2
>
> It allows 200*2 = 400 Pgs per OSD before disabling the creation of new
> pgs.
>
> Above are the defaults (for Luminous, maybe other versions too)
> You can check your current settings with:
>
> ceph daemon mon.ceph-mon01 config show |grep pg_per_osd
>
> Since your current pgs per osd ratio is way higher then the default you
> could set them to for instance:
>
> mon max pg per osd = 1000
> osd max pg per osd hard ratio = 5
>
> Which allow for 5000 pgs per osd before disabling creation of new pgs.
>
> You'll need to inject the setting into the mons/osds and restart mgrs to
> make them active.
>
> ceph tell mon.* injectargs ‘--mon_max_pg_per_osd 1000’
> ceph tell mon.* injectargs ‘--osd_max_pg_per_osd_hard_ratio 5’
> ceph tell osd.* injectargs ‘--mon_max_pg_per_osd 1000’
> ceph tell osd.* injectargs ‘--osd_max_pg_per_osd_hard_ratio 5’
> restart mgrs
>
> Kind regards,
> Caspar
>
>
> Op vr 4 jan. 2019 om 04:28 schreef Arun POONIA <
> arun.poo...@nuagenetworks.net>:
>
>> Hi Chris,
>>
>> Indeed that's what happened. I didn't set noout flag either and I did
>> zapped disk on new server every time. In my cluster status fre201 is only
>> new server.
>>
>> Current Status after enabling 3 OSDs on fre201 host.
>>
>> [root@fre201 ~]# ceph osd tree
>> ID  CLASS WEIGHT   TYPE NAME   STATUS REWEIGHT PRI-AFF
>>  -1   70.92137 root default
>>  -25.45549 host fre101
>>   0   hdd  1.81850 osd.0   up  1.0 1.0
>>   1   hdd  1.81850 osd.1   up  1.0 1.0
>>   2   hdd  1.81850 osd.2   up  1.0 1.0
>>  -95.45549 host fre103
>>   3   hdd  1.81850 osd.3   up  1.0 1.0
>>   4   hdd  1.81850 osd.4   up  1.0 1.0
>>   5   hdd  1.81850 osd.5   up  1.0 1.0
>>  -35.45549 host fre105
>>   6   hdd  1.81850 osd.6   up  1.0 1.0
>>   7   hdd  1.81850 osd.7   up  1.0 1.0
>>   8   hdd  1.81850 osd.8   up  

Re: [ceph-users] Help Ceph Cluster Down

[Caspar Smit]

Hi Arun,

How did you end up with a 'working' cluster with so many pgs per OSD?

"too many PGs per OSD (2968 > max 200)"

To (temporarily) allow this kind of pgs per osd you could try this:

Change these values in the global section in your ceph.conf:

mon max pg per osd = 200
osd max pg per osd hard ratio = 2

It allows 200*2 = 400 Pgs per OSD before disabling the creation of new
pgs.

Above are the defaults (for Luminous, maybe other versions too)
You can check your current settings with:

ceph daemon mon.ceph-mon01 config show |grep pg_per_osd

Since your current pgs per osd ratio is way higher then the default you
could set them to for instance:

mon max pg per osd = 1000
osd max pg per osd hard ratio = 5

Which allow for 5000 pgs per osd before disabling creation of new pgs.

You'll need to inject the setting into the mons/osds and restart mgrs to
make them active.

ceph tell mon.* injectargs ‘--mon_max_pg_per_osd 1000’
ceph tell mon.* injectargs ‘--osd_max_pg_per_osd_hard_ratio 5’
ceph tell osd.* injectargs ‘--mon_max_pg_per_osd 1000’
ceph tell osd.* injectargs ‘--osd_max_pg_per_osd_hard_ratio 5’
restart mgrs

Kind regards,
Caspar


Op vr 4 jan. 2019 om 04:28 schreef Arun POONIA <
arun.poo...@nuagenetworks.net>:

> Hi Chris,
>
> Indeed that's what happened. I didn't set noout flag either and I did
> zapped disk on new server every time. In my cluster status fre201 is only
> new server.
>
> Current Status after enabling 3 OSDs on fre201 host.
>
> [root@fre201 ~]# ceph osd tree
> ID  CLASS WEIGHT   TYPE NAME   STATUS REWEIGHT PRI-AFF
>  -1   70.92137 root default
>  -25.45549 host fre101
>   0   hdd  1.81850 osd.0   up  1.0 1.0
>   1   hdd  1.81850 osd.1   up  1.0 1.0
>   2   hdd  1.81850 osd.2   up  1.0 1.0
>  -95.45549 host fre103
>   3   hdd  1.81850 osd.3   up  1.0 1.0
>   4   hdd  1.81850 osd.4   up  1.0 1.0
>   5   hdd  1.81850 osd.5   up  1.0 1.0
>  -35.45549 host fre105
>   6   hdd  1.81850 osd.6   up  1.0 1.0
>   7   hdd  1.81850 osd.7   up  1.0 1.0
>   8   hdd  1.81850 osd.8   up  1.0 1.0
>  -45.45549 host fre107
>   9   hdd  1.81850 osd.9   up  1.0 1.0
>  10   hdd  1.81850 osd.10  up  1.0 1.0
>  11   hdd  1.81850 osd.11  up  1.0 1.0
>  -55.45549 host fre109
>  12   hdd  1.81850 osd.12  up  1.0 1.0
>  13   hdd  1.81850 osd.13  up  1.0 1.0
>  14   hdd  1.81850 osd.14  up  1.0 1.0
>  -65.45549 host fre111
>  15   hdd  1.81850 osd.15  up  1.0 1.0
>  16   hdd  1.81850 osd.16  up  1.0 1.0
>  17   hdd  1.81850 osd.17  up  0.7 1.0
>  -75.45549 host fre113
>  18   hdd  1.81850 osd.18  up  1.0 1.0
>  19   hdd  1.81850 osd.19  up  1.0 1.0
>  20   hdd  1.81850 osd.20  up  1.0 1.0
>  -85.45549 host fre115
>  21   hdd  1.81850 osd.21  up  1.0 1.0
>  22   hdd  1.81850 osd.22  up  1.0 1.0
>  23   hdd  1.81850 osd.23  up  1.0 1.0
> -105.45549 host fre117
>  24   hdd  1.81850 osd.24  up  1.0 1.0
>  25   hdd  1.81850 osd.25  up  1.0 1.0
>  26   hdd  1.81850 osd.26  up  1.0 1.0
> -115.45549 host fre119
>  27   hdd  1.81850 osd.27  up  1.0 1.0
>  28   hdd  1.81850 osd.28  up  1.0 1.0
>  29   hdd  1.81850 osd.29  up  1.0 1.0
> -125.45549 host fre121
>  30   hdd  1.81850 osd.30  up  1.0 1.0
>  31   hdd  1.81850 osd.31  up  1.0 1.0
>  32   hdd  1.81850 osd.32  up  1.0 1.0
> -135.45549 host fre123
>  33   hdd  1.81850 osd.33  up  1.0 1.0
>  34   hdd  1.81850 osd.34  up  1.0 1.0
>  35   hdd  1.81850 osd.35  up  1.0 1.0
> -275.45549 host fre201
>  36   hdd  1.81850 osd.36  up  1.0 1.0
>  37   hdd  1.81850 osd.37  up  1.0 1.0
>  38   hdd  1.81850 osd.38  up  1.0 1.0
> [root@fre201 ~]#
> [root@fre201 ~]#
> [root@fre201 ~]#
> [root@fre201 ~]#
> [root@fre201 ~]#
> [root@fre201 ~]# ceph -s
>   cluster:
> id: adb9ad8e-f458-4124-bf58-7963a8d1391f
> health: HEALTH_ERR
> 3 pools have many more objects per pg than average
> 585791/12391450 objects misplaced (4.727%)
> 2 scrub errors
> 2374 PGs pending on creation
> Reduced data availability: 6578 pgs inactive, 2025 pgs down,
> 74 pgs peering, 1234 pgs stale
> Possible 

[ceph-users] Fwd: CephFS MDS optimal setup on Google Cloud

[Mahmoud Ismail]

Forgot to mention that i'm using Ceph 13.2.2 on Centos 7.
Any ideas?

-- Forwarded message -
From: Mahmoud Ismail 
Date: Fri, Dec 21, 2018 at 4:44 PM
Subject: CephFS MDS optimal setup on Google Cloud
To: 


Hello,

I'm doing benchmarks for metadata operations on CephFS, HDFS, and HopsFS on
Google Cloud. In my current setup, i'm using 32 vCPU machines with 29 GB
memory, and i have 1 MDS, 1 MON and 3 OSDs. The MDS and the MON nodes are
co-located on one vm, while each of the OSDs is on a separate vm with 1 SSD
disk attached. I'm using the default configuration for MDS, and OSDs.

I'm running 300 clients on 10 machines (16 vCPU), each client creates a
CephFileSystem using the CephFS hadoop plugin, and then writes empty files
for 30 seconds followed by reading the empty files for another 30 seconds.
The aggregated throughput is around 2000 file create operations/sec and
1 file read operations/sec. However, the MDS is not fully utilizing the
32 cores on the machine, is there any configuration that i should consider
to fully utilize the machine?.

Also, i noticed that running more than 20-30 clients (on different threads)
per machine degrade the aggregated throughput for read, is there a
limitation on CephFileSystem and libceph on the number of clients created
per machine?

Another issue,  Are the MDS operations single threaded as pointed here "
https://www.slideshare.net/XiaoxiChen3/cephfs-jewel-mds-performance-benchmark
"?
Regarding the MDS global lock, is it it a single lock per MDS or is it a
global distributed lock for all MDSs?

Regards,
Mahmoud
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com