Re: [ceph-users] Ceph-mgr Python error with prometheus plugin

[Konstantin Shalygin]

On 02/18/2018 09:20 AM, Sergey Malinin wrote:

All I got with script replacement is the following:



Oooh. Try something earlier, without this function 
https://github.com/ceph/ceph/blob/1adf4325455ccdbf03da63046c510bb041ac8520/src/pybind/mgr/prometheus/module.py


To be clear: prometheus module was never work out of box at 12.2.2



k
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph-mgr Python error with prometheus plugin

[Sergey Malinin]

All I got with script replacement is the following:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cherrypy/_cprequest.py", line 670, in 
respond
response.body = self.handler()
  File "/usr/lib/python2.7/dist-packages/cherrypy/lib/encoding.py", line 217, 
in __call__
self.body = self.oldhandler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cherrypy/_cpdispatch.py", line 61, in 
__call__
return self.callable(*self.args, **self.kwargs)
  File "/usr/lib/ceph/mgr/prometheus/module.py", line 409, in metrics
if global_instance().have_mon_connection():
AttributeError: 'Module' object has no attribute 'have_mon_connection'


I'm not familiar with Python, so any advise is much appreciated.


On Friday, February 16, 2018 at 12:10, Konstantin Shalygin wrote:

> > i just try to get the prometheus plugin up and runing
> 
> 
> 
> Use module from master.
> 
> From this commit should work with 12.2.2, just wget it and replace 
> stock module.
> 
> https://github.com/ceph/ceph/blob/d431de74def1b8889ad568ab99436362833d063e/src/pybind/mgr/prometheus/module.py
> 
> 
> 
> 
> k
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com (mailto:ceph-users@lists.ceph.com)
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cephfs fsal + nfs-ganesha + el7/centos7

[Oliver Freyermuth]

Hi together,

many thanks for the RPMs provided at:
  http://download.ceph.com/nfs-ganesha/
They are very much appreciated! 


Since the statement was that they will also be maintained in the future, and 
NFS Ganesha seems an important project for the future of Ceph, 
let me do the first "packaging" bug report. 

It seems that the current packages do not play so well with SELinux. I'm 
currently using an SELinux module with the following allows, found by 
iterative use of audit2allow (full ".te" module added at the end of the mail):

allow ganesha_t cyphesis_port_t:tcp_socket name_connect;
allow ganesha_t proc_net_t:file { getattr open read };
allow ganesha_t self:capability dac_override;
allow ganesha_t self:capability setuid;
allow ganesha_t self:capability setgid;

"cyphesis_port_t" is probably needed since its range (tcp: 6767, 6769, 
6780-6799) overlaps with the default ports
recommended for use by OSDs and nfs-ganesha uses libcephfs to talk to them, the 
other caps appear to be needed by nfs-ganesha itself. 

With these in place, it seems my setup is working well. Without the "setgid" 
cap, for example, nfs-ganesha just segfaults after the permission denied 
failure. 
Of course, it would be best if they were installed by the package (potentially, 
more restrictive allows are possible with some care). 


Please include me in replies, I am not subscribed to the list. 

Cheers and all the best,
Oliver



module nfs_ganesha-fix-perms 1.0;

require {
type proc_net_t;
type cyphesis_port_t;
type ganesha_t;
class capability setuid;
class capability setgid;
class capability dac_override;
class tcp_socket name_connect;
class file { getattr open read };
}

#= ganesha_t ==
allow ganesha_t cyphesis_port_t:tcp_socket name_connect;
allow ganesha_t proc_net_t:file { getattr open read };
allow ganesha_t self:capability dac_override;
allow ganesha_t self:capability setuid;
allow ganesha_t self:capability setgid;




smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Understanding/correcting sudden onslaught of unfound objects

[David Zafman]

The commits below came after v12.2.2 and may impact this issue. When a 
pg is active+clean+inconsistent means that scrub has detected issues 
with 1 or more replicas of 1 or more objects .  An unfound object is a 
potentially temporary state in which the current set of available OSDs 
doesn't allow an object to be recovered/backfilled/repaired.  When the 
primary OSD restarts, any unfound objects ( an in memory structure) are 
reset so that the new set of peered OSDs can determine again what 
objects are unfound.


I'm not clear in this scenario whether recovery failed to start, 
recovery hung before due to a bug or if recovery stopped (as designed) 
because of the unfound object.  The new recovery_unfound and 
backfill_unfound states indicates that recovery has stopped due to 
unfound objects.



commit 64047e1bac2e775a06423a03cfab69b88462538c
Author: David Zafman 
Date:   Wed Jan 10 13:30:41 2018 -0800

    osd: Don't start recovery for missing until active pg state set

    I was seeing recovery hang when it is started before 
_activate_committed()
    The state machine passes into "Active" but this transitions to 
activating

    pg state and after commmitted into "active" pg state.

    Signed-off-by: David Zafman 

commit 7f8b0ce9e681f727d8217e3ed74a1a3355f364f3
Author: David Zafman 
Date:   Mon Oct 9 08:19:21 2017 -0700

    osd, mon: Add new pg states recovery_unfound and backfill_unfound

    Signed-off-by: David Zafman 



On 2/16/18 1:40 PM, Gregory Farnum wrote:

On Fri, Feb 16, 2018 at 12:17 PM Graham Allan  wrote:


On 02/16/2018 12:31 PM, Graham Allan wrote:

If I set debug rgw=1 and demug ms=1 before running the "object stat"
command, it seems to stall in a loop of trying communicate with osds for
pool 96, which is .rgw.control


10.32.16.93:0/2689814946 --> 10.31.0.68:6818/8969 --
osd_op(unknown.0.0:541 96.e 96:7759931f:::notify.3:head [watch ping
cookie 139709246356176] snapc 0=[] ondisk+write+known_if_redirected
e507695) v8 -- 0x7f10ac033610 con 0
10.32.16.93:0/2689814946 <== osd.38 10.31.0.68:6818/8969 59 
osd_op_reply(541 notify.3 [watch ping cookie 139709246356176] v0'0
uv3933745 ondisk = 0) v8  152+0+0 (2536111836 <(253)%20611-1836> 0

0) 0x7f1158003e20

con 0x7f117afd8390

Prior to that, probably more relevant, this was the only communication
logged with the primary osd of the pg:


10.32.16.93:0/1552085932 --> 10.31.0.71:6838/66301 --
osd_op(unknown.0.0:96 70.438s0
70:1c20c157:::default.325674.85_bellplants_images%2f1042066.jpg:head
[getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e507695)
v8 -- 0x7fab79889fa0 con 0
10.32.16.93:0/1552085932 <== osd.175 10.31.0.71:6838/66301 1 
osd_backoff(70.438s0 block id 1


[70:1c20c157:::default.325674.85_bellplants_images%2f1042066.jpg:head,70:1c20c157:::default.325674.85_bellplants_images%2f1042066.jpg:head)

e507695) v1  209+0+0 (1958971312 0 0) 0x7fab5003d3c0 con
0x7fab79885980
210.32.16.93:0/1552085932 --> 10.31.0.71:6838/66301 --
osd_backoff(70.438s0 ack-block id 1


[70:1c20c157:::default.325674.85_bellplants_images%2f1042066.jpg:head,70:1c20c157:::default.325674.85_bellplants_images%2f1042066.jpg:head)

e507695) v1 -- 0x7fab48065420 con 0

so I guess the backoff message above is saying the object is
unavailable. OK, that certainly makes sense. Not sure that it helps me
understand how to fix the inconsistencies

If I restart the primary osd for the pg, that makes it forget its state
and return to active+clean+inconsistent. I can then download the
previously-unfound objects again, as well as run "radosgw-admin object
stat".

So the interesting bit is probably figuring out why it decides these
objects are unfound, when clearly they aren't.

What would be the best place to enable additional logging to understand
this - perhaps the primary osd?


David, this sounds like one of the bugs where an OSD can mark objects as
inconsistent locally but then doesn't actually trigger recovery on them. Or
it doesn't like any copy but doesn't persist that.
Do any known issues around that apply to 12.2.2?
-Greg



Thanks for all your help,

Graham
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] High Load and High Apply Latency

[Marc Roos]

But that is already the default not? (on CentOS7 rpms)

[@c03 ~]# cat /etc/sysconfig/ceph
# /etc/sysconfig/ceph
#
# Environment file for ceph daemon systemd unit files.
#

# Increase tcmalloc cache size
TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728
 



-Original Message-
From: John Petrini [mailto:jpetr...@coredial.com] 
Sent: zaterdag 17 februari 2018 1:06
To: David Turner
Cc: ceph-users
Subject: Re: [ceph-users] High Load and High Apply Latency

I thought I'd follow up on this just in case anyone else experiences 
similar issues. We ended up increasing the tcmalloc thread cache size 
and saw a huge improvement in latency. This got us out of the woods 
because we were finally in a state where performance was good enough 
that it was no longer impacting services. 

The tcmalloc issues are pretty well documented on this mailing list and 
I don't believe they impact newer versions of Ceph but I thought I'd at 
least give a data point. After making this change our average apply 
latency dropped to 3.46ms during peak business hours. To give you an 
idea of how significant that is here's a graph of the apply latency 
prior to the change: https://imgur.com/KYUETvD


This however did not resolve all of our issues. We were still seeing 
high iowait (repeated spikes up to 400ms) on three of our OSD nodes on 
all disks. We tried replacing the RAID controller (PERC H730) on these 
nodes and while this resolved the issue on one server the two others 
remained problematic. These two nodes were configured differently than 
the rest. They'd been configured in non-raid mode while the others were 
configured as individual raid-0. This turned out to be the problem. We 
ended up removing the two nodes one at a time and rebuilding them with 
their disks configured in independent raid-0 instead of non-raid. After 
this change iowait rarely spikes above 15ms and averages <1ms.


I was really surprised at the performance impact when using non-raid 
mode. While I realize non-raid bypasses the controller cache I still 
would have never expected such high latency. Dell has a whitepaper that 
recommends using individual raid-0 but their own tests show only a small 
performance advantage over non-raid. Note that we are running SAS disks, 
they actually recommend non-raid mode for SATA but I have not tested 
this. You can view the whtiepaper here: 
http://en.community.dell.com/techcenter/cloud/m/dell_cloud_resources/20442913/download


I hope this helps someone.


John Petrini



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Restoring keyring capabilities

[Eugen Block]

Hi,

I'm not sure if this is applicable to the admin user, but there's a  
import function in the ceph-authtool which allows to import caps from  
a file. I used it once in a test cluster to restore keyrings for a  
specific user. But it wasn't the admin's caps, so I'm not sure if this  
will work, but worth a shot, I guess.


Regards


Zitat von Nico Schottelius :


It seems your monitor capabilities are different to mine:

root@server3:/opt/ungleich-tools# ceph -k  
/var/lib/ceph/mon/ceph-server3/keyring -n mon. auth list
2018-02-16 20:34:59.257529 7fe0d5c6b700  0 librados: mon.  
authentication error (13) Permission denied

[errno 13] error connecting to the cluster
root@server3:/opt/ungleich-tools# cat /var/lib/ceph/mon/ceph-server3/keyring
[mon.]
key = AQCp9IVa2GmYARAAVvCGfNpXfxOoUf119KAq1g==

Where you have


root@ceph-mon1:/# cat /var/lib/ceph/mon/ceph-ceph-mon1/keyring
[mon.]
key = AQD1y3RapVDCNxAAmInc8D3OPZKuTVeUcNsPug==
caps mon = "allow *"


Which probably explains why it works for you, but not for me.

--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




--
Eugen Block voice   : +49-40-559 51 75
NDE Netzdesign und -entwicklung AG  fax : +49-40-559 51 77
Postfach 61 03 15
D-22423 Hamburg e-mail  : ebl...@nde.ag

Vorsitzende des Aufsichtsrates: Angelika Mozdzen
  Sitz und Registergericht: Hamburg, HRB 90934
  Vorstand: Jens-U. Mozdzen
   USt-IdNr. DE 814 013 983

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com