Re: [ceph-users] Strange CEPH_ARGS problems

[Toby Darling]

On 15/11/2019 06:46, Rainer Krienke wrote:

export CEPH_ARGS=="-n client.rz --keyring=/etc
/ceph/ceph.client.user.keyring"


It's probably the '==', try '=' instead.

Cheers
Toby
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Strange CEPH_ARGS problems

[Rainer Krienke]

Hello,

I try to use CEPH_ARGS in order to use eg rbd with a non client.admin
user and keyring without extra parameters. On a ceph-client with Ubuntu
18.04.3 I get this:

# unset CEPH_ARGS
# rbd --name=client.user --keyring=/etc/ceph/ceph.client.user.keyring ls
a
b
c

# export CEPH_ARGS=="-n client.rz --keyring=/etc
/ceph/ceph.client.user.keyring"
# rbd ls

rbd: couldn't connect to the cluster!
rbd: listing images failed: (22) Invalid argument

# export CEPH_ARGS=="--keyring=/etc/ceph/ceph.client.user.keyring"
# rbd -n client.user ls
a
b
c

Is this the desired behavior? I would like to set both user name and
keyring to be used, so that I can run rbd without any parameters.

How do you do this?

Thanks
Rainer

-- 
Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1
56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312
Web: http://userpages.uni-koblenz.de/~krienke
PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Large OMAP Object

[JC Lopez]

Hi

this probably comes from your RGW which is a big consumer/producer of OMAP for 
bucket indexes.

Have a look at this previous post and just adapt the pool name to match the one 
where it’s detected: https://www.spinics.net/lists/ceph-users/msg51681.html

Regards
JC

> On Nov 14, 2019, at 15:23, dhils...@performair.com wrote:
> 
> All;
> 
> We had a warning about a large OMAP object pop up in one of our clusters 
> overnight.  The cluster is configured for CephFS, but nothing mounts a 
> CephFS, at this time.
> 
> The cluster mostly uses RGW.  I've checked the cluster log, the MON log, and 
> the MGR log on one of the mons, with no useful references to the pool / pg 
> where the large OMAP objects resides.
> 
> Is my only option to find this large OMAP object to go through the OSD logs 
> for the individual OSDs in the cluster?
> 
> Thank you,
> 
> Dominic L. Hilsbos, MBA 
> Director - Information Technology 
> Perform Air International Inc.
> dhils...@performair.com 
> www.PerformAir.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Large OMAP Object

[DHilsbos]

All;

We had a warning about a large OMAP object pop up in one of our clusters 
overnight.  The cluster is configured for CephFS, but nothing mounts a CephFS, 
at this time.

The cluster mostly uses RGW.  I've checked the cluster log, the MON log, and 
the MGR log on one of the mons, with no useful references to the pool / pg 
where the large OMAP objects resides.

Is my only option to find this large OMAP object to go through the OSD logs for 
the individual OSDs in the cluster?

Thank you,

Dominic L. Hilsbos, MBA 
Director - Information Technology 
Perform Air International Inc.
dhils...@performair.com 
www.PerformAir.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Create containers/buckets in a custom rgw pool

[soumya tr]

Thanks Lopez !!

On Tue, Nov 12, 2019 at 2:07 PM JC Lopez  wrote:

> Hi Soumya,
>
> have a look at this page that will show you how to map your special pool
> from the RADOS Gateway perspective.
>
> L uminous:
> https://docs.ceph.com/docs/luminous/radosgw/placement/
> Mimic: https://docs.ceph.com/docs/mimic/radosgw/placement/
> Master: https://docs.ceph.com/docs/master/radosgw/placement/
>
> As for mixing RGW data with RBD data this is not best practice and should
> be avoided.
>
> Best regards
> JC
>
> On Nov 11, 2019, at 16:20, soumya tr  wrote:
>
> Hey,
>
> By default, there are some custom pools created for rgw in a ceph cluster.
> And when buckets/containers are created using OpenStack horizon or swift
> CLI, it gets created in the default pool.
> --
> 1 default.rgw.buckets.data
> 2 default.rgw.control
> 3 default.rgw.data.root
> 4 default.rgw.gc
> 5 default.rgw.log
> 6 default.rgw.intent-log
> 7 default.rgw.meta
> 8 default.rgw.usage
> 9 default.rgw.users.keys
> 10 default.rgw.users.email
> 11 default.rgw.users.swift
> 12 default.rgw.users.uid
> 13 default.rgw.buckets.extra
> 14 default.rgw.buckets.index
> 15 .rgw.root
> --
>
> I created a custom pool and associated it with application rgw. But not
> sure how to make use of it. I didn't get many references for the same.
>
> Is there any way to have the buckets/containers created in a custom rbd
> pool?
>
> --
> Regards,
> Soumya
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>

-- 
Regards,
Soumya


"I like the dreams of the future better than the history of the past."
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph cluster works UNTIL the OSDs are rebooted

[Richard Geoffrion]

I had a working ceph cluster running nautilus in a test lab just a few 
months ago.   Now that I'm trying to take ceph live on production 
hardware, I can't seem to get the cluster to stay up and available even 
though all three OSDs are UP and IN.


I believe the problem is that the OSDs don't mount their volumes after a 
reboot.   The ceph-deploy routine can install an OSD node, format the 
disk and bring it online, and it can get all the OSD nodes UP and IN and 
reach a quorum BUT, once an OSD gets rebooted, all the PGs related to 
that OSD go "stuck inactive...current state unknown, last acting".


I've found and resolved all my hostname and firewall errors, and I'm 
comfortable that I've ruled out network issues. For grins and giggles, I 
reconfigured the OSDs to be on the same 'public' network with the MON 
servers and the OSDs still drop their disks from the cluster after a reboot.


What do I need to do next?

Below is a pastebin link to some log file data where you can see some 
traceback errors.



[2019-10-30 14:52:10,201][ceph_volume][ERROR ] exception caught by decorator
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", 
line 59, in newfunc

return f(*a, **kw)


Some of these errors might be due to the system seeing the three other 
setup attempts that are no longer available. A 'ceph-deploy purge' and 
'ceph-deploy purgedata' doesn't seem to get rid of EVERYTHING. I've 
learned since that /var/lib/ceph retains some data.  I'll be sure to 
remove the data from that directory when I next attempt to start fresh.


What do I need to be looking at to correct this "OSD not remounting it's 
disk" issue?


https://pastebin.com/NMXvYBcZ
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Revert a CephFS snapshot?

[Sage Weil]

On Thu, 14 Nov 2019, Patrick Donnelly wrote:
> On Wed, Nov 13, 2019 at 6:36 PM Jerry Lee  wrote:
> >
> > On Thu, 14 Nov 2019 at 07:07, Patrick Donnelly  wrote:
> > >
> > > On Wed, Nov 13, 2019 at 2:30 AM Jerry Lee  wrote:
> > > > Recently, I'm evaluating the snpahsot feature of CephFS from kernel
> > > > client and everthing works like a charm.  But, it seems that reverting
> > > > a snapshot is not available currently.  Is there some reason or
> > > > technical limitation that the feature is not provided?  Any insights
> > > > or ideas are appreciated.
> > >
> > > Please provide more information about what you tried to do (commands
> > > run) and how it surprised you.
> >
> > The thing I would like to do is to rollback a snapped directory to a
> > previous version of snapshot.  It looks like the operation can be done
> > by over-writting all the current version of files/directories from a
> > previous snapshot via cp.  But cp may take lots of time when there are
> > many files and directories in the target directory.  Is there any
> > possibility to achieve the goal much faster from the CephFS internal
> > via command like "ceph fs   snap rollback
> > " (just a example)?  Thank you!
> 
> RADOS doesn't support rollback of snapshots so it needs to be done
> manually. The best tool to do this would probably be rsync of the
> .snap directory with appropriate options including deletion of files
> that do not exist in the source (snapshot).

rsync is the best bet now, yeah.

RADOS does have a rollback operation that uses clone where it can, but 
it's a per-object operation, so something still needs to walk the 
hierarchy and roll back each file's content.  The MDS could do this more 
efficiently than rsync give what it knows about the snapped inodes 
(skipping untouched inodes or, eventually, entire subtrees) but it's a 
non-trivial amount of work to implement.

sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Revert a CephFS snapshot?

[Patrick Donnelly]

On Wed, Nov 13, 2019 at 6:36 PM Jerry Lee  wrote:
>
> On Thu, 14 Nov 2019 at 07:07, Patrick Donnelly  wrote:
> >
> > On Wed, Nov 13, 2019 at 2:30 AM Jerry Lee  wrote:
> > > Recently, I'm evaluating the snpahsot feature of CephFS from kernel
> > > client and everthing works like a charm.  But, it seems that reverting
> > > a snapshot is not available currently.  Is there some reason or
> > > technical limitation that the feature is not provided?  Any insights
> > > or ideas are appreciated.
> >
> > Please provide more information about what you tried to do (commands
> > run) and how it surprised you.
>
> The thing I would like to do is to rollback a snapped directory to a
> previous version of snapshot.  It looks like the operation can be done
> by over-writting all the current version of files/directories from a
> previous snapshot via cp.  But cp may take lots of time when there are
> many files and directories in the target directory.  Is there any
> possibility to achieve the goal much faster from the CephFS internal
> via command like "ceph fs   snap rollback
> " (just a example)?  Thank you!

RADOS doesn't support rollback of snapshots so it needs to be done
manually. The best tool to do this would probably be rsync of the
.snap directory with appropriate options including deletion of files
that do not exist in the source (snapshot).

-- 
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RGW performance with low object sizes

[Christian]

Hi,

I used https://github.com/dvassallo/s3-benchmark to measure some
performance values for the rgws and got some unexpected results.
Everything above 64K has excellent performance but below it drops down to a
fraction of the speed and responsiveness resulting in even 256K objects
being faster than anything below 64K.
Does anyone observe similar effects while running this benchmark? Is it the
benchmarks fault or are there some options to tweak performance for low
object sizes?

s3-benchmark results below.

Best,
Christian

$ ./s3-benchmark -region us-east-1 -threads-min=8 -threads-max=8
-payloads-min=4 -payloads-max=15 -samples=1000 -endpoint
https://mymungedrgwendpoint.de -bucket-name=bench -create-bucket=false

--- SETUP


Uploading 8 KB objects
 100% ||  [0s:0s]
[...munged...]

--- BENCHMARK



 
+-+
   |Time to First Byte (ms)
|Time to Last Byte (ms)  |
+-++++
| Threads | Throughput |  avg   min   p25   p50   p75   p90   p99   max
|  avg   min   p25   p50   p75   p90   p99   max |
+-++++
|   8 |   1.8 MB/s |   32 1 74142424243
|   32 1 74142424243 |
+-++++

Download performance with 16 KB objects

+-+
   |Time to First Byte (ms)
|Time to Last Byte (ms)  |
+-++++
| Threads | Throughput |  avg   min   p25   p50   p75   p90   p99   max
|  avg   min   p25   p50   p75   p90   p99   max |
+-++++
|   8 |   3.7 MB/s |   30 1 54141424345
|   30 2 54141424345 |
+-++++

Download performance with 32 KB objects

+-+
   |Time to First Byte (ms)
|Time to Last Byte (ms)  |
+-++++
| Threads | Throughput |  avg   min   p25   p50   p75   p90   p99   max
|  avg   min   p25   p50   p75   p90   p99   max |
+-++++
|   8 |   6.3 MB/s |   34 2414141424343
|   35 2414142424343 |
+-++++

Download performance with 64 KB objects

+-+
   |Time to First Byte (ms)
|Time to Last Byte (ms)  |
+-++++
| Threads | Throughput |  avg   min   p25   p50   p75   p90   p99   max
|  avg   min   p25   p50   p75   p90   p99   max |
+-++++
|   8 | 156.2 MB/s |2 1 2 2 3 4 5 5
|3 2 2 3 3 5 5 6 |
+-++++

Download performance with 128 KB objects

+-+
   |Time to First Byte (ms)
|Time to Last Byte (ms)  |
+-++++
| Thread

Re: [ceph-users] dashboard hangs

[thoralf schulze]

hi Lenz,

On 11/13/19 6:38 PM, Lenz Grimmer wrote:
> there have been several reports about Ceph mgr modules (not just the
> dashboard) experiencing hangs and freezes recently. The thread "mgr
> daemons becoming unresponsive" might give you some additional insight.
> 
> Is the "device health metrics" module enabled on your cluster? Could you
> try disabling it to see if that fixes the issue?

thank you for your answer … i should have mentioned that we tried with
nautilus 14.2.2 and 14.2.4, with and without the patch to
src/pybind/mgr/devicehealth/module.py provided by Sage in the thread
mentioned above. while the patch apparently fixed the issue for other
people, it didn't help in our case.

regarding the modules: currently, we have dashboard, iostat,
pg_autoscaler, prometheus and restful enabled. disabling them one by one
until only dashboard is left helps, albeit for a short while only - i
guess this is due to the mgr respawning itself.

with kind regards,
t.



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] problem returning mon back to cluster

[Nikola Ciprich]

Hi,

just wanted to add some info..

1) I was able to workaround the problem (as advised by Harald) by increasing
mon_lease to 50s, waiting for monitor to join the cluster (it took hours!) and
decreasing it again.

2) since then we got hit by the same problem on different cluster. same 
symptoms,
same workaround.

3) I was able to 100% reproduce the problem on cleanly installed ceph 
environment
in virtual machines with same addresation and copied monitor data. for anyone 
of developers interested, I could give direct SSH access.

shall I fill a bug report for this?

thanks

nik



On Tue, Oct 15, 2019 at 07:17:38AM +0200, Nikola Ciprich wrote:
> On Tue, Oct 15, 2019 at 06:50:31AM +0200, Nikola Ciprich wrote:
> > 
> > 
> > On Mon, Oct 14, 2019 at 11:52:55PM +0200, Paul Emmerich wrote:
> > > How big is the mon's DB?  As in just the total size of the directory you 
> > > copied
> > > 
> > > FWIW I recently had to perform mon surgery on a 14.2.4 (or was it
> > > 14.2.2?) cluster with 8 GB mon size and I encountered no such problems
> > > while syncing a new mon which took 10 minutes or so.
> > Hi Paul,
> > 
> > yup I forgot to mention this.. It doesn't seem to be too big, just about
> > 100MB. I also noticed that while third monitor tries to join the cluster,
> > leader starts flapping between "leader" and "electing", so I suppose it's
> > quorum forming problem.. I tried bumping debug_ms and debug_paxos but
> > couldn't make head or tails of it.. can paste the logs somewhere if it
> > can help
> 
> btw I just noticed, that on test cluster, third mon finally managed to join
> the cluster and forum got formed.. after more then 6 hours.. knowing that 
> during
> it, the IO blocks for clients, it's pretty scary
> 
> now I can stop/start monitors without problems on it.. so it somehow got 
> "fixed"
> 
> still dunno what to do with this production cluster though, so I'll just 
> prepare
> test environment again and try digging more into it
> 
> BR
> 
> nik
> 
> 
> 
> 
> 
> > 
> > BR
> > 
> > nik
> > 
> > 
> > 
> > > 
> > > Paul
> > > 
> > > -- 
> > > Paul Emmerich
> > > 
> > > Looking for help with your Ceph cluster? Contact us at https://croit.io
> > > 
> > > croit GmbH
> > > Freseniusstr. 31h
> > > 81247 München
> > > www.croit.io
> > > Tel: +49 89 1896585 90
> > > 
> > > On Mon, Oct 14, 2019 at 9:41 PM Nikola Ciprich
> > >  wrote:
> > > >
> > > > On Mon, Oct 14, 2019 at 04:31:22PM +0200, Nikola Ciprich wrote:
> > > > > On Mon, Oct 14, 2019 at 01:40:19PM +0200, Harald Staub wrote:
> > > > > > Probably same problem here. When I try to add another MON, "ceph
> > > > > > health" becomes mostly unresponsive. One of the existing ceph-mon
> > > > > > processes uses 100% CPU for several minutes. Tried it on 2 test
> > > > > > clusters (14.2.4, 3 MONs, 5 storage nodes with around 2 hdd osds
> > > > > > each). To avoid errors like "lease timeout", I temporarily increase
> > > > > > "mon lease", from 5 to 50 seconds.
> > > > > >
> > > > > > Not sure how bad it is from a customer PoV. But it is a problem by
> > > > > > itself to be several minutes without "ceph health", when there is an
> > > > > > increased risk of losing the quorum ...
> > > > >
> > > > > Hi Harry,
> > > > >
> > > > > thanks a lot for your reply! not sure we're experiencing the same 
> > > > > issue,
> > > > > i don't have it on any other cluster.. when this is happening to you, 
> > > > > does
> > > > > only ceph health stop working, or it also blocks all clients IO?
> > > > >
> > > > > BR
> > > > >
> > > > > nik
> > > > >
> > > > >
> > > > > >
> > > > > >  Harry
> > > > > >
> > > > > > On 13.10.19 20:26, Nikola Ciprich wrote:
> > > > > > >dear ceph users and developers,
> > > > > > >
> > > > > > >on one of our production clusters, we got into pretty unpleasant 
> > > > > > >situation.
> > > > > > >
> > > > > > >After rebooting one of the nodes, when trying to start monitor, 
> > > > > > >whole cluster
> > > > > > >seems to hang, including IO, ceph -s etc. When this mon is stopped 
> > > > > > >again,
> > > > > > >everything seems to continue. Traying to spawn new monitor leads 
> > > > > > >to the same problem
> > > > > > >(even on different node).
> > > > > > >
> > > > > > >I had to give up after minutes of outage, since it's unacceptable. 
> > > > > > >I think we had this
> > > > > > >problem once in the past on this cluster, but after some (but much 
> > > > > > >shorter) time, monitor
> > > > > > >joined and it worked fine since then.
> > > > > > >
> > > > > > >All cluster nodes are centos 7 machines, I have 3 monitors (so 2 
> > > > > > >are now running), I'm
> > > > > > >using ceph 13.2.6
> > > > > > >
> > > > > > >Network connection seems to be fine.
> > > > > > >
> > > > > > >Anyone seen similar problem? I'd be very grateful for tips on how 
> > > > > > >to debug and solve this..
> > > > > > >
> > > > > > >for those interested, here's log of one of running monitors with 
> > > > > > >debug_mon set to 10/10:
> > > > > > >
> > > > > > >https://s