Re: [ceph-users] slow requests are blocked

[LOPEZ Jean-Charles]

Hi Grigory,

looks like osd.16 is having a hard time acknowledging the write request (for 
bucket resharding operations from what it looks like) as it takes about 15 
seconds for osd.16 to receive the commit confirmation from osd.21 on subop 
communication.

Have a go and check at the journal device for osd.21 or if the machine where 
osd.21 is running is either overloaded or has a network issue.

Regards
JC

> On 15 May 2018, at 19:49, Grigory Murashov  wrote:
> 
> Hello guys!
> 
> I collected output of ceph daemon osd.16 dump_ops_in_flight and ceph daemon 
> osd.16 dump_historic_ops.
> 
> Here is the output of ceph heath details in the moment of problem
> 
> HEALTH_WARN 20 slow requests are blocked > 32 sec
> REQUEST_SLOW 20 slow requests are blocked > 32 sec
> 20 ops are blocked > 65.536 sec
> osds 16,27,29 have blocked requests > 65.536 sec
> So I grab logs from osd.16.
> 
> The file is attached.  Could you please help to translate?
> 
> Thanks in advance.
> Grigory Murashov
> Voximplant
> 14.05.2018 18:14, Grigory Murashov пишет:
>> Hello David!
>> 
>> 2. I set it up 10/10
>> 
>> 3. Thanks, my problem was I did it on host where was no osd.15 daemon.
>> 
>> Could you please help to read osd logs?
>> 
>> Here is a part from ceph.log
>> 
>> 2018-05-14 13:46:32.644323 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0 
>> 553895 : cluster [INF] Cluster is now healthy
>> 2018-05-14 13:46:43.741921 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0 
>> 553896 : cluster [WRN] Health check failed: 21 slow requests are blocked > 
>> 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:46:49.746994 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0 
>> 553897 : cluster [WRN] Health check update: 23 slow requests are blocked > 
>> 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:46:55.752314 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0 
>> 553900 : cluster [WRN] Health check update: 3 slow requests are blocked > 32 
>> sec (REQUEST_SLOW)
>> 2018-05-14 13:47:01.030686 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0 
>> 553901 : cluster [WRN] Health check update: 4 slow requests are blocked > 32 
>> sec (REQUEST_SLOW)
>> 2018-05-14 13:47:07.764236 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0 
>> 553903 : cluster [WRN] Health check update: 32 slow requests are blocked > 
>> 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:47:13.770833 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0 
>> 553904 : cluster [WRN] Health check update: 21 slow requests are blocked > 
>> 32 sec (REQUEST_SLOW)
>> 2018-05-14 13:47:17.774530 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0 
>> 553905 : cluster [INF] Health check cleared: REQUEST_SLOW (was: 12 slow 
>> requests are blocked > 32 sec)
>> 2018-05-14 13:47:17.774582 mon.storage-ru1-osd1 mon.0 185.164.149.2:6789/0 
>> 553906 : cluster [INF] Cluster is now healthy
>> At 13-47 I had a problem with osd.21
>> 
>> 1. Ceph Health (storage-ru1-osd1.voximplant.com:ceph.health): HEALTH_WARN
>> {u'REQUEST_SLOW': {u'severity': u'HEALTH_WARN', u'summary': {u'message': u'4 
>> slow requests are blocked > 32 sec'}}}
>> HEALTH_WARN 4 slow requests are blocked > 32 sec
>> REQUEST_SLOW 4 slow requests are blocked > 32 sec
>> 2 ops are blocked > 65.536 sec
>> 2 ops are blocked > 32.768 sec
>> osd.21 has blocked requests > 65.536 sec
>> Here is a part from ceph-osd.21.log
>> 2018-05-14 13:47:06.891399 7fb806dd6700 10 osd.21 pg_epoch: 236 pg[2.0( v 
>> 236'297 (0'0,236'297] local-lis/les=223/224 n=1 ec=119/119 lis/c 223/223 
>> les/c/f 224/224/0 223/223/212) [21,29,15]
>> r=0 lpr=223 crt=236'297 lcod 236'296 mlcod 236'296 active+clean]  dropping 
>> ondisk_read_lock
>> 2018-05-14 13:47:06.891435 7fb806dd6700 10 osd.21 236 dequeue_op 
>> 0x56453b753f80 finish
>> 2018-05-14 13:47:07.111388 7fb8185f9700 10 osd.21 236 tick
>> 2018-05-14 13:47:07.111398 7fb8185f9700 10 osd.21 236 do_waiters -- start
>> 2018-05-14 13:47:07.111401 7fb8185f9700 10 osd.21 236 do_waiters -- finish
>> 2018-05-14 13:47:07.800421 7fb817df8700 10 osd.21 236 tick_without_osd_lock
>> 2018-05-14 13:47:07.800444 7fb817df8700 10 osd.21 236 
>> promote_throttle_recalibrate 0 attempts, promoted 0 objects and 0  bytes; 
>> target 25 obj/sec or 5120 k bytes/sec
>> 2018-05-14 13:47:07.800449 7fb817df8700 10 osd.21 236 
>> promote_throttle_recalibrate  actual 0, actual/prob ratio 1, adjusted 
>> new_prob 1000, prob 1000 -> 1000
>> 2018-05-14 13:47:08.111470 7fb8185f9700 10 osd.21 236 tick
>> 2018-05-14 13:47:08.111483 7fb8185f9700 10 osd.21 236 do_waiters -- start
>> 2018-05-14 13:47:08.111485 7fb8185f9700 10 osd.21 236 do_waiters -- finish
>> 2018-05-14 13:47:08.181070 7fb8055d3700 10 osd.21 236 dequeue_op 
>> 0x564539651000 prio 63 cost 0 latency 0.000143 
>> osd_op(client.2597258.0:213844298 6.1d4 6.4079fd4 (undecoded) ondisk+read+kno
>> wn_if_redirected e236) v8 pg pg[6.1d4( v 236'20882 (236'19289,236'20882] 
>> local-lis/les=223/224 n=20791 ec=145/132 lis/c 223/223 les/c/f 224/224/0 
>> 223/223/212) [21,29,17] r=0 lpr=223 crt=236

Re: [ceph-users] Erasure code profile

[LOPEZ Jean-Charles]

Hi,

the default failure domain if not specified on the CLI at the moment you create 
your EC profile is set to HOST. So you need 14 OSDs spread across 14 different 
nodes by default. And you only have 8 different nodes.

Regards
JC

> On 23 Oct 2017, at 21:13, Karun Josy <karunjo...@gmail.com> wrote:
> 
> Thank you for the reply.
> 
> There are 8 OSD nodes with 23 OSDs in total. (However, they are not 
> distributed equally on all nodes)
> 
> So it satisfies that criteria, right?
> 
> 
> 
> Karun Josy
> 
> On Tue, Oct 24, 2017 at 12:30 AM, LOPEZ Jean-Charles <jelo...@redhat.com 
> <mailto:jelo...@redhat.com>> wrote:
> Hi,
> 
> yes you need as many OSDs that k+m is equal to. In your example you need a 
> minimum of 14 OSDs for each PG to become active+clean.
> 
> Regards
> JC
> 
>> On 23 Oct 2017, at 20:29, Karun Josy <karunjo...@gmail.com 
>> <mailto:karunjo...@gmail.com>> wrote:
>> 
>> Hi,
>> 
>> While creating a pool with erasure code profile k=10, m=4, I get PG status as
>> "200 creating+incomplete"
>> 
>> While creating pool with profile k=5, m=3 it works fine.
>> 
>> Cluster has 8 OSDs with total 23 disks.
>> 
>> Is there any requirements for setting the first profile ?
>> 
>> Karun 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
>> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
> 
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Erasure code profile

[LOPEZ Jean-Charles]

Hi,

yes you need as many OSDs that k+m is equal to. In your example you need a 
minimum of 14 OSDs for each PG to become active+clean.

Regards
JC

> On 23 Oct 2017, at 20:29, Karun Josy  wrote:
> 
> Hi,
> 
> While creating a pool with erasure code profile k=10, m=4, I get PG status as
> "200 creating+incomplete"
> 
> While creating pool with profile k=5, m=3 it works fine.
> 
> Cluster has 8 OSDs with total 23 disks.
> 
> Is there any requirements for setting the first profile ?
> 
> Karun 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS: mount fs - single posing of failure

[LOPEZ Jean-Charles]

Hi Oscar,

the mount command accepts multiple MON addresses.

mount -t ceph monhost1,monhost2,monhost3:/ /mnt/foo

If not specified the port by default is 6789.

JC

> On Aug 28, 2017, at 13:54, Oscar Segarra  wrote:
> 
> Hi,
> 
> In Ceph, by design there is no single point of failure I  terms of server 
> roles, nevertheless, from the client point of view, it might exist.
> 
> In my environment:
> Mon1: 192.168.100.101:6789 
> Mon2: 192.168.100.102:6789 
> Mon3: 192.168.100.103:6789 
> 
> Client: 192.168.100.104
> 
> I have created a line in /etc/fstab referencing Mon1 but, of course, if Mon1 
> fails, the mount point gets stuck. 
> 
> I'd like to create a vip assigned to any host with tcp port 6789 UP and, in 
> the client, mount the CephFS using that VIP.
> 
> Is there any way to achieve this? 
> 
> Thanks a lot in advance! 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Health_Warn recovery stuck / crushmap problem?

[LOPEZ Jean-Charles]

Hi Jonas,

In your current CRUSH map your root ssd contains 2 nodes but those two nodes 
contain no osds and this is causing the problem.

Look like you forgot to set the parameter osd_crush_update_on_start = false 
before applying your special CRUSH Map. Hence when you restarted the OSD they 
wen back the default behaviour of attaching themselves to the host they run on.

To get it back to healthy for now, set the parameter above in your ceph.conf on 
your OSD nodes, restart your OSDs then re-apply your customized CRUSH map.

As an alternative you can also use the CRUSH location hook to automate the 
placement of your OSDs 
(http://docs.ceph.com/docs/master/rados/operations/crush-map/#custom-location-hooks
 
).
 

Regards
JC

> On 24 Jan 2017, at 07:42, Jonas Stunkat  wrote:
> 
> All OSD´s and Monitors are up from what I can see.
> I read through the troubleshooting like mentioned in the ceph documentation 
> for PGs and came to the conclusion that nothing there would help me, so I 
> didn´t try anything - except restarting / rebooting OSD´s and Monitors.
> 
> How do I recover from this, it looks to me that the data itself should be 
> safe for now, but why is it not restoring?
> I guess the problem may be the crushmap.
> 
> Here are some outputs:
> 
> #ceph health detail
> 
> HEALTH_WARN 475 pgs degraded; 640 pgs stale; 475 pgs stuck degraded; 640 pgs 
> stuck stale; 640 pgs stuck unclean; 475 pgs stuck undersized; 475 pgs 
> undersized; recovery 104812/279550 objects degraded (37.493%); recovery 
> 69926/279550 objects misplaced (25.014%)
> pg 3.ec is stuck unclean for 3326815.935321, current state 
> stale+active+remapped, last acting [7,6]
> pg 3.ed is stuck unclean for 3288818.682456, current state 
> stale+active+remapped, last acting [6,7]
> pg 3.ee is stuck unclean for 409973.052061, current state 
> stale+active+undersized+degraded, last acting [7]
> pg 3.ef is stuck unclean for 3357894.554762, current state 
> stale+active+undersized+degraded, last acting [7]
> pg 3.e8 is stuck unclean for 384815.518837, current state 
> stale+active+undersized+degraded, last acting [6]
> pg 3.e9 is stuck unclean for 3274554.591000, current state 
> stale+active+remapped, last acting [6,7]
> ..
> 
> 
> 
> This is the crushmap I created and intended to use and thought I used for the 
> past 2 months:
> - pvestorage1-ssd and pvestorage1-platter are the same hosts, it seems like 
> this is not possible but I never noticed
> - likewise with pvestorage2
> 
> # begin crush map
> tunable choose_local_tries 0
> tunable choose_local_fallback_tries 0
> tunable choose_total_tries 50
> tunable chooseleaf_descend_once 1
> tunable straw_calc_version 1
> 
> # devices
> device 0 osd.0
> device 1 osd.1
> device 2 osd.2
> device 3 osd.3
> device 4 osd.4
> device 5 osd.5
> device 6 osd.6
> device 7 osd.7
> 
> # types
> type 0 osd
> type 1 host
> type 2 chassis
> type 3 rack
> type 4 row
> type 5 pdu
> type 6 pod
> type 7 room
> type 8 datacenter
> type 9 region
> type 10 root
> 
> # buckets
> host pvestorage1-ssd {
> id -2   # do not change unnecessarily
> # weight 1.740
> alg straw
> hash 0  # rjenkins1
> item osd.0 weight 0.870
> item osd.1 weight 0.870
> }
> host pvestorage2-ssd {
> id -3   # do not change unnecessarily
> # weight 1.740
> alg straw
> hash 0  # rjenkins1
> item osd.2 weight 0.870
> item osd.3 weight 0.870
> }
> host pvestorage1-platter {
> id -4   # do not change unnecessarily
> # weight 4
> alg straw
> hash 0  # rjenkins1
> item osd.4 weight 2.000
> item osd.5 weight 2.000
> }
> host pvestorage2-platter {
> id -5   # do not change unnecessarily
> # weight 4
> alg straw
> hash 0  # rjenkins1
> item osd.6 weight 2.000
> item osd.7 weight 2.000
> }
> 
> root ssd {
> id -1   # do not change unnecessarily
> # weight 3.480
> alg straw
> hash 0  # rjenkins1
> item pvestorage1-ssd weight 1.740
> item pvestorage2-ssd weight 1.740
> }
> 
> root platter {
> id -6   # do not change unnecessarily
> # weight 8
> alg straw
> hash 0  # rjenkins1
> item pvestorage1-platter weight 4.000
> item pvestorage2-platter weight 4.000
> }
> 
> # rules
> rule ssd {
> ruleset 0
> type replicated
> min_size 1
> max_size 10
> step take ssd
> step chooseleaf firstn 0 type host
> step emit
> }
> 
> rule platter {
> ruleset 1
> type replicated
> min_size 1
> max_size 10
> step take platter
> step chooseleaf firstn 0 type host
> step 

Re: [ceph-users] How exactly does rgw work?

[LOPEZ Jean-Charles]

Hi Gerald,

for the s3 and swift case, the clients are not accessing the ceph cluster. They 
are s3 and swift clients and only discuss with the RGW over HTTP. The RGW is 
the ceph client that does all the interaction with the ceph cluster.

Best
JC

> On Dec 21, 2016, at 07:27, Gerald Spencer  wrote:
> 
> I was under the impression that when a client talks to the cluster, it grabs 
> the osd map and computes the crush algorithm to determine where it stores the 
> object. Does the rgw server do this for clients? If I had 12 clients all 
> talking through one gateway, would that server have to pass all of the 
> objects from the clients to the cluster?
> 
> 
> And 48 osd nodes, each with 12 x 6TB drives and a PCIe write journal. That 
> would be 576 osds in the cluster, with about 3.4PB raw...
> 
> 
> On Tue, Dec 20, 2016 at 1:12 AM Wido den Hollander  > wrote:
> 
> 
> > Op 20 december 2016 om 3:24 schreef Gerald Spencer  > >:
> 
> >
> 
> >
> 
> > Hello all,
> 
> >
> 
> > We're currently waiting on a delivery of equipment for a small 50TB proof
> 
> > of concept cluster, and I've been lurking/learning a ton from you. Thanks
> 
> > for how active everyone is.
> 
> >
> 
> > Question(s):
> 
> > How does the raids gateway work exactly?
> 
> 
> 
> The RGW doesn't do any RAID. It chunks up larger objects into smaller RADOS 
> chunks. The first chunk is always 512k (IIRC) and then it chunks up into 4MB 
> RADOS objects.
> 
> 
> 
> > Does it introduce a single point of failure?
> 
> 
> 
> It does if you deploy only one RGW. Always deploy multiple with loadbalancing 
> in front.
> 
> 
> 
> > Does all of the traffic go through the host running the rgw server?
> 
> 
> 
> Yes it does.
> 
> 
> 
> >
> 
> > I just don't fully understand that side of things. As for architecture our
> 
> > poc will have:
> 
> > - 1 monitor
> 
> > - 4 OSDs with 12 x 6TB drives, 1 x 800 PCIe journal
> 
> >
> 
> 
> 
> Underscaled machines, go for less disks per machine but more machines. More 
> smaller machines works a lot better with Ceph then a few big machines.
> 
> 
> 
> > I'd all goes as planned, this will scale up to:
> 
> > - 3 monitors
> 
> 
> 
> Always run with 3 MONs. Otherwise it is a serious SPOF.
> 
> 
> 
> > - 48 osds
> 
> >
> 
> > This should give us enough storage (~1.2PB) wth enough throughput to handle
> 
> > the data requirements of our machines to saturate our 100Gb link...
> 
> >
> 
> 
> 
> That won't happen with just 4 machines. Replica 3x taken into account is 
> well. You will need a lot more machines to get the 100Gb link fully utilized.
> 
> 
> 
> Wido
> 
> 
> 
> >
> 
> >
> 
> >
> 
> >
> 
> > Cheers,
> 
> > G
> 
> > ___
> 
> > ceph-users mailing list
> 
> > ceph-users@lists.ceph.com 
> 
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
> > 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Very Small Cluster

[LOPEZ Jean-Charles]

mon_osd_min_down_reporters by default set to 2

I guess you’ll have to set it to 1 in your case

JC

> On Sep 29, 2016, at 08:16, Gregory Farnum  wrote:
> 
> I think the problem is that Ceph requires a certain number of OSDs or a 
> certain number of reports of failure before it marks an OSD down. These 
> thresholds are not tuned for a 2-OSD cluster; you probably want to set them 
> to 1.
> Also keep in mind that the OSDs provide a grace period of 20-30 seconds 
> before they'll report somebody down; this helps prevent spurious recovery but 
> means you will get paused IO on an unclean shutdown.
> 
> I can't recall the exact config options off-hand, but it's something like 
> "mon osd min down reports". Search the docs for that. :)
> -Greg
> 
> On Thursday, September 29, 2016, Peter Maloney 
>  > wrote:
> On 09/29/16 14:07, Ranjan Ghosh wrote:
> > Wow. Amazing. Thanks a lot!!! This works. 2 (hopefully) last questions
> > on this issue:
> >
> > 1) When the first node is coming back up, I can just call "ceph osd up
> > 0" and Ceph will start auto-repairing everything everything, right?
> > That is, if there are e.g. new files that were created during the time
> > the first node was down, they will (sooner or later) get replicated
> > there?
> Nope, there is no "ceph osd up "; you just start the osd, and it
> already gets recognized as up. (if you don't like this, you set it out,
> not just down; and there is a "ceph osd in " to undo that.)
> >
> > 2) If I don't call "osd down" manually (perhaps at the weekend when
> > I'm not at the office) when a node dies - did I understand correctly
> > that the "hanging" I experienced is temporary and that after a few
> > minutes (don't want to try out now) the node should also go down
> > automatically?
> I believe so, yes.
> 
> Also, FYI, RBD images don't seem to have this issue, and work right away
> on a 3 osd cluster. Maybe cephfs would also work better with a 3rd osd,
> even an empty one (weight=0). (and I had an unresolved issue testing the
> same with cephfs on my virtual test cluster)
> >
> > BR,
> > Ranjan
> >
> >
> > Am 29.09.2016 um 13:00 schrieb Peter Maloney:
> >>
> >> And also you could try:
> >>  ceph osd down 
> >
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] help on keystone v3 ceph.conf in Jewel

[LOPEZ Jean-Charles]

Hi,

from the log file it looks like librbd.so doesn’t contain a specific entry 
point that needs to be called. See my comment inline.

Have you upgraded the ceph client packages on the cinder node and on the nova 
compute node? Or you just did the upgrade on the ceph nodes?

JC

> On Sep 9, 2016, at 09:37, Robert Duncan  wrote:
> 
> Hi,
> 
> I have deployed the Mirantis distribution of OpenStack Mitaka which comes 
> with Ceph Hammer, since I want to use keystone v3 with radosgw I added the 
> Ubuntu cloud archive for Mitaka on Trusty.
> And then followed the upgrade instructions (had to remove the mos sources 
> from sources.list)
> 
> Anyway the upgrade looks to have gone okay and I am now on jewel, but rdb and 
> rgw have stopped working in the cloud - is this down to my ceph.conf?
> 
> There are no clues on keystone logs
> 
> 
> 
> [global]
> fsid = 5d587e15-5904-4fd2-84db-b4038c18e327
> mon_initial_members = node-10
> mon_host = 172.25.80.4
> auth_cluster_required = cephx
> auth_service_required = cephx
> auth_client_required = cephx
> filestore_xattr_use_omap = true
> log_to_syslog_level = info
> log_to_syslog = True
> osd_pool_default_size = 2
> osd_pool_default_min_size = 1
> osd_pool_default_pg_num = 64
> public_network = 172.25.80.0/24
> log_to_syslog_facility = LOG_LOCAL0
> osd_journal_size = 2048
> auth_supported = cephx
> osd_pool_default_pgp_num = 64
> osd_mkfs_type = xfs
> cluster_network = 172.25.80.0/24
> osd_recovery_max_active = 1
> osd_max_backfills = 1
> setuser match path = /var/lib/ceph/$type/$cluster-$id
> 
> [client]
> rbd_cache_writethrough_until_flush = True
> rbd_cache = True
> 
> [client.radosgw.gateway]
> rgw_keystone_accepted_roles = _member_, Member, admin, swiftoperator
> keyring = /etc/ceph/keyring.radosgw.gateway
> rgw_frontends = fastcgi socket_port=9000 socket_host=127.0.0.1
> rgw_socket_path = /tmp/radosgw.sock
> rgw_keystone_revocation_interval = 100
> rgw_keystone_url = http://172.25.90.5:35357
> rgw_keystone_admin_token = iaUKRVcU6dSa8xuJvJiZYkEZ
> host = node-10
> rgw_dns_name = *.domain.local
> rgw_print_continue = True
> rgw_keystone_token_cache_size = 10
> rgw_data = /var/lib/ceph/radosgw
> user = www-data
> 
> Cinder throws the following error:
> 
> 9 16:01:26 node-10 cinder-volume: 2016-09-09 16:01:26.026 3759 ERROR 
> oslo_messaging.rpc.dispatcher [req-c88086a3-3d6b-42a3-9670-c4c92909423c 
> 9f4bf81c57214f88bced5e233061e71e 1cb2488ad03541df8f122b6f4907c820 - - -] 
> Exception during message handling: /usr/lib/librbd.so.1: undefined symbol: 
> _ZN8librados5Rados15aio_watch_flushEPNS_13AioCompletionE
> 2016-09-09 16:01:26.026 3759 ERROR oslo_messaging.rpc.dispatcher Traceback 
> (most recent call last):
> 2016-09-09 16:01:26.026 3759 ERROR oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 
> 138, in _dispatch_and_reply
> 2016-09-09 16:01:26.026 3759 ERROR oslo_messaging.rpc.dispatcher 
> incoming.message))
> 2016-09-09 16:01:26.026 3759 ERROR oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 
> 185, in _dispatch
> 2016-09-09 16:01:26.026 3759 ERROR oslo_messaging.rpc.dispatcher return 
> self._do_dispatch(endpoint, method, ctxt, args)
> 2016-09-09 16:01:26.026 3759 ERROR oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 
> 127, in _do_dispatch
> 2016-09-09 16:01:26.026 3759 ERROR oslo_messaging.rpc.dispatcher result = 
> func(ctxt, **new_args)
> 2016-09-09 16:01:26.026 3759 ERROR oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 631, in 
> create_volume
> 2016-09-09 16:01:26.026 3759 ERROR oslo_messaging.rpc.dispatcher 
> _run_flow()
> 2016-09-09 16:01:26.026 3759 ERROR oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 618, in 
> _run_flow
> 2016-09-09 16:01:26.026 3759 ERROR oslo_messaging.rpc.dispatcher 
> flow_engine.run()
> 2016-09-09 16:01:26.026 3759 ERROR oslo_messaging.rpc.dispatcher   File 
> "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", 
> line 224, in run
> 2016-09-09 16:01:26.026 3759 ERROR oslo_messaging.rpc.dispatcher for 
> _state in self.run_iter():
> <155>Sep  9 16:01:26 node-10 cinder-scheduler: 2016-09-09 16:01:26.167 4008 
> ERROR cinder.scheduler.filter_scheduler 
> [req-c88086a3-3d6b-42a3-9670-c4c92909423c 9f4bf81c57214f88bced5e233061e71e 
> 1cb2488ad03541df8f122b6f4907c820 - - -] Error scheduling None from last 
> vol-service: rbd:volumes@RBD-backend#RBD-backend : [u'Traceback (most recent 
> call last):\n', u'  File 
> "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py",
>  line 82, in _execute_task\nresult = task.execute(**arguments)\n', u'  
> File 
> 

Re: [ceph-users] Changing Replication count

[LOPEZ Jean-Charles]

Hi,

the stray replicas will be automatically removed in the background.

JC

> On Sep 6, 2016, at 17:58, Vlad Blando  wrote:
> 
> Sorry bout that
> 
> It's all set now, i thought that was replica count as it is also 4 and 5 :)
> 
> I can see the changes now
> 
> [root@controller-node ~]# ceph osd dump | grep 'replicated size'
> pool 4 'images' replicated size 2 min_size 2 crush_ruleset 0 object_hash 
> rjenkins pg_num 1024 pgp_num 1024 last_change 19641 flags hashpspool 
> stripe_width 0
> pool 5 'volumes' replicated size 3 min_size 2 crush_ruleset 0 object_hash 
> rjenkins pg_num 512 pgp_num 512 last_change 19640 flags hashpspool 
> stripe_width 0
> [root@controller-node ~]#
> 
> 
> To my other question, will it remove the excess replicas?
> 
> ​/vlad
> 
> On Wed, Sep 7, 2016 at 8:51 AM, Jeff Bailey  > wrote:
> 
> 
> On 9/6/2016 8:41 PM, Vlad Blando wrote:
> Hi,
> 
> My replication count now is this
> 
> [root@controller-node ~]# ceph osd lspools
> 4 images,5 volumes,
> 
> Those aren't replica counts they're pool ids.
> 
> [root@controller-node ~]#
> 
> and I made adjustment and made it to 3 for images and 2 to volumes to 3, it's 
> been 30 mins now and the values did not change, how do I know if it was 
> really changed.
> 
> this is the command I executed
> 
>  ceph osd pool set images size 2
>  ceph osd pool set volumes size 3
> 
> ceph osd pool set images min_size 2
> ceph osd pool set images min_size 2
> 
> 
> Another question, since the previous replication count for images is 4 and 
> volumes to 5, it will delete the excess replication right?
> 
> Thanks for the help
> 
> 
> /vlad
> ᐧ
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
> 
> 
> ᐧ
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Reading payload from rados_watchcb2_t callback

[LOPEZ Jean-Charles]

Hi Nick,

a good read to see what’s in it.

http://dachary.org/?p=1904

JC

> On Aug 18, 2016, at 08:28, Nick Fisk  wrote:
> 
> Just to answer myself in case anyone stumbles across this in the future. I 
> was on the right track, but I think there are null
> characters before the text payload which was tricking printf.
> 
> In the end I managed to work it out and came up with this:
> 
> char *temp = (char*)data+4;
> 
> Which skips the 1st few bytes of the payload.no idea what they are, but 
> skipping 4 bytes takes you straight to the start of the
> text part that you send with notify.
> 
>> -Original Message-
>> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
>> Nick Fisk
>> Sent: 17 August 2016 21:49
>> To: 'ceph-users' 
>> Subject: [ceph-users] Reading payload from rados_watchcb2_t callback
>> 
>> Hi All,
>> 
>> I'm writing a small piece of code to call fsfreeze/unfreeze that can be 
>> invoked by a RADOS notify. I have the basic watch/notify
>> functionality working but I need to be able to determine if the notify 
>> message is to freeze or unfreeze, or maybe something
>> completely unrelated.
>> 
>> I'm looking at the rados_watchcb2_t callback and can see that the data 
>> payload is returned as a void pointer. This is where it all
> starts
>> to go a little pear shaped for my basic C skills. I think I have to cast the 
>> pointer to a (char *) but I still can't seem to get
> anything useful
>> from it.
>> 
>> I've been following some of the tests in the Ceph source and they seem to 
>> use some sort of typedef called a bufferlist, is this
> what I
>> need to try and look into?
>> 
>> Does anyone have any pointers (excuse the pun) as to how I would read the 
>> text part of the payload from it?
>> 
>> void watch_notify2_cb(void *arg, uint64_t notify_id, uint64_t cookie, 
>> uint64_t notifier_gid, void *data, size_t data_len)
>> 
>> Many Thanks,
>> Nick
>> 
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RGW integration with keystone

[LOPEZ Jean-Charles]

Hi,

what can not get started?

have you checked the log files for any critical error message of the “can’t get 
started component"?

what does the config file you have created look like: The good one and the bad 
one?

in your particular case what is your Keystone configuration?

And may be providing the community with some links to the log file and 
configuration file capture could really help.
Regards
JC


> On Jun 9, 2016, at 24:16, fridifree  wrote:
> 
> Hi Everyone,
> I install cluster ceph and while I install it using the keyston options in 
> the ceph.conf cannot get started,but if I remove those options about keystone 
> in the ceph.conf everything is going good.
> 
> 
> Any ideas?
> 
> 
> Thanks
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] no osds in jewel

[LOPEZ Jean-Charles]

Hi,

first I’f have one remark. You run both a ceph-deploy mon create-initial then a 
"ceph-deploy mon create + ceph-deploy gatherkeys". Choose one or the other not 
both.

Then, I notice that you are zapping and deploying using drive /dev/sda which is 
usually the system disks. So next question is: Is /dev/sda really empty and 
available for OSDs

I guess that what happened is that because sda is not free and empty, the zap 
failed as it does not have the force option and then the old create failed 
because sda could not be prepared/formatted.

Regards
JC

> On Jun 5, 2016, at 10:22, Jaemyoun Lee  wrote:
> 
> Hi, 
> 
> When I run below script to install Ceph (10.2.0), I met an error "no osds".
> Hammer was installed by the script.
> So I think I miss new thing, which was released since Hammer.
> 
> Do you know what I miss?
> 
> --- The script ---
> #!/bin/sh
> 
> set -x
> 
> ceph-deploy new csElsa
> echo "osd pool default size = 1" >> ceph.conf
> ceph-deploy install csElsa csAnt csBull csCat
> ceph-deploy mon create-initial
> ceph-deploy mon create csElsa
> ceph-deploy gatherkeys csElsa
> ceph-deploy disk zap csAnt:sda
> ceph-deploy disk zap csBull:sda
> ceph-deploy disk zap csCat:sda
> ceph-deploy osd create csAnt:sda csBull:sda csCat:sda
> ceph-deploy admin csElsa csElsa csAnt csBull csCat
> sudo chmod +r /etc/ceph/ceph.client.admin.keyring
> ceph health
> --- end ---
> 
> --- The result of "ceph -w" ---
> # I blocked the IP
> jae@csElsa:~/git/ceph$ ceph -w
> 
> cluster 8b2816e9-1953-4157-aaf7-95e9e668fe46
>  health HEALTH_ERR
> 64 pgs are stuck inactive for more than 300 seconds
> 64 pgs stuck inactive
> no osds
>  monmap e1: 1 mons at {csElsa=1xx.1xx.2xx.1:6789/0}
> election epoch 3, quorum 0 csElsa
>  osdmap e1: 0 osds: 0 up, 0 in
> flags sortbitwise
>   pgmap v2: 64 pgs, 1 pools, 0 bytes data, 0 objects
> 0 kB used, 0 kB / 0 kB avail
>   64 creating
> 
> 2016-06-06 01:59:08.054985 mon.0 [INF] from='client.? 
> 1xx.1xx.2xx.1:0/115687' entity='client.admin' cmd='[{"prefix": "auth 
> get-or-create", "entity": "client.bootstrap-mds", "caps": ["mon", "allow 
> profile bootstrap-mds"]}]': finished
> --- end ---
> 
> Best regards,
> Jae
> 
> -- 
>   Jaemyoun Lee
>   
>   CPS Lab. (Cyber-Physical Systems Laboratory in Hanyang University) 
>   E-mail : jaemy...@hanyang.ac.kr 
>   Website : http://cpslab.hanyang.ac.kr 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Client not finding keyring

[LOPEZ Jean-Charles]

Looks like I missed the paste: 
http://docs.ceph.com/docs/master/man/8/ceph/#options

There you have the options available from the command line.

In your case the user id is radosgw-rgw0 so the command line should be 
radosgw-admin --id radosgw.rgw0 usage show or radosgw-admin --name 
client.radosgw.rgw0 usage show

If you do this it will cause the library to apply the parameters located in the 
[client.radosgw.rgw0] section of your config file which includes the location 
of the keyring.

Assuming the keyring file pointed to by the config file is valid. You can still 
share its content via gist so we can check if it’s valid

JC

> On Jun 1, 2016, at 14:37, RJ Nowling <rnowl...@redhat.com> wrote:
> 
> I did use ceph-ansible to deploy the gateway -- using the default settings.  
> It should work out of the box but does not.
> 
> So... can the radosgw-admin CLI utility take a keyring path in the conf file 
> or does the path need to be manually specified? 
> 
> And secondly, after copying the keyring to one of the default locations, I 
> tried using the client.admin and radosgw.rgw ids you mentioned but neither 
> worked:
> 
> [vagrant@ceph-rgw0 ~]$ sudo radosgw-admin --id radosgw.rgw usage show
> 2016-06-01 21:34:20.884525 7f2e5e01aa40  0 librados: client.radosgw.rgw 
> authentication error (22) Invalid argument
> couldn't init storage provider
> 
> [vagrant@ceph-rgw0 ~]$ sudo radosgw-admin --id client.admin usage show
> 2016-06-01 21:34:31.699323 7fe774f96a40  0 librados: client.client.admin 
> authentication error (22) Invalid argument
> couldn't init storage provider 
> 
> And what page were you referring to?  I didn't see a link in the email.
> 
> On Wed, Jun 1, 2016 at 4:30 PM, LOPEZ Jean-Charles <jelo...@redhat.com 
> <mailto:jelo...@redhat.com>> wrote:
> Hi,
> 
> radosgw-admin is not radosgw. It’s the RADDOS Gateway cli admin utility.
> 
> All ceph components by default use the client.admin user name to connect to 
> the Ceph cluster. If you deployed the radosgw, the gateway itself was 
> properly configured by Ansible and the files were placed where they have to 
> be.
> 
> To override, look at the following page and use the appropriate command line 
> option to specify the proper user name or user id (--id or --name): e.g. 
> radosgw-admin --id radosgw.rgw usage show
> 
> Regards
> JC
> 
> 
>> On Jun 1, 2016, at 14:07, RJ Nowling <rnowl...@redhat.com 
>> <mailto:rnowl...@redhat.com>> wrote:
>> 
>> Hi all,
>> 
>> I'm trying to set up a Ceph cluster with an S3 gateway using the 
>> ceph-ansible playbooks.  I'm running into an issue where the radosgw-admin 
>> client can't find the keyring.  The path to the keyring is listed in the 
>> ceph.conf file.  I confirmed with strace that the client opens the conf file 
>> but doesn't open the path given in the conf file -- only the default keyring 
>> paths.
>> 
>> Here's the output:
>> 
>> [vagrant@ceph-rgw0 ~]$ sudo radosgw-admin usage show
>> 2016-06-01 13:36:17.294935 7fe45ac80a40 -1 auth: unable to find a keyring on 
>> /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin:
>>  (2) No such file or directory
>> 2016-06-01 13:36:17.300732 7fe45ac80a40 -1 monclient(hunting): authenticate 
>> NOTE: no keyring found; disabled cephx authentication
>> 2016-06-01 13:36:17.300737 7fe45ac80a40  0 librados: client.admin 
>> authentication error (95) Operation not supported
>> couldn't init storage provider
>> 
>> Here's the relevant part of the config file:
>> 
>> [client.radosgw.rgw0]
>> host = ceph-rgw0
>> keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-rgw0/keyring
>> rgw socket path = /tmp/radosgw-ceph-rgw0.sock
>> log file = /var/log/ceph/ceph-rgw-ceph-rgw0.log
>> rgw data = /var/lib/ceph/radosgw/ceph-rgw.ceph-rgw0
>> rgw frontends = civetweb port=10.0.2.15:8080 <http://10.0.2.15:8080/> 
>> num_threads=50
>> 
>> When I copy the keyring to /etc/ceph/keyring, the keyring is found but I get 
>> a different error:
>> 
>> [vagrant@ceph-rgw0 ~]$ sudo radosgw-admin usage show
>> 2016-06-01 21:07:22.677525 7f62f2e38a40  0 librados: client.admin 
>> authentication error (22) Invalid argument
>> couldn't init storage provider
>> 
>> Any insights would be greatly appreciated!
>> 
>> Thanks!
>> RJ
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
>> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
> 
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Client not finding keyring

[LOPEZ Jean-Charles]

Hi,

radosgw-admin is not radosgw. It’s the RADDOS Gateway cli admin utility.

All ceph components by default use the client.admin user name to connect to the 
Ceph cluster. If you deployed the radosgw, the gateway itself was properly 
configured by Ansible and the files were placed where they have to be.

To override, look at the following page and use the appropriate command line 
option to specify the proper user name or user id (--id or --name): e.g. 
radosgw-admin --id radosgw.rgw usage show

Regards
JC


> On Jun 1, 2016, at 14:07, RJ Nowling  wrote:
> 
> Hi all,
> 
> I'm trying to set up a Ceph cluster with an S3 gateway using the ceph-ansible 
> playbooks.  I'm running into an issue where the radosgw-admin client can't 
> find the keyring.  The path to the keyring is listed in the ceph.conf file.  
> I confirmed with strace that the client opens the conf file but doesn't open 
> the path given in the conf file -- only the default keyring paths.
> 
> Here's the output:
> 
> [vagrant@ceph-rgw0 ~]$ sudo radosgw-admin usage show
> 2016-06-01 13:36:17.294935 7fe45ac80a40 -1 auth: unable to find a keyring on 
> /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin:
>  (2) No such file or directory
> 2016-06-01 13:36:17.300732 7fe45ac80a40 -1 monclient(hunting): authenticate 
> NOTE: no keyring found; disabled cephx authentication
> 2016-06-01 13:36:17.300737 7fe45ac80a40  0 librados: client.admin 
> authentication error (95) Operation not supported
> couldn't init storage provider
> 
> Here's the relevant part of the config file:
> 
> [client.radosgw.rgw0]
> host = ceph-rgw0
> keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-rgw0/keyring
> rgw socket path = /tmp/radosgw-ceph-rgw0.sock
> log file = /var/log/ceph/ceph-rgw-ceph-rgw0.log
> rgw data = /var/lib/ceph/radosgw/ceph-rgw.ceph-rgw0
> rgw frontends = civetweb port=10.0.2.15:8080  
> num_threads=50
> 
> When I copy the keyring to /etc/ceph/keyring, the keyring is found but I get 
> a different error:
> 
> [vagrant@ceph-rgw0 ~]$ sudo radosgw-admin usage show
> 2016-06-01 21:07:22.677525 7f62f2e38a40  0 librados: client.admin 
> authentication error (22) Invalid argument
> couldn't init storage provider
> 
> Any insights would be greatly appreciated!
> 
> Thanks!
> RJ
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw s3 errors after installation quickstart

[LOPEZ Jean-Charles]

Hi,

in order to use s3cmd, just make sure you have the rgw_dns_name = 
{bucket_fqdn_suffix} in your config file in the RGW section. In your case I’d 
say rgw_dns_name = xen1

And you should be good to go.

JC

> On May 31, 2016, at 04:51, hp cre  wrote:
> 
> Hello,
> 
> I created a test cluster of 3 OSD hosts (xen1,2,3) based on Ubuntu Xenial, 
> ceph 10.2.1 using the quick start steps in the docs master branch.
> 
> After jumping through a few problems, mainly from the inconsistent details in 
> the docs, i got a stable cluster running with RGW.
> 
> Running s3boto test script to create a new bucket works fine. however, when I 
> use ay other tool to PUT files, I get a strange error stating "host not 
> found". There is nothing in the gateway logs that would suggest why this 
> happens, i only get the list of get requests from the cliient(s) I use.
> 
> sample:
> ===
> 2016-05-31 13:30:21.953366 7fb2f37be700  1 civetweb: 0x7fb32800cbc0: 10.0.0.1 
> - - [31/May/2016:13:30:21 +0200] "GET / HTTP/1.1" 200 0 - CrossFTP/1.97.6 
> (Linux/4.4.0-21-generic; amd64; en; JVM 1.8.0_91)
> 2016-05-31 13:30:22.975609 7fb2f2fbd700  1 == starting new request 
> req=0x7fb2f2fb77e0 =
> 2016-05-31 13:30:22.978613 7fb2f2fbd700  1 == req done req=0x7fb2f2fb77e0 
> op status=0 http_status=200 ==
> 2016-05-31 13:30:22.978710 7fb2f2fbd700  1 civetweb: 0x7fb330016690: 10.0.0.1 
> - - [31/May/2016:13:30:22 +0200] "GET / HTTP/1.1" 200 0 - CrossFTP/1.97.6 
> (Linux/4.4.0-21-generic; amd64; en; JVM 1.8.0_91)
> 2016-05-31 13:32:04.032800 7fb2f27bc700  1 == starting new request 
> req=0x7fb2f27b67e0 =
> 2016-05-31 13:32:04.034847 7fb2f27bc700  1 == req done req=0x7fb2f27b67e0 
> op status=0 http_status=200 ==
> 2016-05-31 13:32:04.034895 7fb2f27bc700  1 civetweb: 0x7fb32c005910: 10.0.0.1 
> - - [31/May/2016:13:32:04 +0200] "GET / HTTP/1.1" 200 0 - DragonDisk 1.05 ( 
> http://www.dragondisk.com  )
> ==
> 
> example error message when I try a PUT operation on DragonDisk client to a 
> bucket I created (myb), file called testfile
> 
> 
> 2
> Operation: Copy
> /home/wes/testfile -> http://myb.xen1/testfile 
> Host not found
> 
> 
> example error I got from using CrossFTP client to upload a file
> ==
> L2] LIST All Buckets
> [R1] LIST /myb
> [R1] LIST All Buckets (cached)
> [R1] Succeeded
> [R1] LIST All Buckets
> [R1] Succeeded
> [L2] Succeeded
>  Secure random seed initialized.
> [L2] S3 Error: -1 (null) error: Request Error: myb.xen1; XML Error Message: 
> null
> [L2] -1 (null) error: Request Error: myb.xen1; XML Error Message: null
> [R1] S3 Error: -1 (null) error: Request Error: myb.xen1: unknown error; XML 
> Error Message: null
> [R1] -1 (null) error: Request Error: myb.xen1: unknown error; XML Error 
> Message: null
> 
> 
> example put operation for file "ceph-deploy-ceph.log" using s3cmd client on 
> the gateway node (xen1)
> 
> =
> root@xen1:/home/cl# s3cmd put ceph-deploy-ceph.log s3://myb
> upload: 'ceph-deploy-ceph.log' -> 's3://myb/ceph-deploy-ceph.log'  [1 of 1]
>   0 of 641045 0% in0s 0.00 B/s  failed
> WARNING: Retrying failed request: /ceph-deploy-ceph.log ([Errno -2] Name or 
> service not known)
> WARNING: Waiting 3 sec...
> upload: 'ceph-deploy-ceph.log' -> 's3://myb/ceph-deploy-ceph.log'  [1 of 1]
>   0 of 641045 0% in0s 0.00 B/s  failed
> WARNING: Retrying failed request: /ceph-deploy-ceph.log ([Errno -2] Name or 
> service not known)
> WARNING: Waiting 6 sec...
> ===
> 
> 
> 
> Here is my ceph.conf
> =
> [global]
> fsid = 77dbb949-8eed-4eea-b0ff-0c612e7e2991
> mon_initial_members = xen1, xen2, xen3
> mon_host = 10.0.0.10,10.0.0.11,10.0.0.12
> auth_cluster_required = cephx
> auth_service_required = cephx
> auth_client_required = cephx
> osd_pool_default_size = 2
> 
> 
> [client.radosgw.gateway]
> rgw_frontends = "civetweb port=80"
> host = xen1
> keyring = /etc/ceph/ceph.client.radosgw.keyring
> rgw socket path = /var/run/ceph/ceph.radosgw.gateway.sock
> rgw print continue = false
> =
> 
> Any troubleshooting help will be appreciated.
> 
> Thanks,
> hpcre
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] New user questions with radosgw with Jewel 10.2.1

[LOPEZ Jean-Charles]

Hi Sylvain,

this is probably related to the fact that the systemd unit file for the RGW is 
configured to run as user ceph. As ceph is not a privileged user, it can not 
bind to lower port numbers.

Modify the ceph-radosgw unit file and make sure the set user is set for root.

To verify this is the root cause, manually start the ceph-radosgw process from 
the command line being connected as root. If it works and you can query your 
RGW on port 80, then it is this problem.

Cheers
JC

> On May 23, 2016, at 10:03, Sylvain, Eric  wrote:
> 
>  
> Hello,
>  
> I’m a fairly new user and I am trying to bring up radosgw.
>  
> I am following this page: 
> http://docs.ceph.com/docs/master/install/install-ceph-gateway/ 
> 
>  
> I have Jewel 10.2.1 installed with a co-located admin/mon host and a separate 
> osd host
>  
> First a question: Can I run radosgw on a co-located admin/monitor host?
>  
> And then my issue: I cannot seem to get the frontent (civetweb) to run on 
> port 80.
> (I do the following)
> su – ceph
> cd my-cluster
> sudo ceph-deploy install –rgw p6-os1-mon2
> ceph-deploy --overwrite-conf rgw create p6-os1-mon
>(Note it says: The Ceph Object Gateway (RGW) is now running on host 
> p6-os1-mon2 and default port 7480)
>(Testing to http://p6-os1-mon2:7480  show the 
> correct info)
> Edit ~ceph/my-cluster/ceph.conf and add:
>[client.radosgw.p6-os1-mon2]
>rgw_frontends = "civetweb port=80”
> Reboot p6-os1/mon2
> After everything is back up, try http://p6-os1-mon2:80 
>No luck, it appears it is still running at 7480…
> What am I missing that would make radosgw get the correct port?
> (Running by hand, I see “listening_ports”
> 2016-05-23 12:13:43.479136 7fc0793f7a40 20 civetweb config: decode_url: no
> 2016-05-23 12:13:43.479138 7fc0793f7a40 20 civetweb config: 
> enable_keep_alive: yes
> 2016-05-23 12:13:43.479138 7fc0793f7a40 20 civetweb config: listening_ports: 
> 7480
> 2016-05-23 12:13:43.479139 7fc0793f7a40 20 civetweb config: num_threads: 100
> 2016-05-23 12:13:43.479140 7fc0793f7a40 20 civetweb config: run_as_user: ceph
>  
> I suspected maybe the keyring? So I did:
>   $ sudo rm -rf /etc/ceph/ceph.client.radosgw.keyring
>   $ sudo ceph-authtool --create-keyring 
> /etc/ceph/ceph.client.radosgw.keyring
>   $ sudo chmod +r /etc/ceph/ceph.client.radosgw.keyring
>   $ sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n 
> client.radosgw.p6-os1-mon2
>   $ sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n 
> client.radosgw.p6-os1-mon2 --gen-key
>   $ sudo ceph-authtool -n client.radosgw.p6-os1-mon2 --cap osd 'allow 
> rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring
>   $ sudo ceph -k /etc/ceph/ceph.client.admin.keyring auth add 
> client.radosgw.p6-os1-mon2 -i /etc/ceph/ceph.client.radosgw.keyring
>   (Also copied to to my osd node…)
>   Reboot everything, still no luck civetweb remains on 7480
>  
> Ideas? Comments?
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Upgrade to Jewel... More interesting things...

[LOPEZ Jean-Charles]

Hi Tu,

from what version were you upgrading from?

In Jewel, all Ceph processes run with the ceph user and not the root user and 
may be you should investigate into the permissions of the /var/lib/ceph/osd 
subdirectories? So if you have upgraded from hammer it could likely be the 
problem.

Regards
JC


> On May 13, 2016, at 11:00, Tu Holmes  wrote:
> 
> So I'm updating a trusty cluster to Jewel and updating the kernel at the same 
> time. 
> 
> Got around some mon issues, and that seems ok, but after upgrading one of my 
> OSD nodes, I'm getting these errors in the old log on that node.
> 
> 
> 
> 0 ceph version 10.2.1 (3a66dd4f30852819c1bdaa8ec23c795d4ad77269), process 
> ceph-osd, pid 22401 
> 2016-05-13 10:51:39.416511 7fe486e73800 0 pidfile_write: ignore empty 
> --pid-file 
> 2016-05-13 10:51:39.423336 7fe486e73800 -1 
> filestore(/var/lib/ceph/osd/ceph-6) FileStore::mount: unable to access 
> basedir '/var/lib/ceph/osd/ceph-6': (13) Permission denied 
> 2016-05-13 10:51:39.423342 7fe486e73800 -1 osd.6 0 OSD:init: unable to mount 
> object store 
> 2016-05-13 10:51:39.423348 7fe486e73800 -1 ** ERROR: osd init failed: (13) 
> Permission denied 
> 2016-05-13 10:51:39.572099 7f4d86a68800 0 set uid:gid to 1000:1000 
> (ceph:ceph) 
> 2016-05-13 10:51:39.572126 7f4d86a68800 0 ceph version 10.2.1 
> (3a66dd4f30852819c1bdaa8ec23c795d4ad77269), process ceph-osd, pid 22530 
> 2016-05-13 10:51:39.573744 7f4d86a68800 0 pidfile_write: ignore empty 
> --pid-file 
> 2016-05-13 10:51:39.580543 7f4d86a68800 -1 
> filestore(/var/lib/ceph/osd/ceph-6) FileStore::mount: unable to access 
> basedir '/var/lib/ceph/osd/ceph-6': (13) Permission denied 
> 2016-05-13 10:51:39.580549 7f4d86a68800 -1 osd.6 0 OSD:init: unable to mount 
> object store 
> 2016-05-13 10:51:39.580554 7f4d86a68800 -1 ** ERROR: osd init failed: (13) 
> Permission denied 
> 2016-05-13 10:51:39.798196 7fc0daf80800 0 set uid:gid to 1000:1000 
> (ceph:ceph) 
> 2016-05-13 10:51:39.798222 7fc0daf80800 0 ceph version 10.2.1 
> (3a66dd4f30852819c1bdaa8ec23c795d4ad77269), process ceph-osd, pid 22724 
> 2016-05-13 10:51:39.799923 7fc0daf80800 0 pidfile_write: ignore empty 
> --pid-file 
> 2016-05-13 10:51:39.806382 7fc0daf80800 -1 
> filestore(/var/lib/ceph/osd/ceph-6) FileStore::mount: unable to access 
> basedir '/var/lib/ceph/osd/ceph-6': (13) Permission denied 
> 2016-05-13 10:51:39.806387 7fc0daf80800 -1 osd.6 0 OSD:init: unable to mount 
> object store 
> 2016-05-13 10:51:39.806390 7fc0daf80800 -1 ** ERROR: osd init failed: (13) 
> Permission denied 
> 2016-05-13 10:51:39.954085 7feb28741800 0 set uid:gid to 1000:1000 
> (ceph:ceph) 
> 2016-05-13 10:51:39.954112 7feb28741800 0 ceph version 10.2.1 
> (3a66dd4f30852819c1bdaa8ec23c795d4ad77269), process ceph-osd, pid 22888 
> 2016-05-13 10:51:39.955839 7feb28741800 0 pidfile_write: ignore empty 
> --pid-file 
> 2016-05-13 10:51:39.962785 7feb28741800 -1 
> filestore(/var/lib/ceph/osd/ceph-6) FileStore::mount: unable to access 
> basedir '/var/lib/ceph/osd/ceph-6': (13) Permission denied 
> 2016-05-13 10:51:39.962791 7feb28741800 -1 osd.6 0 OSD:init: unable to mount 
> object store 
> 2016-05-13 10:51:39.962796 7feb28741800 -1 ** ERROR: osd init failed: (13) 
> Permission denied
> 
> 
> The OSDs are all mounted as expected:
> 
> 
> 
> /dev/sdl1 3.7T 1.8T 1.9T 49% /var/lib/ceph/osd/ceph-6 
> /dev/sdn1 3.7T 2.0T 1.7T 55% /var/lib/ceph/osd/ceph-18 
> /dev/sdb1 3.7T 2.2T 1.5T 61% /var/lib/ceph/osd/ceph-30 
> /dev/sdf1 3.7T 2.0T 1.7T 54% /var/lib/ceph/osd/ceph-54 
> /dev/sdh1 3.7T 1.9T 1.8T 52% /var/lib/ceph/osd/ceph-66 
> /dev/sde1 3.7T 1.9T 1.8T 51% /var/lib/ceph/osd/ceph-48 
> /dev/sdd1 3.7T 1.8T 1.9T 49% /var/lib/ceph/osd/ceph-42 
> /dev/sdk1 3.7T 1.7T 2.0T 46% /var/lib/ceph/osd/ceph-0 
> /dev/sda1 3.7T 1.9T 1.8T 51% /var/lib/ceph/osd/ceph-24 
> /dev/sdm1 3.7T 1.9T 1.8T 52% /var/lib/ceph/osd/ceph-12 
> /dev/sdc1 3.7T 1.7T 2.0T 47% /var/lib/ceph/osd/ceph-36 
> /dev/sdg1 3.7T 1.8T 1.9T 49% /var/lib/ceph/osd/ceph-60
> 
> 
> Any ideas as to what could be going on?
> 
> //Tu Holmes
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] about available space

[LOPEZ Jean-Charles]

Hi

you can use the pool quota feature to limit the usage of a particular pool.

ceph osd pool set-quota  [max_objects ] [max_bytes 
]

To remove a quota, set its value to 0.

Cheers
JC

> On May 11, 2016, at 19:49, Geocast Networks  wrote:
> 
> Hi,
> 
> my ceph df output as following,
> 
> # ceph df 
> GLOBAL:
> SIZE AVAIL RAW USED %RAW USED 
> 911T  911T 121G  0.01 
> POOLS:
> NAME ID USED %USED MAX AVAIL OBJECTS 
> block7 0 0  303T   0 
> image8 0 0  303T   0 
> metadata 9 0 0  303T   0 
> backup   100 0  303T   0 
> 
> all four pools MAX AVAIL is 303T.
> 
> can the four pools' space be setup with weighted?
> For example, I expect block pool has 70% of space, each of the other pools 
> has 10% of storage space.
> 
> Thanks.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] fibre channel as ceph storage interconnect

[LOPEZ Jean-Charles]

Hi,

your easiest way here if you want to use your FC hardware is to do IP over FC 
so that you can leverage the existing FC HBA in your servers but stick to IP as 
a communication layer. FC here would just be a low latency 
transport/encapsulation layer.

I’ve played with this gazillion years ago (early 2000’s) when I was with 
working with EMC and Brocade.

The question here is that I’m not sure who still supports IP over FC and it 
would be dependent on your FC HBAs, FC switches/directors.

For information, here is a link for AIX setup 
https://www.ibm.com/support/knowledgecenter/#!/ssw_aix_71/com.ibm.aix.networkcomm/fibrechan_intro.htm


> On Apr 21, 2016, at 20:12, Schlacta, Christ  wrote:
> 
> Is it possible?  Can I use fibre channel to interconnect my ceph OSDs?
> Intuition tells me it should be possible, yet experience (Mostly with
> fibre channel) tells me no.  I don't know enough about how ceph works
> to know for sure.  All my googling returns results about using ceph as
> a BACKEND for exporting fibre channel LUNs, which is, sadly, not what
> I'm looking for at the moment.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

JC

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mons die with mon/OSDMonitor.cc: 125: FAILED assert(version >= osdmap.epoch)...

[LOPEZ Jean-Charles]

Hi,

looks like one of your OSDs has been marked as out. Just make sure it’s in so 
you can read '67 osds: 67 up, 67 in' rather than '67 osds: 67 up, 66 in’ in the 
‘ceph -s’ output

You can quickly check which one is not in with the ‘ceph old tree’ command

JC

> On Apr 12, 2016, at 11:21, Joao Eduardo Luis  wrote:
> 
> On 04/12/2016 07:16 PM, Eric Hall wrote:
>> Removed mon on mon1, added mon on mon1 via ceph-deply.  mons now have
>> quorum.
>> 
>> I am left with:
>>cluster 5ee52b50-838e-44c4-be3c-fc596dc46f4e
>>  health HEALTH_WARN 1086 pgs peering; 1086 pgs stuck inactive; 1086
>> pgs stuck unclean; pool vms has too few pgs
>>  monmap e5: 3 mons at
>> {cephsecurestore1=172.16.250.7:6789/0,cephsecurestore2=172.16.250.8:6789/0,cephsecurestore3=172.16.250.9:6789/0},
>> election epoch 28, quorum 0,1,2
>> cephsecurestore1,cephsecurestore2,cephsecurestore3
>>  mdsmap e2: 0/0/1 up
>>  osdmap e38769: 67 osds: 67 up, 66 in
>>   pgmap v33886066: 7688 pgs, 24 pools, 4326 GB data, 892 kobjects
>> 11620 GB used, 8873 GB / 20493 GB avail
>>3 active+clean+scrubbing+deep
>> 1086 peering
>> 6599 active+clean
>> 
>> All OSDs are up/in as reported.  But I see no recovery I/O for those in
>> inactive/peering/unclean.
> 
> Someone else will probably be able to chime in with more authority than me, 
> but I would first try to restart the osds to which those stuck pgs are being 
> mapped.
> 
>  -Joao
> 
>> 
>> Thanks,
>> --
>> Eric
>> 
>> On 4/12/16 1:14 PM, Joao Eduardo Luis wrote:
>>> On 04/12/2016 06:38 PM, Eric Hall wrote:
 Ok, mon2 and mon3 are happy together, but mon1 dies with
 mon/MonitorDBStore.h: 287: FAILED assert(0 == "failed to write to db")
 
 I take this to mean mon1:store.db is corrupt as I see no permission
 issues.
 
 So... remove mon1 and add a mon?
 
 Nothing special to worry about re-adding a mon on mon1, other than rm/mv
 the current store.db path, correct?
>>> 
>>> You'll actually need to recreate the mon with 'ceph-mon --mkfs' for that
>>> to work, and that will likely require you to rm/mv the mon data
>>> directory.
>>> 
>>> You *could* copy the mon dir from one of the other monitors and use that
>>> instead. But given you have a functioning quorum, I don't think there's
>>> any reason to resort to that.
>>> 
>>> Follow the docs on removing monitors[1] and recreate the monitor from
>>> scratch, adding it to the cluster. It will sync up from scratch from the
>>> other monitors. That'll make them happy.
>>> 
>>>   -Joao
>>> 
>>> [1]
>>> http://docs.ceph.com/docs/master/rados/operations/add-or-rm-mons/#removing-monitors
>>> 
>>> 
>>> 
 
 Thanks again,
 --
 Eric
 
 On 4/12/16 11:18 AM, Joao Eduardo Luis wrote:
> On 04/12/2016 05:06 PM, Joao Eduardo Luis wrote:
>> On 04/12/2016 04:27 PM, Eric Hall wrote:
>>> On 4/12/16 9:53 AM, Joao Eduardo Luis wrote:
>>> 
 So this looks like the monitors didn't remove version 1, but this
 may
 just be a red herring.
 
 What matters, really, is the values in 'first_committed' and
 'last_committed'. If either first or last_committed happens to be
 '1',
 then there may be a bug somewhere in the code, but I doubt that.
 This
 seems just an artefact.
 
 So, it would be nice if you could provide the value of both
 'osdmap:first_committed' and 'osdmap:last_committed'.
>>> 
>>> mon1:
>>> (osdmap, last_committed)
>>>  : 01 00 00 00 00 00 00 00 : 
>>> (osdmap, fist_committed) does not exist
>>> 
>>> mon2:
>>> (osdmap, last_committed)
>>>  : 01 00 00 00 00 00 00 00 : 
>>> (osdmap, fist_committed) does not exist
>>> 
>>> mon3:
>>> (osdmap, last_committed)
>>>  : 01 00 00 00 00 00 00 00 : 
>>> (osdmap, first_committed)
>>>  : b8 94 00 00 00 00 00 00
>> 
>> Wow! This is unexpected, but fits the assertion just fine.
>> 
>> The solution, I think, will be rewriting first_committed and
>> last_committed on all monitors - except on mon1.
> 
> Let me clarify this a bit: the easy way out for mon1 would be to fix
> the
> other two monitors and recreate mon1.
> 
> If you prefer to also fix mon1, you can simply follow the same steps on
> the previous email for all the monitors, but ensuring
> osdmap:full_latest
> on mon1 reflects the last available full_ version on its store.
> 
>   -Joao
>>> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com

Re: [ceph-users] Replication between regions?

[LOPEZ Jean-Charles]

Hi,

this is where it is discussed : 
http://docs.ceph.com/docs/hammer/radosgw/federated-config/

JC

> On Feb 18, 2016, at 15:14, Alexandr Porunov  
> wrote:
> 
> Is it possible to replicate objects across the regions. How can we create 
> such clusters?
> 
> Could you suggest me helpful articles/books about Ceph Cooking? I want to 
> know is it possible to create multi master data centers with data replication 
> among them.
> 
> Sincerely
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Problem with radosgw

[LOPEZ Jean-Charles]

Hi,

first checks you can do:
- Check the RADOSGW process is running
- Check the output of ceph auth list for typos in permissions for the RADOSGW 
user
- Check you have the keyring file for the user you created on the RADOSGW node
- Check the output of ceph df to verify the RADOSGW was able to create its pools
- Check the execute permission on the FCGI script file
- Check the content of your ceph.conf file on the RADOSGW node and check for 
typos.

Feel free to post the result of those checks (ceph.conf file, ls -l, ceph df 
output, ps -ef | grep radosgw output) remove any key

JC

> On Feb 16, 2016, at 08:08, Alexandr Porunov  
> wrote:
> 
> I have problem with radosgw. I have pass this tutorial but without success: 
> http://docs.ceph.com/docs/hammer/radosgw/config/ 
> 
> 
> When I try:
> curl http://porunov.com 
> 
> I always get the same page:
> ...
> 500 Internal Server Error
> ...
> 
> /var/log/httpd/error.log shows:
> ...
> [Tue Feb 16 17:32:37.413558 2016] [:error] [pid 6377] (13)Permission denied: 
> [client 192.168.56.80:41121 ] FastCGI: failed to 
> connect to server "/var/www/html/s3gw.fcgi": connect() failed
> [Tue Feb 16 17:32:37.413596 2016] [:error] [pid 6377] [client 
> 192.168.56.80:41121 ] FastCGI: incomplete 
> headers (0 bytes) recived from server "/var/www/html/s3gw.fcgi"
> 
> /var/log/httpd/access.log shows:
> ...
> 192.168.56.80 - - [16/Feb/2016:17:32:37 + 0200] "GET / HTTP/1.1" 500 530 "-" 
> "curl/7.29.0"
> 
> I have 6 nodes:
> node1 (ip: 192.168.56.101) - mon, osd
> node2 (ip: 192.168.56.102) - mon, osd
> node3 (ip: 192.168.56.103) - mon, osd
> admin-node (ip: 192.168.56.100)
> ns1 (ip: 192.168.56.50) - dns server (bind 9)
> ceph-rgw (ip: 192.168.56.80) - Ceph Gateway Node
> 
> Dns server have this zone file:
> $TTL 86400
> @IN SOA porunov.com . admin.porunov.com 
> . (
> 2016021000
> 43200
> 3600
> 360
> 2592000 )
> ;
> @IN NS ns1.porunov.com .
> @IN A 192.168.56.80
> *  IN CNAME @
> 
> /var/www/html/s3gw.fcgi contains:
> #!/bin/sh
> exec /usr/bin/radosgw -c /etc/ceph/ceph.conf -n client.radosgw.gateway
> 
> /etc/httpd/conf.d/rgw.conf contains:
> FastCgiExternalServer /var/www/html/s3gw.fcgi -socket 
> /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock
> 
>   ServerName porunov.com 
>   ServerAlias *.porunov.com 
>   ServerAdmin ad...@porunov.com 
>   DocumentRoot /var/www/html
>   RewriteEngine On
>   RewriteRule ^/(.*) /s3gw.fcgi?%{QUERY_STRING} 
> [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
>   
> 
>   Options +ExecCGI
>   AllowOverride All
>   SetHandler fastcgi-script
>   Order allow,deny
>   Allow from all
>   AuthBasicAuthoritative Off
> 
>   
>   AllowEncodedSlashes On
>   ErrorLog /var/log/httpd/error.log
>   CustomLog /var/log/httpd/access.log combined
>   ServerSignature Off
> 
> 
> I use CentOS 7 on all nodes. Also I can not start radosgw with this command:
> systemctl start ceph-radosgw
> because it shows:
> Failed to start ceph-radosgw.service: Unit ceph-radosgw.service failed to 
> load: No such file or directory.
> 
> But this command seems to work:
> systemctl start ceph-radosgw@radosgw.gateway.service
> 
> httpd and ceph-radosgw@radosgw.gateway service is: active (running)
> 
> Please help me to figure out how to repair it.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] add new monitor doesn't update ceph.conf in hammer with ceph-deploy.

[LOPEZ Jean-Charles]

Hi Stefan,

update the ceph.conf file on your ceph-deploy node (~/ceph-deploy/ceph.conf) 
and then push the updated config file to other machines in the cluster as well 
as clients (if your config file is generic between cluster nodes and client 
nodes). If client config file is different you’ll have to perform the update on 
the clients a different way.

Pushing the config from your ceph-deploy machine: ceph-deploy 
--overwrite-config config push node1 node2 node3 ...

JC

> On Oct 20, 2015, at 07:54, Stefan Eriksson  wrote:
> 
> Hi
> 
> I’m using cep-deploy with hammer and recently added a new monitor, I used 
> this: http://docs.ceph.com/docs/hammer/rados/deployment/ceph-deploy-mon/ 
> 
> But it doesn’t say anything about adding conf manually to 
> /etc/ceph/ceph.conf, should we add the new monitor to either of:
> 
> mon_initial_members=
> mon_host=
> 
> my monmap is showing the new monitor and its active if I look at ceph -s, but 
> I just want to know best practice, when using cep-deploy to add a new 
> monitor, should we manually add new entries to /etc/ceph/ceph.conf and push 
> these out to the other monitors through ”ceph config push” ?
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] add new monitor doesn't update ceph.conf in hammer with ceph-deploy.

[LOPEZ Jean-Charles]

And forgot.

Yes, update both lines with the new mon node information
mon_initial_members and mon_host

JC

> On Oct 20, 2015, at 07:54, Stefan Eriksson  wrote:
> 
> Hi
> 
> I’m using cep-deploy with hammer and recently added a new monitor, I used 
> this: http://docs.ceph.com/docs/hammer/rados/deployment/ceph-deploy-mon/ 
> 
> But it doesn’t say anything about adding conf manually to 
> /etc/ceph/ceph.conf, should we add the new monitor to either of:
> 
> mon_initial_members=
> mon_host=
> 
> my monmap is showing the new monitor and its active if I look at ceph -s, but 
> I just want to know best practice, when using cep-deploy to add a new 
> monitor, should we manually add new entries to /etc/ceph/ceph.conf and push 
> these out to the other monitors through ”ceph config push” ?
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cinder + CEPH Storage Full Scenario

[LOPEZ Jean-Charles]

Hi,

when an OSD gets full, any write operation to the entire cluster will be 
disabled.

As a result, creating a single RBD will become impossible and all VMs that need 
to write to one of their Ceph back RBDs will suffer the same pain.

Usually, this ends up as a bad sorry for the VMs.

The best practice is to monitor the disk space usage for the OSDs and as a 
matter of fact RHCS 1.# includes a cep old df command to do this. You can also 
use the output of the cep old report command to grab the appropriate info to 
compute it or rely on external SNMP monitoring tools to grab the usage details 
of the particular OSD disk drives.

Have a great day.
JC

> On Oct 19, 2015, at 02:32, Bharath Krishna  wrote:
> 
> I mean cluster OSDs are physically full.
> 
> I understand its not a pretty way to operate CEPH allowing to become full,
> but I just wanted to know the boundary condition if it becomes full.
> 
> Will cinder create volume operation creates new volume at all or error is
> thrown at Cinder API level itself stating that no space available?
> 
> When IO stalls, will I be able to read the data from CEPH cluster I.e can
> I still read data from existing volumes created from CEPH cluster?
> 
> Thanks for the quick reply.
> 
> Regards
> M Bharath Krishna
> 
> On 10/19/15, 2:51 PM, "Jan Schermer"  wrote:
> 
>> Do you mean when the CEPH cluster (OSDs) is physically full or when the
>> quota is reached?
>> 
>> If CEPH becomes full it just stalls all IO (maybe just write IO, but
>> effectively same thing) - not pretty and you must never ever let it
>> become full.
>> 
>> Jan
>> 
>> 
>>> On 19 Oct 2015, at 11:15, Bharath Krishna 
>>> wrote:
>>> 
>>> Hi
>>> 
>>> What happens when Cinder service with CEPH backend storage cluster
>>> capacity is FULL?
>>> 
>>> What would be the out come of new cinder create volume request?
>>> 
>>> Will volume be created with space not available for use or an error
>>> thrown from Cinder API stating no space available for new volume.
>>> 
>>> I could not try this in my environment and fill up the cluster.
>>> 
>>> Please reply if you have ever tried and tested this.
>>> 
>>> Thank you.
>>> 
>>> Regards,
>>> M Bharath Krishna
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cache-tier problem when cache becomes full

[LOPEZ Jean-Charles]

Hi Xavier

see comments inline

JC

 On 16 Apr 2015, at 23:02, Xavier Serrano xserrano+c...@ac.upc.edu wrote:
 
 Hello all,
 
 We are trying to run some tests on a cache-tier Ceph cluster, but
 we are encountering serious problems, which eventually lead the cluster
 unusable.
 
 We are apparently doing something wrong, but we have no idea of
 what it could be. We'd really appreciate if someone could point us what
 to do.
 
 We are running Ceph version:
 ceph version 0.87.1 (283c2e7cfa2457799f534744d7d549f83ea1335e)
 
 All nodes are Ubuntu 14.04.2 LTS (GNU/Linux 3.16.0-34-generic x86_64)
 
 Our test cluster is:
 * disk-host-1: monitor, with 128 GB RAM
 
 * disk-brick-3, disk-brick-4, disk-brick-5: each node has:
   - 32 GB RAM
   - /dev/sda and /dev/sdb: 2 TB spinning HDDs
   - /dev/sdu: 400 GB SSD
 
 * disk-host-5: client, with 128 GB RAM
 
 Please, find the ceph.conf file and the decompiled CRUSH map at the end
 of this e-mail.
 
 
 This is what we do:
 
 (1) Create a pool named cache_pool:
 sudo ceph osd pool create cache_pool 32 32
 sudo ceph osd pool set cache_pool crush_ruleset 4
 
 (2) Increase PGs of the default rbd pool before putting anything in it:
 sudo ceph osd pool set rbd pg_num 256
 sudo ceph osd pool set rbd pgp_num 256
 
 (3) Create cache-tier having the new cache_pool over rbd:
 sudo ceph osd tier add rbd cache_pool
 sudo ceph osd tier cache-mode cache_pool writeback
 sudo ceph osd tier set-overlay rbd cache_pool
 
 (4) Configure some parameters for cache_pool:
 sudo ceph osd pool set cache_pool hit_set_type bloom
 sudo ceph osd pool set cache_pool hit_set_count 1
 sudo ceph osd pool set cache_pool hit_set_period 300
 sudo ceph osd pool set cache_pool cache_min_flush_age 300
 sudo ceph osd pool set cache_pool cache_min_evict_age 300
I would lower this parameter. If the cache pool really gets full you will not 
be able to evict an object if it’s not more than 5 minute old.
—-
 sudo ceph osd pool set cache_pool target_max_bytes 0
 sudo ceph osd pool set cache_pool target_max_objects 0
—-
One or both above parameters between the dashed lines must be set
 sudo ceph osd pool set cache_pool cache_target_dirty_ratio .4 —— The ratio 
 here are expressed as a proportional value of target_max_bytes and/or 
 target_max_objects
 sudo ceph osd pool set cache_pool cache_target_full_ratio .8  —— The ratio 
 here are expressed as a proportional value of target_max_bytes and/or 
 target_max_objects
—-
 
 (5) Create a 2 TB object to run our tests:
 sudo rbd create fiobench --size 2048000
 
 (6) In the client (disk-host-5), map and mount the object:
 sudo rbd map --image fiobench --- result is /dev/rbd0
 mkfs.xfs /dev/rbd0
 mkdir /mnt/fio
 mount /dev/rbd0 /mnt/fio
 
 (7) Run the fio tests (http://packages.ubuntu.com/trusty/fio)
 in the client. Please, find the fiobench.sh script at the end of
 this e-mail with all the details.
 
 fio creates 64 files of 30 GB each on the /mnt/fio filesystem
 (built on top of a RADOS image) prior to its measurements. Creating the
 files works OK, and the benchmark begins.
 
 After a while, the benchmark becomes stalled. Read and write tests
 were completed, but random read tests just hung. Inspecting the cluster,
 we see that one OSD in the cache_pool has become full, and ceph has
 marked it down.
 
 From that point, it is not possible to resume the benchmark,
 and we are not able to get the cluster healthy (HEALTH_OK) back again.
 
 Any ideas will be very much appreciated.
 
 Thank you very much for your time and your help.
 
 Best regards,
 - Xavier Serrano
 - LCAC, Laboratori de Càlcul
 - Departament d'Arquitectura de Computadors
 - UPC, Universitat Politècnica de Catalunya, BarcelonaTECH
 
 
 The /etc/ceph/ceph.conf file is:
 
 [global]
 fsid = 726babd1-c7df-4fed-8b5f-c5a70d35c4a0
 mon_initial_members = disk-host-1
 mon_host = 192.168.31.65
 auth_cluster_required = cephx
 auth_service_required = cephx
 auth_client_required = cephx
 filestore_xattr_use_omap = true
 public_network = 192.168.31.0/24
 
 
 The CRUSH map looks like this:
 
 # begin crush map
 tunable choose_local_tries 0
 tunable choose_local_fallback_tries 0
 tunable choose_total_tries 50
 tunable chooseleaf_descend_once 1
 
 # devices
 device 0 osd.0
 device 1 osd.1
 device 2 osd.2
 device 3 osd.3
 device 4 osd.4
 device 5 osd.5
 device 6 osd.6
 device 7 osd.7
 device 8 osd.8
 
 # types
 type 0 osd
 type 1 host
 type 2 chassis
 type 3 rack
 type 4 row
 type 5 pdu
 type 6 pod
 type 7 room
 type 8 datacenter
 type 9 region
 type 10 root
 
 # buckets
 host disk-brick-4 {
   id -2   # do not change unnecessarily
   # weight 3.980
   alg straw
   hash 0  # rjenkins1
   item osd.0 weight 1.810
   item osd.3 weight 1.810
   item osd.6 weight 0.360
 }
 host disk-brick-5 {
   id -3   # do not change unnecessarily
   # weight 3.980
   alg straw
   hash 

Re: [ceph-users] switching journal location

[LOPEZ Jean-Charles]

Hi Tom,

you will have to stop the OSD, flush the existing journal to ensure data 
consistency at the OSD level and then switch over to the new journal location 
(initialise journal then start the OSD).

Visit this link for step by step from Sébastien : 
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-August/042085.html 
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-August/042085.html 
An old ML post actually

Cheers
JC

 On 16 Apr 2015, at 15:17, Deneau, Tom tom.den...@amd.com wrote:
 
 If my cluster is quiet and on one node I want to switch the location of the 
 journal from
 the default location to a file on an SSD drive (or vice versa), what is the
 quickest way to do that?  Can I make a soft link to the new location and
 do it without restarting the OSDs?
 
 -- Tom Deneau, AMD
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] deep scrubbing causes osd down

[LOPEZ Jean-Charles]

Hi Andrei,

1) what ceph version are you running?
2) what distro and version are you running?
3) have you checked the disk elevator for the OSD devices to be set to cfq?
4) Have have you considered exploring the following  parameters to further tune
- osd_scrub_chunk_min lower the default value of 5. e.g. = 1
- osd_scrub_chunk_max lower the default value of 25. e.g. = 5
- osd_deep_scrub_stride If you have lowered parameters above, you can play with 
this one to fit best your physical disk behaviour.
- osd_scrub_sleep introduce a half second sleep between 2 scrubs; e.g. = 0.5 to 
start with a half second delay


Cheers
JC


 On 10 Apr 2015, at 12:01, Andrei Mikhailovsky and...@arhont.com wrote:
 
 Hi guys,
 
 I was wondering if anyone noticed that the deep scrubbing process causes some 
 osd to go down?
 
 I have been keeping an eye on a few remaining stability issues in my test 
 cluster. One of the unsolved issues is the occasional reporting of osd(s) 
 going down and coming back up after about 20-30 seconds. This happens to 
 various osds throughout the cluster. I have a small cluster of just 2 osd 
 servers with 9 osds each. 
 
 The common trend that i see week after week is that whenever there is a long 
 deep scrubbing activity on the cluster it triggers one or more osds to go 
 down for a short period of time. After the osd is marked down, it goes back 
 up after about 20 seconds. Obviously there is a repair process that kicks in 
 which causes more load on the cluster. While looking at the logs, i've not 
 seen the osds being marked down when the cluster is not deep scrubbing. It 
 _always_ happens when there is a deep scrub activity. I am seeing the reports 
 of osds going down about 3-4 times a week.
 
 The latest happened just recently with the following log entries:
 
 2015-04-10 19:32:48.330430 mon.0 192.168.168.13:6789/0 3441533 : cluster 
 [INF] pgmap v50849466: 8508 pgs: 8506 active+clean, 2 
 active+clean+scrubbing+deep; 13213 GB data, 26896 GB used, 23310 GB / 50206 
 GB avail; 1005 B/s rd, 1005 
 B/s wr, 0 op/s
 2015-04-10 19:32:52.950633 mon.0 192.168.168.13:6789/0 3441542 : cluster 
 [INF] osd.6 192.168.168.200:6816/3738 failed (5 reports from 5 peers after 
 60.747890 = grace 46.701350)
 2015-04-10 19:32:53.121904 mon.0 192.168.168.13:6789/0 3441544 : cluster 
 [INF] osdmap e74309: 18 osds: 17 up, 18 in
 2015-04-10 19:32:53.231730 mon.0 192.168.168.13:6789/0 3441545 : cluster 
 [INF] pgmap v50849467: 8508 pgs: 599 stale+active+clean, 7907 active+clean, 1 
 stale+active+clean+scrubbing+deep, 1 active+clean+scrubbing+deep; 13213 GB 
 data, 26896 GB used, 23310 GB / 50206 GB avail; 375 B/s rd, 0 op/s
 
 
 osd.6 logs around the same time are:
 
 2015-04-10 19:16:29.110617 7fad6d5ec700  0 log_channel(default) log [INF] : 
 5.3d7 deep-scrub ok
 2015-04-10 19:27:47.561389 7fad6bde9700  0 log_channel(default) log [INF] : 
 5.276 deep-scrub ok
 2015-04-10 19:31:11.611321 7fad6d5ec700  0 log_channel(default) log [INF] : 
 5.287 deep-scrub ok
 2015-04-10 19:31:53.339881 7fad7ce0b700  1 heartbeat_map is_healthy 
 'OSD::osd_op_tp thread 0x7fad735f8700' had timed out after 15
 2015-04-10 19:31:53.339887 7fad7ce0b700  1 heartbeat_map is_healthy 
 'OSD::osd_op_tp thread 0x7fad745fa700' had timed out after 15
 2015-04-10 19:31:53.339890 7fad7ce0b700  1 heartbeat_map is_healthy 
 'OSD::osd_op_tp thread 0x7fad705f2700' had timed out after 15
 2015-04-10 19:31:53.340050 7fad7e60e700  1 heartbeat_map is_healthy 
 'OSD::osd_op_tp thread 0x7fad735f8700' had timed out after 15
 2015-04-10 19:31:53.340053 7fad7e60e700  1 heartbeat_map is_healthy 
 'OSD::osd_op_tp thread 0x7fad745fa700' had timed out after 15
 
 [.] 
 
 2015-04-10 19:32:53.010609 7fad7e60e700  1 heartbeat_map is_healthy 
 'FileStore::op_tp thread 0x7fad86132700' had timed out after 60
 2015-04-10 19:32:53.010611 7fad7e60e700  1 heartbeat_map is_healthy 
 'FileStore::op_tp thread 0x7fad88937700' had timed out after 60
 2015-04-10 19:32:53.111470 7fad66ed2700  0 -- 192.168.168.200:6817/3738  
 192.168.168.201:6837/4409 pipe(0x2b793b80 sd=179 :6817 s=2 pgs=5 cs=1 l=0 
 c=0x21e8b420).fault with nothing to send, going to standby
 2015-04-10 19:32:53.111496 7fad6329d700  0 -- 192.168.168.200:6817/3738  
 192.168.168.201:6827/4208 pipe(0x2b793600 sd=172 :6817 s=2 pgs=7 cs=1 l=0 
 c=0x1791ab00).fault with nothing to send, going to standby
 2015-04-10 19:32:53.111463 7fad55bd0700  0 -- 192.168.168.200:6817/3738  
 192.168.168.201:6822/3910 pipe(0x2cb55dc0 sd=262 :6817 s=2 pgs=8 cs=1 l=0 
 c=0xe7802c0).fault with nothing to send, going to standby
 2015-04-10 19:32:53.121815 7fad6218c700  0 -- 192.168.168.200:6817/3738  
 192.168.168.201:6807/3575 pipe(0x2cf8e080 sd=294 :6817 s=2 pgs=4 cs=1 l=0 
 c=0x138669a0).fault with nothing to send, going to standby
 2015-04-10 19:32:53.121856 7fad67bdf700  0 -- 192.168.168.200:6817/3738  
 192.168.168.201:6842/4442 pipe(0x2b792580 sd=190 :6817 s=2 pgs=9 cs=1 l=0 
 c=0x138922c0).fault with nothing to send, going 

Re: [ceph-users] Radosgw GC parallelization

[LOPEZ Jean-Charles]

Hi,

the following parameters can be used to have more GC processing and more 
efficient
- rgw_gc_max_objs defaults to 32
- rgw_gc_obj_min_wait defaults to 2 * 3600
- rgw_gc_processor_max_time defaults to 3600 
- rgw_gc_processor_period defaults to 3600

It is recommended to set rgw_gc_max_objs with a prime number rather than the 
default value of 32 to have a better dispersion on the objects across the GC 
buckets. The minimum action here is to set this value to either 31 or to a 
higher prime number such as 97 for example.

You can then play with the other parameters so that objects are picked up 
faster by the GC process and let the GC process run for longer period of time 
rather than the default values.

This way you can adjust to your particular environment

JC

 On 8 Apr 2015, at 07:12, c...@jack.fr.eu.org wrote:
 
 Hi,
 
 I have a Ceph cluster, used through radosgw.
 In that cluster, I write files each seconds: input files are known,
 predictible and stable, there is always the same number of new
 fiexd-size files, each second.
 
 Theses files are kept a few days, then remove after a fixed duration.
 And thus, I want to remove the same stable number of files each second
 (I want to remove them faster than store them, or my cluster will grow
 to death)
 
 On a daily-basis, I look at my files, remove the useless ones, and force
 the gc process.
 The first part of the process is quite fast (couple of minutes).
 The last part (gc process) is, however, a bit slow.
 
 Is there a way to speed up the garbage collection ?
 Is the GC dedicated to a radosgw (meaning : if I remove  gc process
 from multiple radosgw, will the process be faster ?)
 From my experiment, multithreading the gc process using some stuff like
 parallel is not efficient (I guess gc processes are locking each others
 or something): is that true ?
 
 Thanks for advises!
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Recovering incomplete PGs with ceph_objectstore_tool

[LOPEZ Jean-Charles]

Hi Chris,

according to your ceph osd tree capture, although the OSD reweight is set to 1, 
the OSD CRUSH weight is set to 0 (2nd column). You need to assign the OSD a 
CRUSH weight so that it can be selected by CRUSH: ceph osd crush reweight 
osd.30 x.y (where 1.0=1TB)

Only when this is done will you see if it joins.

JC

 On 2 Apr 2015, at 19:36, Chris Kitzmiller ckitzmil...@hampshire.edu wrote:
 
 I have a cluster running 0.80.9 on Ubuntu 14.04. A couple nights ago I lost 
 two disks from a pool with size=2. :(
 
 I replaced the two failed OSDs and I now have two PGs which are marked as 
 incomplete in an otherwise healthy cluster. Following this page ( 
 https://ceph.com/community/incomplete-pgs-oh-my/ 
 https://ceph.com/community/incomplete-pgs-oh-my/ ) I was able to set up 
 another node and install Giant 0.87.1, mount one of my failed OSD drives and 
 successfully export the two PGs. I set up another OSD on my new node, 
 weighted it to zero, and imported the two PGs.
 
 I'm still stuck though. It seems as though the new OSD just doesn't want to 
 share with the other OSDs. Is there any way for me to ask an OSD which PGs it 
 has (rather than ask the MON which OSDs a PG is on) to verify that my import 
 was good? Help!
 
 0 and 15 were the OSDs I lost. 30 is the new OSD. I've currently got size = 
 2, min_size = 1.
 
 root@storage1:~# ceph pg dump | grep incomplete | column -t
 dumped all in format plain
 3.102  0  0  0  0  0  0  0  incomplete  2015-04-02  20:49:32.529594  0'0  
 15730:21  [0,15]  0   [0,15]  0   13985'53107  2015-03-29  21:17:15.568125  
 13985'49195  2015-03-24  18:38:08.244769
 3.c7   0  0  0  0  0  0  0  incomplete  2015-04-02  20:49:32.968841  0'0  
 15730:17  [15,0]  15  [15,0]  15  13985'54076  2015-03-31  19:14:22.721695  
 13985'54076  2015-03-31  19:14:22.721695
 
 root@storage1:~# ceph health detail
 HEALTH_WARN 2 pgs incomplete; 2 pgs stuck inactive; 2 pgs stuck unclean; 1 
 requests are blocked  32 sec; 1 osds have slow requests
 pg 3.c7 is stuck inactive since forever, current state incomplete, last 
 acting [15,0]
 pg 3.102 is stuck inactive since forever, current state incomplete, last 
 acting [0,15]
 pg 3.c7 is stuck unclean since forever, current state incomplete, last acting 
 [15,0]
 pg 3.102 is stuck unclean since forever, current state incomplete, last 
 acting [0,15]
 pg 3.102 is incomplete, acting [0,15]
 pg 3.c7 is incomplete, acting [15,0]
 1 ops are blocked  8388.61 sec
 1 ops are blocked  8388.61 sec on osd.15
 1 osds have slow requests
 
 root@storage1:~# ceph osd tree
 # idweight  type name   up/down reweight
 -1  81.65   root default
 -2  81.65   host storage1
 -3  13.63   journal storage1-journal1
 1   2.72osd.1   up  1
 4   2.72osd.4   up  1
 2   2.73osd.2   up  1
 3   2.73osd.3   up  1
 0   2.73osd.0   up  1
 -4  13.61   journal storage1-journal2
 5   2.72osd.5   up  1
 6   2.72osd.6   up  1
 8   2.72osd.8   up  1
 9   2.72osd.9   up  1
 7   2.73osd.7   up  1
 -5  13.6journal storage1-journal3
 11  2.72osd.11  up  1
 12  2.72osd.12  up  1
 13  2.72osd.13  up  1
 14  2.72osd.14  up  1
 10  2.72osd.10  up  1
 -6  13.61   journal storage1-journal4
 16  2.72osd.16  up  1
 17  2.72osd.17  up  1
 18  2.72osd.18  up  1
 19  2.72osd.19  up  1
 15  2.73osd.15  up  1
 -7  13.6journal storage1-journal5
 20  2.72osd.20  up  1
 21  2.72osd.21  up  1
 22  2.72osd.22  up  1
 23  2.72osd.23  up  1
 24  2.72osd.24  up  1
 -8  13.6journal storage1-journal6
 25  2.72osd.25  up  1
 26  2.72osd.26  up  1
 27  2.72osd.27  up  1
 28  2.72osd.28  up  1
 29  2.72osd.29  up  1
 -9  0   host ithome
 30  0   osd.30  up  1
 
 ___
 ceph-users mailing list
 

Re: [ceph-users] Single node cluster

[LOPEZ Jean-Charles]

Hi,

just make sure you modify your CRUSH map so that each copy of the objects are 
just dispatched on different OSDs rather than on different hosts.

Follow these steps:
ceph osd getcrushmap -o /tmp/cm
crushtool -i /tmp/cm -o /tmp/cm.txt

Edit the /tmp/cm.txt file. Locate the crush rule ID 0 at the end of the text 
file and replace chooseleaf firstn 0 type host with chooseleaf firstn 0 type 
osd

crushtool -c /tmp/cm.txt -o /tmp/cm.new
ceph osd setcrushmap -i /tmp/cm.new

And this should do the trick.

Cheers
JC

 On 18 Mar 2015, at 09:57, Khalid Ahsein kahs...@gmail.com wrote:
 
 Hello everybody,
 
 I want to build a new architecture with Ceph for storage backend.
 For the moment I’ve got only one server with this specs : 
 
 1 RAID-1 SSD : OS + OSD journals
 12x 4To : OSD daemons.
 
 I never reached the « clean state » on my cluster and I’m always in 
 HEALTH_WARN mode like this :
   health HEALTH_WARN 25 pgs degraded; 24 pgs incomplete; 24 pgs stuck 
 inactive; 64 pgs stuck unclean; 25 pgs undersized
 
 I tried to add 3 — 12 OSD but it’s always the same problem.
 
 What is the right configuration to have a valid cluster please ?
 
 # cat ceph.conf
 [global]
 fsid = 588595a0-3570-44bb-af77-3c0eaa28fbdb
 mon_initial_members = drt-marco
 mon_host = 172.16.21.4
 auth_cluster_required = cephx
 auth_service_required = cephx
 auth_client_required = cephx
 filestore_xattr_use_omap = true
 public network = 172.16.21.0/24
 
 [osd]
 osd journal size = 1
 osd crush chooseleaf type = 0
 osd pool default size = 1
 
 NB : I use ceph-deploy for debian wheezy to deploy the services.
 
 Thank you so much for your help !
 k.
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] what means active+clean+scrubbing+deep

[LOPEZ Jean-Charles]

Hi Ryan,

it means that the PG is in good health (clean), is available (active) and that 
deep scrubbing is currently being performed  (scrubbing+deep)

JC

 On 13 Mar 2015, at 17:59, ryan_h...@supercluster.cn wrote:
 
 Hi all,
 
 Anyone knows what means 'active+clean+scrubbing+deep' ?
 
 ryan_h...@supercluster.cn
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] what means active+clean+scrubbing+deep

[LOPEZ Jean-Charles]

Hi Ryan,

it means that the OSDs are physically reading the whole content of the PG, 
recalculating the checksum of each object to verify that the content of the PG 
is identical on each OSD protecting the PG. This is to make sure the data that 
was written to the underlying filesystem of each OSD is identical and hasn't 
been altered.

So to make it short the OSDs are checking the coherence of the data stored in 
the PG.

Cheers
JC

 On 13 Mar 2015, at 18:53, ryan_h...@supercluster.cn wrote:
 
 Hi Jean,
 
 Actually, I want to ask you that what means deep scrubbing? What was ceph 
 doing at that time?
 
 ryan_h...@supercluster.cn
  
 From: LOPEZ Jean-Charles mailto:jelo...@redhat.com
 Date: 2015-03-13 15:34
 To: ryan_h...@supercluster.cn mailto:ryan_h...@supercluster.cn
 CC: LOPEZ Jean-Charles mailto:jelo...@redhat.com; ceph-users 
 mailto:ceph-us...@ceph.com
 Subject: Re: [ceph-users] what means active+clean+scrubbing+deep
 Hi Ryan,
  
 it means that the PG is in good health (clean), is available (active) and 
 that deep scrubbing is currently being performed  (scrubbing+deep)
  
 JC
  
  On 13 Mar 2015, at 17:59, ryan_h...@supercluster.cn wrote:
 
  Hi all,
 
  Anyone knows what means 'active+clean+scrubbing+deep' ?
 
  ryan_h...@supercluster.cn
  ___
  ceph-users mailing list
  ceph-users@lists.ceph.com
  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
  

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CephFS: stripe_unit=65536 + object_size=1310720 = pipe.fault, server, going to standby

[LOPEZ Jean-Charles]

Hi Florent

What are the « rules » for stripe_unit  object_size ? - stripe_unit * 
stripe_count = object_size

So in your case set stripe_unit = 2

JC


 On 11 Mar 2015, at 19:59, Florent B flor...@coppint.com wrote:
 
 Hi all,
 
 I'm testing CephFS with Giant and I have a problem when I set these attrs :
 
 setfattr -n ceph.dir.layout.stripe_unit -v 65536 pool_cephfs01/
 setfattr -n ceph.dir.layout.stripe_count -v 1 pool_cephfs01/
 setfattr -n ceph.dir.layout.object_size -v 1310720 pool_cephfs01/
 setfattr -n ceph.dir.layout.pool -v cephfs01 pool_cephfs01/ 
 
 When a client writes files in pool_cephfs01/, It got failed: Transport 
 endpoint is not connected (107) and these errors on MDS :
 
 10.111.0.6:6801/41706  10.111.17.118:0/9384 pipe(0x5e3a580 sd=27 :6801 s=2 
 pgs=2 cs=1 l=0 c=0x6a8d1e0).fault, server, going to standby
 
 When I set stripe_unit=1048576  object_size=1048576, it seems working.
 
 What are the rules for stripe_unit  object_size ?
 
 Thank you.
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph cache tier pool objects not evicted automatically even when reaching full ratio

[LOPEZ Jean-Charles]

Hi,

you need to set the max dirty bytes and/or max dirty objects as these 2 
parameters will default to 0 for your cache pool.

ceph osd pool set cache_pool_name target_max_objects x
ceph osd pool set cache_pool_name target_max_bytes x

The ratios you already set (dirty_ratio = 0.4 and full_ratio = 0.7) will be 
applied based on the above 2 values. The first one reached (objects and/or 
bytes) will trigger flush and eviction

e.g.
You set target_max_bytes = 1
- Flushing will start when your cache pool reaches 4 dirty bytes
- Evicting will start when your cache pool reaches 7 dirty bytes

Cheers
JC



 On 10 Mar 2015, at 22:41, Kamil Kuramshin kamil.kurams...@tatar.ru wrote:
 
 hi, folks! I'm testing cache tier for erasure coded pool and with RBD image 
 on it. And now I'm facing a problem with full cache pool and object are not 
 evicted automatically, Only if I run manually rados -p cache 
 cache-flush-evict-all
 
 client side is:
 superuser@share:~$ uname -a
 Linux share 3.16-2-amd64 #1 SMP Debian 3.16.3-2 (2014-09-20) x86_64 GNU/Linux
 
 
 ceph node all are debian wheezy
 superuser~$ dpkg -l | grep ceph
 ii  ceph   0.87-1~bpo70+1   amd64 
distributed storage and file system
 ii  ceph-common0.87-1~bpo70+1   amd64 
common utilities to mount and interact with a ceph storage cluster
 ii  ceph-fs-common 0.87-1~bpo70+1   amd64 
common utilities to mount and interact with a ceph file system
 ii  ceph-mds   0.87-1~bpo70+1   amd64 
metadata server for the ceph distributed file system
 ii  libcephfs1 0.87-1~bpo70+1   amd64 
Ceph distributed file system client library
 ii  libcurl3-gnutls:amd64  7.29.0-1~bpo70+1.cephamd64 
easy-to-use client-side URL transfer library (GnuTLS flavour)
 ii  python-ceph0.87-1~bpo70+1   amd64 
Python libraries for the Ceph distributed filesystem
 
 
 
 There are all steps to reproduce (excepting creation of pools):
 
 
 
 superuser@admin:~$ ceph osd pool get ec_backup-storage erasure_code_profile
 erasure_code_profile: default
 superuser@admin:~$ ceph osd erasure-code-profile get default
 directory=/usr/lib/ceph/erasure-code
 k=2
 m=1
 plugin=jerasure
 technique=reed_sol_van
 
 
 *** ADMIN NODE OPERATIONS 
  
 superuser@admin:~$ ceph df
 GLOBAL:
 SIZE AVAIL RAW USED %RAW USED
 242T  224T6092G  2.46
 POOLS:
 NAME  ID USED  %USED MAX AVAIL OBJECTS
 ec_backup-storage 4  0 0  147T   0
 cache 5  0 0  185G   0
 block-devices 6  1948G  0.7975638G  498771
 superuser@admin:~$ rados df
 pool name   category KB  objects   clones 
 degraded  unfound   rdrd KB   wrwr KB
 block-devices   - 2042805201   4987710
 0   067127259320535  2070571   2403248346
 cache   -  000
 0   060496247235411   966553499544074
 ec_backup-storage -  000  
   0   0   156988537227276   400355819838985
   total used  6388431372   498771
   total avail   240559782780
   total space   260163775608
  
 * 'cache' pool is replicated pool, 'ec_backup-storage' - Erasure Encoded 
 pool *
 * running simple script for enabling cache tiering:
 
 
 * There is my simple script for enabling cache tier:
  
 superuser@admin:~$ ./enable_cache_tier.sh cache ec_backup-storage
 pool 'cache' is now (or already was) a tier of 'ec_backup-storage'
 set cache-mode for pool 'cache' to writeback
 overlay for 'ec_backup-storage' is now (or already was) 'cache'
 set pool 5 hit_set_type to bloom
 set pool 5 cache_target_dirty_ratio to 0.4
 set pool 5 cache_target_full_ratio to 0.7
 set pool 5 cache_min_flush_age to 10
 set pool 5 cache_min_evict_age to 10
 
 * Displaying some cache pool parameters:
 
 superuser@admin:~$ for param in cache_target_dirty_ratio 
 cache_target_full_ratio target_max_bytes target_max_objects 
 cache_min_flush_age cache_min_evict_age; do  ceph osd pool get cache $param; 
 done
 cache_target_dirty_ratio: 0.4
 cache_target_full_ratio: 0.7
 target_max_bytes: 0
 target_max_objects: 0
 cache_min_flush_age: 10
 cache_min_evict_age: 10
 
 
 *** END ADMIN NODE OPERATIONS 
  
 *** CEPH CLIENT OPERATIONS 
  
 superuser@share:~$ rbd create -p  ec_backup-storage ec_image.img --size 
 50 --image-format 2
 superuser@share:~$ rbd -p 

Re: [ceph-users] Updating monmap

[LOPEZ Jean-Charles]

Hi,

use the following command line: ceph-mon -i {monitor_id} --inject-monmap 
{updated_monmap_file}

JC

 On 18 Feb 2015, at 11:15, SUNDAY A. OLUTAYO olut...@sadeeb.com wrote:
 
 How do I update the ceph monmap after extracting and removing unwanted an ip 
 in the monmap to the clean monmap?
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD capacity variance ?

[LOPEZ Jean-Charles]

Hi Howard,

be default each OSD is weighed based on its capacity automatically. So the 
smaller OSDs will receive less data than the bigger ones.

Be careful though in this case to properly monitor the utilization rate of all 
OSDs in your cluster so that one of them does not reach the odd_full ratio

Read this link that will help you get e better view on Ceph data placement 
mechanisms.

Cheers
JC


 On Jan 31, 2015, at 14:39, Howard Thomson h...@thomsons.co.uk wrote:
 
 Hi All,
 
 I am developing a custom disk storage backend for the Bacula backup
 system, and am in the process of setting up a trial Ceph system,
 intending to use a direct interface to RADOS.
 
 I have a variety of 1Tb, 250Mb and 160Mb disk drives that I would like
 to use, but it is not [as yet] obvious as to whether having differences
 in capacity at different OSDs matters.
 
 Can anyone comment, or point me in the right direction on
 docs.ceph.com ?
 
 Thanks,
 
 Howard
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Cache Settings

[LOPEZ Jean-Charles]

Hi Nick

it is correct that the ratios are relative to the size directives, 
target_max_bytes and target_max_objects which ever is crossed first in case 
they are both set. Those parameters are cache pool specific so you can create 
multiple cache pools, all using the same OSDs (same CRUSH rule assignment), but 
using different settings for size, flushing and eviction. In this case, all 
cache pool PGs will be hosted by the same OSDs hence will “compete” for space. 
Therefore, the sum of each max size directive for all your cache pools should 
not exceed that capacity of the OSDs hosting them.

See comments inline

Cheers
JC


 On Feb 7, 2015, at 12:23, Nick Fisk n...@fisk.me.uk wrote:
 
 Hi All,
 
 Time for a little Saturday evening Ceph related quiz.
 
 From this documentation page
 
 http://ceph.com/docs/master/rados/operations/cache-tiering/
 
 It seems to indicate that you can either flush/evict using relative sizing
 (cache_target_dirty_ratio)  or absolute sizing (target_max_bytes). But the
 two are separate methods and are mutually exclusive. Ie flush at 80% or when
 you hit the number of bytes specified
 
 The same goes for the max_age parameters, ie it will flush all objects older
 than 300 seconds no matter how full the pool is.
 
 However this documentation page
 
 https://ceph.com/docs/master/dev/cache-pool/
 
 Seems to indicate that the target_max_bytes is actually the number of bytes
 that the cache_target_dirty_ratio uses to calculate the size of the cache
 pool it has to work with. And that the max_age parameters just make sure
 objects aren't evicted too quickly.
 
 1. Maybe, I'm reading it wrong but they appear conflicting to me, Which is
 correct?
 
 The following questions may be invalid depending on the answer to #1
 
 2. Assuming link #1 is correct, is it possible to have multiple cache pools
 on group of SSD's and how does Ceph work out capacity for each pool?
 3. Assuming #2 is correct, I can also specify min_age variables without
 overriding the target_max_bytes and cache_target_dirty_ratio?
 
 
 So assuming link #2 is correct (which makes more sense to me), if I had the
 following configuration
 
 target_max_bytes = 10,000,000,000
 cache_target_dirty_ratio = .4
 cache_full_dirty_ratio = .8
 cache_min_flush_age = 60
 cache_min_evict_age = 120
 
 Then are the following assumptions true:-
 
 1. I have a cache pool that is 10G total in size, regardless of the actual
 size of the pool

Yes and remember that the pool does not have a size expressed, only a number of 
PGs protected by the OSDs chosen according to your CRUSH rule

 2. When the pool has 4G of dirty bytes in it, it will start trying to flush
 them as long as they are older than 60 seconds

Yes flushing will kick in at 40% usage of your max value

 3. When the pool is 8G full it will start evicting all objects that are
 older than 120 seconds in a LRU order

Yes eviction will kick in at 80% usage of your max value

 4. If I manage to fill the pool up to 10G, Ceph will block until free space
 becomes available from evictions

Yes

 5. If I had a 100G worth of SSD capacity after replication, I could have 10
 of these cache pools (disregard performance concerns)

Yes

 
 Many Thanks for any answers,
 Nick
 
 
 
 
 
 
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com