[ceph-users] radosgw crash - Infernalis

[Ben Hines]

Is this a known one? Ceph 9.2.1. Can provide more logs if needed.

2> 2016-04-26 22:07:59.662702 7f49aeffd700  1 == req done
req=0x7f49c4138be0 http_status=200 ==
   -11> 2016-04-26 22:07:59.662752 7f49aeffd700  1 civetweb:
0x7f49c4001280: 10.30.1.221 - - [26/Apr/2016:22:07:59 -0700] "HEAD
/HTTP/1.1" 200 0 - Boto/2.32.1 Python/2.7.8 Windows/2008ServerR2
   -10> 2016-04-26 22:07:59.672109 7f49aeffd700  1 == starting new
request req=0x7f49c4148d00 =
-9> 2016-04-26 22:07:59.672131 7f49aeffd700  2 req 212:0.22::GET
::initializing for trans_id =
tx000d4-005720492f-2ac65f1-default
-8> 2016-04-26 22:07:59.672137 7f49aeffd700 10 host=sm-cephrgw3.scea.com
-7> 2016-04-26 22:07:59.672159 7f49aeffd700 10 s->object=
s->bucket=
-6> 2016-04-26 22:07:59.672165 7f49aeffd700  2 req 212:0.56:s3:GET
::getting op
-5> 2016-04-26 22:07:59.672169 7f49aeffd700  2 req 212:0.61:s3:GET
:authorizing
-4> 2016-04-26 22:07:59.672199 7f49aeffd700 10 get_canon_resource():
dest=
-3> 2016-04-26 22:07:59.672203 7f49aeffd700 10 auth_hdr:
GET


Wed, 27 Apr 2016 05:07:59 GMT

-2> 2016-04-26 22:07:59.672240 7f49aeffd700  2 req 212:0.000131:s3:GET
:reading permissions
-1> 2016-04-26 22:07:59.672338 7f49aeffd700  1 -- 10.30.1.8:0/4080085251
--> 10.30.1.6:6800/10350 -- osd_op(client.44852756.0:79
default.42048218. [getxattrs,stat,read 0~524288] 12.aa730416
ack+read+known_if_redirected e100207) v6 -- ?+0 0x7f49c41880b0 con
0x7f49c4145eb0
 0> 2016-04-26 22:07:59.685615 7f49a07f0700 -1 *** Caught signal
(Segmentation fault) **
 in thread 7f49a07f0700

 ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd)
 1: (()+0x30b0a2) [0x7f4c4907f0a2]
 2: (()+0xf100) [0x7f4c44f7a100]
 NOTE: a copy of the executable, or `objdump -rdS ` is needed
to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   0/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 keyvaluestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 5 heartbeatmap
   1/ 5 perfcounter
  10/10 rgw
  10/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 newstore
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)

  max_recent 1
  max_new 1000
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] how ceph mon works

[Christian Balzer]

On Tue, 26 Apr 2016 23:31:39 +0200 (CEST) Wido den Hollander wrote:

> 
> > Op 26 april 2016 om 23:24 schreef yang sheng :
> > 
> > 
> > Hi
> > 
> > according to ceph docs, it recommend 3 monitors as least. All the
> > clients will contact monitors first to get the ceph map and connect
> > the osd.
> > 
> 
> Yes, indeed.
> 
> > I am curious that if I have 3 monitors, are these monitors run in
> > master-master mode or master-slave mode?  In another word, will clients
> > talk to any of these 3 monitors or all clients can only talk to one
> > monitor and if that monitor is unreachable, than they talk to the
> > second monitor?
> > 
> 
> During the eleOn 04/26/2016 04:59 PM, M.Z.Medina wrote:
> anti mosquito ction process one monitor will become the 'leader' and
> others will become 'peons'.
> 
This is also by default the monitor with the lowest IP address, so until
one day that may become configurable, keep this in mind and put dedicated
MONs with faster HW at the low IP address(es).

> A client can talk to each monitor, but the peons will forward commands
> to the leader which makes the decisions inside the cluster.
> 
AFAIK this only goes for changes (i.e. adding an OSD, changing weights),
getting information will be handled by the MON in question with no need
for it to contact the leader.

> Clients are aware of all the Monitors in a cluster, so if a monitor goes
> down they can contact the other one.
> 
While true, clients basically contact MONs in a round-robin fashion
(randomized or not, I don't know). 

While this works, it of course introduces delays when they try to reach a
dead MON.
I've always felt that something more elegant would be nice here, as in not
sending any real requests to a MON that hasn't been responsive and retry
that MON with a dummy request every minute or so and if successful add it
back to the to "good" pool.

Regards,

Christian
> Wido
> 
> > 
> > thanks
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 


-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Any Docs to configure NFS to access RADOSGW buckets on Jewel

[WD_Hwang]

Hello:
  Are there any documents or examples to explain the configuration of NFS to 
access RADOSGW buckets on Jewel?
Thanks a lot.

Best Regards,
WD

---
This email contains confidential or legally privileged information and is for 
the sole use of its intended recipient. 
Any unauthorized review, use, copying or distribution of this email or the 
content of this email is strictly prohibited.
If you are not the intended recipient, you may reply to the sender and should 
delete this e-mail immediately.
---
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Any docs for replication in Jewel radosgw?

[Richard Chan]

Hi Cephers

I'm interested in the new features of active active and bidirectional
replication.

https://github.com/ceph/ceph/blob/master/doc/radosgw/multisite.rst

has setup information but nothing on replication.

Thanks.

-- 
Richard Chan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Hammer broke after adding 3rd osd server

[Alwin Antreich]

Hi Andrei,

are you using Jumbo Frames? My experience, I had a driver issues where
one NIC wouldn't accept the MTU set for the interface and the cluster
ran into a very similar behavior as you are describing. After I have set
the MTU for all NICs and servers to the working value of my troubling
NIC, everything went back to normal.

Regards,
Alwin

On 04/26/2016 05:18 PM, Wido den Hollander wrote:
> 
>> Op 26 april 2016 om 22:31 schreef Andrei Mikhailovsky :
>>
>>
>> Hi Wido,
>>
>> Thanks for your reply. We have a very simple ceph network. A single 40gbit/s 
>> infiniband switch where the osd servers and hosts are connected to. There 
>> are no default gates on the storage network. The IB is used only for ceph; 
>> everything else goes over the ethernet. 
>>
>> I've checked the stats on the IB interfaces of osd servers and there are no 
>> errors. The ipoib interface has a very small number of dropped packets 
>> (0.0003%).
>>
>> What kind of network tests would you suggest that I run? What do you mean by 
>> "I would suggest that you check if the network towards clients is also OK."? 
>> By clients do you mean the host servers?
>>
> 
> With clients I mean you verify if the hosts talking to the Ceph cluster can 
> reach each machine running OSDs.
> 
> In my case there was packet loss from certain clients which caused the issues 
> to occur.
> 
> Wido
> 
>> Many thanks
>>
>> Andrei
>>
>> - Original Message -
>>> From: "Wido den Hollander" 
>>> To: "ceph-users" , "Andrei Mikhailovsky" 
>>> 
>>> Sent: Tuesday, 26 April, 2016 21:17:59
>>> Subject: Re: [ceph-users] Hammer broke after adding 3rd osd server
>>
 Op 26 april 2016 om 17:52 schreef Andrei Mikhailovsky :


 Hello everyone,

 I've recently performed a hardware upgrade on our small two osd server ceph
 cluster, which seems to have broke the ceph cluster. We are using ceph for
 cloudstack rbd images for vms.All of our servers are Ubuntu 14.04 LTS with
 latest updates and kernel 4.4.6 from ubuntu repo.

 Previous hardware:

 2 x osd servers with 9 sas osds, 32gb ram and 12 core Intel cpu 2620 @ 
 2Ghz each
 and 2 consumer SSDs for journal. Infiniband 40gbit/s networking using 
 IPoIB.

 The following things were upgraded:

 1. journal ssds were upgraded from consumer ssd to Intel 3710 200gb. We 
 now have
 5 osds per single ssd.
 2. added additional osd server with 64gb ram, 10 osds, Intel 2670 cpu @ 
 2.6Ghz
 3. Upgraded ram on osd servers to become 64gb
 4. Installed additional osd disk to have 10 osds per server.

 After adding the third osd server and finishing the initial sync, the 
 cluster
 worked okay for 1-2 days. No issues were noticed. On a third day my 
 monitoring
 system started reporting a bunch of issues from the ceph cluster as well as
 from our virtual machines. This tend to happen between 7:20am and 7:40am 
 and
 lasts for about 2-3 hours before things become normal again. I've checked 
 the
 osd servers and there is nothing that I could find in cron or otherwise 
 that
 starts around 7:20am.

 The problem is as follows: the new osd server's load goes to 400+ with 
 ceph-osd
 processes consuming all cpu resources. The ceph -w shows a high number of 
 slow
 requests which relate to osds belonging to the new osd server. The log 
 files
 show the following:

 2016-04-20 07:39:04.346459 osd.7 192.168.168.200:6813/2650 2 : cluster 
 [WRN]
 slow request 30.032033 seconds old, received at 2016-04-20 07:38:34.314014:
 osd_op(client.140476549.0:13203438 
 rbd_data.2c9de71520eedd1.0621
 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 
 2572288~4096]
 5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) currently waiting 
 for
 subops from 22
 2016-04-20 07:39:04.346465 osd.7 192.168.168.200:6813/2650 3 : cluster 
 [WRN]
 slow request 30.031878 seconds old, received at 2016-04-20 07:38:34.314169:
 osd_op(client.140476549.0:13203439 
 rbd_data.2c9de71520eedd1.0621
 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 
 1101824~8192]
 5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) currently waiting 
 for
 rw locks



 There are practically every osd involved in the slow requests and they 
 tend to
 be between the old two osd servers and the new one. There were no issues 
 as far
 as I can see between the old two servers.

 The first thing i've checked is the networking. No issue was identified 
 from
 running ping -i .1  as well as using hping3 for the tcp 
 connection
 checks. The network tests were running for over a week and not a single 
 packet
 was lost. The slow requests took place while the network tests were 
 running.

 I've 

Re: [ceph-users] how ceph mon works

[Wido den Hollander]

> Op 26 april 2016 om 23:24 schreef yang sheng :
> 
> 
> Hi
> 
> according to ceph docs, it recommend 3 monitors as least. All the clients
> will contact monitors first to get the ceph map and connect the osd.
> 

Yes, indeed.

> I am curious that if I have 3 monitors, are these monitors run in
> master-master mode or master-slave mode?  In another word, will clients
> talk to any of these 3 monitors or all clients can only talk to one monitor
> and if that monitor is unreachable, than they talk to the second monitor?
> 

During the election process one monitor will become the 'leader' and others 
will become 'peons'.

A client can talk to each monitor, but the peons will forward commands to the 
leader which makes the decisions inside the cluster.

Clients are aware of all the Monitors in a cluster, so if a monitor goes down 
they can contact the other one.

Wido

> 
> thanks
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] how ceph mon works

[yang sheng]

Hi

according to ceph docs, it recommend 3 monitors as least. All the clients
will contact monitors first to get the ceph map and connect the osd.

I am curious that if I have 3 monitors, are these monitors run in
master-master mode or master-slave mode?  In another word, will clients
talk to any of these 3 monitors or all clients can only talk to one monitor
and if that monitor is unreachable, than they talk to the second monitor?


thanks
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Hammer broke after adding 3rd osd server

[Wido den Hollander]

> Op 26 april 2016 om 22:31 schreef Andrei Mikhailovsky :
> 
> 
> Hi Wido,
> 
> Thanks for your reply. We have a very simple ceph network. A single 40gbit/s 
> infiniband switch where the osd servers and hosts are connected to. There are 
> no default gates on the storage network. The IB is used only for ceph; 
> everything else goes over the ethernet. 
> 
> I've checked the stats on the IB interfaces of osd servers and there are no 
> errors. The ipoib interface has a very small number of dropped packets 
> (0.0003%).
> 
> What kind of network tests would you suggest that I run? What do you mean by 
> "I would suggest that you check if the network towards clients is also OK."? 
> By clients do you mean the host servers?
> 

With clients I mean you verify if the hosts talking to the Ceph cluster can 
reach each machine running OSDs.

In my case there was packet loss from certain clients which caused the issues 
to occur.

Wido

> Many thanks
> 
> Andrei
> 
> - Original Message -
> > From: "Wido den Hollander" 
> > To: "ceph-users" , "Andrei Mikhailovsky" 
> > 
> > Sent: Tuesday, 26 April, 2016 21:17:59
> > Subject: Re: [ceph-users] Hammer broke after adding 3rd osd server
> 
> >> Op 26 april 2016 om 17:52 schreef Andrei Mikhailovsky :
> >> 
> >> 
> >> Hello everyone,
> >> 
> >> I've recently performed a hardware upgrade on our small two osd server ceph
> >> cluster, which seems to have broke the ceph cluster. We are using ceph for
> >> cloudstack rbd images for vms.All of our servers are Ubuntu 14.04 LTS with
> >> latest updates and kernel 4.4.6 from ubuntu repo.
> >> 
> >> Previous hardware:
> >> 
> >> 2 x osd servers with 9 sas osds, 32gb ram and 12 core Intel cpu 2620 @ 
> >> 2Ghz each
> >> and 2 consumer SSDs for journal. Infiniband 40gbit/s networking using 
> >> IPoIB.
> >> 
> >> The following things were upgraded:
> >> 
> >> 1. journal ssds were upgraded from consumer ssd to Intel 3710 200gb. We 
> >> now have
> >> 5 osds per single ssd.
> >> 2. added additional osd server with 64gb ram, 10 osds, Intel 2670 cpu @ 
> >> 2.6Ghz
> >> 3. Upgraded ram on osd servers to become 64gb
> >> 4. Installed additional osd disk to have 10 osds per server.
> >> 
> >> After adding the third osd server and finishing the initial sync, the 
> >> cluster
> >> worked okay for 1-2 days. No issues were noticed. On a third day my 
> >> monitoring
> >> system started reporting a bunch of issues from the ceph cluster as well as
> >> from our virtual machines. This tend to happen between 7:20am and 7:40am 
> >> and
> >> lasts for about 2-3 hours before things become normal again. I've checked 
> >> the
> >> osd servers and there is nothing that I could find in cron or otherwise 
> >> that
> >> starts around 7:20am.
> >> 
> >> The problem is as follows: the new osd server's load goes to 400+ with 
> >> ceph-osd
> >> processes consuming all cpu resources. The ceph -w shows a high number of 
> >> slow
> >> requests which relate to osds belonging to the new osd server. The log 
> >> files
> >> show the following:
> >> 
> >> 2016-04-20 07:39:04.346459 osd.7 192.168.168.200:6813/2650 2 : cluster 
> >> [WRN]
> >> slow request 30.032033 seconds old, received at 2016-04-20 07:38:34.314014:
> >> osd_op(client.140476549.0:13203438 
> >> rbd_data.2c9de71520eedd1.0621
> >> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 
> >> 2572288~4096]
> >> 5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) currently waiting 
> >> for
> >> subops from 22
> >> 2016-04-20 07:39:04.346465 osd.7 192.168.168.200:6813/2650 3 : cluster 
> >> [WRN]
> >> slow request 30.031878 seconds old, received at 2016-04-20 07:38:34.314169:
> >> osd_op(client.140476549.0:13203439 
> >> rbd_data.2c9de71520eedd1.0621
> >> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 
> >> 1101824~8192]
> >> 5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) currently waiting 
> >> for
> >> rw locks
> >> 
> >> 
> >> 
> >> There are practically every osd involved in the slow requests and they 
> >> tend to
> >> be between the old two osd servers and the new one. There were no issues 
> >> as far
> >> as I can see between the old two servers.
> >> 
> >> The first thing i've checked is the networking. No issue was identified 
> >> from
> >> running ping -i .1  as well as using hping3 for the tcp 
> >> connection
> >> checks. The network tests were running for over a week and not a single 
> >> packet
> >> was lost. The slow requests took place while the network tests were 
> >> running.
> >> 
> >> I've also checked the osd and ssd disks and I was not able to identify 
> >> anything
> >> problematic.
> >> 
> >> Stopping all osds on the new server causes no issues between the old two 
> >> osd
> >> servers. I've left the new server disconnected for a few days and had no 
> >> issues
> >> with the cluster.
> >> 
> >> I am a bit lost on what else to try and how to debug the issue. Could 
> >> someone
> >> please help m

Re: [ceph-users] CEPH All OSD got segmentation fault after CRUSH edit

[Wido den Hollander]

> Op 26 april 2016 om 19:39 schreef Samuel Just :
> 
> 
> I think?  Probably worth reproducing on a vstart cluster to validate
> the fix.  Didn't we introduce something in the mon to validate new
> crushmaps?  Hammer maybe?

I ended up injecting a fixed CRUSHMap into osdmap 1432 and 1433 on this cluster.

For future reference:

$ for i in {1392..1450}; do find -name "osdmap*${i}*" -exec osdmaptool 
--export-crush /tmp/crush.${i} {} \;; crushtool -d /tmp/crush.${i} -o 
/tmp/crush.${i}.txt; done

That is how I extracted the CRUSHMap out of the OSDMap. Tracing the logs I 
found that it went wrong somewhere around OSDMap 1430. I inspected those maps 
manually and traced the root-cause to 1432 and 1433.

I fixed the CRUSHmap and compiled it again:

$ crushtool -c /tmp/crush.1432.txt -o /tmp/crush.1432.new

Afterwards I injected it again

$ find /var/lib/ceph/osd/ceph-*/current/meta -name 'osdmap.1432*'|xargs -n 1 
osdmaptool --import-crush /tmp/crush.1432.new

The OSDs now start and keep running. The cluster isn't stable yet, but I at 
least have the OSDs back up and running.

Wido

> -Sam
> 
> On Tue, Apr 26, 2016 at 8:09 AM, Wido den Hollander  wrote:
> >
> >> Op 26 april 2016 om 16:58 schreef Samuel Just :
> >>
> >>
> >> Can you attach the OSDMap (ceph osd getmap -o )?
> >> -Sam
> >>
> >
> > Henrik contacted me to look at this and this is what I found:
> >
> > 0x00b18b81 in crush_choose_firstn (map=map@entry=0x1f00200, 
> > bucket=0x0, weight=weight@entry=0x1f2b880, weight_max=weight_max@entry=30, 
> > x=x@entry=1731224833, numrep=2, type=1, out=0x7fffdc036508, outpos=0, 
> > out_size=2, tries=51, recurse_tries=1, local_retries=0,
> > local_fallback_retries=0, recurse_to_leaf=1, vary_r=0, 
> > out2=0x7fffdc036510, parent_r=0) at crush/mapper.c:345
> > 345 crush/mapper.c: No such file or directory.
> >
> > A bit more output from GDB:
> >
> > #0  0x00b18b81 in crush_choose_firstn (map=map@entry=0x1f00200, 
> > bucket=0x0, weight=weight@entry=0x1f2b880, weight_max=weight_max@entry=30, 
> > x=x@entry=1731224833, numrep=2, type=1, out=0x7fffdc036508, outpos=0, 
> > out_size=2, tries=51, recurse_tries=1, local_retries=0,
> > local_fallback_retries=0, recurse_to_leaf=1, vary_r=0, 
> > out2=0x7fffdc036510, parent_r=0) at crush/mapper.c:345
> > #1  0x00b194cb in crush_do_rule (map=0x1f00200, ruleno= > out>, x=1731224833, result=0x7fffdc036520, result_max=, 
> > weight=0x1f2b880, weight_max=30, scratch=) at 
> > crush/mapper.c:794
> > #2  0x00a61680 in do_rule (weight=std::vector of length 30, 
> > capacity 30 = {...}, maxout=2, out=std::vector of length 0, capacity 0, 
> > x=1731224833, rule=0, this=0x1f72340) at ./crush/CrushWrapper.h:939
> > #3  OSDMap::_pg_to_osds (this=this@entry=0x1f46800, pool=..., pg=..., 
> > osds=osds@entry=0x7fffdc036600, primary=primary@entry=0x7fffdc0365ec, 
> > ppps=0x7fffdc0365f4) at osd/OSDMap.cc:1417
> >
> > It seems that CRUSH can't find entries in the CRUSHMap. In this case the 
> > 'root default' was removed while the default ruleset still refers to it.
> >
> > The cluster is running 0.80.11
> >
> > I extracted the CRUSHMaps from the OSDMaps on osd.0:
> >
> > $ for i in {1392..1450}; do find -name "osdmap*${i}*" -exec osdmaptool 
> > --export-crush /tmp/crush.${i} {} \;; crushtool -d /tmp/crush.${i} -o 
> > /tmp/crush.${i}.txt; done
> >
> > Here I see that in map 1433 the root 'default' doesn't exist, but the crush 
> > ruleset refers to 'bucket0'. This crushmap is attached.
> >
> > rule replicated_ruleset {
> > ruleset 0
> > type replicated
> > min_size 1
> > max_size 10
> > step take bucket0
> > step chooseleaf firstn 0 type host
> > step emit
> > }
> >
> > The root bucket0 doesn't exist.
> >
> > bucket0 seems like something which was created by Ceph/CRUSH and not by the 
> > user.
> >
> > I'm thinking about injecting a fixed CRUSHMap into this OSDMap where 
> > bucket0 does exist. Does that seem like a sane thing to do?
> >
> > Wido
> >
> >
> >> On Tue, Apr 26, 2016 at 2:07 AM, Henrik Svensson 
> >>  >> > wrote:
> >>
> >> > Hi!
> >> >
> >> > We got a three node CEPH cluster with 10 OSD each.
> >> >
> >> > We bought 3 new machines with additional 30 disks that should reside in
> >> > another location.
> >> > Before adding these machines we modified the default CRUSH table.
> >> >
> >> > After modifying the (default) crush table with these commands the cluster
> >> > went down:
> >> >
> >> > 
> >> > ceph osd crush add-bucket dc1 datacenter
> >> > ceph osd crush add-bucket dc2 datacenter
> >> > ceph osd crush add-bucket availo datacenter
> >> > ceph osd crush move dc1 root=default
> >> > ceph osd crush move lkpsx0120 root=default datacenter=dc1
> >> > ceph osd crush move lkpsx0130 root=default datacenter=dc1
> >> > ceph osd crush move lkpsx0140 root=default datacenter=dc1
> >> > ceph osd crush move dc2 root=default
> >> > ceph osd crush move availo root=default
> >> 

Re: [ceph-users] Hammer broke after adding 3rd osd server

[Andrei Mikhailovsky]

Hi Wido,

Thanks for your reply. We have a very simple ceph network. A single 40gbit/s 
infiniband switch where the osd servers and hosts are connected to. There are 
no default gates on the storage network. The IB is used only for ceph; 
everything else goes over the ethernet. 

I've checked the stats on the IB interfaces of osd servers and there are no 
errors. The ipoib interface has a very small number of dropped packets 
(0.0003%).

What kind of network tests would you suggest that I run? What do you mean by "I 
would suggest that you check if the network towards clients is also OK."? By 
clients do you mean the host servers?

Many thanks

Andrei

- Original Message -
> From: "Wido den Hollander" 
> To: "ceph-users" , "Andrei Mikhailovsky" 
> 
> Sent: Tuesday, 26 April, 2016 21:17:59
> Subject: Re: [ceph-users] Hammer broke after adding 3rd osd server

>> Op 26 april 2016 om 17:52 schreef Andrei Mikhailovsky :
>> 
>> 
>> Hello everyone,
>> 
>> I've recently performed a hardware upgrade on our small two osd server ceph
>> cluster, which seems to have broke the ceph cluster. We are using ceph for
>> cloudstack rbd images for vms.All of our servers are Ubuntu 14.04 LTS with
>> latest updates and kernel 4.4.6 from ubuntu repo.
>> 
>> Previous hardware:
>> 
>> 2 x osd servers with 9 sas osds, 32gb ram and 12 core Intel cpu 2620 @ 2Ghz 
>> each
>> and 2 consumer SSDs for journal. Infiniband 40gbit/s networking using IPoIB.
>> 
>> The following things were upgraded:
>> 
>> 1. journal ssds were upgraded from consumer ssd to Intel 3710 200gb. We now 
>> have
>> 5 osds per single ssd.
>> 2. added additional osd server with 64gb ram, 10 osds, Intel 2670 cpu @ 
>> 2.6Ghz
>> 3. Upgraded ram on osd servers to become 64gb
>> 4. Installed additional osd disk to have 10 osds per server.
>> 
>> After adding the third osd server and finishing the initial sync, the cluster
>> worked okay for 1-2 days. No issues were noticed. On a third day my 
>> monitoring
>> system started reporting a bunch of issues from the ceph cluster as well as
>> from our virtual machines. This tend to happen between 7:20am and 7:40am and
>> lasts for about 2-3 hours before things become normal again. I've checked the
>> osd servers and there is nothing that I could find in cron or otherwise that
>> starts around 7:20am.
>> 
>> The problem is as follows: the new osd server's load goes to 400+ with 
>> ceph-osd
>> processes consuming all cpu resources. The ceph -w shows a high number of 
>> slow
>> requests which relate to osds belonging to the new osd server. The log files
>> show the following:
>> 
>> 2016-04-20 07:39:04.346459 osd.7 192.168.168.200:6813/2650 2 : cluster [WRN]
>> slow request 30.032033 seconds old, received at 2016-04-20 07:38:34.314014:
>> osd_op(client.140476549.0:13203438 rbd_data.2c9de71520eedd1.0621
>> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 
>> 2572288~4096]
>> 5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) currently waiting for
>> subops from 22
>> 2016-04-20 07:39:04.346465 osd.7 192.168.168.200:6813/2650 3 : cluster [WRN]
>> slow request 30.031878 seconds old, received at 2016-04-20 07:38:34.314169:
>> osd_op(client.140476549.0:13203439 rbd_data.2c9de71520eedd1.0621
>> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 
>> 1101824~8192]
>> 5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) currently waiting for
>> rw locks
>> 
>> 
>> 
>> There are practically every osd involved in the slow requests and they tend 
>> to
>> be between the old two osd servers and the new one. There were no issues as 
>> far
>> as I can see between the old two servers.
>> 
>> The first thing i've checked is the networking. No issue was identified from
>> running ping -i .1  as well as using hping3 for the tcp 
>> connection
>> checks. The network tests were running for over a week and not a single 
>> packet
>> was lost. The slow requests took place while the network tests were running.
>> 
>> I've also checked the osd and ssd disks and I was not able to identify 
>> anything
>> problematic.
>> 
>> Stopping all osds on the new server causes no issues between the old two osd
>> servers. I've left the new server disconnected for a few days and had no 
>> issues
>> with the cluster.
>> 
>> I am a bit lost on what else to try and how to debug the issue. Could someone
>> please help me?
>> 
> 
> I would still say this is a network issue.
> 
> "currently waiting for rw locks" is usually a network problem.
> 
> I found this out myself a few weeks ago:
> http://blog.widodh.nl/2016/01/slow-requests-with-ceph-waiting-for-rw-locks/
> 
> The problem there was a wrong gateway on some machines.
> 
> In that situation the OSDs could talk just fine, but they had problems with
> sending traffic back to the clients which lead to buffers filling up.
> 
> I would suggest that you check if the network towards clients is also OK.
> 
> Wido
> 
>> Many thanks
>> 
>> Andrei
>> 
>

Re: [ceph-users] Hammer broke after adding 3rd osd server

[Wido den Hollander]

> Op 26 april 2016 om 17:52 schreef Andrei Mikhailovsky :
> 
> 
> Hello everyone, 
> 
> I've recently performed a hardware upgrade on our small two osd server ceph 
> cluster, which seems to have broke the ceph cluster. We are using ceph for 
> cloudstack rbd images for vms.All of our servers are Ubuntu 14.04 LTS with 
> latest updates and kernel 4.4.6 from ubuntu repo. 
> 
> Previous hardware: 
> 
> 2 x osd servers with 9 sas osds, 32gb ram and 12 core Intel cpu 2620 @ 2Ghz 
> each and 2 consumer SSDs for journal. Infiniband 40gbit/s networking using 
> IPoIB. 
> 
> The following things were upgraded: 
> 
> 1. journal ssds were upgraded from consumer ssd to Intel 3710 200gb. We now 
> have 5 osds per single ssd. 
> 2. added additional osd server with 64gb ram, 10 osds, Intel 2670 cpu @ 
> 2.6Ghz 
> 3. Upgraded ram on osd servers to become 64gb 
> 4. Installed additional osd disk to have 10 osds per server. 
> 
> After adding the third osd server and finishing the initial sync, the cluster 
> worked okay for 1-2 days. No issues were noticed. On a third day my 
> monitoring system started reporting a bunch of issues from the ceph cluster 
> as well as from our virtual machines. This tend to happen between 7:20am and 
> 7:40am and lasts for about 2-3 hours before things become normal again. I've 
> checked the osd servers and there is nothing that I could find in cron or 
> otherwise that starts around 7:20am. 
> 
> The problem is as follows: the new osd server's load goes to 400+ with 
> ceph-osd processes consuming all cpu resources. The ceph -w shows a high 
> number of slow requests which relate to osds belonging to the new osd server. 
> The log files show the following: 
> 
> 2016-04-20 07:39:04.346459 osd.7 192.168.168.200:6813/2650 2 : cluster [WRN] 
> slow request 30.032033 seconds old, received at 2016-04-20 07:38:34.314014: 
> osd_op(client.140476549.0:13203438 rbd_data.2c9de71520eedd1.0621 
> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 
> 2572288~4096] 5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) 
> currently waiting for subops from 22 
> 2016-04-20 07:39:04.346465 osd.7 192.168.168.200:6813/2650 3 : cluster [WRN] 
> slow request 30.031878 seconds old, received at 2016-04-20 07:38:34.314169: 
> osd_op(client.140476549.0:13203439 rbd_data.2c9de71520eedd1.0621 
> [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 
> 1101824~8192] 5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) 
> currently waiting for rw locks 
> 
> 
> 
> There are practically every osd involved in the slow requests and they tend 
> to be between the old two osd servers and the new one. There were no issues 
> as far as I can see between the old two servers. 
> 
> The first thing i've checked is the networking. No issue was identified from 
> running ping -i .1  as well as using hping3 for the tcp 
> connection checks. The network tests were running for over a week and not a 
> single packet was lost. The slow requests took place while the network tests 
> were running. 
> 
> I've also checked the osd and ssd disks and I was not able to identify 
> anything problematic. 
> 
> Stopping all osds on the new server causes no issues between the old two osd 
> servers. I've left the new server disconnected for a few days and had no 
> issues with the cluster. 
> 
> I am a bit lost on what else to try and how to debug the issue. Could someone 
> please help me? 
> 

I would still say this is a network issue.

"currently waiting for rw locks" is usually a network problem.

I found this out myself a few weeks ago: 
http://blog.widodh.nl/2016/01/slow-requests-with-ceph-waiting-for-rw-locks/

The problem there was a wrong gateway on some machines.

In that situation the OSDs could talk just fine, but they had problems with 
sending traffic back to the clients which lead to buffers filling up.

I would suggest that you check if the network towards clients is also OK.

Wido

> Many thanks 
> 
> Andrei 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cache tier, flushed objects does not appear to be written on disk

[Gregory Farnum]

You've probably got some issues with the exact commands you're running and
how they interact with read-only caching — that's a less-common cache type.
You'll need to get somebody who's experienced using those cache types or
who has worked with it recently to help out, though.
-Greg

On Tue, Apr 26, 2016 at 4:38 AM, Benoît LORIOT 
wrote:

> Hi Greg,
>
> yes directory is hashed four levels deep and contain files
>
>
> # ls -l /var/lib/ceph/osd/ceph-1/current/1.0_head/DIR_0/DIR_0/DIR_0/DIR_0/
>
> total 908
>
> -rw-r--r--. 1 root root 601 Mar 15 15:01
> 1021bdf.__head_E5BD__1
>
> -rw-r--r--. 1 root root 178571 Mar 15 15:06
> 1026de5.__head_7E28__1
>
> -rw-r--r--. 1 root root 7853 Mar 15 15:16
> 1030a58.__head_0DA4__1
>
> -rw-r--r--. 1 root root 0 Mar 17 12:51
> 10c1ccc.__head_D073__1
>
> -rw-r--r--. 1 root root 0 Mar 17 17:23
> 10fc00c.__head_8302__1
>
> -rw-r--r--. 1 root root 0 Mar 17 17:23
> 10fcae7.__head_C3A7__1
>
> [...]
>
>
>
> 2016-04-25 17:27 GMT+02:00 Gregory Farnum :
>
>> On Thursday, April 21, 2016, Benoît LORIOT 
>> wrote:
>>
>>> Hello,
>>>
>>> we want to disable readproxy cache tier but before doing so we would
>>> like to make sure we won't loose data.
>>>
>>> Is there a way to confirm that flush actually write objects to disk ?
>>>
>>> We're using ceph version 0.94.6.
>>>
>>>
>>> I tried that, with cephfs_data_ro_cache being the hot storage pool and
>>> cephfs_data being the cold storage pool
>>>
>>> # rados -p cephfs_data_ro_cache ls
>>>
>>> then choose a random object in the list : 14d6142.
>>>
>>> Find the object on cache disk :
>>>
>>> # ceph osd map cephfs_data_ro_cache 14d6142.
>>> osdmap e301 pool 'cephfs_data_ro_cache' (6) object
>>> '14d6142.' -> pg 6.d01 (6.0) -> up ([4,5,8], p4) acting
>>> ([4,5,8], p4)
>>>
>>> Object is in the pg 6.0 on OSD 4, 5 and 8, I can find the file on disk.
>>>
>>> # ls -l
>>> /var/lib/ceph/osd/ceph-4/current/6.0_head/DIR_0/DIR_0/DIR_0/DIR_0/14d6142.__head_0D01__6
>>> -rw-r--r--. 1 root root 0 Apr  8 19:36
>>> /var/lib/ceph/osd/ceph-4/current/6.0_head/DIR_0/DIR_0/DIR_0/DIR_0/14d6142.__head_0D01__6
>>>
>>> Flush the object :
>>>
>>> # rados -p cephfs_data_ro_cache cache-try-flush 14d6142.
>>>
>>> Find the object on disk :
>>>
>>> # ceph osd map cephfs_data 14d6142.
>>> osdmap e301 pool 'cephfs_data' (1) object '14d6142.' -> pg
>>> 1.d01 (1.0) -> up ([1,7,2], p1) acting ([1,7,2], p1)
>>>
>>> Object is in the pg 1.0 on OSD 1, 7 and 2, I can't find the file on disk
>>> on any of the 3 OSDs
>>>
>>> # ls -l
>>> /var/lib/ceph/osd/ceph-1/current/1.0_head/DIR_0/DIR_0/DIR_0/DIR_0/14d6142.*
>>> ls: cannot access
>>> /var/lib/ceph/osd/ceph-1/current/1.0_head/DIR_0/DIR_0/DIR_0/DIR_0/14d6142.*:
>>> No such file or directory
>>>
>>>
>>>
>>> What am I doing wrong ? To me it seems that nothing is actually flushed
>>> to disk.
>>>
>>>
>> Is the directory actually hashed four levels deep? Or is it shallower and
>> you're looking too far down?
>> -Greg
>>
>>
>>> Thank you,
>>> Ben.
>>>
>>
>
>
> --
> 
>
> Cordialement,
>
>
>
> *Benoit LORIOT*
>
>
>
> *Ligne directe: 07 85 22 44 57*
> 
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CEPH All OSD got segmentation fault after CRUSH edit

[Samuel Just]

I think?  Probably worth reproducing on a vstart cluster to validate
the fix.  Didn't we introduce something in the mon to validate new
crushmaps?  Hammer maybe?
-Sam

On Tue, Apr 26, 2016 at 8:09 AM, Wido den Hollander  wrote:
>
>> Op 26 april 2016 om 16:58 schreef Samuel Just :
>>
>>
>> Can you attach the OSDMap (ceph osd getmap -o )?
>> -Sam
>>
>
> Henrik contacted me to look at this and this is what I found:
>
> 0x00b18b81 in crush_choose_firstn (map=map@entry=0x1f00200, 
> bucket=0x0, weight=weight@entry=0x1f2b880, weight_max=weight_max@entry=30, 
> x=x@entry=1731224833, numrep=2, type=1, out=0x7fffdc036508, outpos=0, 
> out_size=2, tries=51, recurse_tries=1, local_retries=0,
> local_fallback_retries=0, recurse_to_leaf=1, vary_r=0, 
> out2=0x7fffdc036510, parent_r=0) at crush/mapper.c:345
> 345 crush/mapper.c: No such file or directory.
>
> A bit more output from GDB:
>
> #0  0x00b18b81 in crush_choose_firstn (map=map@entry=0x1f00200, 
> bucket=0x0, weight=weight@entry=0x1f2b880, weight_max=weight_max@entry=30, 
> x=x@entry=1731224833, numrep=2, type=1, out=0x7fffdc036508, outpos=0, 
> out_size=2, tries=51, recurse_tries=1, local_retries=0,
> local_fallback_retries=0, recurse_to_leaf=1, vary_r=0, 
> out2=0x7fffdc036510, parent_r=0) at crush/mapper.c:345
> #1  0x00b194cb in crush_do_rule (map=0x1f00200, ruleno= out>, x=1731224833, result=0x7fffdc036520, result_max=, 
> weight=0x1f2b880, weight_max=30, scratch=) at 
> crush/mapper.c:794
> #2  0x00a61680 in do_rule (weight=std::vector of length 30, capacity 
> 30 = {...}, maxout=2, out=std::vector of length 0, capacity 0, x=1731224833, 
> rule=0, this=0x1f72340) at ./crush/CrushWrapper.h:939
> #3  OSDMap::_pg_to_osds (this=this@entry=0x1f46800, pool=..., pg=..., 
> osds=osds@entry=0x7fffdc036600, primary=primary@entry=0x7fffdc0365ec, 
> ppps=0x7fffdc0365f4) at osd/OSDMap.cc:1417
>
> It seems that CRUSH can't find entries in the CRUSHMap. In this case the 
> 'root default' was removed while the default ruleset still refers to it.
>
> The cluster is running 0.80.11
>
> I extracted the CRUSHMaps from the OSDMaps on osd.0:
>
> $ for i in {1392..1450}; do find -name "osdmap*${i}*" -exec osdmaptool 
> --export-crush /tmp/crush.${i} {} \;; crushtool -d /tmp/crush.${i} -o 
> /tmp/crush.${i}.txt; done
>
> Here I see that in map 1433 the root 'default' doesn't exist, but the crush 
> ruleset refers to 'bucket0'. This crushmap is attached.
>
> rule replicated_ruleset {
> ruleset 0
> type replicated
> min_size 1
> max_size 10
> step take bucket0
> step chooseleaf firstn 0 type host
> step emit
> }
>
> The root bucket0 doesn't exist.
>
> bucket0 seems like something which was created by Ceph/CRUSH and not by the 
> user.
>
> I'm thinking about injecting a fixed CRUSHMap into this OSDMap where bucket0 
> does exist. Does that seem like a sane thing to do?
>
> Wido
>
>
>> On Tue, Apr 26, 2016 at 2:07 AM, Henrik Svensson > > wrote:
>>
>> > Hi!
>> >
>> > We got a three node CEPH cluster with 10 OSD each.
>> >
>> > We bought 3 new machines with additional 30 disks that should reside in
>> > another location.
>> > Before adding these machines we modified the default CRUSH table.
>> >
>> > After modifying the (default) crush table with these commands the cluster
>> > went down:
>> >
>> > 
>> > ceph osd crush add-bucket dc1 datacenter
>> > ceph osd crush add-bucket dc2 datacenter
>> > ceph osd crush add-bucket availo datacenter
>> > ceph osd crush move dc1 root=default
>> > ceph osd crush move lkpsx0120 root=default datacenter=dc1
>> > ceph osd crush move lkpsx0130 root=default datacenter=dc1
>> > ceph osd crush move lkpsx0140 root=default datacenter=dc1
>> > ceph osd crush move dc2 root=default
>> > ceph osd crush move availo root=default
>> > ceph osd crush add-bucket sectra root
>> > ceph osd crush move dc1 root=sectra
>> > ceph osd crush move dc2 root=sectra
>> > ceph osd crush move dc3 root=sectra
>> > ceph osd crush move availo root=sectra
>> > ceph osd crush remove default
>> > 
>> >
>> > I tried to revert the CRUSH map but no luck:
>> >
>> > 
>> > ceph osd crush add-bucket default root
>> > ceph osd crush move lkpsx0120 root=default
>> > ceph osd crush move lkpsx0130 root=default
>> > ceph osd crush move lkpsx0140 root=default
>> > ceph osd crush remove sectra
>> > 
>> >
>> > After trying to restart the cluster (and even the machines) no OSD started
>> > up again.
>> > But ceph osd tree gave this output, stating certain OSD:s are up (but the
>> > processes are not running):
>> >
>> > 
>> > # id weight type name up/down reweight
>> > -1 163.8 root default
>> > -2 54.6 host lkpsx0120
>> > 0 5.46 osd.0 down 0
>> > 1 5.46 osd.1 down 0
>> > 2 5.46 osd.2 down 0
>> > 3 5.46 osd.3 down 0
>> > 4 5.46 osd.4 down 0
>> > 5 5.46 osd.5 down 0
>> > 6 5.46 osd.6 down 0
>> > 7 5.46 osd.7 down 0
>> > 8 5.46 

Re: [ceph-users] krbd map on Jewel, sysfs write failed when rbd map

[Ilya Dryomov]

On Tue, Apr 26, 2016 at 5:45 PM, Somnath Roy  wrote:
> By default image format is 2 in jewel which is not supported by krbd..try
> creating image with --image-format 1 and it should be resolved..

With the default striping pattern (no --stripe-unit or --stripe-count
at create time), --image-format 2 is supported by krbd since kernel
3.10.

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RadosGW not start after upgrade to Jewel

[Yehuda Sadeh-Weinraub]

On Tue, Apr 26, 2016 at 6:50 AM, Abhishek Lekshmanan  wrote:
>
> Ansgar Jazdzewski writes:
>
>> Hi,
>>
>> After plaing with the setup i got some output that looks wrong
>>
>> # radosgw-admin zone get
>>
>> "placement_pools": [
>> {
>> "key": "default-placement",
>> "val": {
>> "index_pool": ".eu-qa.rgw.buckets.inde",
>> "data_pool": ".eu-qa.rgw.buckets.dat",
>> "data_extra_pool": ".eu-qa.rgw.buckets.non-e",
>> "index_type": 0
>> }
>> }
>> ],
>>
>> i think it sould be
>>
>> index_pool = .eu-qa.rgw.buckets.index.
>> data_pool = .eu-qa.rgw.buckets
>> data_extra_pool = .eu-qa.rgw.buckets.extra
>>
>> how can i fix it?
>
> Not sure how it reached this state, but given a zone get json, you can

There's an issue now when doing radosgw-admin zone set, and the pool
names start with a period (http://tracker.ceph.com/issues/15597). The
pool name is getting truncated by one character. We will have this
fixed for the next point release, but the workaround now would be to
add an extra character in each pool name before running the zone set
command.

Yehuda

> edit this and set it back using zone set for eg
> # radosgw-admin zone get > zone.json # now edit this file
> # radosgw-admin zone set --rgw-zone="eu-qa" < zone.json
>>
>> Thanks
>> Ansgar
>>
>> 2016-04-26 13:07 GMT+02:00 Ansgar Jazdzewski :
>>> Hi all,
>>>
>>> i got an answer, that pointed me to:
>>> https://github.com/ceph/ceph/blob/master/doc/radosgw/multisite.rst
>>>
>>> 2016-04-25 16:02 GMT+02:00 Karol Mroz :
 On Mon, Apr 25, 2016 at 02:23:28PM +0200, Ansgar Jazdzewski wrote:
> Hi,
>
> we test Jewel in our  QA environment (from Infernalis to Hammer) the
> upgrade went fine but the Radosgw did not start.
>
> the error appears also with radosgw-admin
>
> # radosgw-admin user info --uid="images" --rgw-region=eu --rgw-zone=eu-qa
> 2016-04-25 12:13:33.425481 7fc757fad900  0 error in read_id for id  :
> (2) No such file or directory
> 2016-04-25 12:13:33.425494 7fc757fad900  0 failed reading zonegroup
> info: ret -2 (2) No such file or directory
> couldn't init storage provider
>
> do i have to change some settings, also for upgrade of the radosgw?

 Hi,

 Testing a recent master build (with only default region and zone),
 I'm able to successfully run the command you specified:

 % ./radosgw-admin user info --uid="testid" --rgw-region=default 
 --rgw-zone=default
 ...
 {
 "user_id": "testid",
 "display_name": "M. Tester",
 ...
 }

 Are you certain the region and zone you specified exist?

 What do the following report:

 radosgw-admin zone list
 radosgw-admin region list

 --
 Regards,
 Karol
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
> --
> Abhishek Lekshmanan
> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 
> 21284 (AG Nürnberg)
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can Jewel read Hammer radosgw buckets?

[Richard Chan]

Summary of Yehuda's script on Hammer -> Jewel upgrade:

1. It works: users, buckets, objects now accessible: the zonegroup and zone
have been set to "default" ( previouslly zone = "" and region = "")

2. s3cmd needed to be upgraded to 1.6 to work

Thanks.



On Tue, Apr 26, 2016 at 8:06 AM, Yehuda Sadeh-Weinraub 
wrote:

> I managed to reproduce the issue, and there seem to be multiple
> problems. Specifically we have an issue when upgrading a default
> cluster that hasn't had a zone (and region) explicitly configured
> before. There is another bug that I found
> (http://tracker.ceph.com/issues/15597) that makes things even a bit
> more complicated.
>
> I created the following script that might be able to fix things for you:
>
> https://raw.githubusercontent.com/yehudasa/ceph/wip-fix-default-zone/src/fix-zone
>
> For future reference, this script shouldn't be used if there are any
> zones configured other than the default one. It also makes some ninja
> patching to the zone config because of a bug that exists currently,
> but will probably not apply to any next versions.
>
> Please let me know if you have any issues, or if this actually does its
> magic.
>
> Thanks,
> Yehuda
>
> On Mon, Apr 25, 2016 at 4:10 PM, Richard Chan
>  wrote:
> >
> >> > How do you actually do that?
> >>
> >> What does 'radosgw-admin zone get' return?
> >>
> >> Yehuda
> >
> >
> >
> > [root@node1 ceph]# radosgw-admin zone get
> > unable to initialize zone: (2) No such file or directory
> >
> > (I don't have any rgw configuration in /etc/ceph/ceph.conf; this is from
> a
> > clean
> >
> > ceph-deploy rgw create node1
> >
> > ## user created under Hammer
> > [root@node1 ceph]# radosgw-admin user info --uid=testuser
> > 2016-04-26 07:07:06.159497 7f410c33ca40  0 RGWZoneParams::create(): error
> > creating default zone params: (17) File exists
> > could not fetch user info: no user info saved
> >
> > "rgw_max_chunk_size": "524288",
> > "rgw_max_put_size": "5368709120",
> > "rgw_override_bucket_index_max_shards": "0",
> > "rgw_bucket_index_max_aio": "8",
> > "rgw_enable_quota_threads": "true",
> > "rgw_enable_gc_threads": "true",
> > "rgw_data": "\/var\/lib\/ceph\/radosgw\/ceph-rgw.node1",
> > "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin",
> > "rgw_cache_enabled": "true",
> > "rgw_cache_lru_size": "1",
> > "rgw_socket_path": "",
> > "rgw_host": "",
> > "rgw_port": "",
> > "rgw_dns_name": "",
> > "rgw_dns_s3website_name": "",
> > "rgw_content_length_compat": "false",
> > "rgw_script_uri": "",
> > "rgw_request_uri": "",
> > "rgw_swift_url": "",
> > "rgw_swift_url_prefix": "swift",
> > "rgw_swift_auth_url": "",
> > "rgw_swift_auth_entry": "auth",
> > "rgw_swift_tenant_name": "",
> > "rgw_swift_account_in_url": "false",
> > "rgw_swift_enforce_content_length": "false",
> > "rgw_keystone_url": "",
> > "rgw_keystone_admin_token": "",
> > "rgw_keystone_admin_user": "",
> > "rgw_keystone_admin_password": "",
> > "rgw_keystone_admin_tenant": "",
> > "rgw_keystone_admin_project": "",
> > "rgw_keystone_admin_domain": "",
> > "rgw_keystone_api_version": "2",
> > "rgw_keystone_accepted_roles": "Member, admin",
> > "rgw_keystone_token_cache_size": "1",
> > "rgw_keystone_revocation_interval": "900",
> > "rgw_keystone_verify_ssl": "true",
> > "rgw_keystone_implicit_tenants": "false",
> > "rgw_s3_auth_use_rados": "true",
> > "rgw_s3_auth_use_keystone": "false",
> > "rgw_ldap_uri": "ldaps:\/\/",
> > "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com",
> > "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com",
> > "rgw_ldap_dnattr": "uid",
> > "rgw_ldap_secret": "\/etc\/openldap\/secret",
> > "rgw_s3_auth_use_ldap": "false",
> > "rgw_admin_entry": "admin",
> > "rgw_enforce_swift_acls": "true",
> > "rgw_swift_token_expiration": "86400",
> > "rgw_print_continue": "true",
> > "rgw_remote_addr_param": "REMOTE_ADDR",
> > "rgw_op_thread_timeout": "600",
> > "rgw_op_thread_suicide_timeout": "0",
> > "rgw_thread_pool_size": "100",
> > "rgw_num_control_oids": "8",
> > "rgw_num_rados_handles": "1",
> > "rgw_nfs_lru_lanes": "5",
> > "rgw_nfs_lru_lane_hiwat": "911",
> > "rgw_nfs_fhcache_partitions": "3",
> > "rgw_nfs_fhcache_size": "2017",
> > "rgw_zone": "",
> > "rgw_zone_root_pool": ".rgw.root",
> > "rgw_default_zone_info_oid": "default.zone",
> > "rgw_region": "",
> > "rgw_default_region_info_oid": "default.region",
> > "rgw_zonegroup": "",
> > "rgw_zonegroup_root_pool": ".rgw.root",
> > "rgw_default_zonegroup_info_oid": "default.zonegroup",
> > "rgw_realm": "",
> > "rgw_realm_root_pool": ".rgw.root",
> > "rgw_default_realm_info_oid": "default.realm",
> > "rgw_period_root_pool": ".rgw.root",
> > "rgw_period_latest_epoch_info_oid": ".latest_epoch

[ceph-users] Hammer broke after adding 3rd osd server

[Andrei Mikhailovsky]

Hello everyone, 

I've recently performed a hardware upgrade on our small two osd server ceph 
cluster, which seems to have broke the ceph cluster. We are using ceph for 
cloudstack rbd images for vms.All of our servers are Ubuntu 14.04 LTS with 
latest updates and kernel 4.4.6 from ubuntu repo. 

Previous hardware: 

2 x osd servers with 9 sas osds, 32gb ram and 12 core Intel cpu 2620 @ 2Ghz 
each and 2 consumer SSDs for journal. Infiniband 40gbit/s networking using 
IPoIB. 

The following things were upgraded: 

1. journal ssds were upgraded from consumer ssd to Intel 3710 200gb. We now 
have 5 osds per single ssd. 
2. added additional osd server with 64gb ram, 10 osds, Intel 2670 cpu @ 2.6Ghz 
3. Upgraded ram on osd servers to become 64gb 
4. Installed additional osd disk to have 10 osds per server. 

After adding the third osd server and finishing the initial sync, the cluster 
worked okay for 1-2 days. No issues were noticed. On a third day my monitoring 
system started reporting a bunch of issues from the ceph cluster as well as 
from our virtual machines. This tend to happen between 7:20am and 7:40am and 
lasts for about 2-3 hours before things become normal again. I've checked the 
osd servers and there is nothing that I could find in cron or otherwise that 
starts around 7:20am. 

The problem is as follows: the new osd server's load goes to 400+ with ceph-osd 
processes consuming all cpu resources. The ceph -w shows a high number of slow 
requests which relate to osds belonging to the new osd server. The log files 
show the following: 

2016-04-20 07:39:04.346459 osd.7 192.168.168.200:6813/2650 2 : cluster [WRN] 
slow request 30.032033 seconds old, received at 2016-04-20 07:38:34.314014: 
osd_op(client.140476549.0:13203438 rbd_data.2c9de71520eedd1.0621 
[stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2572288~4096] 
5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) currently waiting for 
subops from 22 
2016-04-20 07:39:04.346465 osd.7 192.168.168.200:6813/2650 3 : cluster [WRN] 
slow request 30.031878 seconds old, received at 2016-04-20 07:38:34.314169: 
osd_op(client.140476549.0:13203439 rbd_data.2c9de71520eedd1.0621 
[stat,set-alloc-hint object_size 4194304 write_size 4194304,write 1101824~8192] 
5.6c3bece2 ack+ondisk+write+known_if_redirected e83912) currently waiting for 
rw locks 



There are practically every osd involved in the slow requests and they tend to 
be between the old two osd servers and the new one. There were no issues as far 
as I can see between the old two servers. 

The first thing i've checked is the networking. No issue was identified from 
running ping -i .1  as well as using hping3 for the tcp connection 
checks. The network tests were running for over a week and not a single packet 
was lost. The slow requests took place while the network tests were running. 

I've also checked the osd and ssd disks and I was not able to identify anything 
problematic. 

Stopping all osds on the new server causes no issues between the old two osd 
servers. I've left the new server disconnected for a few days and had no issues 
with the cluster. 

I am a bit lost on what else to try and how to debug the issue. Could someone 
please help me? 

Many thanks 

Andrei 
















___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] krbd map on Jewel, sysfs write failed when rbd map

[Somnath Roy]

By default image format is 2 in jewel which is not supported by krbd..try 
creating image with --image-format 1 and it should be resolved..

Thanks
Somnath

Sent from my iPhone

On Apr 25, 2016, at 9:38 PM, 
"wd_hw...@wistron.com" 
mailto:wd_hw...@wistron.com>> wrote:

Dear Cephers:
  I got the same issue under Ubuntu 14.04, even I try to use the image format 
‘1’.
# modinfo rbd
filename:   /lib/modules/3.13.0-85-generic/kernel/drivers/block/rbd.ko
license:GPL
author: Jeff Garzik mailto:j...@garzik.org>>
description:rados block device
author: Yehuda Sadeh 
mailto:yeh...@hq.newdream.net>>
author: Sage Weil mailto:s...@newdream.net>>
author: Alex Elder mailto:el...@inktank.com>>
srcversion: 48BFBD5C3D31D799F01D218
depends:libceph
intree: Y
vermagic:   3.13.0-85-generic SMP mod_unload modversions
signer: Magrathea: Glacier signing key
sig_key:C6:33:E9:BF:A6:CA:49:D5:3D:2E:B5:25:6A:35:87:7D:04:F1:64:F8
sig_hashalgo:   sha512

##
# rbd info block_data/data01
rbd image 'data01':
size 10240 MB in 2560 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.540752ae8944a
format: 2
features:
flags:
# rbd map block_data/data01
rbd: sysfs write failed
rbd: map failed: (5) Input/output error

##
# rbd info block_data/data02
rbd image 'data02':
size 10240 MB in 2560 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.2aac0238e1f29
format: 2
features: layering
   flags:
# rbd map block_data/data02
rbd: sysfs write failed
rbd: map failed: (5) Input/output error

  Is there any new idea to solve this issue?

Thanks a lot,
WD

---

This email contains confidential or legally privileged information and is for 
the sole use of its intended recipient.

Any unauthorized review, use, copying or distribution of this email or the 
content of this email is strictly prohibited.

If you are not the intended recipient, you may reply to the sender and should 
delete this e-mail immediately.

---

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
PLEASE NOTE: The information contained in this electronic mail message is 
intended only for the use of the designated recipient(s) named above. If the 
reader of this message is not the intended recipient, you are hereby notified 
that you have received this message in error and that any review, 
dissemination, distribution, or copying of this message is strictly prohibited. 
If you have received this communication in error, please notify the sender by 
telephone or e-mail (as shown above) immediately and destroy any and all copies 
of this message in your possession (whether hard copies or electronically 
stored copies).
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CEPH All OSD got segmentation fault after CRUSH edit

[Wido den Hollander]

> Op 26 april 2016 om 16:58 schreef Samuel Just :
> 
> 
> Can you attach the OSDMap (ceph osd getmap -o )?
> -Sam
> 

Henrik contacted me to look at this and this is what I found:

0x00b18b81 in crush_choose_firstn (map=map@entry=0x1f00200, bucket=0x0, 
weight=weight@entry=0x1f2b880, weight_max=weight_max@entry=30, 
x=x@entry=1731224833, numrep=2, type=1, out=0x7fffdc036508, outpos=0, 
out_size=2, tries=51, recurse_tries=1, local_retries=0, 
local_fallback_retries=0, recurse_to_leaf=1, vary_r=0, out2=0x7fffdc036510, 
parent_r=0) at crush/mapper.c:345
345 crush/mapper.c: No such file or directory.

A bit more output from GDB:

#0  0x00b18b81 in crush_choose_firstn (map=map@entry=0x1f00200, 
bucket=0x0, weight=weight@entry=0x1f2b880, weight_max=weight_max@entry=30, 
x=x@entry=1731224833, numrep=2, type=1, out=0x7fffdc036508, outpos=0, 
out_size=2, tries=51, recurse_tries=1, local_retries=0, 
local_fallback_retries=0, recurse_to_leaf=1, vary_r=0, out2=0x7fffdc036510, 
parent_r=0) at crush/mapper.c:345
#1  0x00b194cb in crush_do_rule (map=0x1f00200, ruleno=, 
x=1731224833, result=0x7fffdc036520, result_max=, 
weight=0x1f2b880, weight_max=30, scratch=) at crush/mapper.c:794
#2  0x00a61680 in do_rule (weight=std::vector of length 30, capacity 30 
= {...}, maxout=2, out=std::vector of length 0, capacity 0, x=1731224833, 
rule=0, this=0x1f72340) at ./crush/CrushWrapper.h:939
#3  OSDMap::_pg_to_osds (this=this@entry=0x1f46800, pool=..., pg=..., 
osds=osds@entry=0x7fffdc036600, primary=primary@entry=0x7fffdc0365ec, 
ppps=0x7fffdc0365f4) at osd/OSDMap.cc:1417

It seems that CRUSH can't find entries in the CRUSHMap. In this case the 'root 
default' was removed while the default ruleset still refers to it.

The cluster is running 0.80.11

I extracted the CRUSHMaps from the OSDMaps on osd.0:

$ for i in {1392..1450}; do find -name "osdmap*${i}*" -exec osdmaptool 
--export-crush /tmp/crush.${i} {} \;; crushtool -d /tmp/crush.${i} -o 
/tmp/crush.${i}.txt; done

Here I see that in map 1433 the root 'default' doesn't exist, but the crush 
ruleset refers to 'bucket0'. This crushmap is attached.

rule replicated_ruleset {
ruleset 0
type replicated
min_size 1
max_size 10
step take bucket0
step chooseleaf firstn 0 type host
step emit
}

The root bucket0 doesn't exist.

bucket0 seems like something which was created by Ceph/CRUSH and not by the 
user.

I'm thinking about injecting a fixed CRUSHMap into this OSDMap where bucket0 
does exist. Does that seem like a sane thing to do?

Wido


> On Tue, Apr 26, 2016 at 2:07 AM, Henrik Svensson  > wrote:
> 
> > Hi!
> >
> > We got a three node CEPH cluster with 10 OSD each.
> >
> > We bought 3 new machines with additional 30 disks that should reside in
> > another location.
> > Before adding these machines we modified the default CRUSH table.
> >
> > After modifying the (default) crush table with these commands the cluster
> > went down:
> >
> > 
> > ceph osd crush add-bucket dc1 datacenter
> > ceph osd crush add-bucket dc2 datacenter
> > ceph osd crush add-bucket availo datacenter
> > ceph osd crush move dc1 root=default
> > ceph osd crush move lkpsx0120 root=default datacenter=dc1
> > ceph osd crush move lkpsx0130 root=default datacenter=dc1
> > ceph osd crush move lkpsx0140 root=default datacenter=dc1
> > ceph osd crush move dc2 root=default
> > ceph osd crush move availo root=default
> > ceph osd crush add-bucket sectra root
> > ceph osd crush move dc1 root=sectra
> > ceph osd crush move dc2 root=sectra
> > ceph osd crush move dc3 root=sectra
> > ceph osd crush move availo root=sectra
> > ceph osd crush remove default
> > 
> >
> > I tried to revert the CRUSH map but no luck:
> >
> > 
> > ceph osd crush add-bucket default root
> > ceph osd crush move lkpsx0120 root=default
> > ceph osd crush move lkpsx0130 root=default
> > ceph osd crush move lkpsx0140 root=default
> > ceph osd crush remove sectra
> > 
> >
> > After trying to restart the cluster (and even the machines) no OSD started
> > up again.
> > But ceph osd tree gave this output, stating certain OSD:s are up (but the
> > processes are not running):
> >
> > 
> > # id weight type name up/down reweight
> > -1 163.8 root default
> > -2 54.6 host lkpsx0120
> > 0 5.46 osd.0 down 0
> > 1 5.46 osd.1 down 0
> > 2 5.46 osd.2 down 0
> > 3 5.46 osd.3 down 0
> > 4 5.46 osd.4 down 0
> > 5 5.46 osd.5 down 0
> > 6 5.46 osd.6 down 0
> > 7 5.46 osd.7 down 0
> > 8 5.46 osd.8 down 0
> > 9 5.46 osd.9 down 0
> > -3 54.6 host lkpsx0130
> > 10 5.46 osd.10 down 0
> > 11 5.46 osd.11 down 0
> > 12 5.46 osd.12 down 0
> > 13 5.46 osd.13 down 0
> > 14 5.46 osd.14 down 0
> > 15 5.46 osd.15 down 0
> > 16 5.46 osd.16 down 0
> > 17 5.46 osd.17 down 0
> > 18 5.46 osd.18 up 1
> > 19 5.46 osd.19 up 1
> > -4 54.6 host lkpsx0140
> > 20 5.46 osd.20 up 1
> > 21 5.46 osd.21 down 0
> > 22 5.

Re: [ceph-users] CEPH All OSD got segmentation fault after CRUSH edit

[Samuel Just]

Can you attach the OSDMap (ceph osd getmap -o )?
-Sam

On Tue, Apr 26, 2016 at 2:07 AM, Henrik Svensson  wrote:

> Hi!
>
> We got a three node CEPH cluster with 10 OSD each.
>
> We bought 3 new machines with additional 30 disks that should reside in
> another location.
> Before adding these machines we modified the default CRUSH table.
>
> After modifying the (default) crush table with these commands the cluster
> went down:
>
> 
> ceph osd crush add-bucket dc1 datacenter
> ceph osd crush add-bucket dc2 datacenter
> ceph osd crush add-bucket availo datacenter
> ceph osd crush move dc1 root=default
> ceph osd crush move lkpsx0120 root=default datacenter=dc1
> ceph osd crush move lkpsx0130 root=default datacenter=dc1
> ceph osd crush move lkpsx0140 root=default datacenter=dc1
> ceph osd crush move dc2 root=default
> ceph osd crush move availo root=default
> ceph osd crush add-bucket sectra root
> ceph osd crush move dc1 root=sectra
> ceph osd crush move dc2 root=sectra
> ceph osd crush move dc3 root=sectra
> ceph osd crush move availo root=sectra
> ceph osd crush remove default
> 
>
> I tried to revert the CRUSH map but no luck:
>
> 
> ceph osd crush add-bucket default root
> ceph osd crush move lkpsx0120 root=default
> ceph osd crush move lkpsx0130 root=default
> ceph osd crush move lkpsx0140 root=default
> ceph osd crush remove sectra
> 
>
> After trying to restart the cluster (and even the machines) no OSD started
> up again.
> But ceph osd tree gave this output, stating certain OSD:s are up (but the
> processes are not running):
>
> 
> # id weight type name up/down reweight
> -1 163.8 root default
> -2 54.6 host lkpsx0120
> 0 5.46 osd.0 down 0
> 1 5.46 osd.1 down 0
> 2 5.46 osd.2 down 0
> 3 5.46 osd.3 down 0
> 4 5.46 osd.4 down 0
> 5 5.46 osd.5 down 0
> 6 5.46 osd.6 down 0
> 7 5.46 osd.7 down 0
> 8 5.46 osd.8 down 0
> 9 5.46 osd.9 down 0
> -3 54.6 host lkpsx0130
> 10 5.46 osd.10 down 0
> 11 5.46 osd.11 down 0
> 12 5.46 osd.12 down 0
> 13 5.46 osd.13 down 0
> 14 5.46 osd.14 down 0
> 15 5.46 osd.15 down 0
> 16 5.46 osd.16 down 0
> 17 5.46 osd.17 down 0
> 18 5.46 osd.18 up 1
> 19 5.46 osd.19 up 1
> -4 54.6 host lkpsx0140
> 20 5.46 osd.20 up 1
> 21 5.46 osd.21 down 0
> 22 5.46 osd.22 down 0
> 23 5.46 osd.23 down 0
> 24 5.46 osd.24 down 0
> 25 5.46 osd.25 up 1
> 26 5.46 osd.26 up 1
> 27 5.46 osd.27 up 1
> 28 5.46 osd.28 up 1
> 29 5.46 osd.29 up 1
> 
>
> The monitor starts/restarts OK (only one monitor exists).
> But when starting one OSD with ceph -w nothing shows.
>
> Here is the ceph mon_status:
>
> 
> { "name": "lkpsx0120",
>   "rank": 0,
>   "state": "leader",
>   "election_epoch": 1,
>   "quorum": [
> 0],
>   "outside_quorum": [],
>   "extra_probe_peers": [],
>   "sync_provider": [],
>   "monmap": { "epoch": 4,
>   "fsid": "9244194a-5e10-47ae-9287-507944612f95",
>   "modified": "0.00",
>   "created": "0.00",
>   "mons": [
> { "rank": 0,
>   "name": "lkpsx0120",
>   "addr": "10.15.2.120:6789\/0"}]}}
> 
>
> Here is the ceph.conf file
>
> 
> [global]
> fsid = 9244194a-5e10-47ae-9287-507944612f95
> mon_initial_members = lkpsx0120
> mon_host = 10.15.2.120
> #debug osd = 20
> #debug ms = 1
> auth_cluster_required = cephx
> auth_service_required = cephx
> auth_client_required = cephx
> filestore_xattr_use_omap = true
> osd_crush_chooseleaf_type = 1
> osd_pool_default_size = 2
> public_network = 10.15.2.0/24
> cluster_network = 10.15.4.0/24
> rbd_cache = true
> rbd_cache_size = 67108864
> rbd_cache_max_dirty = 50331648
> rbd_cache_target_dirty = 33554432
> rbd_cache_max_dirty_age = 2
> rbd_cache_writethrough_until_flush = true
> 
>
> Here is the decompiled crush map:
>
> 
> # begin crush map
> tunable choose_local_tries 0
> tunable choose_local_fallback_tries 0
> tunable choose_total_tries 50
> tunable chooseleaf_descend_once 1
>
> # devices
> device 0 osd.0
> device 1 osd.1
> device 2 osd.2
> device 3 osd.3
> device 4 osd.4
> device 5 osd.5
> device 6 osd.6
> device 7 osd.7
> device 8 osd.8
> device 9 osd.9
> device 10 osd.10
> device 11 osd.11
> device 12 osd.12
> device 13 osd.13
> device 14 osd.14
> device 15 osd.15
> device 16 osd.16
> device 17 osd.17
> device 18 osd.18
> device 19 osd.19
> device 20 osd.20
> device 21 osd.21
> device 22 osd.22
> device 23 osd.23
> device 24 osd.24
> device 25 osd.25
> device 26 osd.26
> device 27 osd.27
> device 28 osd.28
> device 29 osd.29
>
> # types
> type 0 osd
> type 1 host
> type 2 chassis
> type 3 rack
> type 4 row
> type 5 pdu
> type 6 pod
> type 7 room
> type 8 datacenter
> type 9 region
> type 10 root
>
> # buckets
> host lkpsx0120 {
> id -2 # do not change unnecessarily
> # weight 54.600
> alg straw
> hash 0 # rjenkins1
> item osd.0 weight 5.460
> item osd.1 weight 5.460
> item osd.2 weight 5.460
> item osd.3 weight 5.460

Re: [ceph-users] Can Jewel read Hammer radosgw buckets?

[Richard Chan]

My bad: s3cmd errors were unrelated to Jewel upgrade and Yehuda's script:

It required an upgrade from s3cmd from 1.5 to 1.6 - sorry for the noise.

Will try to replicate the upgrade.



On Tue, Apr 26, 2016 at 9:27 PM, Richard Chan 
wrote:

> Also s3cmd is unable to create new buckets:
>
> # s3cmd -c jewel.cfg mb s3://test.3
> ERROR: S3 error: None
>
>
> On Tue, Apr 26, 2016 at 8:06 AM, Yehuda Sadeh-Weinraub 
> wrote:
>
>> I managed to reproduce the issue, and there seem to be multiple
>> problems. Specifically we have an issue when upgrading a default
>> cluster that hasn't had a zone (and region) explicitly configured
>> before. There is another bug that I found
>> (http://tracker.ceph.com/issues/15597) that makes things even a bit
>> more complicated.
>>
>> I created the following script that might be able to fix things for you:
>>
>> https://raw.githubusercontent.com/yehudasa/ceph/wip-fix-default-zone/src/fix-zone
>>
>> For future reference, this script shouldn't be used if there are any
>> zones configured other than the default one. It also makes some ninja
>> patching to the zone config because of a bug that exists currently,
>> but will probably not apply to any next versions.
>>
>> Please let me know if you have any issues, or if this actually does its
>> magic.
>>
>> Thanks,
>> Yehuda
>>
>> On Mon, Apr 25, 2016 at 4:10 PM, Richard Chan
>>  wrote:
>> >
>> >> > How do you actually do that?
>> >>
>> >> What does 'radosgw-admin zone get' return?
>> >>
>> >> Yehuda
>> >
>> >
>> >
>> > [root@node1 ceph]# radosgw-admin zone get
>> > unable to initialize zone: (2) No such file or directory
>> >
>> > (I don't have any rgw configuration in /etc/ceph/ceph.conf; this is
>> from a
>> > clean
>> >
>> > ceph-deploy rgw create node1
>> >
>> > ## user created under Hammer
>> > [root@node1 ceph]# radosgw-admin user info --uid=testuser
>> > 2016-04-26 07:07:06.159497 7f410c33ca40  0 RGWZoneParams::create():
>> error
>> > creating default zone params: (17) File exists
>> > could not fetch user info: no user info saved
>> >
>> > "rgw_max_chunk_size": "524288",
>> > "rgw_max_put_size": "5368709120",
>> > "rgw_override_bucket_index_max_shards": "0",
>> > "rgw_bucket_index_max_aio": "8",
>> > "rgw_enable_quota_threads": "true",
>> > "rgw_enable_gc_threads": "true",
>> > "rgw_data": "\/var\/lib\/ceph\/radosgw\/ceph-rgw.node1",
>> > "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin",
>> > "rgw_cache_enabled": "true",
>> > "rgw_cache_lru_size": "1",
>> > "rgw_socket_path": "",
>> > "rgw_host": "",
>> > "rgw_port": "",
>> > "rgw_dns_name": "",
>> > "rgw_dns_s3website_name": "",
>> > "rgw_content_length_compat": "false",
>> > "rgw_script_uri": "",
>> > "rgw_request_uri": "",
>> > "rgw_swift_url": "",
>> > "rgw_swift_url_prefix": "swift",
>> > "rgw_swift_auth_url": "",
>> > "rgw_swift_auth_entry": "auth",
>> > "rgw_swift_tenant_name": "",
>> > "rgw_swift_account_in_url": "false",
>> > "rgw_swift_enforce_content_length": "false",
>> > "rgw_keystone_url": "",
>> > "rgw_keystone_admin_token": "",
>> > "rgw_keystone_admin_user": "",
>> > "rgw_keystone_admin_password": "",
>> > "rgw_keystone_admin_tenant": "",
>> > "rgw_keystone_admin_project": "",
>> > "rgw_keystone_admin_domain": "",
>> > "rgw_keystone_api_version": "2",
>> > "rgw_keystone_accepted_roles": "Member, admin",
>> > "rgw_keystone_token_cache_size": "1",
>> > "rgw_keystone_revocation_interval": "900",
>> > "rgw_keystone_verify_ssl": "true",
>> > "rgw_keystone_implicit_tenants": "false",
>> > "rgw_s3_auth_use_rados": "true",
>> > "rgw_s3_auth_use_keystone": "false",
>> > "rgw_ldap_uri": "ldaps:\/\/",
>> > "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com",
>> > "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com",
>> > "rgw_ldap_dnattr": "uid",
>> > "rgw_ldap_secret": "\/etc\/openldap\/secret",
>> > "rgw_s3_auth_use_ldap": "false",
>> > "rgw_admin_entry": "admin",
>> > "rgw_enforce_swift_acls": "true",
>> > "rgw_swift_token_expiration": "86400",
>> > "rgw_print_continue": "true",
>> > "rgw_remote_addr_param": "REMOTE_ADDR",
>> > "rgw_op_thread_timeout": "600",
>> > "rgw_op_thread_suicide_timeout": "0",
>> > "rgw_thread_pool_size": "100",
>> > "rgw_num_control_oids": "8",
>> > "rgw_num_rados_handles": "1",
>> > "rgw_nfs_lru_lanes": "5",
>> > "rgw_nfs_lru_lane_hiwat": "911",
>> > "rgw_nfs_fhcache_partitions": "3",
>> > "rgw_nfs_fhcache_size": "2017",
>> > "rgw_zone": "",
>> > "rgw_zone_root_pool": ".rgw.root",
>> > "rgw_default_zone_info_oid": "default.zone",
>> > "rgw_region": "",
>> > "rgw_default_region_info_oid": "default.region",
>> > "rgw_zonegroup": "",
>> > "rgw_zonegroup_root_pool": ".rgw.root",
>> > "rgw_default_zonegroup_info_oid": "default.zonegroup",
>> >  

Re: [ceph-users] RadosGW not start after upgrade to Jewel

[Abhishek Lekshmanan]

Ansgar Jazdzewski writes:

> Hi,
>
> After plaing with the setup i got some output that looks wrong
>
> # radosgw-admin zone get
>
> "placement_pools": [
> {
> "key": "default-placement",
> "val": {
> "index_pool": ".eu-qa.rgw.buckets.inde",
> "data_pool": ".eu-qa.rgw.buckets.dat",
> "data_extra_pool": ".eu-qa.rgw.buckets.non-e",
> "index_type": 0
> }
> }
> ],
>
> i think it sould be
>
> index_pool = .eu-qa.rgw.buckets.index.
> data_pool = .eu-qa.rgw.buckets
> data_extra_pool = .eu-qa.rgw.buckets.extra
>
> how can i fix it?

Not sure how it reached this state, but given a zone get json, you can
edit this and set it back using zone set for eg
# radosgw-admin zone get > zone.json # now edit this file
# radosgw-admin zone set --rgw-zone="eu-qa" < zone.json
>
> Thanks
> Ansgar
>
> 2016-04-26 13:07 GMT+02:00 Ansgar Jazdzewski :
>> Hi all,
>>
>> i got an answer, that pointed me to:
>> https://github.com/ceph/ceph/blob/master/doc/radosgw/multisite.rst
>>
>> 2016-04-25 16:02 GMT+02:00 Karol Mroz :
>>> On Mon, Apr 25, 2016 at 02:23:28PM +0200, Ansgar Jazdzewski wrote:
 Hi,

 we test Jewel in our  QA environment (from Infernalis to Hammer) the
 upgrade went fine but the Radosgw did not start.

 the error appears also with radosgw-admin

 # radosgw-admin user info --uid="images" --rgw-region=eu --rgw-zone=eu-qa
 2016-04-25 12:13:33.425481 7fc757fad900  0 error in read_id for id  :
 (2) No such file or directory
 2016-04-25 12:13:33.425494 7fc757fad900  0 failed reading zonegroup
 info: ret -2 (2) No such file or directory
 couldn't init storage provider

 do i have to change some settings, also for upgrade of the radosgw?
>>>
>>> Hi,
>>>
>>> Testing a recent master build (with only default region and zone),
>>> I'm able to successfully run the command you specified:
>>>
>>> % ./radosgw-admin user info --uid="testid" --rgw-region=default 
>>> --rgw-zone=default
>>> ...
>>> {
>>> "user_id": "testid",
>>> "display_name": "M. Tester",
>>> ...
>>> }
>>>
>>> Are you certain the region and zone you specified exist?
>>>
>>> What do the following report:
>>>
>>> radosgw-admin zone list
>>> radosgw-admin region list
>>>
>>> --
>>> Regards,
>>> Karol
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


--
Abhishek Lekshmanan
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 
(AG Nürnberg)
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Can Jewel read Hammer radosgw buckets?

[Richard Chan]

Also s3cmd is unable to create new buckets:

# s3cmd -c jewel.cfg mb s3://test.3
ERROR: S3 error: None


On Tue, Apr 26, 2016 at 8:06 AM, Yehuda Sadeh-Weinraub 
wrote:

> I managed to reproduce the issue, and there seem to be multiple
> problems. Specifically we have an issue when upgrading a default
> cluster that hasn't had a zone (and region) explicitly configured
> before. There is another bug that I found
> (http://tracker.ceph.com/issues/15597) that makes things even a bit
> more complicated.
>
> I created the following script that might be able to fix things for you:
>
> https://raw.githubusercontent.com/yehudasa/ceph/wip-fix-default-zone/src/fix-zone
>
> For future reference, this script shouldn't be used if there are any
> zones configured other than the default one. It also makes some ninja
> patching to the zone config because of a bug that exists currently,
> but will probably not apply to any next versions.
>
> Please let me know if you have any issues, or if this actually does its
> magic.
>
> Thanks,
> Yehuda
>
> On Mon, Apr 25, 2016 at 4:10 PM, Richard Chan
>  wrote:
> >
> >> > How do you actually do that?
> >>
> >> What does 'radosgw-admin zone get' return?
> >>
> >> Yehuda
> >
> >
> >
> > [root@node1 ceph]# radosgw-admin zone get
> > unable to initialize zone: (2) No such file or directory
> >
> > (I don't have any rgw configuration in /etc/ceph/ceph.conf; this is from
> a
> > clean
> >
> > ceph-deploy rgw create node1
> >
> > ## user created under Hammer
> > [root@node1 ceph]# radosgw-admin user info --uid=testuser
> > 2016-04-26 07:07:06.159497 7f410c33ca40  0 RGWZoneParams::create(): error
> > creating default zone params: (17) File exists
> > could not fetch user info: no user info saved
> >
> > "rgw_max_chunk_size": "524288",
> > "rgw_max_put_size": "5368709120",
> > "rgw_override_bucket_index_max_shards": "0",
> > "rgw_bucket_index_max_aio": "8",
> > "rgw_enable_quota_threads": "true",
> > "rgw_enable_gc_threads": "true",
> > "rgw_data": "\/var\/lib\/ceph\/radosgw\/ceph-rgw.node1",
> > "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin",
> > "rgw_cache_enabled": "true",
> > "rgw_cache_lru_size": "1",
> > "rgw_socket_path": "",
> > "rgw_host": "",
> > "rgw_port": "",
> > "rgw_dns_name": "",
> > "rgw_dns_s3website_name": "",
> > "rgw_content_length_compat": "false",
> > "rgw_script_uri": "",
> > "rgw_request_uri": "",
> > "rgw_swift_url": "",
> > "rgw_swift_url_prefix": "swift",
> > "rgw_swift_auth_url": "",
> > "rgw_swift_auth_entry": "auth",
> > "rgw_swift_tenant_name": "",
> > "rgw_swift_account_in_url": "false",
> > "rgw_swift_enforce_content_length": "false",
> > "rgw_keystone_url": "",
> > "rgw_keystone_admin_token": "",
> > "rgw_keystone_admin_user": "",
> > "rgw_keystone_admin_password": "",
> > "rgw_keystone_admin_tenant": "",
> > "rgw_keystone_admin_project": "",
> > "rgw_keystone_admin_domain": "",
> > "rgw_keystone_api_version": "2",
> > "rgw_keystone_accepted_roles": "Member, admin",
> > "rgw_keystone_token_cache_size": "1",
> > "rgw_keystone_revocation_interval": "900",
> > "rgw_keystone_verify_ssl": "true",
> > "rgw_keystone_implicit_tenants": "false",
> > "rgw_s3_auth_use_rados": "true",
> > "rgw_s3_auth_use_keystone": "false",
> > "rgw_ldap_uri": "ldaps:\/\/",
> > "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com",
> > "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com",
> > "rgw_ldap_dnattr": "uid",
> > "rgw_ldap_secret": "\/etc\/openldap\/secret",
> > "rgw_s3_auth_use_ldap": "false",
> > "rgw_admin_entry": "admin",
> > "rgw_enforce_swift_acls": "true",
> > "rgw_swift_token_expiration": "86400",
> > "rgw_print_continue": "true",
> > "rgw_remote_addr_param": "REMOTE_ADDR",
> > "rgw_op_thread_timeout": "600",
> > "rgw_op_thread_suicide_timeout": "0",
> > "rgw_thread_pool_size": "100",
> > "rgw_num_control_oids": "8",
> > "rgw_num_rados_handles": "1",
> > "rgw_nfs_lru_lanes": "5",
> > "rgw_nfs_lru_lane_hiwat": "911",
> > "rgw_nfs_fhcache_partitions": "3",
> > "rgw_nfs_fhcache_size": "2017",
> > "rgw_zone": "",
> > "rgw_zone_root_pool": ".rgw.root",
> > "rgw_default_zone_info_oid": "default.zone",
> > "rgw_region": "",
> > "rgw_default_region_info_oid": "default.region",
> > "rgw_zonegroup": "",
> > "rgw_zonegroup_root_pool": ".rgw.root",
> > "rgw_default_zonegroup_info_oid": "default.zonegroup",
> > "rgw_realm": "",
> > "rgw_realm_root_pool": ".rgw.root",
> > "rgw_default_realm_info_oid": "default.realm",
> > "rgw_period_root_pool": ".rgw.root",
> > "rgw_period_latest_epoch_info_oid": ".latest_epoch",
> > "rgw_log_nonexistent_bucket": "false",
> > "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n",
> > "rgw_log_object_name_utc": "false",
> > 

Re: [ceph-users] Can Jewel read Hammer radosgw buckets?

[Richard Chan]

Result:
1. user and buckets recognised;
2. radosgw-admin bucket list --bucket test.1
shows objects

but
3. s3cmd cannot list contents of buckets

# s3cmd -c jewel.cfg ls
2016-04-25 15:57  s3://test.1
2016-04-25 15:58  s3://test.2

# s3cmd -c jewel.cfg ls s3://test.1/
ERROR: S3 error: None
s3cmd -c jewel.cfg ls s3://test.2/
ERROR: S3 error: None

ceph version 10.2.0 (3a9fba20ec743699b69bd0181dd6c54dc01c64b9), process
radosgw, pid 18737
starting handler: civetweb
starting handler: fastcgi
-- 192.168.122.111:0/1100370813 submit_message mon_subscribe({osdmap=57})
v2 remote, 192.168.122.
monclient: hunting for new mon
ERROR: no socket server point defined, cannot start fcgi frontend
rgw period pusher: The new period does not contain my zonegroup!
== starting new request req=0x7f6e12f6f690 =
== req done req=0x7f6e12f6f690 op status=0 http_status=403 ==
civetweb: 0x7f6e5c010ba0: 192.168.122.110 - - [26/Apr/2016:21:24:23 +0800]
"GET /test.2/ HTTP/1.1







On Tue, Apr 26, 2016 at 10:20 AM, Richard Chan  wrote:

> Quick questions:
>
> 1. Should this script be run on a pre-Jewel setup (e.g. revert test VMs)
> or
>  *after* Jewel attempted to read the no-zone/no-region Hammer  and created
> the default.* pools?
>
> 2. Should the radosgw daemon be running when executing the script?
>
> Thanks!
>
>
>
> On Tue, Apr 26, 2016 at 8:06 AM, Yehuda Sadeh-Weinraub 
> wrote:
>
>> I managed to reproduce the issue, and there seem to be multiple
>> problems. Specifically we have an issue when upgrading a default
>> cluster that hasn't had a zone (and region) explicitly configured
>> before. There is another bug that I found
>> (http://tracker.ceph.com/issues/15597) that makes things even a bit
>> more complicated.
>>
>> I created the following script that might be able to fix things for you:
>>
>> https://raw.githubusercontent.com/yehudasa/ceph/wip-fix-default-zone/src/fix-zone
>>
>> For future reference, this script shouldn't be used if there are any
>> zones configured other than the default one. It also makes some ninja
>> patching to the zone config because of a bug that exists currently,
>> but will probably not apply to any next versions.
>>
>> Please let me know if you have any issues, or if this actually does its
>> magic.
>>
>> Thanks,
>> Yehuda
>>
>> On Mon, Apr 25, 2016 at 4:10 PM, Richard Chan
>>  wrote:
>> >
>> >> > How do you actually do that?
>> >>
>> >> What does 'radosgw-admin zone get' return?
>> >>
>> >> Yehuda
>> >
>> >
>> >
>> > [root@node1 ceph]# radosgw-admin zone get
>> > unable to initialize zone: (2) No such file or directory
>> >
>> > (I don't have any rgw configuration in /etc/ceph/ceph.conf; this is
>> from a
>> > clean
>> >
>> > ceph-deploy rgw create node1
>> >
>> > ## user created under Hammer
>> > [root@node1 ceph]# radosgw-admin user info --uid=testuser
>> > 2016-04-26 07:07:06.159497 7f410c33ca40  0 RGWZoneParams::create():
>> error
>> > creating default zone params: (17) File exists
>> > could not fetch user info: no user info saved
>> >
>> > "rgw_max_chunk_size": "524288",
>> > "rgw_max_put_size": "5368709120",
>> > "rgw_override_bucket_index_max_shards": "0",
>> > "rgw_bucket_index_max_aio": "8",
>> > "rgw_enable_quota_threads": "true",
>> > "rgw_enable_gc_threads": "true",
>> > "rgw_data": "\/var\/lib\/ceph\/radosgw\/ceph-rgw.node1",
>> > "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin",
>> > "rgw_cache_enabled": "true",
>> > "rgw_cache_lru_size": "1",
>> > "rgw_socket_path": "",
>> > "rgw_host": "",
>> > "rgw_port": "",
>> > "rgw_dns_name": "",
>> > "rgw_dns_s3website_name": "",
>> > "rgw_content_length_compat": "false",
>> > "rgw_script_uri": "",
>> > "rgw_request_uri": "",
>> > "rgw_swift_url": "",
>> > "rgw_swift_url_prefix": "swift",
>> > "rgw_swift_auth_url": "",
>> > "rgw_swift_auth_entry": "auth",
>> > "rgw_swift_tenant_name": "",
>> > "rgw_swift_account_in_url": "false",
>> > "rgw_swift_enforce_content_length": "false",
>> > "rgw_keystone_url": "",
>> > "rgw_keystone_admin_token": "",
>> > "rgw_keystone_admin_user": "",
>> > "rgw_keystone_admin_password": "",
>> > "rgw_keystone_admin_tenant": "",
>> > "rgw_keystone_admin_project": "",
>> > "rgw_keystone_admin_domain": "",
>> > "rgw_keystone_api_version": "2",
>> > "rgw_keystone_accepted_roles": "Member, admin",
>> > "rgw_keystone_token_cache_size": "1",
>> > "rgw_keystone_revocation_interval": "900",
>> > "rgw_keystone_verify_ssl": "true",
>> > "rgw_keystone_implicit_tenants": "false",
>> > "rgw_s3_auth_use_rados": "true",
>> > "rgw_s3_auth_use_keystone": "false",
>> > "rgw_ldap_uri": "ldaps:\/\/",
>> > "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com",
>> > "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com",
>> > "rgw_ldap_dnattr": "uid",
>> > "rgw_ldap_secret": "\/etc\/openldap\/secret",
>> > "rg

[ceph-users] RadosGW and X-Storage-Url

[Paweł Sadowski]

Hi,

I'm testing RadosGW on Infernalis (9.2.1) and have two questions
regarding X-Storage-Url header.
First thing is that it always returns something like below:

X-Storage-Url: http://my.example.domain:0/swift/v1

While docs say it should return "... {api version}/{account} prefix"

Second thing is that in above header I'm getting port specified as 0
while it should be 80 (or not present).
How can I force RadosGW to return correct port?

My RadosGW config:


[global]
  fsid = d8b82ea1-3c51-40ca-92ae-6c62f434
  mon host = .

  auth cluster required = cephx
  auth service required = cephx
  auth client required = cephx
  keyring = /etc/ceph/ceph.client.rgw_user.keyring

  rgw s3 auth use keystone = false
  rgw region = ovh-fr
  rgw region root pool = .rgw.root
  rgw zone = ovh-fr-1
  rgw zone root pool = .rgw.root

  host = my.example.domain
  rgw frontends = civetweb port=127.0.0.1:9000
  rgw swift url = http://my.example.domain

  rgw_swift_account_in_url = true

  admin socket = /var/run/rgw.ovh-fr-1.9000.asok
  log file = /var/log/radosgw/rgw.ovh-fr-1.9000.log


Regards,

-- 
PS
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RadosGW not start after upgrade to Jewel

[Ansgar Jazdzewski]

Hi,

After plaing with the setup i got some output that looks wrong

# radosgw-admin zone get

"placement_pools": [
{
"key": "default-placement",
"val": {
"index_pool": ".eu-qa.rgw.buckets.inde",
"data_pool": ".eu-qa.rgw.buckets.dat",
"data_extra_pool": ".eu-qa.rgw.buckets.non-e",
"index_type": 0
}
}
],

i think it sould be

index_pool = .eu-qa.rgw.buckets.index.
data_pool = .eu-qa.rgw.buckets
data_extra_pool = .eu-qa.rgw.buckets.extra

how can i fix it?

Thanks
Ansgar

2016-04-26 13:07 GMT+02:00 Ansgar Jazdzewski :
> Hi all,
>
> i got an answer, that pointed me to:
> https://github.com/ceph/ceph/blob/master/doc/radosgw/multisite.rst
>
> 2016-04-25 16:02 GMT+02:00 Karol Mroz :
>> On Mon, Apr 25, 2016 at 02:23:28PM +0200, Ansgar Jazdzewski wrote:
>>> Hi,
>>>
>>> we test Jewel in our  QA environment (from Infernalis to Hammer) the
>>> upgrade went fine but the Radosgw did not start.
>>>
>>> the error appears also with radosgw-admin
>>>
>>> # radosgw-admin user info --uid="images" --rgw-region=eu --rgw-zone=eu-qa
>>> 2016-04-25 12:13:33.425481 7fc757fad900  0 error in read_id for id  :
>>> (2) No such file or directory
>>> 2016-04-25 12:13:33.425494 7fc757fad900  0 failed reading zonegroup
>>> info: ret -2 (2) No such file or directory
>>> couldn't init storage provider
>>>
>>> do i have to change some settings, also for upgrade of the radosgw?
>>
>> Hi,
>>
>> Testing a recent master build (with only default region and zone),
>> I'm able to successfully run the command you specified:
>>
>> % ./radosgw-admin user info --uid="testid" --rgw-region=default 
>> --rgw-zone=default
>> ...
>> {
>> "user_id": "testid",
>> "display_name": "M. Tester",
>> ...
>> }
>>
>> Are you certain the region and zone you specified exist?
>>
>> What do the following report:
>>
>> radosgw-admin zone list
>> radosgw-admin region list
>>
>> --
>> Regards,
>> Karol
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] RadosGW not start after upgrade to Jewel

[Ansgar Jazdzewski]

Hi all,

i got an answer, that pointed me to:
https://github.com/ceph/ceph/blob/master/doc/radosgw/multisite.rst

2016-04-25 16:02 GMT+02:00 Karol Mroz :
> On Mon, Apr 25, 2016 at 02:23:28PM +0200, Ansgar Jazdzewski wrote:
>> Hi,
>>
>> we test Jewel in our  QA environment (from Infernalis to Hammer) the
>> upgrade went fine but the Radosgw did not start.
>>
>> the error appears also with radosgw-admin
>>
>> # radosgw-admin user info --uid="images" --rgw-region=eu --rgw-zone=eu-qa
>> 2016-04-25 12:13:33.425481 7fc757fad900  0 error in read_id for id  :
>> (2) No such file or directory
>> 2016-04-25 12:13:33.425494 7fc757fad900  0 failed reading zonegroup
>> info: ret -2 (2) No such file or directory
>> couldn't init storage provider
>>
>> do i have to change some settings, also for upgrade of the radosgw?
>
> Hi,
>
> Testing a recent master build (with only default region and zone),
> I'm able to successfully run the command you specified:
>
> % ./radosgw-admin user info --uid="testid" --rgw-region=default 
> --rgw-zone=default
> ...
> {
> "user_id": "testid",
> "display_name": "M. Tester",
> ...
> }
>
> Are you certain the region and zone you specified exist?
>
> What do the following report:
>
> radosgw-admin zone list
> radosgw-admin region list
>
> --
> Regards,
> Karol
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph OSD down+out =>health ok => remove =>PGsbackfilling... ?

[Burkhard Linke]

Hi,

On 04/26/2016 12:32 PM, SCHAER Frederic wrote:


Hi,

One simple/quick question.

In my ceph cluster, I had a disk wich was in predicted failure. It was 
so much in predicted failure that the ceph OSD daemon crashed.


After the OSD crashed, ceph moved data correctly (or at least that’s 
what I thought), and a ceph –s was giving a “HEALTH_OK”.


Perfect.

I tride to tell ceph to mark the OSD down : it told me the OSD was 
already down… fine.


Then I ran this :

ID=43 ; ceph osd down $ID ; ceph auth del osd.$ID ; ceph osd rm $ID ; 
ceph osd crush remove osd.$ID




*snipsnap*

Removing the dead OSD entry changed the CRUSH weight for the host, 
resulting in a redistribution of the data.


The easiest way to prevent this is setting the CRUSH weight to 0.0 
before removing the OSD. This will trigger backfilling only once.


Regards,
Burkhard.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph OSD down+out =>health ok => remove => PGs backfilling... ?

[SCHAER Frederic]

Hi,

One simple/quick question.
In my ceph cluster, I had a disk wich was in predicted failure. It was so much 
in predicted failure that the ceph OSD daemon crashed.

After the OSD crashed, ceph moved data correctly (or at least that's what I 
thought), and a ceph -s was giving a "HEALTH_OK".
Perfect.
I tride to tell ceph to mark the OSD down : it told me the OSD was already 
down... fine.

Then I ran this :
ID=43 ; ceph osd down $ID ; ceph auth del osd.$ID ; ceph osd rm $ID ; ceph osd 
crush remove osd.$ID

And immediately after this, ceph told me :
# ceph -s
cluster 70ac4a78-46c0-45e6-8ff9-878b37f50fa1
 health HEALTH_WARN
37 pgs backfilling
3 pgs stuck unclean
recovery 12086/355688 objects misplaced (3.398%)
 monmap e2: 3 mons at 
{ceph0=192.54.207.70:6789/0,ceph1=192.54.207.71:6789/0,ceph2=192.54.207.72:6789/0}
election epoch 938, quorum 0,1,2 ceph0,ceph1,ceph2
 mdsmap e64: 1/1/1 up {0=ceph1=up:active}, 1 up:standby-replay, 1 up:standby
 osdmap e25455: 119 osds: 119 up, 119 in; 35 remapped pgs
  pgmap v5473702: 3212 pgs, 10 pools, 378 GB data, 97528 objects
611 GB used, 206 TB / 207 TB avail
12086/355688 objects misplaced (3.398%)
3175 active+clean
  37 active+remapped+backfilling
  client io 192 kB/s rd, 1352 kB/s wr, 117 op/s

Off course, I'm sure the OSD 43 was the one that was down ;)
My question therefore is :

If ceph successfully and automatically migrated data off the down/out OSD, why 
is there even anything happening once I tell ceph to forget about this osd ?
Was the cluster not "HEALTH OK" after all ?

(ceph-0.94.6-0.el7.x86_64 for now)

Thanks && regards

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph cache tier, flushed objects does not appear to be written on disk

[Benoît LORIOT]

Hi Greg,

yes directory is hashed four levels deep and contain files


# ls -l /var/lib/ceph/osd/ceph-1/current/1.0_head/DIR_0/DIR_0/DIR_0/DIR_0/

total 908

-rw-r--r--. 1 root root 601 Mar 15 15:01
1021bdf.__head_E5BD__1

-rw-r--r--. 1 root root 178571 Mar 15 15:06
1026de5.__head_7E28__1

-rw-r--r--. 1 root root 7853 Mar 15 15:16
1030a58.__head_0DA4__1

-rw-r--r--. 1 root root 0 Mar 17 12:51
10c1ccc.__head_D073__1

-rw-r--r--. 1 root root 0 Mar 17 17:23
10fc00c.__head_8302__1

-rw-r--r--. 1 root root 0 Mar 17 17:23
10fcae7.__head_C3A7__1

[...]



2016-04-25 17:27 GMT+02:00 Gregory Farnum :

> On Thursday, April 21, 2016, Benoît LORIOT  wrote:
>
>> Hello,
>>
>> we want to disable readproxy cache tier but before doing so we would like
>> to make sure we won't loose data.
>>
>> Is there a way to confirm that flush actually write objects to disk ?
>>
>> We're using ceph version 0.94.6.
>>
>>
>> I tried that, with cephfs_data_ro_cache being the hot storage pool and
>> cephfs_data being the cold storage pool
>>
>> # rados -p cephfs_data_ro_cache ls
>>
>> then choose a random object in the list : 14d6142.
>>
>> Find the object on cache disk :
>>
>> # ceph osd map cephfs_data_ro_cache 14d6142.
>> osdmap e301 pool 'cephfs_data_ro_cache' (6) object '14d6142.'
>> -> pg 6.d01 (6.0) -> up ([4,5,8], p4) acting ([4,5,8], p4)
>>
>> Object is in the pg 6.0 on OSD 4, 5 and 8, I can find the file on disk.
>>
>> # ls -l
>> /var/lib/ceph/osd/ceph-4/current/6.0_head/DIR_0/DIR_0/DIR_0/DIR_0/14d6142.__head_0D01__6
>> -rw-r--r--. 1 root root 0 Apr  8 19:36
>> /var/lib/ceph/osd/ceph-4/current/6.0_head/DIR_0/DIR_0/DIR_0/DIR_0/14d6142.__head_0D01__6
>>
>> Flush the object :
>>
>> # rados -p cephfs_data_ro_cache cache-try-flush 14d6142.
>>
>> Find the object on disk :
>>
>> # ceph osd map cephfs_data 14d6142.
>> osdmap e301 pool 'cephfs_data' (1) object '14d6142.' -> pg
>> 1.d01 (1.0) -> up ([1,7,2], p1) acting ([1,7,2], p1)
>>
>> Object is in the pg 1.0 on OSD 1, 7 and 2, I can't find the file on disk
>> on any of the 3 OSDs
>>
>> # ls -l
>> /var/lib/ceph/osd/ceph-1/current/1.0_head/DIR_0/DIR_0/DIR_0/DIR_0/14d6142.*
>> ls: cannot access
>> /var/lib/ceph/osd/ceph-1/current/1.0_head/DIR_0/DIR_0/DIR_0/DIR_0/14d6142.*:
>> No such file or directory
>>
>>
>>
>> What am I doing wrong ? To me it seems that nothing is actually flushed
>> to disk.
>>
>>
> Is the directory actually hashed four levels deep? Or is it shallower and
> you're looking too far down?
> -Greg
>
>
>> Thank you,
>> Ben.
>>
>


-- 


Cordialement,



*Benoit LORIOT*



*Ligne directe: 07 85 22 44 57*

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] increase pgnum after adjust reweight osd

[lin zhou]

Thanks Christian.
this cluster has 7 nodes with 69 osds.
I know this version is so old,but its hard to stop service to upgrade.

and I will increase it slowly with step 100.
Thanks again.


2016-04-25 15:55 GMT+08:00 Christian Balzer :
>
> Hello,
>
> On Mon, 25 Apr 2016 13:23:04 +0800 lin zhou wrote:
>
>> Hi,Cephers:
>>
>> Recently,I face a problem of full.and I have using reweight to adjust it.
>> But now I want to increase pgnum before I can add new nodes into the
>> cluster.
>>
> How many more nodes, OSDs?
>
>> current pg_num is 2048,and total OSD is 69.I want to increase to 4096.
>>
>> so what's the recommended steps:one-time increase derectly to 4096 or
>> increase it slowly,eg.increase 200 each time.
>>
>> ceph version is:0.67.8.and I use rbd pool only.
>>
> That's quite old, there are AFAIK some changes in current Ceph versions
> that improve data placement.
>
> Also current versions won't allow you to make large changes to pg_num
> because the massive and prolonged impact that can have.
>
> So you're better off to do it in small steps, unless you can afford your
> cluster having poor performance for a long time.
>
>>  the form of osdid,osd pg_um,osd reweight and osd used is below:
>>
> It's still quite uneven, I'd be worried that any OSD with more that 85%
> utilization might become near_full or even full during the data
> re-balancing.
>
> Christian
>> dumped all in format plain
>> OSD weight  pgnum   used
>> 0   0.89106 83%
>> 1   1   102 73%
>> 2   0.9 104 87%
>> 3   0.9192  107 80%
>> 5   0.89106 85%
>> 6   0.9271  108 82%
>> 7   1   112 77%
>> 8   0.9477  113 82%
>> 9   1   112 78%
>> 10  0.9177  109 79%
>> 11  1   108 76%
>> 12  0.9266  109 84%
>> 13  1   105 75%
>> 14  0.846   103 80%
>> 15  0.91109 80%
>> 16  1   99   68%
>> 17  1   108 79%
>> 18  1   109 77%
>> 19  0.8506  109 84%
>> 20  0.9504  111 79%
>> 21  1   95   71%
>> 22  0.9178  106 76%
>> 23  1   108 76%
>> 24  0.9274  118 82%
>> 25  0.923   117 86%
>> 26  1   107 76%
>> 27  1   111 80%
>> 28  0.9254  101 80%
>> 29  0.9445  104 82%
>> 30  1   115 81%
>> 31  0.9285  105 75%
>> 32  0.7823  105 81%
>> 33  0.9002  111 81%
>> 34  0.8024  106 79%
>> 35  1   100 71%
>> 36  1   117 81%
>> 37  0.7949  106 79%
>> 38  0.9356  108 78%
>> 39  0.866   106 76%
>> 40  0.8322  105 76%
>> 41  0.9297  97   81%
>> 42  1   97   68%
>> 43  0.8393  115 81%
>> 44  0.9355  108 78%
>> 45  0.8429  115 84%
>> 46  1   100 71%
>> 47  1   105 73%
>> 48  0.9476  109 80%
>> 49  1   117 82%
>> 50  0.8642  100 74%
>> 51  1   101 76%
>> 56  1   104 77%
>> 57  1   102 70%
>> 62  1   106 79%
>> 63  0.9332  99 82%
>> 68  1   103 76%
>> 69  1   100 71%
>> 74  1   105 77%
>> 75  1   104 80%
>> 80  1   101 73%
>> 81  1   112 78%
>> 86  0.866   104 76%
>> 87  1   97   70%
>> 92  1   104 79%
>> 93  0.9464  102 75%
>> 98  0.9082  113 80%
>> 99  1   108 77%
>> 104 1   107 79%
>> 105 1   109 77%
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
> --
> Christian BalzerNetwork/Systems Engineer
> ch...@gol.com   Global OnLine Japan/Rakuten Communications
> http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] CEPH All OSD got segmentation fault after CRUSH edit

[Henrik Svensson]

Hi!

We got a three node CEPH cluster with 10 OSD each.

We bought 3 new machines with additional 30 disks that should reside in another 
location.
Before adding these machines we modified the default CRUSH table.

After modifying the (default) crush table with these commands the cluster went 
down:


ceph osd crush add-bucket dc1 datacenter
ceph osd crush add-bucket dc2 datacenter
ceph osd crush add-bucket availo datacenter
ceph osd crush move dc1 root=default
ceph osd crush move lkpsx0120 root=default datacenter=dc1
ceph osd crush move lkpsx0130 root=default datacenter=dc1
ceph osd crush move lkpsx0140 root=default datacenter=dc1
ceph osd crush move dc2 root=default
ceph osd crush move availo root=default
ceph osd crush add-bucket sectra root
ceph osd crush move dc1 root=sectra
ceph osd crush move dc2 root=sectra
ceph osd crush move dc3 root=sectra
ceph osd crush move availo root=sectra
ceph osd crush remove default


I tried to revert the CRUSH map but no luck:


ceph osd crush add-bucket default root
ceph osd crush move lkpsx0120 root=default
ceph osd crush move lkpsx0130 root=default
ceph osd crush move lkpsx0140 root=default
ceph osd crush remove sectra


After trying to restart the cluster (and even the machines) no OSD started up 
again.
But ceph osd tree gave this output, stating certain OSD:s are up (but the 
processes are not running):


# id weight type name up/down reweight
-1 163.8 root default
-2 54.6 host lkpsx0120
0 5.46 osd.0 down 0
1 5.46 osd.1 down 0
2 5.46 osd.2 down 0
3 5.46 osd.3 down 0
4 5.46 osd.4 down 0
5 5.46 osd.5 down 0
6 5.46 osd.6 down 0
7 5.46 osd.7 down 0
8 5.46 osd.8 down 0
9 5.46 osd.9 down 0
-3 54.6 host lkpsx0130
10 5.46 osd.10 down 0
11 5.46 osd.11 down 0
12 5.46 osd.12 down 0
13 5.46 osd.13 down 0
14 5.46 osd.14 down 0
15 5.46 osd.15 down 0
16 5.46 osd.16 down 0
17 5.46 osd.17 down 0
18 5.46 osd.18 up 1
19 5.46 osd.19 up 1
-4 54.6 host lkpsx0140
20 5.46 osd.20 up 1
21 5.46 osd.21 down 0
22 5.46 osd.22 down 0
23 5.46 osd.23 down 0
24 5.46 osd.24 down 0
25 5.46 osd.25 up 1
26 5.46 osd.26 up 1
27 5.46 osd.27 up 1
28 5.46 osd.28 up 1
29 5.46 osd.29 up 1


The monitor starts/restarts OK (only one monitor exists).
But when starting one OSD with ceph -w nothing shows.

Here is the ceph mon_status:


{ "name": "lkpsx0120",
  "rank": 0,
  "state": "leader",
  "election_epoch": 1,
  "quorum": [
0],
  "outside_quorum": [],
  "extra_probe_peers": [],
  "sync_provider": [],
  "monmap": { "epoch": 4,
  "fsid": "9244194a-5e10-47ae-9287-507944612f95",
  "modified": "0.00",
  "created": "0.00",
  "mons": [
{ "rank": 0,
  "name": "lkpsx0120",
  "addr": "10.15.2.120:6789\/0"}]}}


Here is the ceph.conf file


[global]
fsid = 9244194a-5e10-47ae-9287-507944612f95
mon_initial_members = lkpsx0120
mon_host = 10.15.2.120
#debug osd = 20
#debug ms = 1
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
filestore_xattr_use_omap = true
osd_crush_chooseleaf_type = 1
osd_pool_default_size = 2
public_network = 10.15.2.0/24
cluster_network = 10.15.4.0/24
rbd_cache = true
rbd_cache_size = 67108864
rbd_cache_max_dirty = 50331648
rbd_cache_target_dirty = 33554432
rbd_cache_max_dirty_age = 2
rbd_cache_writethrough_until_flush = true


Here is the decompiled crush map:


# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1

# devices
device 0 osd.0
device 1 osd.1
device 2 osd.2
device 3 osd.3
device 4 osd.4
device 5 osd.5
device 6 osd.6
device 7 osd.7
device 8 osd.8
device 9 osd.9
device 10 osd.10
device 11 osd.11
device 12 osd.12
device 13 osd.13
device 14 osd.14
device 15 osd.15
device 16 osd.16
device 17 osd.17
device 18 osd.18
device 19 osd.19
device 20 osd.20
device 21 osd.21
device 22 osd.22
device 23 osd.23
device 24 osd.24
device 25 osd.25
device 26 osd.26
device 27 osd.27
device 28 osd.28
device 29 osd.29

# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root

# buckets
host lkpsx0120 {
id -2 # do not change unnecessarily
# weight 54.600
alg straw
hash 0 # rjenkins1
item osd.0 weight 5.460
item osd.1 weight 5.460
item osd.2 weight 5.460
item osd.3 weight 5.460
item osd.4 weight 5.460
item osd.5 weight 5.460
item osd.6 weight 5.460
item osd.7 weight 5.460
item osd.8 weight 5.460
item osd.9 weight 5.460
}
host lkpsx0130 {
id -3 # do not change unnecessarily
# weight 54.600
alg straw
hash 0 # rjenkins1
item osd.10 weight 5.460
item osd.11 weight 5.460
item osd.12 weight 5.460
item osd.13 weight 5.460
item osd.14 weight 5.460
item osd.15 weight 5.460
item osd.16 weight 5.460
item osd.17 weight 5.460
item osd.18 weight 5.460
item osd.19 w

[ceph-users] How to configure NFS to access RADOSGW buckets

[WD_Hwang]

Hello:
  Are there any documents or examples to explain the configuration of NFS to 
access RADOSGW buckets?
Thanks a lot.

Best Regards,
WD

---
This email contains confidential or legally privileged information and is for 
the sole use of its intended recipient. 
Any unauthorized review, use, copying or distribution of this email or the 
content of this email is strictly prohibited.
If you are not the intended recipient, you may reply to the sender and should 
delete this e-mail immediately.
---
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] krbd map on Jewel, sysfs write failed when rbd map

[WD_Hwang]

Hello:
  Thanks for your suggestion.
  I have upgraded the client kernel to '3.19.0-58-generic'.  Now I can map the 
images, including format 1 and 2.

Best Regards,
WD

From: Stefan Lissmats [mailto:ste...@trimmat.se]
Sent: Tuesday, April 26, 2016 3:13 PM
To: WD Hwang/WHQ/Wistron; ceph-users-boun...@lists.ceph.com
Cc: ceph-users@lists.ceph.com
Subject: SV: [ceph-users] krbd map on Jewel, sysfs write failed when rbd map

Hello again!

I normally map with rbd map data03 -p block_data but your format should work.
Mabye it's woth a try anyway?
Is there any more explaining errors in dmesg after trying to map the image?
Maybe there could be some clue what's happening there?
Also is there any possibility to update the kernel? I see the kernel is quite 
old. In Ubuntu 14.04.4 the kernel version is now at 3.16.0-70-generic.

Regards
Stefan


Från: wd_hw...@wistron.com [wd_hw...@wistron.com]
Skickat: den 26 april 2016 08:30
Till: Stefan Lissmats; 
ceph-users-boun...@lists.ceph.com
Kopia: ceph-users@lists.ceph.com
Ämne: RE: [ceph-users] krbd map on Jewel, sysfs write failed when rbd map
Hello:
  Sorry for that I forgot paste the results of image format 1. And I still 
cannot mount the format 1 or 2 block on Ubuntu 14.04 client, which the kernel 
is 3.13.0-85-generic #129-Ubuntu.

##
# rbd create block_data/data03 -s 10G --image-format 1
rbd: image format 1 is deprecated

# rbd info block_data/data03
2016-04-26 14:21:47.033360 7fb74e438700 -1 librbd::image::OpenRequest: RBD 
image format 1 is deprecated. Please copy this image to image format 2.
rbd image 'data03':
size 10240 MB in 2560 objects
order 22 (4096 kB objects)
block_name_prefix: rb.0.541ab.2ae8944a
format: 1

# rbd map block_data/data03
rbd: sysfs write failed
rbd: map failed: (5) Input/output error

Thanks a lot.
WD

From: Stefan Lissmats [mailto:ste...@trimmat.se]
Sent: Tuesday, April 26, 2016 2:15 PM
To: WD Hwang/WHQ/Wistron; 
ceph-users-boun...@lists.ceph.com
Cc: ceph-users@lists.ceph.com
Subject: SV: [ceph-users] krbd map on Jewel, sysfs write failed when rbd map

Hello!

It seems you referring to an earlier message but i can't find it.
It doesn't look that you have created image format 1 images.
I have created images in Jewel (10.2.0 and also som erlier releases) with the 
switch --image-format 1 and seems to work perfectly even if it's a depreciated 
switch and causes warnings i have used it successfully with krbd and Ubuntu 
14.04.
The image format is going to stop workning i future releases as I understand så 
using this should probably just be used if you don't plan to use them for 
production for an extended time.

Regards
Stefan


Från: ceph-users [ceph-users-boun...@lists.ceph.com] för 
wd_hw...@wistron.com [wd_hw...@wistron.com]
Skickat: den 26 april 2016 04:32
Till: 
ceph-users-boun...@lists.ceph.com
Kopia: ceph-users@lists.ceph.com
Ämne: Re: [ceph-users] krbd map on Jewel, sysfs write failed when rbd map
Dear Cephers:
  I got the same issue under Ubuntu 14.04, even I try to use the image format 
'1'.
# modinfo rbd
filename:   /lib/modules/3.13.0-85-generic/kernel/drivers/block/rbd.ko
license:GPL
author: Jeff Garzik mailto:j...@garzik.org>>
description:rados block device
author: Yehuda Sadeh 
mailto:yeh...@hq.newdream.net>>
author: Sage Weil mailto:s...@newdream.net>>
author: Alex Elder mailto:el...@inktank.com>>
srcversion: 48BFBD5C3D31D799F01D218
depends:libceph
intree: Y
vermagic:   3.13.0-85-generic SMP mod_unload modversions
signer: Magrathea: Glacier signing key
sig_key:C6:33:E9:BF:A6:CA:49:D5:3D:2E:B5:25:6A:35:87:7D:04:F1:64:F8
sig_hashalgo:   sha512

##
# rbd info block_data/data01
rbd image 'data01':
size 10240 MB in 2560 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.540752ae8944a
format: 2
features:
flags:
# rbd map block_data/data01
rbd: sysfs write failed
rbd: map failed: (5) Input/output error

##
# rbd info block_data/data02
rbd image 'data02':
size 10240 MB in 2560 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.2aac0238e1f29
format: 2
features: layering
   flags:
# rbd map block_data/data02
rbd: sysfs write failed
rbd: map failed: (5) Input/output error

  Is there any new idea to solve this issue?

Thanks a lot,
WD

---

This email contains confidential or legally p

Re: [ceph-users] krbd map on Jewel, sysfs write failed when rbd map

[Stefan Lissmats]

Hello again!

I normally map with rbd map data03 -p block_data but your format should work.
Mabye it's woth a try anyway?
Is there any more explaining errors in dmesg after trying to map the image?
Maybe there could be some clue what's happening there?
Also is there any possibility to update the kernel? I see the kernel is quite 
old. In Ubuntu 14.04.4 the kernel version is now at 3.16.0-70-generic.

Regards
Stefan


Från: wd_hw...@wistron.com [wd_hw...@wistron.com]
Skickat: den 26 april 2016 08:30
Till: Stefan Lissmats; ceph-users-boun...@lists.ceph.com
Kopia: ceph-users@lists.ceph.com
Ämne: RE: [ceph-users] krbd map on Jewel, sysfs write failed when rbd map

Hello:
  Sorry for that I forgot paste the results of image format 1. And I still 
cannot mount the format 1 or 2 block on Ubuntu 14.04 client, which the kernel 
is 3.13.0-85-generic #129-Ubuntu.

##
# rbd create block_data/data03 -s 10G --image-format 1
rbd: image format 1 is deprecated

# rbd info block_data/data03
2016-04-26 14:21:47.033360 7fb74e438700 -1 librbd::image::OpenRequest: RBD 
image format 1 is deprecated. Please copy this image to image format 2.
rbd image 'data03':
size 10240 MB in 2560 objects
order 22 (4096 kB objects)
block_name_prefix: rb.0.541ab.2ae8944a
format: 1

# rbd map block_data/data03
rbd: sysfs write failed
rbd: map failed: (5) Input/output error

Thanks a lot.
WD

From: Stefan Lissmats [mailto:ste...@trimmat.se]
Sent: Tuesday, April 26, 2016 2:15 PM
To: WD Hwang/WHQ/Wistron; ceph-users-boun...@lists.ceph.com
Cc: ceph-users@lists.ceph.com
Subject: SV: [ceph-users] krbd map on Jewel, sysfs write failed when rbd map

Hello!

It seems you referring to an earlier message but i can't find it.
It doesn't look that you have created image format 1 images.
I have created images in Jewel (10.2.0 and also som erlier releases) with the 
switch --image-format 1 and seems to work perfectly even if it's a depreciated 
switch and causes warnings i have used it successfully with krbd and Ubuntu 
14.04.
The image format is going to stop workning i future releases as I understand så 
using this should probably just be used if you don't plan to use them for 
production for an extended time.

Regards
Stefan


Från: ceph-users [ceph-users-boun...@lists.ceph.com] för 
wd_hw...@wistron.com [wd_hw...@wistron.com]
Skickat: den 26 april 2016 04:32
Till: 
ceph-users-boun...@lists.ceph.com
Kopia: ceph-users@lists.ceph.com
Ämne: Re: [ceph-users] krbd map on Jewel, sysfs write failed when rbd map
Dear Cephers:
  I got the same issue under Ubuntu 14.04, even I try to use the image format 
‘1’.
# modinfo rbd
filename:   /lib/modules/3.13.0-85-generic/kernel/drivers/block/rbd.ko
license:GPL
author: Jeff Garzik mailto:j...@garzik.org>>
description:rados block device
author: Yehuda Sadeh 
mailto:yeh...@hq.newdream.net>>
author: Sage Weil mailto:s...@newdream.net>>
author: Alex Elder mailto:el...@inktank.com>>
srcversion: 48BFBD5C3D31D799F01D218
depends:libceph
intree: Y
vermagic:   3.13.0-85-generic SMP mod_unload modversions
signer: Magrathea: Glacier signing key
sig_key:C6:33:E9:BF:A6:CA:49:D5:3D:2E:B5:25:6A:35:87:7D:04:F1:64:F8
sig_hashalgo:   sha512

##
# rbd info block_data/data01
rbd image 'data01':
size 10240 MB in 2560 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.540752ae8944a
format: 2
features:
flags:
# rbd map block_data/data01
rbd: sysfs write failed
rbd: map failed: (5) Input/output error

##
# rbd info block_data/data02
rbd image 'data02':
size 10240 MB in 2560 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.2aac0238e1f29
format: 2
features: layering
   flags:
# rbd map block_data/data02
rbd: sysfs write failed
rbd: map failed: (5) Input/output error

  Is there any new idea to solve this issue?

Thanks a lot,
WD

---

This email contains confidential or legally privileged information and is for 
the sole use of its intended recipient.

Any unauthorized review, use, copying or distribution of this email or the 
content of this email is strictly prohibited.

If you are not the intended recipient, you may reply to the sender and should 
delete this e-mail immediately.

---
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph