Re: [ceph-users] Upgrading and lost OSDs

[Brent Kennedy]

Update on this:

 

I was able to link the block softlink back to the original for each offline 
drive.  I used the ceph-bluestore-tool with show-label  “ceph-bluestore-tool 
show-label --dev /dev/disk/by-partuuid/” on each drive ( apparently the newer 
commands link them as ceph-uuid, but these were created with luminous and 
ceph-deploy 1.5.9 ).  I added the keyring, fsid, ceph-fsid, type and whomai 
files to each directory and set the owner as ceph.  The bluestore tool output 
has all the required data for each of those files.  Then just started the 
service.  All 12 disks came back up this way.

 

What worries me here is that there are other files in the 
/var/lib/ceph/osd/ceph- folders on other servers of the same version 
creation date.  Most of them contain simple content.  Wonder why files are used 
instead of a single json config file?

 

Thoughts?

 

-Brent

 

 

 

From: ceph-users  On Behalf Of Brent Kennedy
Sent: Friday, November 22, 2019 6:47 PM
To: 'Alfredo Deza' ; 'Bob R' 
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Upgrading and lost OSDs

 

I just ran into this today with a server we rebooted.  The server has been 
upgraded to Nautilus 14.2.2 for a few months.  Was originally installed as 
Jewel, then upgraded to Luminous ( then Nautilus ).   I have a whole server 
where all 12 OSDs have empty folders.  I recreated the keyring file and the 
type file, but now I have the “bluestore(/var/lib/ceph/osd/ceph-80/block) 
_read_bdev_label failed to open /var/lib/ceph/osd/ceph-80/block: (2) No such 
file or directory”  error.

 

Nov 22 23:10:58 ukosdhost15 systemd: Starting Ceph object storage daemon 
osd.80...

Nov 22 23:10:58 ukosdhost15 systemd: Started Ceph object storage daemon osd.80.

Nov 22 23:10:58 ukosdhost15 ceph-osd: 2019-11-22 23:10:58.662 7f86ebe92d80 -1 
bluestore(/var/lib/ceph/osd/ceph-80/block) _read_bdev_label failed to open 
/var/lib/ceph/osd/ceph-80/block: (2) No such file or directory

Nov 22 23:10:58 ukosdhost15 ceph-osd: 2019-11-22 23:10:58.662 7f86ebe92d80 -1 
#033[0;31m ** ERROR: unable to open OSD superblock on 
/var/lib/ceph/osd/ceph-80: (2) No such file or directory#033[0m

Nov 22 23:10:58 ukosdhost15 systemd:  <mailto:ceph-osd@80.service> 
ceph-osd@80.service: main process exited, code=exited, status=1/FAILURE

 

Were you able to restore those OSDs?  I was adding 24 more OSDs when a network 
issue occurred and this server was rebooted as part of that ( and the OSDs died 
on it ).

 

-Brent

 

From: ceph-users < <mailto:ceph-users-boun...@lists.ceph.com> 
ceph-users-boun...@lists.ceph.com> On Behalf Of Alfredo Deza
Sent: Friday, July 26, 2019 12:48 PM
To: Bob R < <mailto:b...@drinksbeer.org> b...@drinksbeer.org>
Cc:  <mailto:ceph-users@lists.ceph.com> ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Upgrading and lost OSDs

 

 

 

On Thu, Jul 25, 2019 at 7:00 PM Bob R mailto:b...@drinksbeer.org> > wrote:

I would try 'mv /etc/ceph/osd{,.old}' then run 'ceph-volume  simple scan' 
again. We had some problems upgrading due to OSDs (perhaps initially installed 
as firefly?) missing the 'type' attribute and iirc the 'ceph-volume simple 
scan' command refused to overwrite existing json files after I made some 
changes to ceph-volume. 

 

Ooof. I could swear that this issue was fixed already and it took me a while to 
find out that it wasn't at all. We saw this a few months ago in our Long 
Running Cluster used for dogfooding. 

 

I've created a ticket to track this work at http://tracker.ceph.com/issues/40987

 

But what you've done is exactly why we chose to persist the JSON files in 
/etc/ceph/osd/*.json, so that an admin could tell if anything is missing (or 
incorrect like in this case) and make the changes needed.

 

 

 

Bob

 

On Wed, Jul 24, 2019 at 1:24 PM Alfredo Deza mailto:ad...@redhat.com> > wrote:

 

 

On Wed, Jul 24, 2019 at 4:15 PM Peter Eisch mailto:peter.ei...@virginpulse.com> > wrote:

Hi,

 

I appreciate the insistency that the directions be followed.  I wholly agree.  
The only liberty I took was to do a ‘yum update’ instead of just ‘yum update 
ceph-osd’ and then reboot.  (Also my MDS runs on the MON hosts, so it got 
update a step early.)

 

As for the logs:

 

[2019-07-24 15:07:22,713][ceph_volume.main][INFO  ] Running command: 
ceph-volume  simple scan

[2019-07-24 15:07:22,714][ceph_volume.process][INFO  ] Running command: 
/bin/systemctl show --no-pager --property=Id --state=running ceph-osd@*

[2019-07-24 15:07:27,574][ceph_volume.main][INFO  ] Running command: 
ceph-volume  simple activate --all

[2019-07-24 15:07:27,575][ceph_volume.devices.simple.activate][INFO  ] 
activating OSD specified in 
/etc/ceph/osd/0-93fb5f2f-0273-4c87-a718-886d7e6db983.json

[2019-07-24 15:07:27,576][ceph_volume.devices.simple.activate][ERROR ] Required 
devices (block and data) not present for bluestore

[2019-07-24 15:07:27,576][ceph_volume.devices.simple.activate][ERROR ] 

Re: [ceph-users] Upgrading and lost OSDs

[Brent Kennedy]

I just ran into this today with a server we rebooted.  The server has been 
upgraded to Nautilus 14.2.2 for a few months.  Was originally installed as 
Jewel, then upgraded to Luminous ( then Nautilus ).   I have a whole server 
where all 12 OSDs have empty folders.  I recreated the keyring file and the 
type file, but now I have the “bluestore(/var/lib/ceph/osd/ceph-80/block) 
_read_bdev_label failed to open /var/lib/ceph/osd/ceph-80/block: (2) No such 
file or directory”  error.

 

Nov 22 23:10:58 ukosdhost15 systemd: Starting Ceph object storage daemon 
osd.80...

Nov 22 23:10:58 ukosdhost15 systemd: Started Ceph object storage daemon osd.80.

Nov 22 23:10:58 ukosdhost15 ceph-osd: 2019-11-22 23:10:58.662 7f86ebe92d80 -1 
bluestore(/var/lib/ceph/osd/ceph-80/block) _read_bdev_label failed to open 
/var/lib/ceph/osd/ceph-80/block: (2) No such file or directory

Nov 22 23:10:58 ukosdhost15 ceph-osd: 2019-11-22 23:10:58.662 7f86ebe92d80 -1 
#033[0;31m ** ERROR: unable to open OSD superblock on 
/var/lib/ceph/osd/ceph-80: (2) No such file or directory#033[0m

Nov 22 23:10:58 ukosdhost15 systemd: ceph-osd@80.service: main process exited, 
code=exited, status=1/FAILURE

 

Were you able to restore those OSDs?  I was adding 24 more OSDs when a network 
issue occurred and this server was rebooted as part of that ( and the OSDs died 
on it ).

 

-Brent

 

From: ceph-users  On Behalf Of Alfredo Deza
Sent: Friday, July 26, 2019 12:48 PM
To: Bob R 
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Upgrading and lost OSDs

 

 

 

On Thu, Jul 25, 2019 at 7:00 PM Bob R mailto:b...@drinksbeer.org> > wrote:

I would try 'mv /etc/ceph/osd{,.old}' then run 'ceph-volume  simple scan' 
again. We had some problems upgrading due to OSDs (perhaps initially installed 
as firefly?) missing the 'type' attribute and iirc the 'ceph-volume simple 
scan' command refused to overwrite existing json files after I made some 
changes to ceph-volume. 

 

Ooof. I could swear that this issue was fixed already and it took me a while to 
find out that it wasn't at all. We saw this a few months ago in our Long 
Running Cluster used for dogfooding. 

 

I've created a ticket to track this work at http://tracker.ceph.com/issues/40987

 

But what you've done is exactly why we chose to persist the JSON files in 
/etc/ceph/osd/*.json, so that an admin could tell if anything is missing (or 
incorrect like in this case) and make the changes needed.

 

 

 

Bob

 

On Wed, Jul 24, 2019 at 1:24 PM Alfredo Deza mailto:ad...@redhat.com> > wrote:

 

 

On Wed, Jul 24, 2019 at 4:15 PM Peter Eisch mailto:peter.ei...@virginpulse.com> > wrote:

Hi,

 

I appreciate the insistency that the directions be followed.  I wholly agree.  
The only liberty I took was to do a ‘yum update’ instead of just ‘yum update 
ceph-osd’ and then reboot.  (Also my MDS runs on the MON hosts, so it got 
update a step early.)

 

As for the logs:

 

[2019-07-24 15:07:22,713][ceph_volume.main][INFO  ] Running command: 
ceph-volume  simple scan

[2019-07-24 15:07:22,714][ceph_volume.process][INFO  ] Running command: 
/bin/systemctl show --no-pager --property=Id --state=running ceph-osd@*

[2019-07-24 15:07:27,574][ceph_volume.main][INFO  ] Running command: 
ceph-volume  simple activate --all

[2019-07-24 15:07:27,575][ceph_volume.devices.simple.activate][INFO  ] 
activating OSD specified in 
/etc/ceph/osd/0-93fb5f2f-0273-4c87-a718-886d7e6db983.json

[2019-07-24 15:07:27,576][ceph_volume.devices.simple.activate][ERROR ] Required 
devices (block and data) not present for bluestore

[2019-07-24 15:07:27,576][ceph_volume.devices.simple.activate][ERROR ] 
bluestore devices found: [u'data']

[2019-07-24 15:07:27,576][ceph_volume][ERROR ] exception caught by decorator

Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 59, 
in newfunc

return f(*a, **kw)

  File "/usr/lib/python2.7/site-packages/ceph_volume/main.py", line 148, in main

terminal.dispatch(self.mapper, subcommand_args)

  File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in 
dispatch

instance.main()

  File "/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/main.py", 
line 33, in main

terminal.dispatch(self.mapper, self.argv)

  File "/usr/lib/python2.7/site-packages/ceph_volume/terminal.py", line 182, in 
dispatch

instance.main()

  File 
"/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/activate.py", line 
272, in main

self.activate(args)

  File "/usr/lib/python2.7/site-packages/ceph_volume/decorators.py", line 16, 
in is_root

return func(*a, **kw)

  File 
"/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/activate.py", line 
131, in activate

self.validate_devices(osd_metadata)

  File 
"/usr/lib/python2.7/site-packages/ceph_volume/devices/simple/activate.py", line 
62, in validate_devices

raise RuntimeError('Unable to activate bluestore OSD due to 

Re: [ceph-users] NFS

[Brent Kennedy]

Awesome, thank you giving an overview of these features, sounds like the 
correct direction then!

-Brent

-Original Message-
From: Daniel Gryniewicz  
Sent: Thursday, October 3, 2019 8:20 AM
To: Brent Kennedy 
Cc: Marc Roos ; ceph-users 
Subject: Re: [ceph-users] NFS

So, Ganesha is an NFS gateway, living in userspace.  It provides access via NFS 
(for any NFS client) to a number of clustered storage systems, or to local 
filesystems on it's host.  It can run on any system that has access to the 
cluster (ceph in this case).  One Ganesha instance can serve quite a few 
clients (the limit typically being either memory on the Ganesha node or network 
bandwidth).
Ganesha's configuration lives in /etc/ganesha/ganesha.conf.  There should be 
man pages related to Ganesha and it's configuration installed when Ganesha is 
installed.

Ganesha has a number of FSALs (File System Abstraction Layers) that work with a 
number of different clustered storage systems.  For Ceph, Ganesha has 2 FSALs: 
FSAL_CEPH works on top of CephFS, and FSAL_RGW works on top of RadosGW.  
FSAL_CEPH provides full NFS semantics, sinces CephFS is a full POSIX 
filesystem; FSAL_RGW provides slightly limited semantics, since RGW itself it 
not POSIX and doesn't provide everything.  For example, you cannot write into 
an arbitrary location within a file, you can only overwrite the entire file.

Anything you can store in the underlying storage (CephFS or RadosGW) can be 
stored/accessed by Ganesha.  So, 20+GB files should work fine on either one.

Daniel

On Tue, Oct 1, 2019 at 10:45 PM Brent Kennedy  wrote:
>
> We might have to backup a step here so I can understand.  Are you 
> saying stand up a new VM with just those packages installed, then 
> configure the export file  ( the file location isn’t mentioned in the 
> ceph docs ) and supposedly a client can connect to them?  ( only linux 
> clients or any NFS client? )
>
> I don’t use cephFS, so being that it will be an object storage backend, will 
> that be ok with multiple hosts accessing files through the NFS one gateway or 
> should I configure multiple gateways ( one for each share )?
>
> I was hoping to save large files( 20+ GB ), should I stand up cephFS instead 
> for this?
>
> I am used to using a NAS storage appliance server(or freeNAS ), so 
> using ceph as a NAS backend is new to me ( thus I might be over 
> thinking this )  :)
>
> -Brent
>
> -Original Message-
> From: Daniel Gryniewicz 
> Sent: Tuesday, October 1, 2019 8:20 AM
> To: Marc Roos ; bkennedy 
> ; ceph-users 
> Subject: Re: [ceph-users] NFS
>
> Ganesha can export CephFS or RGW.  It cannot export anything else (like iscsi 
> or RBD).  Config for RGW looks like this:
>
> EXPORT
> {
>  Export_ID=1;
>  Path = "/";
>  Pseudo = "/rgw";
>  Access_Type = RW;
>  Protocols = 4;
>  Transports = TCP;
>  FSAL {
>  Name = RGW;
>  User_Id = "testuser";
>  Access_Key_Id ="";
>  Secret_Access_Key = "";
>  }
> }
>
> RGW {
>  ceph_conf = "//ceph.conf";
>  # for vstart cluster, name = "client.admin"
>  name = "client.rgw.foohost";
>  cluster = "ceph";
> #   init_args = "-d --debug-rgw=16";
> }
>
>
> Daniel
>
> On 9/30/19 3:01 PM, Marc Roos wrote:
> >
> > Just install these
> >
> > http://download.ceph.com/nfs-ganesha/
> > nfs-ganesha-rgw-2.7.1-0.1.el7.x86_64
> > nfs-ganesha-vfs-2.7.1-0.1.el7.x86_64
> > libnfsidmap-0.25-19.el7.x86_64
> > nfs-ganesha-mem-2.7.1-0.1.el7.x86_64
> > nfs-ganesha-xfs-2.7.1-0.1.el7.x86_64
> > nfs-ganesha-2.7.1-0.1.el7.x86_64
> > nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64
> >
> >
> > And export your cephfs like this:
> > EXPORT {
> >  Export_Id = 10;
> >  Path = /nfs/cblr-repos;
> >  Pseudo = /cblr-repos;
> >  FSAL { Name = CEPH; User_Id = "cephfs.nfs.cblr"; 
> > Secret_Access_Key = "xxx"; }
> >  Disable_ACL = FALSE;
> >  CLIENT { Clients = 192.168.10.2; access_type = "RW"; }
> >  CLIENT { Clients = 192.168.10.253; } }
> >
> >
> > -Original Message-
> > From: Brent Kennedy [mailto:bkenn...@cfl.rr.com]
> > Sent: maandag 30 september 2019 20:56
> > To: 'ceph-users'
> > Subject: [ceph-users] NFS
> >
> > Wondering if there are any documents for standing up NFS with an 
> > existing ceph cluster.  We don’t use ceph-ansible or any other tools 
> > besides ceph-deploy.  The iscsi dir

Re: [ceph-users] NFS

[Brent Kennedy]

We might have to backup a step here so I can understand.  Are you saying stand 
up a new VM with just those packages installed, then configure the export file  
( the file location isn’t mentioned in the ceph docs ) and supposedly a client 
can connect to them?  ( only linux clients or any NFS client? )

I don’t use cephFS, so being that it will be an object storage backend, will 
that be ok with multiple hosts accessing files through the NFS one gateway or 
should I configure multiple gateways ( one for each share )?  

I was hoping to save large files( 20+ GB ), should I stand up cephFS instead 
for this?

I am used to using a NAS storage appliance server(or freeNAS ), so using ceph 
as a NAS backend is new to me ( thus I might be over thinking this )  :)

-Brent

-Original Message-
From: Daniel Gryniewicz  
Sent: Tuesday, October 1, 2019 8:20 AM
To: Marc Roos ; bkennedy ; 
ceph-users 
Subject: Re: [ceph-users] NFS

Ganesha can export CephFS or RGW.  It cannot export anything else (like iscsi 
or RBD).  Config for RGW looks like this:

EXPORT
{
 Export_ID=1;
 Path = "/";
 Pseudo = "/rgw";
 Access_Type = RW;
 Protocols = 4;
 Transports = TCP;
 FSAL {
 Name = RGW;
 User_Id = "testuser";
 Access_Key_Id ="";
 Secret_Access_Key = "";
 }
}

RGW {
 ceph_conf = "//ceph.conf";
 # for vstart cluster, name = "client.admin"
 name = "client.rgw.foohost";
 cluster = "ceph";
#   init_args = "-d --debug-rgw=16";
}


Daniel

On 9/30/19 3:01 PM, Marc Roos wrote:
>   
> Just install these
> 
> http://download.ceph.com/nfs-ganesha/
> nfs-ganesha-rgw-2.7.1-0.1.el7.x86_64
> nfs-ganesha-vfs-2.7.1-0.1.el7.x86_64
> libnfsidmap-0.25-19.el7.x86_64
> nfs-ganesha-mem-2.7.1-0.1.el7.x86_64
> nfs-ganesha-xfs-2.7.1-0.1.el7.x86_64
> nfs-ganesha-2.7.1-0.1.el7.x86_64
> nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64
> 
> 
> And export your cephfs like this:
> EXPORT {
>  Export_Id = 10;
>  Path = /nfs/cblr-repos;
>  Pseudo = /cblr-repos;
>  FSAL { Name = CEPH; User_Id = "cephfs.nfs.cblr"; 
> Secret_Access_Key = "xxx"; }
>      Disable_ACL = FALSE;
>  CLIENT { Clients = 192.168.10.2; access_type = "RW"; }
>  CLIENT { Clients = 192.168.10.253; } }
> 
> 
> -Original Message-
> From: Brent Kennedy [mailto:bkenn...@cfl.rr.com]
> Sent: maandag 30 september 2019 20:56
> To: 'ceph-users'
> Subject: [ceph-users] NFS
> 
> Wondering if there are any documents for standing up NFS with an 
> existing ceph cluster.  We don’t use ceph-ansible or any other tools 
> besides ceph-deploy.  The iscsi directions were pretty good once I got 
> past the dependencies.
> 
>   
> 
> I saw the one based on Rook, but it doesn’t seem to apply to our setup 
> of ceph vms with physical hosts doing OSDs.  The official ceph 
> documents talk about using ganesha but doesn’t seem to dive into the 
> details of what the process is for getting it online.  We don’t use 
> cephfs, so that’s not setup either.  The basic docs seem to note this is 
> required.
>   Seems my google-fu is failing me when I try to find a more 
> definitive guide.
> 
>   
> 
> The servers are all centos 7 with the latest updates.
> 
>   
> 
> Any guidance would be greatly appreciated!
> 
>   
> 
> Regards,
> 
> -Brent
> 
>   
> 
> Existing Clusters:
> 
> Test: Nautilus 14.2.2 with 3 osd servers, 1 mon/man, 1 gateway, 2 
> iscsi gateways ( all virtual on nvme )
> 
> US Production(HDD): Nautilus 14.2.2 with 13 osd servers, 3 mons, 4 
> gateways, 2 iscsi gateways
> 
> UK Production(HDD): Nautilus 14.2.2 with 25 osd servers, 3 mons/man, 3 
> gateways behind
> 
> US Production(SSD): Nautilus 14.2.2 with 6 osd servers, 3 mons/man, 3 
> gateways, 2 iscsi gateways
> 
>   
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] NFS

[Brent Kennedy]

Wondering if there are any documents for standing up NFS with an existing
ceph cluster.  We don't use ceph-ansible or any other tools besides
ceph-deploy.  The iscsi directions were pretty good once I got past the
dependencies.  

 

I saw the one based on Rook, but it doesn't seem to apply to our setup of
ceph vms with physical hosts doing OSDs.  The official ceph documents talk
about using ganesha but doesn't seem to dive into the details of what the
process is for getting it online.  We don't use cephfs, so that's not setup
either.  The basic docs seem to note this is required.  Seems my google-fu
is failing me when I try to find a more definitive guide.

 

The servers are all centos 7 with the latest updates.

 

Any guidance would be greatly appreciated!

 

Regards,

-Brent

 

Existing Clusters:

Test: Nautilus 14.2.2 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi
gateways ( all virtual on nvme )

US Production(HDD): Nautilus 14.2.2 with 13 osd servers, 3 mons, 4 gateways,
2 iscsi gateways

UK Production(HDD): Nautilus 14.2.2 with 25 osd servers, 3 mons/man, 3
gateways behind

US Production(SSD): Nautilus 14.2.2 with 6 osd servers, 3 mons/man, 3
gateways, 2 iscsi gateways

 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Health Check error ( havent seen before ) [EXT]

[Brent Kennedy]

If the cluster I am working on now has the same issue, I will try that
instead.  I honestly didn’t try that first because two of the monitors
already had python-pip installed, so it was just running the “pip install
requests==2.6.0” command to fix those two monitors.   I did not check the
version before running that command though, but on the cluster I am working
on now, it does show as 2.9.1.  Since the cluster monitors I fixed were just
upgraded to 16.04, it makes me wonder if there is a problem with 2.9.1.

I am almost done with the OSDs on the other cluster I am upgrading to
nautilus, so once I fix the bluestore legacy message on all of those, I can
restart the gateways and try installing the new dashboard.  

The monitors that generated that error message about SSLv3 had been
installed at Ubuntu 12 originally.  I think the ones I am working on now
were originally Ubuntu 14.  Both sets of cluster monitors were upgraded to
16.04 though.  We plan to move them to CentOS once nautilus is fully in
place though.

-Brent



-Original Message-
From: Matthew Vernon  
Sent: Tuesday, July 30, 2019 5:01 AM
To: Brent Kennedy ; 'ceph-users'

Subject: Re: [ceph-users] Ceph Health Check error ( havent seen before )
[EXT]

On 29/07/2019 23:24, Brent Kennedy wrote:
> Apparently sent my email too quickly.  I had to install python-pip on 
> the mgr nodes and run “pip install requests==2.6.0” to fix the missing 
> module and then reboot all three monitors.  Now the dashboard enables 
> no issue.

I'm a bit confused as to why installing the python-requests package wasn't
the correct answer? 16.04 has 2.9.1

Regards,

Matthew



--
 The Wellcome Sanger Institute is operated by Genome Research  Limited, a
charity registered in England with number 1021457 and a  company registered
in England with number 2742969, whose registered  office is 215 Euston Road,
London, NW1 2BE. 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph Health Check error ( havent seen before )

[Brent Kennedy]

Apparently sent my email too quickly.  I had to install python-pip on the
mgr nodes and run "pip install requests==2.6.0" to fix the missing module
and then reboot all three monitors.  Now the dashboard enables no issue.  

 

This is apparently a known Ubuntu issue.  ( yet another reason to move to
CentOS, not that I mean to disparage any Ubuntu users out there J )

 

-Brent

 

From: ceph-users  On Behalf Of Brent
Kennedy
Sent: Monday, July 29, 2019 6:11 PM
To: 'ceph-users' 
Subject: [ceph-users] Ceph Health Check error ( havent seen before )

 

I upgraded a cluster to 14.2.1 a month ago and installed the
ceph-mgr-dashboard so I could see and use the new dashboard.  I then
upgraded the cluster again from 14.2.1 to 14.2.1 this past week and after
clearing all the usual notifications and upgrading the dashboard via apt, I
tried to enable the dashboard again, but now the health check shows an
error.  I also had to use the force function:

 

Error Message:

MGR_MODULE_DEPENDENCY Module 'dashboard' has failed dependency: 'module'
object has no attribute 'PROTOCOL_SSLv3'

Module 'dashboard' has failed dependency: 'module' object has no
attribute 'PROTOCOL_SSLv3'

 

I checked openssl and I am on version 1.0.2g  ( march 2016 ) on all three
monitor nodes.  The monitor nodes are running 16.04 with all packages
updated.  The open ssl site is showing 1.0.2s, so the version difference is
small, but there is one.  It appears Ubuntu wants us to go to 18, but I
would rather not do that.  All of the osd host servers are CentOS( we are
moving to centos from Ubuntu, slowly ).

 

I used "ceph mgr module disable dashboard" to disable the module, but the
error message persists.  I tried to enable the module with force, then
disable SSL with "ceph config set mgr mgr/dashboard/ssl false" but that
resulted in "Error EINVAL: unrecognized config option 'mgr/dashboard/ssl'".

 

Not sure if this will help(shows ssl disabled):

ceph config-key dump

{

"config-history/1/": "<<< binary blob of length 12 >>>",

"config-history/2/": "<<< binary blob of length 12 >>>",

"config-history/2/+mgr/mgr/devicehealth/enable_monitoring": "true",

"config-history/3/": "<<< binary blob of length 12 >>>",

"config-history/3/+mgr/mgr/dashboard/ssl": "false",

"config-history/4/": "<<< binary blob of length 12 >>>",

"config-history/4/+mgr/mgr/dashboard/ssl": "true",

"config-history/4/-mgr/mgr/dashboard/ssl": "false",

"config-history/5/": "<<< binary blob of length 12 >>>",

"config-history/5/+mgr/mgr/dashboard/ssl": "false",

"config-history/5/-mgr/mgr/dashboard/ssl": "true",

"config-history/6/": "<<< binary blob of length 12 >>>",

"config-history/6/+mgr/mgr/dashboard/RGW_API_ACCESS_KEY": "",

"config-history/7/": "<<< binary blob of length 12 >>>",

"config-history/7/+mgr/mgr/dashboard/RGW_API_SECRET_KEY": "",

"config/mgr/mgr/dashboard/RGW_API_ACCESS_KEY": "",

"config/mgr/mgr/dashboard/RGW_API_SECRET_KEY": "",

"config/mgr/mgr/dashboard/ssl": "false",

"config/mgr/mgr/devicehealth/enable_monitoring": "true",

"mgr/dashboard/accessdb_v1": "{\"version\": 1, \"users\": {\"ceph\":
{\"username\": \" \", \"lastUpdate\": 1560736662, \"name\": null, \"roles\":
[\"administrator\"], \"password\": \" \", \"email\": null}}, \"roles\":
{}}",

"mgr/dashboard/crt": "-BEGIN CERTIFICATE-END
CERTIFICATE-\n",

"mgr/dashboard/jwt_secret": "",

"mgr/dashboard/key": "-BEGIN PRIVATE KEY-END PRIVATE
KEY-\n",

"mgr/devicehealth/last_scrape": "20190729-000743"

}

 

We have another cluster with centos that was recently upgraded to 14.2.2
from Luminous, no issues with this, but the OS is CentOS 7.6.  so not
exactly the same.

 

At this point, the health is in a health warn.  trying to clear that.

 

Regards,

-Brent

 

Existing Clusters:

Test: Nautilus 14.2.2 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi
gateways ( all virtual on nvme )

US Production(HDD): Nautilus 14.2.2 with 11 osd servers, 3 mons, 4 gateways
behind haproxy LB

UK Production(HDD): Nautilus 14.2.1 with 25 osd servers, 3 mons/man, 3
gateways behind haproxy LB

US Production(SSD): Nautilus 14.2.1 with 6 osd servers, 3 mons/man, 3
gateways behind haproxy LB

 

 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph Health Check error ( havent seen before )

[Brent Kennedy]

I upgraded a cluster to 14.2.1 a month ago and installed the
ceph-mgr-dashboard so I could see and use the new dashboard.  I then
upgraded the cluster again from 14.2.1 to 14.2.1 this past week and after
clearing all the usual notifications and upgrading the dashboard via apt, I
tried to enable the dashboard again, but now the health check shows an
error.  I also had to use the force function:

 

Error Message:

MGR_MODULE_DEPENDENCY Module 'dashboard' has failed dependency: 'module'
object has no attribute 'PROTOCOL_SSLv3'

Module 'dashboard' has failed dependency: 'module' object has no
attribute 'PROTOCOL_SSLv3'

 

I checked openssl and I am on version 1.0.2g  ( march 2016 ) on all three
monitor nodes.  The monitor nodes are running 16.04 with all packages
updated.  The open ssl site is showing 1.0.2s, so the version difference is
small, but there is one.  It appears Ubuntu wants us to go to 18, but I
would rather not do that.  All of the osd host servers are CentOS( we are
moving to centos from Ubuntu, slowly ).

 

I used "ceph mgr module disable dashboard" to disable the module, but the
error message persists.  I tried to enable the module with force, then
disable SSL with "ceph config set mgr mgr/dashboard/ssl false" but that
resulted in "Error EINVAL: unrecognized config option 'mgr/dashboard/ssl'".

 

Not sure if this will help(shows ssl disabled):

ceph config-key dump

{

"config-history/1/": "<<< binary blob of length 12 >>>",

"config-history/2/": "<<< binary blob of length 12 >>>",

"config-history/2/+mgr/mgr/devicehealth/enable_monitoring": "true",

"config-history/3/": "<<< binary blob of length 12 >>>",

"config-history/3/+mgr/mgr/dashboard/ssl": "false",

"config-history/4/": "<<< binary blob of length 12 >>>",

"config-history/4/+mgr/mgr/dashboard/ssl": "true",

"config-history/4/-mgr/mgr/dashboard/ssl": "false",

"config-history/5/": "<<< binary blob of length 12 >>>",

"config-history/5/+mgr/mgr/dashboard/ssl": "false",

"config-history/5/-mgr/mgr/dashboard/ssl": "true",

"config-history/6/": "<<< binary blob of length 12 >>>",

"config-history/6/+mgr/mgr/dashboard/RGW_API_ACCESS_KEY": "",

"config-history/7/": "<<< binary blob of length 12 >>>",

"config-history/7/+mgr/mgr/dashboard/RGW_API_SECRET_KEY": "",

"config/mgr/mgr/dashboard/RGW_API_ACCESS_KEY": "",

"config/mgr/mgr/dashboard/RGW_API_SECRET_KEY": "",

"config/mgr/mgr/dashboard/ssl": "false",

"config/mgr/mgr/devicehealth/enable_monitoring": "true",

"mgr/dashboard/accessdb_v1": "{\"version\": 1, \"users\": {\"ceph\":
{\"username\": \" \", \"lastUpdate\": 1560736662, \"name\": null, \"roles\":
[\"administrator\"], \"password\": \" \", \"email\": null}}, \"roles\":
{}}",

"mgr/dashboard/crt": "-BEGIN CERTIFICATE-END
CERTIFICATE-\n",

"mgr/dashboard/jwt_secret": "",

"mgr/dashboard/key": "-BEGIN PRIVATE KEY-END PRIVATE
KEY-\n",

"mgr/devicehealth/last_scrape": "20190729-000743"

}

 

We have another cluster with centos that was recently upgraded to 14.2.2
from Luminous, no issues with this, but the OS is CentOS 7.6.  so not
exactly the same.

 

At this point, the health is in a health warn.  trying to clear that.

 

Regards,

-Brent

 

Existing Clusters:

Test: Nautilus 14.2.2 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi
gateways ( all virtual on nvme )

US Production(HDD): Nautilus 14.2.2 with 11 osd servers, 3 mons, 4 gateways
behind haproxy LB

UK Production(HDD): Nautilus 14.2.1 with 25 osd servers, 3 mons/man, 3
gateways behind haproxy LB

US Production(SSD): Nautilus 14.2.1 with 6 osd servers, 3 mons/man, 3
gateways behind haproxy LB

 

 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Iscsi in the nautilus Dashboard

[Brent Kennedy]

The devs came through and added a 3.0.1 ceph-iscsi release that works with 
14.2.2 ( detects as version 9 ).  I then went into the dashboard to add a 
target and then hit a wall.  I used the auto generated target IQN, then 
selected the two gateways as the “add portal” option and hit “create target” ( 
not selecting any other options as they were not marked as required ).  I then 
get an error message:  “Failed to execut iscsi:  Gateway creation failed on 
iscsi1.  Could not create target on gateway: Error initializing iSCSI target: 
gateway IP addresses provided do not match any ip on this host”.  

 

I tried adding the iqn with the local domain I am using ( via host files on 
each node ), but that didn’t change the error message.  Oddly enough, its 
adding the targets, but I cant edit any of them.  When I try to save them, it 
says I cannot delete any targets that don’t have any portals assigned ( I was 
trying to edit to add the portal, wasn’t even trying to delete ).

 

I was doing all of this in Chrome and then tried again in firefox but no 
change.  The dashboard is looking pretty fly, but it seems some things are not 
functional yet L 

 

I confirmed that none of the targets that show up in the dashboard show up with 
gwcli ls on the iscsi1 gateway.

 

I was able to create the same target directly via gwcli though ( no errors ) 
and it shows up in the dashboard.

 

-Brent

 

From: ceph-users  On Behalf Of Brent Kennedy
Sent: Tuesday, July 23, 2019 10:30 AM
To: 'Kaspar Bosma' ; 'Paul Emmerich' 

Cc: 'ceph-users' 
Subject: Re: [ceph-users] Iscsi in the nautilus Dashboard

 

I had installed 3.0 and its detected by the dashboard as version 8.  I checked 
the common.py in the source from the ceph-iscsi site and it shows as version 8 
on line 59(meaning I confirmed what the dashboard is saying).  From what I can 
tell, there is no version 9 posted to the ceph-iscsi github.  I looked at 
version 3.1 and version 3.2(newly posted) and they both show as version 10 on 
line 59.  Of course the dashboard wants 9…I was really hoping to see the 
dashboard in action L

 

Maybe I can adjust the version detection to make it work….  Wonder why they 
don’t support version “X and newer” ( probably a programming thing ).

 

-Brent

 

From: Kaspar Bosma mailto:kaspar.bo...@home.nl> > 
Sent: Tuesday, July 23, 2019 1:21 AM
To: Brent Kennedy mailto:bkenn...@cfl.rr.com> >; Paul 
Emmerich mailto:paul.emmer...@croit.io> >
Cc: ceph-users mailto:ceph-users@lists.ceph.com> >
Subject: RE: [ceph-users] Iscsi in the nautilus Dashboard

 

Hi Brent,

As far as I know version 3.0 (which I assume is version 9) is the minimum 
required for the dashboard.

I would go with the latest from Shaman; it won't break the actual iSCSI part of 
the setup, only maybe the iSCSI support in the dashboard. I haven't tried it 
myself, I'm still at version 2.7 (which is version 8 I would gather...)

Kaspar

Op 23 juli 2019 om 4:49 schreef Brent Kennedy mailto:bkenn...@cfl.rr.com> >: 

I posted to the ceph-iscsi github but Dillaman noted that 3.2 was version 10.  
Which means that wouldn’t solve the issue with the version 9 requirement of the 
current 14.2.2 nautilus.   Paul noted 3.1 is “pretty broken”, soo which version 
is version 9?  Or should I hack/patch the dashboard in 14.2.2 to accept version 
10 or 8? 

  

-Brent 

  

From: Kaspar Bosma mailto:kaspar.bo...@home.nl> > 
Sent: Monday, July 22, 2019 8:11 AM
To: Paul Emmerich mailto:paul.emmer...@croit.io> >; 
Brent Kennedy mailto:bkenn...@cfl.rr.com> >
Cc: ceph-users mailto:ceph-users@lists.ceph.com> >
Subject: Re: [ceph-users] Iscsi in the nautilus Dashboard 

 

Hi all,

That was not the most recent. This is it (3.2.4): 
https://2.chacra.ceph.com/r/ceph-iscsi/master/8a3967698257e1b49a9d554847b84418c15da902/centos/7/flavors/default/

Kaspar

Op 22 juli 2019 om 14:01 schreef Kaspar Bosma mailto:kaspar.bo...@home.nl> >:

Hi Brent,

You may want to have a look at the repos at shaman.ceph.com.

The latest (3.2.2) packaged version of Ceph iSCSI is located here:

https://4.chacra.ceph.com/r/ceph-iscsi/master/ff5e6873c43ab6828d3f7264526100b95a7e3954/centos/7/flavors/default/noarch/

You can also find related package repos for the tcmu-runner and ceph-iscsi-cli 
projects.

Regards, Kaspar

Op 22 juli 2019 om 12:52 schreef Paul Emmerich mailto:paul.emmer...@croit.io> >:

Version 9 is the fqdn stuff which was introduced in 3.1.

Use 3.2 as 3.1 is pretty broken.

 

Paul

 

-- 
Paul Emmerich 

Looking for help with your Ceph cluster? Contact us at https://croit.io 

croit GmbH 
Freseniusstr. 31h 
81247 München 
www.croit.io <http://www.croit.io>  
Tel: +49 89 1896585 90

 

 

On Mon, Jul 22, 2019 at 3:24 AM Brent Kennedy < bkenn...@cfl.rr.com 
<mailto:bkenn...@cfl.rr.com> > wrote:

I have a test cluster running centos 7.6 setup with two iscsi gateways ( per 
the requirement ).  I have the dashboard setup in 

Re: [ceph-users] Iscsi in the nautilus Dashboard

[Brent Kennedy]

I had installed 3.0 and its detected by the dashboard as version 8.  I checked 
the common.py in the source from the ceph-iscsi site and it shows as version 8 
on line 59(meaning I confirmed what the dashboard is saying).  From what I can 
tell, there is no version 9 posted to the ceph-iscsi github.  I looked at 
version 3.1 and version 3.2(newly posted) and they both show as version 10 on 
line 59.  Of course the dashboard wants 9…I was really hoping to see the 
dashboard in action L

 

Maybe I can adjust the version detection to make it work….  Wonder why they 
don’t support version “X and newer” ( probably a programming thing ).

 

-Brent

 

From: Kaspar Bosma  
Sent: Tuesday, July 23, 2019 1:21 AM
To: Brent Kennedy ; Paul Emmerich 
Cc: ceph-users 
Subject: RE: [ceph-users] Iscsi in the nautilus Dashboard

 

Hi Brent,

As far as I know version 3.0 (which I assume is version 9) is the minimum 
required for the dashboard.

I would go with the latest from Shaman; it won't break the actual iSCSI part of 
the setup, only maybe the iSCSI support in the dashboard. I haven't tried it 
myself, I'm still at version 2.7 (which is version 8 I would gather...)

Kaspar

Op 23 juli 2019 om 4:49 schreef Brent Kennedy mailto:bkenn...@cfl.rr.com> >: 

I posted to the ceph-iscsi github but Dillaman noted that 3.2 was version 10.  
Which means that wouldn’t solve the issue with the version 9 requirement of the 
current 14.2.2 nautilus.   Paul noted 3.1 is “pretty broken”, soo which version 
is version 9?  Or should I hack/patch the dashboard in 14.2.2 to accept version 
10 or 8? 

  

-Brent 

  

From: Kaspar Bosma mailto:kaspar.bo...@home.nl> > 
Sent: Monday, July 22, 2019 8:11 AM
To: Paul Emmerich mailto:paul.emmer...@croit.io> >; 
Brent Kennedy mailto:bkenn...@cfl.rr.com> >
Cc: ceph-users mailto:ceph-users@lists.ceph.com> >
Subject: Re: [ceph-users] Iscsi in the nautilus Dashboard 

 

Hi all,

That was not the most recent. This is it (3.2.4): 
https://2.chacra.ceph.com/r/ceph-iscsi/master/8a3967698257e1b49a9d554847b84418c15da902/centos/7/flavors/default/

Kaspar

Op 22 juli 2019 om 14:01 schreef Kaspar Bosma mailto:kaspar.bo...@home.nl> >:

Hi Brent,

You may want to have a look at the repos at shaman.ceph.com.

The latest (3.2.2) packaged version of Ceph iSCSI is located here:

https://4.chacra.ceph.com/r/ceph-iscsi/master/ff5e6873c43ab6828d3f7264526100b95a7e3954/centos/7/flavors/default/noarch/

You can also find related package repos for the tcmu-runner and ceph-iscsi-cli 
projects.

Regards, Kaspar

Op 22 juli 2019 om 12:52 schreef Paul Emmerich mailto:paul.emmer...@croit.io> >:

Version 9 is the fqdn stuff which was introduced in 3.1.

Use 3.2 as 3.1 is pretty broken.

 

Paul

 

-- 
Paul Emmerich 

Looking for help with your Ceph cluster? Contact us at https://croit.io 

croit GmbH 
Freseniusstr. 31h 
81247 München 
www.croit.io <http://www.croit.io>  
Tel: +49 89 1896585 90

 

 

On Mon, Jul 22, 2019 at 3:24 AM Brent Kennedy < bkenn...@cfl.rr.com 
<mailto:bkenn...@cfl.rr.com> > wrote:

I have a test cluster running centos 7.6 setup with two iscsi gateways ( per 
the requirement ).  I have the dashboard setup in nautilus ( 14.2.2 ) and I 
added the iscsi gateways via the command.  Both show down and when I go to the 
dashboard it states:

 

“ Unsupported `ceph-iscsi` config version. Expected 9 but found 8.  “

 

Both iscsi gateways were setup from scratch since the latest and greatest 
packages required for ceph iscsi install are not available in the centos 
repositories.  Is 3.0 not considered version 9?  ( did I do something wrong? ) 
Why is it called/detected as version 8 when its version 3?

 

I also wondering, the package versions listed as required in the nautilus 
docs(http://docs.ceph.com/docs/nautilus/rbd/iscsi-target-cli/)  state x.x.x or 
NEWER package, but when I try to add a gateway gwcli complains about the 
tcmu-runner and targetcli versions and I have to use the

Skipchecks=true option when adding them. 

 

Another thing came up, might be doing it wrong as well: 

Added a disk, then added the client, then tried to add the auth using the auth 
command and it states: “Failed to update the client's auth: Invalid password”

 

Actual output:

/iscsi-target...erpix:backup1> auth username=test password=test

CMD: ../hosts/ auth *

username=test, password=test, mutual_username=None, mutual_password=None

CMD: ../hosts/ auth *

auth to be set to username='test', password='test', mutual_username='None', 
mutual_password='None' for 'iqn.2019-07.com.somgthing:backup1'

Failed to update the client's auth: Invalid username

 

Did I miss something in the setup doc?

 

Installed packages:

rtslib:  wget https://github.com/open-iscsi/rtslib-fb/archive/v2.1.fb69.tar.gz

target-cli: wget 
https://github.com/open-iscsi/targetcli-fb/archive/v2.1.fb49.tar.gz

tcmu-runner: wget 
https://github.com/open-iscsi/tcmu-runner/archive/v1.4.1.tar.

Re: [ceph-users] Iscsi in the nautilus Dashboard

[Brent Kennedy]

I posted to the ceph-iscsi github but Dillaman noted that 3.2 was version 10.  
Which means that wouldn’t solve the issue with the version 9 requirement of the 
current 14.2.2 nautilus.   Paul noted 3.1 is “pretty broken”, soo which version 
is version 9?  Or should I hack/patch the dashboard in 14.2.2 to accept version 
10 or 8?

 

-Brent

 

From: Kaspar Bosma  
Sent: Monday, July 22, 2019 8:11 AM
To: Paul Emmerich ; Brent Kennedy 
Cc: ceph-users 
Subject: Re: [ceph-users] Iscsi in the nautilus Dashboard

 

Hi all,

That was not the most recent. This is it (3.2.4): 
https://2.chacra.ceph.com/r/ceph-iscsi/master/8a3967698257e1b49a9d554847b84418c15da902/centos/7/flavors/default/

Kaspar

Op 22 juli 2019 om 14:01 schreef Kaspar Bosma mailto:kaspar.bo...@home.nl> >: 

Hi Brent,

You may want to have a look at the repos at shaman.ceph.com.

The latest (3.2.2) packaged version of Ceph iSCSI is located here:

https://4.chacra.ceph.com/r/ceph-iscsi/master/ff5e6873c43ab6828d3f7264526100b95a7e3954/centos/7/flavors/default/noarch/

You can also find related package repos for the tcmu-runner and ceph-iscsi-cli 
projects.

Regards, Kaspar

Op 22 juli 2019 om 12:52 schreef Paul Emmerich mailto:paul.emmer...@croit.io> >: 

Version 9 is the fqdn stuff which was introduced in 3.1.

Use 3.2 as 3.1 is pretty broken. 

 

Paul

 

-- 
Paul Emmerich 

Looking for help with your Ceph cluster? Contact us at https://croit.io 

croit GmbH 
Freseniusstr. 31h 
81247 München 
www.croit.io <http://www.croit.io>  
Tel: +49 89 1896585 90

 

 

On Mon, Jul 22, 2019 at 3:24 AM Brent Kennedy < bkenn...@cfl.rr.com 
<mailto:bkenn...@cfl.rr.com> > wrote: 

I have a test cluster running centos 7.6 setup with two iscsi gateways ( per 
the requirement ).  I have the dashboard setup in nautilus ( 14.2.2 ) and I 
added the iscsi gateways via the command.  Both show down and when I go to the 
dashboard it states:

 

“ Unsupported `ceph-iscsi` config version. Expected 9 but found 8.  “

 

Both iscsi gateways were setup from scratch since the latest and greatest 
packages required for ceph iscsi install are not available in the centos 
repositories.  Is 3.0 not considered version 9?  ( did I do something wrong? ) 
Why is it called/detected as version 8 when its version 3?

 

I also wondering, the package versions listed as required in the nautilus 
docs(http://docs.ceph.com/docs/nautilus/rbd/iscsi-target-cli/)  state x.x.x or 
NEWER package, but when I try to add a gateway gwcli complains about the 
tcmu-runner and targetcli versions and I have to use the 

Skipchecks=true option when adding them.  

 

Another thing came up, might be doing it wrong as well:  

Added a disk, then added the client, then tried to add the auth using the auth 
command and it states: “Failed to update the client's auth: Invalid password”

 

Actual output:

/iscsi-target...erpix:backup1> auth username=test password=test

CMD: ../hosts/ auth *

username=test, password=test, mutual_username=None, mutual_password=None

CMD: ../hosts/ auth *

auth to be set to username='test', password='test', mutual_username='None', 
mutual_password='None' for 'iqn.2019-07.com.somgthing:backup1'

Failed to update the client's auth: Invalid username

 

Did I miss something in the setup doc?

 

Installed packages:

rtslib:  wget https://github.com/open-iscsi/rtslib-fb/archive/v2.1.fb69.tar.gz

target-cli: wget 
https://github.com/open-iscsi/targetcli-fb/archive/v2.1.fb49.tar.gz 

tcmu-runner: wget 
https://github.com/open-iscsi/tcmu-runner/archive/v1.4.1.tar.gz 

ceph-iscsi: wget https://github.com/ceph/ceph-iscsi/archive/3.0.tar.gz 

configshell: wget 
https://github.com/open-iscsi/configshell-fb/archive/v1.1.fb25.tar.gz 

 

Other bits I installed as part of this:

yum install epel-release python-pip python-devel -y

yum groupinstall "Development Tools" -y

python -m pip install --upgrade pip setuptools wheel

pip install netifaces cryptography flask

 

 

Any helps or pointer would be greatly appreciated!

 

-Brent

 

Existing Clusters:

Test: Nautilus 14.2.2 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi 
gateways ( all virtual on nvme )

US Production(HDD): Nautilus 14.2.1 with 11 osd servers, 3 mons, 4 gateways 
behind haproxy LB

UK Production(HDD): Luminous 12.2.11 with 25 osd servers, 3 mons/man, 3 
gateways behind haproxy LB

US Production(SSD): Luminous 12.2.11 with 6 osd servers, 3 mons/man, 3 gateways 
behind haproxy LB

 

 

___ 
ceph-users mailing list 
ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>  
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___ 
ceph-users mailing list 
ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>  
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 


 

___ ceph-users mailing list 
ceph-users@l

Re: [ceph-users] Changing the release cadence

[Brent Kennedy]

12 months sounds good to me, I like the idea of march as well since we plan
on doing upgrades in June/July each year.  Gives it time to be discussed and
marinate before we decide to upgrade.

-Brent

-Original Message-
From: ceph-users  On Behalf Of Sage Weil
Sent: Wednesday, June 5, 2019 11:58 AM
To: ceph-us...@ceph.com; ceph-de...@vger.kernel.org; d...@ceph.io
Subject: [ceph-users] Changing the release cadence

Hi everyone,

Since luminous, we have had the follow release cadence and policy:   
 - release every 9 months
 - maintain backports for the last two releases
 - enable upgrades to move either 1 or 2 releases heads
   (e.g., luminous -> mimic or nautilus; mimic -> nautilus or octopus; ...)

This has mostly worked out well, except that the mimic release received less
attention that we wanted due to the fact that multiple downstream Ceph
products (from Red Has and SUSE) decided to based their next release on
nautilus.  Even though upstream every release is an "LTS" release, as a
practical matter mimic got less attention than luminous or nautilus.

We've had several requests/proposals to shift to a 12 month cadence. This
has several advantages:

 - Stable/conservative clusters only have to be upgraded every 2 years
   (instead of every 18 months)
 - Yearly releases are more likely to intersect with downstream
   distribution release (e.g., Debian).  In the past there have been 
   problems where the Ceph releases included in consecutive releases of a 
   distro weren't easily upgradeable.
 - Vendors that make downstream Ceph distributions/products tend to
   release yearly.  Aligning with those vendors means they are more likely 
   to productize *every* Ceph release.  This will help make every Ceph 
   release an "LTS" release (not just in name but also in terms of 
   maintenance attention).

So far the balance of opinion seems to favor a shift to a 12 month cycle[1],
especially among developers, so it seems pretty likely we'll make that
shift.  (If you do have strong concerns about such a move, now is the time
to raise them.)

That brings us to an important decision: what time of year should we
release?  Once we pick the timing, we'll be releasing at that time *every
year* for each release (barring another schedule shift, which we want to
avoid), so let's choose carefully!

A few options:

 - November: If we release Octopus 9 months from the Nautilus release
   (planned for Feb, released in Mar) then we'd target this November.  We 
   could shift to a 12 months candence after that.
 - February: That's 12 months from the Nautilus target.
 - March: That's 12 months from when Nautilus was *actually* released.

November is nice in the sense that we'd wrap things up before the holidays.
It's less good in that users may not be inclined to install the new release
when many developers will be less available in December.

February kind of sucked in that the scramble to get the last few things done
happened during the holidays.  OTOH, we should be doing what we can to avoid
such scrambles, so that might not be something we should factor in.  March
may be a bit more balanced, with a solid 3 months before when people are
productive, and 3 months after before they disappear on holiday to address
any post-release issues.

People tend to be somewhat less available over the summer months due to
holidays etc, so an early or late summer release might also be less than
ideal.

Thoughts?  If we can narrow it down to a few options maybe we could do a
poll to gauge user preferences.

Thanks!
sage


[1] https://twitter.com/larsmb/status/1130010208971952129

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Iscsi in the nautilus Dashboard

[Brent Kennedy]

I have a test cluster running centos 7.6 setup with two iscsi gateways ( per
the requirement ).  I have the dashboard setup in nautilus ( 14.2.2 ) and I
added the iscsi gateways via the command.  Both show down and when I go to
the dashboard it states:

 

" Unsupported `ceph-iscsi` config version. Expected 9 but found 8.  "

 

Both iscsi gateways were setup from scratch since the latest and greatest
packages required for ceph iscsi install are not available in the centos
repositories.  Is 3.0 not considered version 9?  ( did I do something wrong?
) Why is it called/detected as version 8 when its version 3?

 

I also wondering, the package versions listed as required in the nautilus
docs(http://docs.ceph.com/docs/nautilus/rbd/iscsi-target-cli/)  state x.x.x
or NEWER package, but when I try to add a gateway gwcli complains about the
tcmu-runner and targetcli versions and I have to use the 

Skipchecks=true option when adding them.  

 

Another thing came up, might be doing it wrong as well:  

Added a disk, then added the client, then tried to add the auth using the
auth command and it states: "Failed to update the client's auth: Invalid
password"

 

Actual output:

/iscsi-target...erpix:backup1> auth username=test password=test

CMD: ../hosts/ auth *

username=test, password=test, mutual_username=None, mutual_password=None

CMD: ../hosts/ auth *

auth to be set to username='test', password='test', mutual_username='None',
mutual_password='None' for 'iqn.2019-07.com.somgthing:backup1'

Failed to update the client's auth: Invalid username

 

Did I miss something in the setup doc?

 

Installed packages:

rtslib:  wget
https://github.com/open-iscsi/rtslib-fb/archive/v2.1.fb69.tar.gz

target-cli: wget
https://github.com/open-iscsi/targetcli-fb/archive/v2.1.fb49.tar.gz 

tcmu-runner: wget
https://github.com/open-iscsi/tcmu-runner/archive/v1.4.1.tar.gz 

ceph-iscsi: wget https://github.com/ceph/ceph-iscsi/archive/3.0.tar.gz 

configshell: wget
https://github.com/open-iscsi/configshell-fb/archive/v1.1.fb25.tar.gz 

 

Other bits I installed as part of this:

yum install epel-release python-pip python-devel -y

yum groupinstall "Development Tools" -y

python -m pip install --upgrade pip setuptools wheel

pip install netifaces cryptography flask

 

 

Any helps or pointer would be greatly appreciated!

 

-Brent

 

Existing Clusters:

Test: Nautilus 14.2.2 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi
gateways ( all virtual on nvme )

US Production(HDD): Nautilus 14.2.1 with 11 osd servers, 3 mons, 4 gateways
behind haproxy LB

UK Production(HDD): Luminous 12.2.11 with 25 osd servers, 3 mons/man, 3
gateways behind haproxy LB

US Production(SSD): Luminous 12.2.11 with 6 osd servers, 3 mons/man, 3
gateways behind haproxy LB

 

 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] New best practices for osds???

[Brent Kennedy]

Went over this with a few clusters built last year, it seems best practices
( at least in the white papers I have read ) only use jbod configuration(
and so have we for all clusters ).  I agree regarding the writeback cache
issues and not using it but that's more due to how we use hardware (
definitely test it with your configuration or have your reseller show you
why they recommend it ).

Jbod is more intensive to manage when a drive fails...  so there is that.

Regards,
-Brent

Existing Clusters:
Test: Nautilus 14.2.1 with 3 osd servers, 1 mon/man, 1 gateway ( all virtual
on SSD )
US Production(HDD): Nautilus 14.2.1 with 11 osd servers, 3 mons, 4 gateways
behind haproxy LB
UK Production(HDD): Luminous 12.2.11 with 25 osd servers, 3 mons/man, 3
gateways behind haproxy LB
US Production(SSD): Luminous 12.2.11 with 6 osd servers, 3 mons/man, 3
gateways behind haproxy LB



-Original Message-
From: ceph-users  On Behalf Of Stolte,
Felix
Sent: Tuesday, July 16, 2019 11:42 AM
To: ceph-users 
Subject: [ceph-users] New best practices for osds???

Hi guys,

our ceph cluster is performing way less than it could, based on the disks we
are using. We could narrow it down to the storage controller (LSI SAS3008
HBA) in combination with an SAS expander. Yesterday we had a meeting with
our hardware reseller and sale representatives of the hardware manufacturer
to resolve the issue.

They told us, that "best practices" for ceph would be to deploy disks as
Raid 0 consisting of one disk using a raid controller with a big writeback
cache. 

Since this "best practice" is new to me, I would like to hear your opinion
on this topic.

Regards Felix


-

-
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr.
Sebastian M. Schmidt

-

-
 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] problems after upgrade to 14.2.1

[Brent Kennedy]

After installing the package on each mgr server and restarting the service,
i disabled the module then enabled the module with the force option.  (
seems I cut that out of the output I pasted )  It was essentially trial and
error.  After doing this, check and make sure you can see the module as
enabled  ( ceph mgr services ).  You should see something in the output at
that point.

 

I also had to fiddle with the SSL bits.

 

-Brent

 

From: ST Wong (ITSC)  
Sent: Friday, June 21, 2019 12:08 AM
To: Brent Kennedy ; ceph-users@lists.ceph.com
Subject: RE: [ceph-users] problems after upgrade to 14.2.1

 

Thanks.  I also didn't encounter the spillover issue on another cluster from
13.2.6 -> 14.2.1.  On that cluster, the dashboard also didn't work but
reconfiguring it similar to what you did worked.  Yes, nice new look. J

 

I commands like yours but it keeps prompting "all mgr daemons do not support
module 'dashboard', pass --force to force enablement".   Restart the mgr
service didn't help. 

 

/st wong

 

From: Brent Kennedy mailto:bkenn...@cfl.rr.com> > 
Sent: Friday, June 21, 2019 11:57 AM
To: ST Wong (ITSC) mailto:s...@itsc.cuhk.edu.hk> >;
ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> 
Subject: RE: [ceph-users] problems after upgrade to 14.2.1

 

Not sure about the spillover stuff, didn't happen to me when I upgraded from
Luminous to 14.2.1.  The dashboard thing did happen to me.  Seems you have
to disable the dashboard and then renable it after installing the separate
dashboard rpm.  Also, make sure to restart the mgr services on each node
before trying that and after the dashboard package install.  I didn't end up
using the SSL certificate bits.  Also, there is a code issue for 14.2.1
where you cannot login ( the login page just refreshes ), the bug report
says its fixed in 14.2.2..

 

Login page Bug Report: https://tracker.ceph.com/issues/40051   ( manual
fix:  https://github.com/ceph/ceph/pull/27942/files )  Make sure to change
the dashboard password after applying the fix.

 

The literal command history before I had it working again.  Love the new
look though!

2046  ceph mgr module enable dashboard

2047  ceph mgr module disable dashboard

2048  ceph config set mgr mgr/dashboard/ssl false

2049  ceph mgr module disable dashboard

2050  ceph mgr module enable dashboard

2051  ceph dashboard create-self-signed-cert

2052  ceph config set mgr mgr/dashboard/ssl true

2053  ceph mgr module disable dashboard

2054  ceph mgr module enable dashboard

2056  systemctl restart ceph-mgr.target

2057  ceph mgr module disable dashboard

2058  ceph mgr module enable dashboard

2059  ceph dashboard set-login-credentials 

 2060  systemctl restart ceph-mgr.target

2063  ceph mgr module disable dashboard

2064  ceph mgr module enable dashboard

2065  ceph dashboard ac-user-set-password

 

-Brent

 

 

From: ceph-users mailto:ceph-users-boun...@lists.ceph.com> > On Behalf Of ST Wong (ITSC)
Sent: Thursday, June 20, 2019 10:24 PM
To: ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> 
Subject: [ceph-users] problems after upgrade to 14.2.1

 

Hi all,

 

We recently upgrade a testing cluster from 13.2.4 to 14.2.1.  We encountered
2 problems:

 

1.   Got warning of BlueFS spillover but the usage is low while it's a
testing cluster without much activity/data:

 

# ceph -s

  cluster:

id: cc795498-5d16-4b84-9584-1788d0458be9

health: HEALTH_WARN

BlueFS spillover detected on 8 OSD(s)

[snipped]

 

# ceph health detail

HEALTH_WARN BlueFS spillover detected on 8 OSD(s)

BLUEFS_SPILLOVER BlueFS spillover detected on 8 OSD(s)

 osd.0 spilled over 48 MiB metadata from 'db' device (17 MiB used of 500
MiB) to slow device

 osd.1 spilled over 41 MiB metadata from 'db' device (6.0 MiB used of
500 MiB) to slow device

 osd.2 spilled over 47 MiB metadata from 'db' device (17 MiB used of 500
MiB) to slow device

 osd.3 spilled over 48 MiB metadata from 'db' device (6.0 MiB used of
500 MiB) to slow device

 osd.4 spilled over 44 MiB metadata from 'db' device (19 MiB used of 500
MiB) to slow device

 osd.5 spilled over 45 MiB metadata from 'db' device (6.0 MiB used of
500 MiB) to slow device

 osd.6 spilled over 46 MiB metadata from 'db' device (14 MiB used of 500
MiB) to slow device

 osd.7 spilled over 43 MiB metadata from 'db' device (6.0 MiB used of
500 MiB) to slow device

 

Is this a bug in 14 like this  <http://tracker.ceph.com/issues/38745>
http://tracker.ceph.com/issues/38745 ?

 

 

 

2.   Dashboard configuration are lost and unable to reconfigure it
again.  

 

The ceph-mgr-dashboard rpm is there, but we can't configure dashboard again:

 

--- cut here --

# ceph mgr module enable dashboard

Error ENOENT: all mgr daemons do not support module 'dashboard', pass
--force to force enablement

 

# ceph mgr module enable dashboard --for

Re: [ceph-users] problems after upgrade to 14.2.1

[Brent Kennedy]

Not sure about the spillover stuff, didn't happen to me when I upgraded from
Luminous to 14.2.1.  The dashboard thing did happen to me.  Seems you have
to disable the dashboard and then renable it after installing the separate
dashboard rpm.  Also, make sure to restart the mgr services on each node
before trying that and after the dashboard package install.  I didn't end up
using the SSL certificate bits.  Also, there is a code issue for 14.2.1
where you cannot login ( the login page just refreshes ), the bug report
says its fixed in 14.2.2..

 

Login page Bug Report: https://tracker.ceph.com/issues/40051   ( manual
fix:  https://github.com/ceph/ceph/pull/27942/files )  Make sure to change
the dashboard password after applying the fix.

 

The literal command history before I had it working again.  Love the new
look though!

2046  ceph mgr module enable dashboard

2047  ceph mgr module disable dashboard

2048  ceph config set mgr mgr/dashboard/ssl false

2049  ceph mgr module disable dashboard

2050  ceph mgr module enable dashboard

2051  ceph dashboard create-self-signed-cert

2052  ceph config set mgr mgr/dashboard/ssl true

2053  ceph mgr module disable dashboard

2054  ceph mgr module enable dashboard

2056  systemctl restart ceph-mgr.target

2057  ceph mgr module disable dashboard

2058  ceph mgr module enable dashboard

2059  ceph dashboard set-login-credentials 

 2060  systemctl restart ceph-mgr.target

2063  ceph mgr module disable dashboard

2064  ceph mgr module enable dashboard

2065  ceph dashboard ac-user-set-password

 

-Brent

 

 

From: ceph-users  On Behalf Of ST Wong
(ITSC)
Sent: Thursday, June 20, 2019 10:24 PM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] problems after upgrade to 14.2.1

 

Hi all,

 

We recently upgrade a testing cluster from 13.2.4 to 14.2.1.  We encountered
2 problems:

 

1.   Got warning of BlueFS spillover but the usage is low while it's a
testing cluster without much activity/data:

 

# ceph -s

  cluster:

id: cc795498-5d16-4b84-9584-1788d0458be9

health: HEALTH_WARN

BlueFS spillover detected on 8 OSD(s)

[snipped]

 

# ceph health detail

HEALTH_WARN BlueFS spillover detected on 8 OSD(s)

BLUEFS_SPILLOVER BlueFS spillover detected on 8 OSD(s)

 osd.0 spilled over 48 MiB metadata from 'db' device (17 MiB used of 500
MiB) to slow device

 osd.1 spilled over 41 MiB metadata from 'db' device (6.0 MiB used of
500 MiB) to slow device

 osd.2 spilled over 47 MiB metadata from 'db' device (17 MiB used of 500
MiB) to slow device

 osd.3 spilled over 48 MiB metadata from 'db' device (6.0 MiB used of
500 MiB) to slow device

 osd.4 spilled over 44 MiB metadata from 'db' device (19 MiB used of 500
MiB) to slow device

 osd.5 spilled over 45 MiB metadata from 'db' device (6.0 MiB used of
500 MiB) to slow device

 osd.6 spilled over 46 MiB metadata from 'db' device (14 MiB used of 500
MiB) to slow device

 osd.7 spilled over 43 MiB metadata from 'db' device (6.0 MiB used of
500 MiB) to slow device

 

Is this a bug in 14 like this  
http://tracker.ceph.com/issues/38745 ?

 

 

 

2.   Dashboard configuration are lost and unable to reconfigure it
again.  

 

The ceph-mgr-dashboard rpm is there, but we can't configure dashboard again:

 

--- cut here --

# ceph mgr module enable dashboard

Error ENOENT: all mgr daemons do not support module 'dashboard', pass
--force to force enablement

 

# ceph mgr module enable dashboard --force

# ceph mgr module ls

{

"enabled_modules": [

"dashboard"

],

 

[snipped]

 

# ceph mgr services

{}

 

# ceph dashboard create-self-signed-cert

Error EINVAL: No handler found for 'dashboard create-self-signed-cert'

 

// repeat the command gives different results

 

#  ceph dashboard create-self-signed-cert

Error EINVAL: Warning: due to ceph-mgr restart, some PG states may not be up
to date

No handler found for 'dashboard create-self-signed-cert'

 

#  ceph dashboard create-self-signed-cert

no valid command found; 10 closest matches:

osd down  [...]

osd require-osd-release luminous|mimic|nautilus {--yes-i-really-mean-it}

osd unset
full|pause|noup|nodown|noout|noin|nobackfill|norebalance|norecover|noscrub|n
odeep-scrub|notieragent|nosnaptrim

osd set
full|pause|noup|nodown|noout|noin|nobackfill|norebalance|norecover|noscrub|n
odeep-scrub|notieragent|nosnaptrim|pglog_hardlimit {--yes-i-really-mean-it}

osd erasure-code-profile ls

osd erasure-code-profile rm 

osd erasure-code-profile get 

osd erasure-code-profile set  { [...]} {--force}

osd unpause

osd pause

Error EINVAL: invalid command

--- cut here --

 

Did we miss anything?

 

Thanks a lot.

Regards

/st wong

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ISCSI Setup

[Brent Kennedy]

Sounds like progress!  Thanks for the update!  I will see if I can get it
working in my test off of the GH site.

-Brent

-Original Message-
From: Michael Christie  
Sent: Wednesday, June 19, 2019 5:24 PM
To: Brent Kennedy ; 'Ceph Users'

Subject: Re: [ceph-users] ISCSI Setup

On 06/19/2019 12:34 AM, Brent Kennedy wrote:
> Recently upgraded a ceph cluster to nautilus 14.2.1 from Luminous, no 
> issues.  One of the reasons for doing so was to take advantage of some 
> of the new ISCSI updates that were added in Nautilus.  I installed 
> CentOS 7.6 and did all the basic stuff to get the server online.  I 
> then tried to use the 
> http://docs.ceph.com/docs/nautilus/rbd/iscsi-target-cli/ document and 
> hit a hard stop.  Apparently, the package versions for the required 
> packages at the top nor the ceph-iscsi exist yet in any repositories.

I am in the process of updating the upstream docs (Aaron wrote up the
changes to the RHCS docs and I am just converting to the upstream docs and
making into patches for a PR, and ceph-ansible
(https://github.com/ceph/ceph-ansible/pull/3977) for the transition from
ceph-iscsi-cli/config to ceph-iscsi.

The upstream GH for ceph-iscsi is here
https://github.com/ceph/ceph-iscsi

and it is built here:
https://shaman.ceph.com/repos/ceph-iscsi/

I think we are just waiting on one last patch for fqdn support from SUSE so
we can make a new ceph-iscsi release.


> Reminds me of when I first tried to setup RGWs.  Is there a hidden 
> repository somewhere that hosts these required packages?  Also, I 
> found a thread talking about those packages and the instructions being 
> off, which concerns me.  Is there a good tutorial online somewhere?  I 
> saw the ceph-ansible bits, but wasn't sure if that would even work 
> because of the package issue.  I use ansible to deploy machines all 
> the time.  I also wonder if the ISCSI bits are considered production 
> or Test ( I see RedHat has a bunch of docs talking about using iscsi, 
> so I would think production ).
> 
>  
> 
> Thoughts anyone?
> 
>  
> 
> Regards,
> 
> -Brent
> 
>  
> 
> Existing Clusters:
> 
> Test: Nautilus 14.2.1 with 3 osd servers, 1 mon/man, 1 gateway ( all 
> virtual on SSD )
> 
> US Production(HDD): Nautilus 14.2.1 with 11 osd servers, 3 mons, 4 
> gateways behind haproxy LB
> 
> UK Production(HDD): Luminous 12.2.11 with 25 osd servers, 3 mons/man, 
> 3 gateways behind haproxy LB
> 
> US Production(SSD): Luminous 12.2.11 with 6 osd servers, 3 mons/man, 3 
> gateways behind haproxy LB
> 
>  
> 
>  
> 
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ISCSI Setup

[Brent Kennedy]

Recently upgraded a ceph cluster to nautilus 14.2.1 from Luminous, no
issues.  One of the reasons for doing so was to take advantage of some of
the new ISCSI updates that were added in Nautilus.  I installed CentOS 7.6
and did all the basic stuff to get the server online.  I then tried to use
the http://docs.ceph.com/docs/nautilus/rbd/iscsi-target-cli/ document and
hit a hard stop.  Apparently, the package versions for the required packages
at the top nor the ceph-iscsi exist yet in any repositories.  Reminds me of
when I first tried to setup RGWs.  Is there a hidden repository somewhere
that hosts these required packages?  Also, I found a thread talking about
those packages and the instructions being off, which concerns me.  Is there
a good tutorial online somewhere?  I saw the ceph-ansible bits, but wasn't
sure if that would even work because of the package issue.  I use ansible to
deploy machines all the time.  I also wonder if the ISCSI bits are
considered production or Test ( I see RedHat has a bunch of docs talking
about using iscsi, so I would think production ).

 

Thoughts anyone?

 

Regards,

-Brent

 

Existing Clusters:

Test: Nautilus 14.2.1 with 3 osd servers, 1 mon/man, 1 gateway ( all virtual
on SSD )

US Production(HDD): Nautilus 14.2.1 with 11 osd servers, 3 mons, 4 gateways
behind haproxy LB

UK Production(HDD): Luminous 12.2.11 with 25 osd servers, 3 mons/man, 3
gateways behind haproxy LB

US Production(SSD): Luminous 12.2.11 with 6 osd servers, 3 mons/man, 3
gateways behind haproxy LB

 

 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 'Missing' capacity

[Brent Kennedy]

That’s good to know as well, I was seeing the same thing.  I hope this is just 
an informational message though.

-Brent

-Original Message-
From: ceph-users  On Behalf Of Mark Schouten
Sent: Tuesday, April 16, 2019 3:15 AM
To: Igor Podlesny ; Sinan Polat 
Cc: Ceph Users 
Subject: Re: [ceph-users] 'Missing' capacity



root@proxmox01:~# ceph osd df tree | sort -n -k8 | tail -1
  1   ssd  0.87000  1.0  889GiB  721GiB  168GiB 81.14 1.50  82 
osd.1  


root@proxmox01:~# ceph osd df tree | grep -c osd
68


68*168=11424

That is closer, thanks. I thought that available was the same as the cluster 
available. But appearantly it is the available on the fullest OSD. Thanks, 
learned someting again!
--

Mark Schouten 

Tuxis, Ede, https://www.tuxis.nl

T: +31 318 200208 
 



- Originele bericht -


Van: Sinan Polat (si...@turka.nl)
Datum: 16-04-2019 06:43
Naar: Igor Podlesny (ceph-u...@poige.ru)
Cc: Mark Schouten (m...@tuxis.nl), Ceph Users (ceph-users@lists.ceph.com)
Onderwerp: Re: [ceph-users] 'Missing' capacity


Probably inbalance of data across your OSDs.

Could you show ceph osd df.

From there take the disk with lowest available space. Multiply that number with 
number of OSDs. How much is it?

Kind regards,
Sinan Polat

> Op 16 apr. 2019 om 05:21 heeft Igor Podlesny  het 
> volgende geschreven:
>
>> On Tue, 16 Apr 2019 at 06:43, Mark Schouten  wrote:
>> [...]
>> So where is the rest of the free space? :X
>
> Makes sense to see:
>
> sudo ceph osd df tree
>
> --
> End of message. Next message?
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Default Pools

[Brent Kennedy]

Yea, that was a cluster created during firefly...

Wish there was a good article on the naming and use of these, or perhaps a way 
I could make sure they are not used before deleting them.  I know RGW will 
recreate anything it uses, but I don’t want to lose data because I wanted a 
clean system.

-Brent

-Original Message-
From: Gregory Farnum  
Sent: Monday, April 15, 2019 5:37 PM
To: Brent Kennedy 
Cc: Ceph Users 
Subject: Re: [ceph-users] Default Pools

On Mon, Apr 15, 2019 at 1:52 PM Brent Kennedy  wrote:
>
> I was looking around the web for the reason for some of the default pools in 
> Ceph and I cant find anything concrete.  Here is our list, some show no use 
> at all.  Can any of these be deleted ( or is there an article my googlefu 
> failed to find that covers the default pools?
>
> We only use buckets, so I took out .rgw.buckets, .users and 
> .rgw.buckets.index…
>
> Name
> .log
> .rgw.root
> .rgw.gc
> .rgw.control
> .rgw
> .users.uid
> .users.email
> .rgw.buckets.extra
> default.rgw.control
> default.rgw.meta
> default.rgw.log
> default.rgw.buckets.non-ec

All of these are created by RGW when you run it, not by the core Ceph system. I 
think they're all used (although they may report sizes of 0, as they mostly 
make use of omap).

> metadata

Except this one used to be created-by-default for CephFS metadata, but that 
hasn't been true in many releases. So I guess you're looking at an old cluster? 
(In which case it's *possible* some of those RGW pools are also unused now but 
were needed in the past; I haven't kept good track of them.) -Greg

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Default Pools

[Brent Kennedy]

I was looking around the web for the reason for some of the default pools in
Ceph and I cant find anything concrete.  Here is our list, some show no use
at all.  Can any of these be deleted ( or is there an article my googlefu
failed to find that covers the default pools?

 

We only use buckets, so I took out .rgw.buckets, .users and
.rgw.buckets.index.

 

Name

.log

.rgw.root

.rgw.gc 

.rgw.control   

.rgw   

.users.uid

.users.email   

.rgw.buckets.extra  

default.rgw.control 

default.rgw.meta

default.rgw.log 

default.rgw.buckets.non-ec

metadata

 

Regards,

-Brent

 

Existing Clusters:

Test: Luminous 12.2.11 with 3 osd servers, 1 mon/man, 1 gateway ( all
virtual on SSD )

US Production(HDD): Luminous 12.2.11 with 11 osd servers, 3 mons, 3 gateways
behind haproxy LB

UK Production(HDD): Luminous 12.2.11 with 15 osd servers, 3 mons/man, 3
gateways behind haproxy LB

US Production(SSD): Luminous 12.2.11 with 6 osd servers, 3 mons/man, 3
gateways behind haproxy LB

 

 

 

 

 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OS Upgrade now monitor wont start

[Brent Kennedy]

Thanks Brad!  I completely forgot about that trick!  I copied the output and 
modified the command as suggested and the monitor came up.  So at least that 
does work, now I just need to figure out why the normal service setup is 
borked.  I was quick concerned that it wouldn’t come back at all and I would 
have to go back to a snapshot!

Regards,
Brent

-Original Message-
From: Brad Hubbard  
Sent: Sunday, March 24, 2019 9:49 PM
To: Brent Kennedy 
Cc: Ceph Users 
Subject: Re: [ceph-users] OS Upgrade now monitor wont start

Do a "ps auwwx" to see how a running monitor was started and use the equivalent 
command to try to start the MON that won't start. "ceph-mon --help" will show 
you what you need. Most important is to get the ID portion right and to add 
"-d" to get it to run in teh foreground and log to stdout. HTH and good luck!

On Mon, Mar 25, 2019 at 11:10 AM Brent Kennedy  wrote:
>
> Upgraded all the OS’s in the cluster to Ubuntu 14.04 LTS from Ubuntu 12.02 
> LTS then finished the upgrade from Firefly to Luminous.
>
>
>
> I then tried to upgrade the first monitor to Ubuntu 16.04 LTS, the OS upgrade 
> went fine, but then the monitor and manager wouldn’t start.  I then used 
> ceph-deploy to install over the existing install to ensure the new packages 
> were installed.  Monitor and manager still wont start.  Oddly enough, it 
> seems that logging wont populate either.  I was trying to find the command to 
> run the monitor manually to see if could read the output since the logging in 
> /var/log/ceph isn’t populating.  I did a file system search to see if a log 
> file was created in another directory, but it appears that’s not the case.  
> Monitor and cluster were healthy before I started the OS upgrade.  Nothing in 
> “Journalctl –xe” other than the services starting up without any errors.  
> Cluster shows 1/3 monitors down in health status though.
>
>
>
> I hope to upgrade all the remaining monitors to 16.04.  I already upgraded 
> the gateways to 16.04 without issue.  All the OSDs are being replaced with 
> newer hardware and going to CentOS 7.6.
>
>
>
>
>
> Regards,
>
> -Brent
>
>
>
> Existing Clusters:
>
> Test: Luminous 12.2.11 with 3 osd servers, 1 mon/man, 1 gateway ( all 
> virtual on SSD )
>
> US Production(HDD): Jewel 10.2.11 with 5 osd servers, 3 mons, 3 
> gateways behind haproxy LB
>
> UK Production(HDD): Luminous 12.2.11 with 15 osd servers, 3 mons/man, 
> 3 gateways behind haproxy LB
>
> US Production(SSD): Luminous 12.2.11 with 6 osd servers, 3 mons/man, 3 
> gateways behind haproxy LB
>
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



--
Cheers,
Brad

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] OS Upgrade now monitor wont start

[Brent Kennedy]

Upgraded all the OS's in the cluster to Ubuntu 14.04 LTS from Ubuntu 12.02
LTS then finished the upgrade from Firefly to Luminous.  

 

I then tried to upgrade the first monitor to Ubuntu 16.04 LTS, the OS
upgrade went fine, but then the monitor and manager wouldn't start.  I then
used ceph-deploy to install over the existing install to ensure the new
packages were installed.  Monitor and manager still wont start.  Oddly
enough, it seems that logging wont populate either.  I was trying to find
the command to run the monitor manually to see if could read the output
since the logging in /var/log/ceph isn't populating.  I did a file system
search to see if a log file was created in another directory, but it appears
that's not the case.  Monitor and cluster were healthy before I started the
OS upgrade.  Nothing in "Journalctl -xe" other than the services starting up
without any errors.  Cluster shows 1/3 monitors down in health status
though.

 

I hope to upgrade all the remaining monitors to 16.04.  I already upgraded
the gateways to 16.04 without issue.  All the OSDs are being replaced with
newer hardware and going to CentOS 7.6.

 

 

Regards,

-Brent

 

Existing Clusters:

Test: Luminous 12.2.11 with 3 osd servers, 1 mon/man, 1 gateway ( all
virtual on SSD )

US Production(HDD): Jewel 10.2.11 with 5 osd servers, 3 mons, 3 gateways
behind haproxy LB

UK Production(HDD): Luminous 12.2.11 with 15 osd servers, 3 mons/man, 3
gateways behind haproxy LB

US Production(SSD): Luminous 12.2.11 with 6 osd servers, 3 mons/man, 3
gateways behind haproxy LB

 

 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OMAP size on disk

[Brent Kennedy]

They released Luminous 12.2.11 and now my large objects are starting to
count down ( after running the rm command suggested in the release notes ).
Seems dynamic sharding will clean up after itself now to!  So case closed!

-Brent

-Original Message-
From: ceph-users  On Behalf Of Brent
Kennedy
Sent: Thursday, October 11, 2018 2:47 PM
To: 'Matt Benjamin' 
Cc: 'Ceph Users' 
Subject: Re: [ceph-users] OMAP size on disk

Does anyone have a good blog entry or explanation of bucket sharding
requirements/commands?  Plus perhaps a howto?  

I upgraded our cluster to Luminous and now I have a warning about 5 large
objects.  The official blog says that sharding is turned on by default but
we upgraded, so I cant quite tell if our existing buckets had sharding
turned on during the upgrade or if that is something I need to do after(blog
doesn't state that).  Also, when I looked into the sharding commands, they
wanted a shard size, which if its automated, why would need to provide that?
Not to mention I don't know what to start with...

I found this:  https://tracker.ceph.com/issues/24457 which talks about the
issue and the #14 says he worked through it, but information seems outside
of my googlefu.

-Brent

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
Matt Benjamin
Sent: Tuesday, October 9, 2018 7:28 AM
To: Luis Periquito 
Cc: Ceph Users 
Subject: Re: [ceph-users] OMAP size on disk

Hi Luis,

There are currently open issues with space reclamation after dynamic bucket
index resharding, esp. http://tracker.ceph.com/issues/34307

Changes are being worked on to address this, and to permit administratively
reclaiming space.

Matt

On Tue, Oct 9, 2018 at 5:50 AM, Luis Periquito  wrote:
> Hi all,
>
> I have several clusters, all running Luminous (12.2.7) proving S3 
> interface. All of them have enabled dynamic resharding and is working.
>
> One of the newer clusters is starting to give warnings on the used 
> space for the OMAP directory. The default.rgw.buckets.index pool is 
> replicated with 3x copies of the data.
>
> I created a new crush ruleset to only use a few well known SSDs, and 
> the OMAP directory size changed as expected: if I set the OSD as out 
> and them tell to compact, the size of the OMAP will shrink. If I set 
> the OSD as in the OMAP will grow to its previous state. And while the 
> backfill is going we get loads of key recoveries.
>
> Total physical space for OMAP in the OSDs that have them is ~1TB, so 
> given a 3x replica ~330G before replication.
>
> The data size for the default.rgw.buckets.data is just under 300G.
> There is one bucket who has ~1.7M objects and 22 shards.
>
> After deleting that bucket the size of the database didn't change - 
> even after running gc process and telling the OSD to compact its 
> database.
>
> This is not happening in older clusters, i.e created with hammer.
> Could this be a bug?
>
> I looked at getting all the OMAP keys and sizes
> (https://ceph.com/geen-categorie/get-omap-keyvalue-size/) and they add 
> up to close the value I expected them to take, looking at the physical 
> storage.
>
> Any ideas where to look next?
>
> thanks for all the help.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>



-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] SSD Recovery Settings

[Brent Kennedy]

Lots of good info there, thank you!  I tend to get options fatigue when trying 
to pick out a new system.  This should help narrow that focus greatly.  

 

-Brent

 

From: Reed Dier  
Sent: Wednesday, March 20, 2019 12:48 PM
To: Brent Kennedy 
Cc: ceph-users 
Subject: Re: [ceph-users] SSD Recovery Settings

 

Grafana <https://grafana.com/>  is the web frontend for creating the graphs.

 

InfluxDB <https://www.influxdata.com/time-series-platform/influxdb/>  holds the 
time series data that Grafana pulls from.

 

To collect data, I am using collectd 
<https://collectd.org/wiki/index.php/Plugin:Ceph>  daemons running on each ceph 
node (mon,mds,osd), as this was my initial way of ingesting metrics.

I am also now using the influx plugin in ceph-mgr 
<http://docs.ceph.com/docs/luminous/mgr/influx/>  to have ceph-mgr directly 
report statistics to InfluxDB.

 

I know two other popular methods of collecting data are Telegraf 
<https://www.influxdata.com/time-series-platform/telegraf/>  and Prometheus 
<https://prometheus.io/> , both of which are popular, both of which have 
ceph-mgr plugins as well here <http://docs.ceph.com/docs/mimic/mgr/telegraf/>  
and here <http://docs.ceph.com/docs/luminous/mgr/prometheus/> .

Influx Data also has a Grafana like graphing front end Chronograf 
<https://www.influxdata.com/time-series-platform/chronograf/> , which some 
prefer to Grafana.

 

Hopefully thats enough to get you headed in the right direction.

I would recommend not going down the CollectD path, as the project doesn't move 
as quickly as Telegraf and Prometheus, and the majority of the metrics I am 
pulling from these days are provided from the ceph-mgr plugin.

 

Hope that helps,

Reed





On Mar 20, 2019, at 11:30 AM, Brent Kennedy mailto:bkenn...@cfl.rr.com> > wrote:

 

Reed:  If you don’t mind me asking, what was the graphing tool you had in the 
post?  I am using the ceph health web panel right now but it doesn’t go that 
deep.

 

Regards,

Brent

 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] SSD Recovery Settings

[Brent Kennedy]

Seems both of you are spot on.  I injected the change and its now moving at
.080 instead of .002.  I did fix the label on the drives from HDD to SDD but
I didn't restart the OSDs due to the recovery process.  Seeing it fly now.
I also restarted the stuck OSDs but I know they are where the data is, so
they keep going back to slow.  I imagine this is part of the pg creation
process and my failure to adjust these when I created them.  Thanks!

 

I wasn't able to pull the config from the daemon(directory not found), but I
used the web panel to look at the setting.  I also found that
"bluestore_bdev_type" was set to "hdd", so I am going to see if there is a
way to change that because when I restarted some of the stuck OSDs, the tag
change I made doesn't seem to affect this setting.  I use ceph-deploy to do
the deployment(after ansible server setup), so it could also be a switch I
need to be using.  This is our first SSD cluster.

 

Reed:  If you don't mind me asking, what was the graphing tool you had in
the post?  I am using the ceph health web panel right now but it doesn't go
that deep.

 

Regards,

Brent

 

From: Reed Dier  
Sent: Wednesday, March 20, 2019 11:01 AM
To: Brent Kennedy 
Cc: ceph-users 
Subject: Re: [ceph-users] SSD Recovery Settings

 

Not sure what your OSD config looks like,

 

When I was moving from Filestore to Bluestore on my SSD OSD's (and NVMe FS
journal to NVMe Bluestore block.db),

I had an issue where the OSD was incorrectly being reported as rotational in
some part of the chain.

Once I overcame that, I had a huge boost in recovery performance (repaving
OSDs).

Might be something useful in there.

 

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-February/025039.htm
l

 

Reed





On Mar 19, 2019, at 11:29 PM, Konstantin Shalygin mailto:k0...@k0ste.ru> > wrote:

 

 

I setup an SSD Luminous 12.2.11 cluster and realized after data had been
added that pg_num was not set properly on the default.rgw.buckets.data pool
( where all the data goes ).  I adjusted the settings up, but recovery is
going really slow ( like 56-110MiB/s ) ticking down at .002 per log
entry(ceph -w).  These are all SSDs on luminous 12.2.11 ( no journal drives
) with a set of 2 10Gb fiber twinax in a bonded LACP config.  There are six
servers, 60 OSDs, each OSD is 2TB.  There was about 4TB of data ( 3 million
objects ) added to the cluster before I noticed the red blinking lights.
 
 
 
I tried adjusting the recovery to:
 
ceph tell 'osd.*' injectargs '--osd-max-backfills 16'
 
ceph tell 'osd.*' injectargs '--osd-recovery-max-active 30'
 
 
 
Which did help a little, but didn't seem to have the impact I was looking
for.  I have used the settings on HDD clusters before to speed things up (
using 8 backfills and 4 max active though ).  Did I miss something or is
this part of the pg expansion process.  Should I be doing something else
with SSD clusters?
 
 
 
Regards,
 
-Brent
 
 
 
Existing Clusters:
 
Test: Luminous 12.2.11 with 3 osd servers, 1 mon/man, 1 gateway ( all
virtual on SSD )
 
US Production(HDD): Jewel 10.2.11 with 5 osd servers, 3 mons, 3 gateways
behind haproxy LB
 
UK Production(HDD): Luminous 12.2.11 with 15 osd servers, 3 mons/man, 3
gateways behind haproxy LB
 
US Production(SSD): Luminous 12.2.11 with 6 osd servers, 3 mons/man, 3
gateways behind haproxy LB

 

Try to lower `osd_recovery_sleep*` options.

You can get your current values from ceph admin socket like this:

```

ceph daemon osd.0 config show | jq 'to_entries[] | if
(.key|test("^(osd_recovery_sleep)(.*)")) then (.) else empty end'

```

 

k

___
ceph-users mailing list
 <mailto:ceph-users@lists.ceph.com> ceph-users@lists.ceph.com
 <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] SSD Recovery Settings

[Brent Kennedy]

I setup an SSD Luminous 12.2.11 cluster and realized after data had been
added that pg_num was not set properly on the default.rgw.buckets.data pool
( where all the data goes ).  I adjusted the settings up, but recovery is
going really slow ( like 56-110MiB/s ) ticking down at .002 per log
entry(ceph -w).  These are all SSDs on luminous 12.2.11 ( no journal drives
) with a set of 2 10Gb fiber twinax in a bonded LACP config.  There are six
servers, 60 OSDs, each OSD is 2TB.  There was about 4TB of data ( 3 million
objects ) added to the cluster before I noticed the red blinking lights.

 

I tried adjusting the recovery to:

ceph tell 'osd.*' injectargs '--osd-max-backfills 16'

ceph tell 'osd.*' injectargs '--osd-recovery-max-active 30'

 

Which did help a little, but didn't seem to have the impact I was looking
for.  I have used the settings on HDD clusters before to speed things up (
using 8 backfills and 4 max active though ).  Did I miss something or is
this part of the pg expansion process.  Should I be doing something else
with SSD clusters?

 

Regards,

-Brent

 

Existing Clusters:

Test: Luminous 12.2.11 with 3 osd servers, 1 mon/man, 1 gateway ( all
virtual on SSD )

US Production(HDD): Jewel 10.2.11 with 5 osd servers, 3 mons, 3 gateways
behind haproxy LB

UK Production(HDD): Luminous 12.2.11 with 15 osd servers, 3 mons/man, 3
gateways behind haproxy LB

US Production(SSD): Luminous 12.2.11 with 6 osd servers, 3 mons/man, 3
gateways behind haproxy LB

 

 

 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Rebuild after upgrade

[Brent Kennedy]

Thanks for taking the time to answer!  It’s a really old cluster, so that
does make sense, thanks for confirming!

-Brent

-Original Message-
From: Hector Martin  
Sent: Monday, March 18, 2019 1:07 AM
To: Brent Kennedy ; 'Ceph Users'

Subject: Re: [ceph-users] Rebuild after upgrade

On 18/03/2019 13:24, Brent Kennedy wrote:
> I finally received approval to upgrade our old firefly(0.8.7) cluster 
> to Luminous.  I started the upgrade, upgrading to hammer(0.94.10), 
> then jewel(10.2.11), but after jewel, I ran the “ceph osd crush 
> tunables optimal” command, then “ceph –s” command showed 60% of the 
> objects were misplaced.  Now the cluster is just churning while it 
> does the recovery for that.
> 
> Is this something that happens when upgrading from firefly up?  I had 
> done a hammer upgrade to Jewel before, no rebalance occurred after 
> issuing that command.

Any time you change the CRUSH tunables, you can expect data movement. 
The exact impact can vary from nothing (if no changes were made or the
changes don't impact your actual pools/CRUSH rules) to a lot of data
movement. This is documented here:

http://docs.ceph.com/docs/master/rados/operations/crush-map/

In particular, you turned on CRUSH_TUNALBLES5, which causes a large amount
of data movement:
http://docs.ceph.com/docs/master/rados/operations/crush-map/#jewel-crush-tun
ables5

Going from Firefly to Hammer has a much smaller impact (see the CRUSH_V4
section).

--
Hector Martin (hec...@marcansoft.com)
Public Key: https://mrcn.st/pub

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Rebuild after upgrade

[Brent Kennedy]

I finally received approval to upgrade our old firefly(0.8.7) cluster to
Luminous.  I started the upgrade, upgrading to hammer(0.94.10), then
jewel(10.2.11), but after jewel, I ran the "ceph osd crush tunables optimal"
command, then "ceph -s" command showed 60% of the objects were misplaced.
Now the cluster is just churning while it does the recovery for that.  

 

Is this something that happens when upgrading from firefly up?  I had done a
hammer upgrade to Jewel before, no rebalance occurred after issuing that
command.

 

Regards,

-Brent

 

Existing Clusters:

Test: Luminous 12.2.10 with 3 osd servers, 1 mon/man, 1 gateway ( all
virtual )

US Production: Jewel 10.2.11 with 5 osd servers, 3 mons, 3 gateways behind
haproxy LB

UK Production: Luminous 12.2.10 with 15 osd servers, 3 mons/man, 3 gateways
behind haproxy LB

US Production all SSD: Luminous 12.2.10 with 6 osd servers, 3 mons/man, 3
gateways behind haproxy LB

 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OMAP size on disk

[Brent Kennedy]

Does anyone have a good blog entry or explanation of bucket sharding
requirements/commands?  Plus perhaps a howto?  

I upgraded our cluster to Luminous and now I have a warning about 5 large
objects.  The official blog says that sharding is turned on by default but
we upgraded, so I cant quite tell if our existing buckets had sharding
turned on during the upgrade or if that is something I need to do after(blog
doesn't state that).  Also, when I looked into the sharding commands, they
wanted a shard size, which if its automated, why would need to provide that?
Not to mention I don't know what to start with...

I found this:  https://tracker.ceph.com/issues/24457 which talks about the
issue and the #14 says he worked through it, but information seems outside
of my googlefu.

-Brent

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
Matt Benjamin
Sent: Tuesday, October 9, 2018 7:28 AM
To: Luis Periquito 
Cc: Ceph Users 
Subject: Re: [ceph-users] OMAP size on disk

Hi Luis,

There are currently open issues with space reclamation after dynamic bucket
index resharding, esp. http://tracker.ceph.com/issues/34307

Changes are being worked on to address this, and to permit administratively
reclaiming space.

Matt

On Tue, Oct 9, 2018 at 5:50 AM, Luis Periquito  wrote:
> Hi all,
>
> I have several clusters, all running Luminous (12.2.7) proving S3 
> interface. All of them have enabled dynamic resharding and is working.
>
> One of the newer clusters is starting to give warnings on the used 
> space for the OMAP directory. The default.rgw.buckets.index pool is 
> replicated with 3x copies of the data.
>
> I created a new crush ruleset to only use a few well known SSDs, and 
> the OMAP directory size changed as expected: if I set the OSD as out 
> and them tell to compact, the size of the OMAP will shrink. If I set 
> the OSD as in the OMAP will grow to its previous state. And while the 
> backfill is going we get loads of key recoveries.
>
> Total physical space for OMAP in the OSDs that have them is ~1TB, so 
> given a 3x replica ~330G before replication.
>
> The data size for the default.rgw.buckets.data is just under 300G.
> There is one bucket who has ~1.7M objects and 22 shards.
>
> After deleting that bucket the size of the database didn't change - 
> even after running gc process and telling the OSD to compact its 
> database.
>
> This is not happening in older clusters, i.e created with hammer.
> Could this be a bug?
>
> I looked at getting all the OMAP keys and sizes
> (https://ceph.com/geen-categorie/get-omap-keyvalue-size/) and they add 
> up to close the value I expected them to take, looking at the physical 
> storage.
>
> Any ideas where to look next?
>
> thanks for all the help.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>



-- 

Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-821-5101
fax.  734-769-8938
cel.  734-216-5309
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] https://ceph-storage.slack.com

[Brent Kennedy]

If this is a thing, I would like to be invited

-Brent

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
Alfredo Daniel Rezinovsky
Sent: Tuesday, September 18, 2018 3:15 PM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] https://ceph-storage.slack.com

Can anyone add me to this slack?

with my email alfrenov...@gmail.com

Thanks.

--
Alfredo Daniel Rezinovsky
Director de Tecnologías de Información y Comunicaciones Facultad de Ingeniería 
- Universidad Nacional de Cuyo

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OMAP warning ( again )

[Brent Kennedy]

I found this discussion between Wido and Florian ( two really good ceph folks 
), but it doesn’t seem to deep dive into sharding ( something I would like to 
know more about ).

https://www.spinics.net/lists/ceph-users/msg24420.html  

None of my clusters are using multi-site sync ( was thinking about it though ). 
 

Matt:  I see you are talking about this bug report:  
http://tracker.ceph.com/issues/34307  

Not sure how I could even find the retired bucket index you are referring to.  
I think I understand what sharding is doing, but as far as the inner workings, 
I am still a newb.

-Brent


-Original Message-
From: Matt Benjamin [mailto:mbenj...@redhat.com] 
Sent: Saturday, September 1, 2018 2:56 PM
To: Brent Kennedy 
Cc: Will Marley ; ceph-users 

Subject: Re: [ceph-users] OMAP warning ( again )

Apparently it is the case presently that when dynamic resharding completes, the 
retired bucket index shards need to be manually deleted.  We plan to change 
this, but it's worth checking for such objects.  Alternatively, though, look 
for other large omap "objects", e.g., sync-error.log, if you are using 
multi-site sync.

Matt

On Sat, Sep 1, 2018 at 1:47 PM, Brent Kennedy  wrote:
> I didn’t want to attempt anything until I had more information.  I 
> have been tied up with secondary stuff, so we are just monitoring for 
> now.  The only thing I could find was a setting to make the warning go 
> away, but that doesn’t seem like a good idea as it was identified as 
> an issue that should be addressed.  What perplexes me is that in 
> newest version of Luminous, it should be automatically resharding.  
> Noted here:  https://ceph.com/releases/v12-2-0-luminous-released/
>
> "RGW now supports dynamic bucket index sharding. This has to be enabled via 
> the rgw dynamic resharding configurable. As the number of objects in a bucket 
> grows, RGW will automatically reshard the bucket index in response.  No user 
> intervention or bucket size capacity planning is required."
>
> The documentation, says its enabled by default though, noted here: 
> http://docs.ceph.com/docs/master/radosgw/dynamicresharding
>
> I ran a few of the commands on the buckets to check and the sharding says -1, 
> so what I believe to be the case is that dynamic resharding is only enabled 
> by default on new buckets created in luminous, its not applied to existing 
> buckets.  So now the question is, can I enable dynamic resharding on a 
> particular bucket and will my cluster go nuts if the bucket is huge.  I would 
> rather the system figure out the sharding though as I am unsure what the 
> shard numbers would be should I have to tell the system to do it manually.
>
> There is an open bug report that I have been following: 
> https://tracker.ceph.com/issues/24457 ( appears we are not the only ones ).
>
> We need defined remediation steps for this. I was thinking of 
> hitting up the IRC room since RGW folks don’t seem to be around :(
>
> -Brent
>
> -Original Message-
> From: Will Marley [mailto:will.mar...@ukfast.co.uk]
> Sent: Friday, August 31, 2018 6:08 AM
> To: Brent Kennedy 
> Subject: RE: [ceph-users] OMAP warning ( again )
>
> Hi Brent,
>
> We're currently facing a similar issue. Did a manual reshard repair this for 
> you? Or do you have any more information to hand regarding a solution with 
> this? We're currently sat around scratching our heads about this as there 
> doesn't seem to be much documentation available.
>
> Kind Regards
>
> Will
> Ceph & Openstack Engineer
>
> UKFast.Net Limited, Registered in England Company Registration Number 
> 3845616 Registered office: UKFast Campus, Birley Fields, Manchester 
> M15 5QJ
>
> -Original Message-
> From: ceph-users  On Behalf Of 
> Brent Kennedy
> Sent: 01 August 2018 23:08
> To: 'Brad Hubbard' 
> Cc: 'ceph-users' 
> Subject: Re: [ceph-users] OMAP warning ( again )
>
> Ceph health detail gives this:
> HEALTH_WARN 1 large omap objects
> LARGE_OMAP_OBJECTS 1 large omap objects
> 1 large objects found in pool '.rgw.buckets.index'
> Search the cluster log for 'Large omap object found' for more details.
>
> The ceph.log file on the monitor server only shows the 1 large omap objects 
> message.
>
> I looked further into the issue again and remembered it was related to bucket 
> sharding.  I then remembered that in Luminous it was supposed to dynamic. I 
> went through the process this time of checking to see what the shards were 
> set to for one of the buckets we have and the max shards is still set to 0.  
> The blog posting about it says that there isn’t anything we have to do, but I 
> am wondering if the same is true for clusters that were upgraded to luminous 
> from

Re: [ceph-users] OMAP warning ( again )

[Brent Kennedy]

I didn’t want to attempt anything until I had more information.  I have been 
tied up with secondary stuff, so we are just monitoring for now.  The only 
thing I could find was a setting to make the warning go away, but that doesn’t 
seem like a good idea as it was identified as an issue that should be 
addressed.  What perplexes me is that in newest version of Luminous, it should 
be automatically resharding.  Noted here:  
https://ceph.com/releases/v12-2-0-luminous-released/ 

"RGW now supports dynamic bucket index sharding. This has to be enabled via the 
rgw dynamic resharding configurable. As the number of objects in a bucket 
grows, RGW will automatically reshard the bucket index in response.  No user 
intervention or bucket size capacity planning is required."

The documentation, says its enabled by default though, noted here: 
http://docs.ceph.com/docs/master/radosgw/dynamicresharding 

I ran a few of the commands on the buckets to check and the sharding says -1, 
so what I believe to be the case is that dynamic resharding is only enabled by 
default on new buckets created in luminous, its not applied to existing 
buckets.  So now the question is, can I enable dynamic resharding on a 
particular bucket and will my cluster go nuts if the bucket is huge.  I would 
rather the system figure out the sharding though as I am unsure what the shard 
numbers would be should I have to tell the system to do it manually.

There is an open bug report that I have been following: 
https://tracker.ceph.com/issues/24457 ( appears we are not the only ones ).

We need defined remediation steps for this. I was thinking of hitting up 
the IRC room since RGW folks don’t seem to be around :(  

-Brent

-Original Message-
From: Will Marley [mailto:will.mar...@ukfast.co.uk] 
Sent: Friday, August 31, 2018 6:08 AM
To: Brent Kennedy 
Subject: RE: [ceph-users] OMAP warning ( again )

Hi Brent,

We're currently facing a similar issue. Did a manual reshard repair this for 
you? Or do you have any more information to hand regarding a solution with 
this? We're currently sat around scratching our heads about this as there 
doesn't seem to be much documentation available.

Kind Regards

Will
Ceph & Openstack Engineer

UKFast.Net Limited, Registered in England Company Registration Number 3845616 
Registered office: UKFast Campus, Birley Fields, Manchester M15 5QJ

-Original Message-
From: ceph-users  On Behalf Of Brent Kennedy
Sent: 01 August 2018 23:08
To: 'Brad Hubbard' 
Cc: 'ceph-users' 
Subject: Re: [ceph-users] OMAP warning ( again )

Ceph health detail gives this:
HEALTH_WARN 1 large omap objects
LARGE_OMAP_OBJECTS 1 large omap objects
1 large objects found in pool '.rgw.buckets.index'
Search the cluster log for 'Large omap object found' for more details.

The ceph.log file on the monitor server only shows the 1 large omap objects 
message.

I looked further into the issue again and remembered it was related to bucket 
sharding.  I then remembered that in Luminous it was supposed to dynamic. I 
went through the process this time of checking to see what the shards were set 
to for one of the buckets we have and the max shards is still set to 0.  The 
blog posting about it says that there isn’t anything we have to do, but I am 
wondering if the same is true for clusters that were upgraded to luminous from 
older versions.

Do I need to run this: radosgw-admin reshard add --bucket= 
--num-shards=  for every bucket to get that going?

When I look at a bucket ( BKTEST ), it shows num_shards as 0:
root@ukpixmon1:/var/log/ceph# radosgw-admin metadata get 
bucket.instance:BKTEST:default.7320.3
{
"key": "bucket.instance:BKTEST:default.7320.3",
"ver": {
"tag": "_JFn84AijvH8aWXWXyvSeKpZ",
"ver": 1
},
"mtime": "2018-01-10 18:50:07.994194Z",
"data": {
"bucket_info": {
"bucket": {
"name": "BKTEST",
"marker": "default.7320.3",
"bucket_id": "default.7320.3",
"tenant": "",
"explicit_placement": {
"data_pool": ".rgw.buckets",
"data_extra_pool": ".rgw.buckets.extra",
"index_pool": ".rgw.buckets.index"
}
},
"creation_time": "2016-03-09 17:23:50.00Z",
"owner": "zz",
"flags": 0,
"zonegroup": "default",
"placement_rule": "default-placement",
"has_instance_obj": "true",
"quota": {
"enabled": fal

Re: [ceph-users] Clock skew

[Brent Kennedy]

For clock skew, I setup NTPD on one of the monitors with a public time server 
to pull from.  Then I setup NTPD on all the servers with them pulling time only 
from the local monitor server.  Restart the time service on each server until 
they get relatively close.  If you have a time server setup already in place, 
that would work as well.  Make sure to eliminate the backup time server entry 
as well.

If this is already in place, then what is usually necessary is a restart of the 
monitor service on the monitor complaining of clock skew.  If any are 
virtualized, make sure the time is not syncing from the host server to the VM, 
this could be causing the skew as well.

-Brent

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
Dominque Roux
Sent: Wednesday, August 15, 2018 5:38 AM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Clock skew

Hi all,

We recently facing clock skews from time to time.
This means that sometimes everything is fine but hours later the warning 
appears again.

NTPD is running and configured with the same pool.

Did someone else already had the same issue and could probably help us to fix 
this?

Thanks a lot!

Dominique
-- 

Your Swiss, Open Source and IPv6 Virtual Machine. Now on www.datacenterlight.ch


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Inconsistent PG could not be repaired

[Brent Kennedy]

Last time I had an inconsistent PG that could not be repaired using the repair 
command, I looked at which OSDs hosted the PG, then restarted them one by 
one(usually stopping, waiting a few seconds, then starting them back up ).  You 
could also stop them, flush the journal, then start them back up.  

 

If that didn’t work, it meant there was data loss and I had to use the 
ceph-objectstore-tool repair tool to export the objects from a location that 
had the latest data and import into the one that had no data.  The 
ceph-objectstore-tool is not a simple thing though and should not be used 
lightly.  When I say data loss, I mean that ceph thinks the last place written 
has the data, that place being the OSD that doesn’t actually have the 
data(meaning it failed to write there).

 

If you want to go that route, let me know, I wrote a how to on it.  Should be 
the last resort though.  I also don’t know your setup, so I would hate to 
recommend something so drastic.

 

-Brent

 

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of 
Arvydas Opulskis
Sent: Monday, August 6, 2018 4:12 AM
To: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Inconsistent PG could not be repaired

 

Hi again,

 

after two weeks I've got another inconsistent PG in same cluster. OSD's are 
different from first PG, object can not be GET as well: 


# rados list-inconsistent-obj 26.821 --format=json-pretty

{

"epoch": 178472,

"inconsistents": [

{

"object": {

"name": 
"default.122888368.52__shadow_.3ubGZwLcz0oQ55-LTb7PCOTwKkv-nQf_7",

"nspace": "",

"locator": "",

"snap": "head",

"version": 118920

},

"errors": [],

"union_shard_errors": [

"data_digest_mismatch_oi"

],

"selected_object_info": 
"26:8411bae4:::default.122888368.52__shadow_.3ubGZwLcz0oQ55-LTb7PCOTwKkv-nQf_7:head(126495'118920
 client.142609570.0:41412640 dirty|data_digest|omap_digest s 4194304 uv 118920 
dd cd142aaa od  alloc_hint [0 0])",

"shards": [

{

"osd": 20,

"errors": [

"data_digest_mismatch_oi"

],

"size": 4194304,

"omap_digest": "0x",

"data_digest": "0x6b102e59"

},

{

"osd": 44,

"errors": [

"data_digest_mismatch_oi"

],

"size": 4194304,

"omap_digest": "0x",

"data_digest": "0x6b102e59"

}

]

}

]

}



# rados -p .rgw.buckets get 
default.122888368.52__shadow_.3ubGZwLcz0oQ55-LTb7PCOTwKkv-nQf_7 test_2pg.file

error getting 
.rgw.buckets/default.122888368.52__shadow_.3ubGZwLcz0oQ55-LTb7PCOTwKkv-nQf_7: 
(5) Input/output error

 

 

Still struggling how to solve it. Any ideas, guys?

 

Thank you

 

 

 

On Tue, Jul 24, 2018 at 10:27 AM, Arvydas Opulskis mailto:zebedie...@gmail.com> > wrote:

Hello, Cephers,

 

after trying different repair approaches I am out of ideas how to repair 
inconsistent PG. I hope, someones sharp eye will notice what I overlooked.

 

Some info about cluster:

Centos 7.4

Jewel 10.2.10 

Pool size 2 (yes, I know it's a very bad choice)

Pool with inconsistent PG: .rgw.buckets 

 

After routine deep-scrub I've found PG 26.c3f in inconsistent status. While 
running "ceph pg repair 26.c3f" command and monitoring "ceph -w" log, I noticed 
these errors:

2018-07-24 08:28:06.517042 osd.36 [ERR] 26.c3f shard 30: soid 
26:fc32a1f1:::default.142609570.87_20180206.093111%2frepositories%2fnuget-local%2fApplication%2fCompany.Application.Api%2fCompany.Application.Api.1.1.1.nupkg.artifactory-metadata%2fproperties.xml:head
 data_digest 0x540e4f8b != data_digest 0x49a34c1f from auth oi 
26:e261561a:::default.168602061.10_team-xxx.xxx-jobs.H6.HADOOP.data-segmentation.application.131.xxx-jvm.cpu.load%2f2018-05-05T03%3a51%3a39+00%3a00.sha1:head(167828'216051
 client.179334015.0:1847715760 dirty|data_digest|omap_digest s 40 uv 216051 dd 
49a34c1f od  alloc_hint [0 0])

 

2018-07-24 08:28:06.517118 osd.36 [ERR] 26.c3f shard 36: soid 
26:fc32a1f1:::default.142609570.87_20180206.093111%2frepositories%2fnuget-local%2fApplication%2fCompany.Application.Api%2fCompany.Application.Api.1.1.1.nupkg.artifactory-metadata%2fproperties.xml:head
 data_digest 0x540e4f8b != data_digest 0x49a34c1f from auth oi 
26:e261561a:::default.168602061.10_team-xxx.xxx-jobs.H6.HADOOP.data-segmentation.application.131.xxx-jvm.cpu.load%2f2018-05-05T03%3a51%3a39+00%3a00.sha1:head(167828'216051
 client.179334015.0:1847715760 dirty|data_digest|omap_digest s 40 uv 216051 dd 
49a34c1f od  alloc_hint [0 0])

 

2018-07-24 08:28:06.517122 

Re: [ceph-users] OMAP warning ( again )

[Brent Kennedy]

Ceph health detail gives this:
HEALTH_WARN 1 large omap objects
LARGE_OMAP_OBJECTS 1 large omap objects
1 large objects found in pool '.rgw.buckets.index'
Search the cluster log for 'Large omap object found' for more details.

The ceph.log file on the monitor server only shows the 1 large omap objects 
message.

I looked further into the issue again and remembered it was related to bucket 
sharding.  I then remembered that in Luminous it was supposed to dynamic. I 
went through the process this time of checking to see what the shards were set 
to for one of the buckets we have and the max shards is still set to 0.  The 
blog posting about it says that there isn’t anything we have to do, but I am 
wondering if the same is true for clusters that were upgraded to luminous from 
older versions.

Do I need to run this: radosgw-admin reshard add --bucket= 
--num-shards=  for every bucket to get that going?

When I look at a bucket ( BKTEST ), it shows num_shards as 0:
root@ukpixmon1:/var/log/ceph# radosgw-admin metadata get 
bucket.instance:BKTEST:default.7320.3
{
"key": "bucket.instance:BKTEST:default.7320.3",
"ver": {
"tag": "_JFn84AijvH8aWXWXyvSeKpZ",
"ver": 1
},
"mtime": "2018-01-10 18:50:07.994194Z",
"data": {
"bucket_info": {
"bucket": {
"name": "BKTEST",
"marker": "default.7320.3",
"bucket_id": "default.7320.3",
"tenant": "",
"explicit_placement": {
"data_pool": ".rgw.buckets",
"data_extra_pool": ".rgw.buckets.extra",
"index_pool": ".rgw.buckets.index"
}
},
"creation_time": "2016-03-09 17:23:50.00Z",
"owner": "zz",
"flags": 0,
"zonegroup": "default",
"placement_rule": "default-placement",
"has_instance_obj": "true",
"quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1024,
"max_size_kb": 0,
"max_objects": -1
},
"num_shards": 0,
"bi_shard_hash_type": 0,
"requester_pays": "false",
"has_website": "false",
"swift_versioning": "false",
"swift_ver_location": "",
"index_type": 0,
"mdsearch_config": [],
"reshard_status": 0,
"new_bucket_instance_id": ""

When I run that shard setting to change the number of shards:
"radosgw-admin reshard add --bucket=BKTEST --num-shards=2"

Then run to get the status:
"radosgw-admin reshard list"

[
{
    "time": "2018-08-01 21:58:13.306381Z",
"tenant": "",
"bucket_name": "BKTEST",
"bucket_id": "default.7320.3",
"new_instance_id": "",
"old_num_shards": 1,
"new_num_shards": 2
}
]

If it was 0, why does it say old_num_shards was 1?

-Brent

-Original Message-
From: Brad Hubbard [mailto:bhubb...@redhat.com] 
Sent: Tuesday, July 31, 2018 9:07 PM
To: Brent Kennedy 
Cc: ceph-users 
Subject: Re: [ceph-users] OMAP warning ( again )

Search the cluster log for 'Large omap object found' for more details.

On Wed, Aug 1, 2018 at 3:50 AM, Brent Kennedy  wrote:
> Upgraded from 12.2.5 to 12.2.6, got a “1 large omap objects” warning 
> message, then upgraded to 12.2.7 and the message went away.  I just 
> added four OSDs to balance out the cluster ( we had some servers with 
> fewer drives in them; jbod config ) and now the “1 large omap objects” 
> warning message is back.  I did some googlefoo to try to figure out 
> what it means and then how to correct it, but the how to correct it is a bit 
> vague.
>
>
>
> We use rados gateways for all storage, so everything is in the 
> .rgw.buckets pool, which I gather from research is why we are getting 
> the warning message ( there are millions of objects in there ).
>
>
>
> Is there an if/then process to clearing this error message?
>
>
>
> Regards,
>
> -Brent
>
>
>
> Existing Clusters:
>
> Test: Luminous 12.2.7 with 3 osd servers, 1 mon/man, 1 gateway ( all 
> virtual
> )
>
> US Production: Firefly with 4 osd servers, 3 mons, 3 gateways behind 
> haproxy LB
>
> UK Production: Luminous 12.2.7 with 8 osd servers, 3 mons/man, 3 
> gateways behind haproxy LB
>
>
>
>
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



--
Cheers,
Brad

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Whole cluster flapping

[Brent Kennedy]

I have had this happen during large data movements.  Stopped happening after
I went to 10Gb though(from 1Gb).  What I had done is injected a setting (
and adjusted the configs ) to give more time before an OSD was marked down.

 

osd heartbeat grace = 200

mon osd down out interval = 900

 

For injecting runtime values/settings( under runtime changes ):

http://docs.ceph.com/docs/luminous/rados/configuration/ceph-conf/ 

 

Probably should check the logs before doing anything to ensure the OSDs or
host is not failing.  

 

-Brent

 

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
CUZA Frédéric
Sent: Tuesday, July 31, 2018 5:06 AM
To: ceph-users@lists.ceph.com
Subject: [ceph-users] Whole cluster flapping

 

Hi Everyone,

 

I just upgrade our cluster to Luminous 12.2.7 and I delete a quite large
pool that we had (120 TB).

Our cluster is made of 14 Nodes with each composed of 12 OSDs (1 HDD -> 1
OSD), we have SDD for journal.

 

After I deleted the large pool my cluster started to flapping on all OSDs.

Osds are marked down and then marked up as follow :

 

2018-07-31 10:42:51.504319 mon.ceph_monitor01 [INF] osd.97
172.29.228.72:6800/95783 boot

2018-07-31 10:42:55.330993 mon.ceph_monitor01 [WRN] Health check update:
5798/5845200 objects misplaced (0.099%) (OBJECT_MISPLACED)

2018-07-31 10:42:55.331065 mon.ceph_monitor01 [WRN] Health check update:
Degraded data redundancy: 221365/5845200 objects degraded (3.787%), 98 pgs
degraded, 317 pgs undersized (PG_DEGRADED)

2018-07-31 10:42:55.331093 mon.ceph_monitor01 [WRN] Health check update: 81
slow requests are blocked > 32 sec (REQUEST_SLOW)

2018-07-31 10:42:55.548385 mon.ceph_monitor01 [WRN] Health check update:
Reduced data availability: 13 pgs inactive, 4 pgs peering (PG_AVAILABILITY)

2018-07-31 10:42:55.610556 mon.ceph_monitor01 [INF] osd.96
172.29.228.72:6803/95830 boot

2018-07-31 10:43:00.331787 mon.ceph_monitor01 [WRN] Health check update: 5
osds down (OSD_DOWN)

2018-07-31 10:43:00.331930 mon.ceph_monitor01 [WRN] Health check update:
5782/5845401 objects misplaced (0.099%) (OBJECT_MISPLACED)

2018-07-31 10:43:00.331950 mon.ceph_monitor01 [WRN] Health check update:
Degraded data redundancy: 167757/5845401 objects degraded (2.870%), 77 pgs
degraded, 223 pgs undersized (PG_DEGRADED)

2018-07-31 10:43:00.331966 mon.ceph_monitor01 [WRN] Health check update: 76
slow requests are blocked > 32 sec (REQUEST_SLOW)

2018-07-31 10:43:01.729891 mon.ceph_monitor01 [WRN] Health check update:
Reduced data availability: 7 pgs inactive, 6 pgs peering (PG_AVAILABILITY)

2018-07-31 10:43:01.753867 mon.ceph_monitor01 [INF] osd.4
172.29.228.246:6812/3144542 boot

2018-07-31 10:43:05.332624 mon.ceph_monitor01 [WRN] Health check update: 4
osds down (OSD_DOWN)

2018-07-31 10:43:05.332691 mon.ceph_monitor01 [WRN] Health check update:
5767/5845569 objects misplaced (0.099%) (OBJECT_MISPLACED)

2018-07-31 10:43:05.332718 mon.ceph_monitor01 [WRN] Health check update:
Degraded data redundancy: 130565/5845569 objects degraded (2.234%), 67 pgs
degraded, 220 pgs undersized (PG_DEGRADED)

2018-07-31 10:43:05.332736 mon.ceph_monitor01 [WRN] Health check update: 83
slow requests are blocked > 32 sec (REQUEST_SLOW)

2018-07-31 10:43:07.004993 mon.ceph_monitor01 [WRN] Health check update:
Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY)

2018-07-31 10:43:10.333548 mon.ceph_monitor01 [WRN] Health check update:
5752/5845758 objects misplaced (0.098%) (OBJECT_MISPLACED)

2018-07-31 10:43:10.333593 mon.ceph_monitor01 [WRN] Health check update:
Degraded data redundancy: 107805/5845758 objects degraded (1.844%), 59 pgs
degraded, 197 pgs undersized (PG_DEGRADED)

2018-07-31 10:43:10.333608 mon.ceph_monitor01 [WRN] Health check update: 95
slow requests are blocked > 32 sec (REQUEST_SLOW)

2018-07-31 10:43:15.334451 mon.ceph_monitor01 [WRN] Health check update:
5738/5845923 objects misplaced (0.098%) (OBJECT_MISPLACED)

2018-07-31 10:43:15.334494 mon.ceph_monitor01 [WRN] Health check update:
Degraded data redundancy: 107807/5845923 objects degraded (1.844%), 59 pgs
degraded, 197 pgs undersized (PG_DEGRADED)

2018-07-31 10:43:15.334510 mon.ceph_monitor01 [WRN] Health check update: 98
slow requests are blocked > 32 sec (REQUEST_SLOW)

2018-07-31 10:43:15.334865 mon.ceph_monitor01 [INF] osd.18 failed
(root=default,room=,host=) (8 reporters from different host after
54.650576 >= grace 54.300663)

2018-07-31 10:43:15.336552 mon.ceph_monitor01 [WRN] Health check update: 5
osds down (OSD_DOWN)

2018-07-31 10:43:17.357747 mon.ceph_monitor01 [WRN] Health check update:
Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY)

2018-07-31 10:43:20.339495 mon.ceph_monitor01 [WRN] Health check update:
5724/5846073 objects misplaced (0.098%) (OBJECT_MISPLACED)

2018-07-31 10:43:20.339543 mon.ceph_monitor01 [WRN] Health check update:
Degraded data redundancy: 122901/5846073 objects degraded (2.102%), 65 pgs
degraded, 201 pgs 

[ceph-users] OMAP warning ( again )

[Brent Kennedy]

Upgraded from 12.2.5 to 12.2.6, got a "1 large omap objects" warning
message, then upgraded to 12.2.7 and the message went away.  I just added
four OSDs to balance out the cluster ( we had some servers with fewer drives
in them; jbod config ) and now the "1 large omap objects" warning message is
back.  I did some googlefoo to try to figure out what it means and then how
to correct it, but the how to correct it is a bit vague.

 

We use rados gateways for all storage, so everything is in the .rgw.buckets
pool, which I gather from research is why we are getting the warning message
( there are millions of objects in there ).

 

Is there an if/then process to clearing this error message?

 

Regards,

-Brent

 

Existing Clusters:

Test: Luminous 12.2.7 with 3 osd servers, 1 mon/man, 1 gateway ( all virtual
)

US Production: Firefly with 4 osd servers, 3 mons, 3 gateways behind haproxy
LB

UK Production: Luminous 12.2.7 with 8 osd servers, 3 mons/man, 3 gateways
behind haproxy LB

 

 

 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] download.ceph.com repository changes

[Brent Kennedy]

It would be nice if ceph-deploy could select the version as well as the
release.  E.G:  --release luminous --version 12.2.7   

Otherwise, I deploy a newest release to a new OSD server, then have to
upgrade the rest of the cluster ( unless the cluster is on a previous
release at the highest level )

Not sure if this adds to this particular discussion though :)

-Brent

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Dan
van der Ster
Sent: Tuesday, July 24, 2018 11:26 AM
To: Alfredo Deza 
Cc: ceph-users ; ceph-de...@vger.kernel.org;
ceph-maintain...@ceph.com
Subject: Re: [ceph-users] download.ceph.com repository changes

On Tue, Jul 24, 2018 at 5:08 PM Dan van der Ster  wrote:
>
> On Tue, Jul 24, 2018 at 4:59 PM Alfredo Deza  wrote:
> >
> > On Tue, Jul 24, 2018 at 10:54 AM, Dan van der Ster 
wrote:
> > > On Tue, Jul 24, 2018 at 4:38 PM Alfredo Deza  wrote:
> > >>
> > >> Hi all,
> > >>
> > >> After the 12.2.6 release went out, we've been thinking on better 
> > >> ways to remove a version from our repositories to prevent users 
> > >> from upgrading/installing a known bad release.
> > >>
> > >> The way our repos are structured today means every single version 
> > >> of the release is included in the repository. That is, for 
> > >> Luminous, every 12.x.x version of the binaries is in the same 
> > >> repo. This is true for both RPM and DEB repositories.
> > >>
> > >> However, the DEB repos don't allow pinning to a given version 
> > >> because our tooling (namely reprepro) doesn't construct the 
> > >> repositories in a way that this is allowed. For RPM repos this is 
> > >> fine, and version pinning works.
> > >>
> > >> To remove a bad version we have to proposals (and would like to 
> > >> hear ideas on other possibilities), one that would involve 
> > >> symlinks and the other one which purges the known bad version from
our repos.
> > >
> > > What we did with our mirror was: `rm -f *12.2.6*; createrepo 
> > > --update .` Took a few seconds. Then disabled the mirror cron.
> >
> > Up until next time when we cut another release and you have to 
> > re-enable the mirror with 12.2.6 in it :(
> >
>
> Right... we re-sync'd 12.2.6 along with 12.2.7 -- but people here 
> mostly grab the highest version.
>
> > This is also fast for RPM repos, but not quite fast for DEB repos.
> > Finally, *if* you are doing this, the metadata changes, and the 
> > repos need to be signed again. I am curious how that --update 
> > operation didn't make installations complain
>
> Good question.. I don't know enough about the repo signatures to 
> comment on this.

I asked our mirror man. Apparently we don't sign the repo, only the rpms. So
not applicable in general I suppose.

Another completely different (and not my) idea, how about we retag the last
good release with z+1. In this case we had 12.2.5 as the last good, and
12.2.6 broken, so we add the v12.2.7 tag on v12.2.5, effectively re-pushing
12.2.5 to the top.

-- dan

> I do know that all clients who had distro-sync'd up to 12.2.6 
> successfully distro-sync'd back to 12.2.5.
> (Our client machines yum distro-sync daily).
>
> -- Dan
>
> >
> > >
> > > -- Dan
> > >
> > >>
> > >> *Symlinking*
> > >> When releasing we would have a "previous" and "latest" symlink 
> > >> that would get updated as versions move forward. It would require 
> > >> separation of versions at the URL level (all versions would no 
> > >> longer be available in one repo).
> > >>
> > >> The URL structure would then look like:
> > >>
> > >> debian/luminous/12.2.3/
> > >> debian/luminous/previous/  (points to 12.2.5)
> > >> debian/luminous/latest/   (points to 12.2.7)
> > >>
> > >> Caveats: the url structure would change from debian-luminous/ to 
> > >> prevent breakage, and the versions would be split. For RPMs it 
> > >> would mean a regression if someone is used to pinning, for 
> > >> example pinning to 12.2.2 wouldn't be possible using the same url.
> > >>
> > >> Pros: Faster release times, less need to move packages around, 
> > >> and easier to remove a bad version
> > >>
> > >>
> > >> *Single version removal*
> > >> Our tooling would need to go and remove the known bad version 
> > >> from the repository, which would require to rebuild the 
> > >> repository again, so that the metadata is updated with the difference
in the binaries.
> > >>
> > >> Caveats: time intensive process, almost like cutting a new 
> > >> release which takes about a day (and sometimes longer). Error 
> > >> prone since the process wouldn't be the same (one off, just when 
> > >> a version needs to be
> > >> removed)
> > >>
> > >> Pros: all urls for download.ceph.com and its structure are kept the
same.
> > >> --
> > >> To unsubscribe from this list: send the line "unsubscribe 
> > >> ceph-devel" in the body of a message to majord...@vger.kernel.org 
> > >> More majordomo info at  
> > >> http://vger.kernel.org/majordomo-info.html

Re: [ceph-users] Omap warning in 12.2.6

[Brent Kennedy]

Thanks for the heads up.  I upgraded the cluster to 12.2.7 and the message went 
away.  No CRC errors luckily.

 

-Brent

 

From: Brady Deetz [mailto:bde...@gmail.com] 
Sent: Thursday, July 19, 2018 3:26 PM
To: Brent Kennedy 
Cc: ceph-users 
Subject: Re: [ceph-users] Omap warning in 12.2.6

 

12.2.6 has a regression. See "v12.2.7 Luminous released" and all of the related 
disaster posts. Also in the release nodes for .7 is a bug disclosure for 12.2.5 
that affects rgw users pretty badly during upgrade. You might take a look there.

 

On Thu, Jul 19, 2018 at 2:13 PM Brent Kennedy mailto:bkenn...@cfl.rr.com> > wrote:

I just upgraded our cluster to 12.2.6 and now I see this warning about 1 large 
omap object.  I looked and it seems this warning was just added in 12.2.6.  I 
found a few discussions on what is was but not much information on addressing 
it properly.  Our cluster uses rgw exclusively with just a few buckets in the 
.rgw.buckets pool.  Our largest bucket has millions of objects in it.

 

Any thoughts or links on this?

 

 

Regards,

-Brent

 

Existing Clusters:

Test: Luminous 12.2.6 with 3 osd servers, 1 mon/man, 1 gateway ( all virtual )

US Production: Firefly with 4 osd servers, 3 mons, 3 gateways behind haproxy LB

UK Production: Luminous 12.2.6 with 8 osd servers, 3 mons/man, 3 gateways 
behind haproxy LB

 

 

___
ceph-users mailing list
ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Omap warning in 12.2.6

[Brent Kennedy]

I just upgraded our cluster to 12.2.6 and now I see this warning about 1
large omap object.  I looked and it seems this warning was just added in
12.2.6.  I found a few discussions on what is was but not much information
on addressing it properly.  Our cluster uses rgw exclusively with just a few
buckets in the .rgw.buckets pool.  Our largest bucket has millions of
objects in it.

 

Any thoughts or links on this?

 

 

Regards,

-Brent

 

Existing Clusters:

Test: Luminous 12.2.6 with 3 osd servers, 1 mon/man, 1 gateway ( all virtual
)

US Production: Firefly with 4 osd servers, 3 mons, 3 gateways behind haproxy
LB

UK Production: Luminous 12.2.6 with 8 osd servers, 3 mons/man, 3 gateways
behind haproxy LB

 

 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 4 incomplete PGs causing RGW to go offline?

[Brent Kennedy]

I marked the PGs complete using the cephstore tool and that fixed the issues 
with the gateways going down.  They have been up for 2 days now without issue 
and made it through testing.  I tried to extract the data from the failing 
server, but I was unable to import it.  The failing server was on Hammer and I 
had upgraded to Jewel then Luminous shortly thereafter( I had all but the 4 PGs 
resolved before upgrade ).  I dont know if the tool supports restoring to newer 
versions, which might be the issue.  I assume this is part of the reasoning 
behind only upgrading healthy clusters.  We may try to extract the data 
directly from the drives at this point, all of it is replaceable though.

 

-Brent

 

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Brent 
Kennedy
Sent: Friday, January 12, 2018 10:27 AM
To: 'David Turner' <drakonst...@gmail.com>
Cc: 'Ceph Users' <ceph-users@lists.ceph.com>
Subject: Re: [ceph-users] 4 incomplete PGs causing RGW to go offline?

 

Rgw.buckets ( which is where the data is being sent ).  I am just surprised 
that a few incomplete PGs would grind three gateways to a halt.  Granted, the 
incomplete part of a large hardware failure situation we had and having a 
min_size setting of 1 didn’t help the situation.  We are not completely 
innocent, but I would hope that the system as a whole would work together to 
skip those incomplete PGs.  Fixing them doesn’t appear to be an easy task at 
this point, hence why we haven’t fixed them yet(I wish that were easier, but I 
understand the counter argument ).

 

-Brent

 

From: David Turner [mailto:drakonst...@gmail.com] 
Sent: Thursday, January 11, 2018 8:22 PM
To: Brent Kennedy <bkenn...@cfl.rr.com <mailto:bkenn...@cfl.rr.com> >
Cc: Ceph Users <ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> >
Subject: Re: [ceph-users] 4 incomplete PGs causing RGW to go offline?

 

Which pools are the incomplete PGs a part of? I would say it's very likely that 
if some of the RGW metadata was incomplete that the daemons wouldn't be happy.

 

On Thu, Jan 11, 2018, 6:17 PM Brent Kennedy <bkenn...@cfl.rr.com 
<mailto:bkenn...@cfl.rr.com> > wrote:

We have 3 RadosGW servers running behind HAProxy to enable clients to connect 
to the ceph cluster like an amazon bucket.  After all the failures and upgrade 
issues were resolved, I cannot get the RadosGW servers to stay online.  They 
were upgraded to luminous, I even upgraded the OS to Ubuntu 16 on them ( before 
upgrading to Luminous ).  They used to have apache on them as they ran Hammer 
and before that firefly.  I removed apache before upgrading to Luminous.  The 
start up and run for about 4-6 hours before all three start to go offline.  
Client traffic is light right now as we are just testing file read/write before 
we reactivate them ( they switched back to amazon while we fix them ).  

 

Could the 4 incomplete PGs be causing them to go offline?  The last time I saw 
an issue like this was when recovery wasn’t working 100%, so it seems related 
since they haven’t been stable since we upgraded( but that was also after the 
failures we had, which is why I am not trying to specifically blame the upgrade 
).

 

When I look at the radosgw log, this is what I see ( the first 2 lines show up 
plenty before this, they are health checks by the haproxy server, the next two 
are file requests that 404 fail I am guessing, then the last one is me 
restarting the service ):

 

2018-01-11 20:14:36.640577 7f5826aa3700  1 == req done req=0x7f5826a9d1f0 
op status=0 http_status=200 ==

2018-01-11 20:14:36.640602 7f5826aa3700  1 civetweb: 0x56202c567000: 
192.168.120.21 - - [11/Jan/2018:20:14:36 +] "HEAD / HTTP/1.0" 1 0 - -

2018-01-11 20:14:36.640835 7f5816282700  1 == req done req=0x7f581627c1f0 
op status=0 http_status=200 ==

2018-01-11 20:14:36.640859 7f5816282700  1 civetweb: 0x56202c61: 
192.168.120.22 - - [11/Jan/2018:20:14:36 +] "HEAD / HTTP/1.0" 1 0 - -

2018-01-11 20:14:36.761917 7f5835ac1700  1 == starting new request 
req=0x7f5835abb1f0 =

2018-01-11 20:14:36.763936 7f5835ac1700  1 == req done req=0x7f5835abb1f0 
op status=0 http_status=404 ==

2018-01-11 20:14:36.763983 7f5835ac1700  1 civetweb: 0x56202c4ce000: 
192.168.120.21 - - [11/Jan/2018:20:14:36 +] "HEAD 
/Jobimages/vendor05/10/3962896/3962896_cover.pdf HTTP/1.1" 1 0 - 
aws-sdk-dotnet-35/2

.0.2.2 .NET Runtime/4.0 .NET Framework/4.0 OS/6.2.9200.0 FileIO

2018-01-11 20:14:36.772611 7f5808266700  1 == starting new request 
req=0x7f58082601f0 =

2018-01-11 20:14:36.773733 7f5808266700  1 == req done req=0x7f58082601f0 
op status=0 http_status=404 ==

2018-01-11 20:14:36.773769 7f5808266700  1 civetweb: 0x56202c6aa000: 
192.168.120.21 - - [11/Jan/2018:20:14:36 +] "HEAD 
/Jobimages/vendor05/10/3962896/3962896_cover.pdf HTTP/1.1" 1 0 - 
aws-sdk-dotnet

Re: [ceph-users] Ceph-objectstore-tool import failure

[Brent Kennedy]

That interesting information, unfortunately our failing server just failed 
completely, so copying the disks may be the only option at this point.  It’s a 
JBOD setup, so we could pull the data off directly from the drives.   It would 
have been nice to restore from the tool nice and clean.  I had to use the tool 
to mark the PGs complete just so we could use the radosgw’s again.

 

-Brent

 

From: Brady Deetz [mailto:bde...@gmail.com] 
Sent: Sunday, January 14, 2018 2:35 AM
To: Brent Kennedy <bkenn...@cfl.rr.com>
Cc: ceph-users <ceph-users@lists.ceph.com>
Subject: Re: [ceph-users] Ceph-objectstore-tool import failure

 

It's not simply a zip. I recently went through an incomplete pg incident as 
well. I'm not sure why your import is failing, but I do know that much. Here's 
a note in slack from our effort to reverse the export. I'm hoping to explore 
this a bit more in the next week. 

 

Data frames appear to have the following format:

ceff DTDT SIZE PAYLOAD

 

Size is probably in bytes? DTDT is frame type, 8-bits, repeated (so BEGIN=1 is 
encoded as 0101).



File format looks like:



Superblock - starts with:   ceff ceff 0200

PG_BEGINceff 0101 <64 bit little-Endian size>

PG_METADATA ceff 0909

OBJECT_BEGINceff 0303 <64 bit little-Endian size> (represents 
first object of a file?)

TYPE_DATA   ceff 0505 (represents an object?)

(repeat TYPE_DATA frames until file completed)

TYPE_ATTRS  ceff 0606

TYPE_OMAP   ceff 0808

OBJECT_END  ceff 0404

(repeat above block for all files?)

 

 

enum {

TYPE_NONE = 0,

TYPE_PG_BEGIN,

TYPE_PG_END,

TYPE_OBJECT_BEGIN,

TYPE_OBJECT_END,

TYPE_DATA,

TYPE_ATTRS,

TYPE_OMAP_HDR,

TYPE_OMAP,

TYPE_PG_METADATA,

TYPE_POOL_BEGIN,

TYPE_POOL_END,

END_OF_TYPES,  //Keep at the end

};

 

 

 

On Jan 14, 2018 1:27 AM, "Brent Kennedy" <bkenn...@cfl.rr.com 
<mailto:bkenn...@cfl.rr.com> > wrote:

I was able to bring a server back online for a short time and perform an export 
of the incomplete PGs I originally posted about last week.  The export showed 
the files it was exporting and then dropped them all to a PGID.export file.  I 
then SCP’ed the four PGID.export files to a server where I had an empty OSD 
weighted to 0.  I stopped that OSD and then tried to import all four PGs.  I 
then got the following messages for all four I tried:

 

finish_remove_pgs 11.720_head removing 11.720

Importing pgid 11.c13

do_import threw exception error buffer::malformed_input: void 
object_stat_sum_t::decode(ceph::buffer::list::iterator&) decode past end of 
struct encoding

Corrupt input for import

 

 

Command I ran: 

ceph-objectstore-tool --op import --data-path /var/lib/ceph/osd/ceph-13 
--journal-path /var/lib/ceph/osd/ceph-13/block --file 11.c13.export

 

The files match the space used by PGs on the disk.  As noted above, I saw it 
copy the PG to the export file successfully.  Both servers are running Ubuntu 
14 with the newest ceph-objectstore-tool installed via the package from here:  
http://download.ceph.com/debian-luminous/pool/main/c/ceph/ceph-test_12.2.2-1trusty_amd64.deb
  ( cluster is Luminous 12.2.2 .  Its possible the PGs in question are on the 
jewel version as I wasn’t able to complete the upgrade to luminous on them.

 

Am I missing something?  Can I just copy the files off the failing server via a 
zip operation locally and then a unzip operation at the destination server?

 

-Brent


___
ceph-users mailing list
ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph-objectstore-tool import failure

[Brent Kennedy]

I was able to bring a server back online for a short time and perform an
export of the incomplete PGs I originally posted about last week.  The
export showed the files it was exporting and then dropped them all to a
PGID.export file.  I then SCP'ed the four PGID.export files to a server
where I had an empty OSD weighted to 0.  I stopped that OSD and then tried
to import all four PGs.  I then got the following messages for all four I
tried:

 

finish_remove_pgs 11.720_head removing 11.720

Importing pgid 11.c13

do_import threw exception error buffer::malformed_input: void
object_stat_sum_t::decode(ceph::buffer::list::iterator&) decode past end of
struct encoding

Corrupt input for import

 

 

Command I ran: 

ceph-objectstore-tool --op import --data-path /var/lib/ceph/osd/ceph-13
--journal-path /var/lib/ceph/osd/ceph-13/block --file 11.c13.export

 

The files match the space used by PGs on the disk.  As noted above, I saw it
copy the PG to the export file successfully.  Both servers are running
Ubuntu 14 with the newest ceph-objectstore-tool installed via the package
from here:
http://download.ceph.com/debian-luminous/pool/main/c/ceph/ceph-test_12.2.2-1
trusty_amd64.deb  ( cluster is Luminous 12.2.2 .  Its possible the PGs in
question are on the jewel version as I wasn't able to complete the upgrade
to luminous on them.

 

Am I missing something?  Can I just copy the files off the failing server
via a zip operation locally and then a unzip operation at the destination
server?

 

-Brent

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 4 incomplete PGs causing RGW to go offline?

[Brent Kennedy]

Rgw.buckets ( which is where the data is being sent ).  I am just surprised 
that a few incomplete PGs would grind three gateways to a halt.  Granted, the 
incomplete part of a large hardware failure situation we had and having a 
min_size setting of 1 didn’t help the situation.  We are not completely 
innocent, but I would hope that the system as a whole would work together to 
skip those incomplete PGs.  Fixing them doesn’t appear to be an easy task at 
this point, hence why we haven’t fixed them yet(I wish that were easier, but I 
understand the counter argument ).

 

-Brent

 

From: David Turner [mailto:drakonst...@gmail.com] 
Sent: Thursday, January 11, 2018 8:22 PM
To: Brent Kennedy <bkenn...@cfl.rr.com>
Cc: Ceph Users <ceph-users@lists.ceph.com>
Subject: Re: [ceph-users] 4 incomplete PGs causing RGW to go offline?

 

Which pools are the incomplete PGs a part of? I would say it's very likely that 
if some of the RGW metadata was incomplete that the daemons wouldn't be happy.

 

On Thu, Jan 11, 2018, 6:17 PM Brent Kennedy <bkenn...@cfl.rr.com 
<mailto:bkenn...@cfl.rr.com> > wrote:

We have 3 RadosGW servers running behind HAProxy to enable clients to connect 
to the ceph cluster like an amazon bucket.  After all the failures and upgrade 
issues were resolved, I cannot get the RadosGW servers to stay online.  They 
were upgraded to luminous, I even upgraded the OS to Ubuntu 16 on them ( before 
upgrading to Luminous ).  They used to have apache on them as they ran Hammer 
and before that firefly.  I removed apache before upgrading to Luminous.  The 
start up and run for about 4-6 hours before all three start to go offline.  
Client traffic is light right now as we are just testing file read/write before 
we reactivate them ( they switched back to amazon while we fix them ).  

 

Could the 4 incomplete PGs be causing them to go offline?  The last time I saw 
an issue like this was when recovery wasn’t working 100%, so it seems related 
since they haven’t been stable since we upgraded( but that was also after the 
failures we had, which is why I am not trying to specifically blame the upgrade 
).

 

When I look at the radosgw log, this is what I see ( the first 2 lines show up 
plenty before this, they are health checks by the haproxy server, the next two 
are file requests that 404 fail I am guessing, then the last one is me 
restarting the service ):

 

2018-01-11 20:14:36.640577 7f5826aa3700  1 == req done req=0x7f5826a9d1f0 
op status=0 http_status=200 ==

2018-01-11 20:14:36.640602 7f5826aa3700  1 civetweb: 0x56202c567000: 
192.168.120.21 - - [11/Jan/2018:20:14:36 +] "HEAD / HTTP/1.0" 1 0 - -

2018-01-11 20:14:36.640835 7f5816282700  1 == req done req=0x7f581627c1f0 
op status=0 http_status=200 ==

2018-01-11 20:14:36.640859 7f5816282700  1 civetweb: 0x56202c61: 
192.168.120.22 - - [11/Jan/2018:20:14:36 +] "HEAD / HTTP/1.0" 1 0 - -

2018-01-11 20:14:36.761917 7f5835ac1700  1 == starting new request 
req=0x7f5835abb1f0 =

2018-01-11 20:14:36.763936 7f5835ac1700  1 == req done req=0x7f5835abb1f0 
op status=0 http_status=404 ==

2018-01-11 20:14:36.763983 7f5835ac1700  1 civetweb: 0x56202c4ce000: 
192.168.120.21 - - [11/Jan/2018:20:14:36 +] "HEAD 
/Jobimages/vendor05/10/3962896/3962896_cover.pdf HTTP/1.1" 1 0 - 
aws-sdk-dotnet-35/2

.0.2.2 .NET Runtime/4.0 .NET Framework/4.0 OS/6.2.9200.0 FileIO

2018-01-11 20:14:36.772611 7f5808266700  1 == starting new request 
req=0x7f58082601f0 =

2018-01-11 20:14:36.773733 7f5808266700  1 == req done req=0x7f58082601f0 
op status=0 http_status=404 ==

2018-01-11 20:14:36.773769 7f5808266700  1 civetweb: 0x56202c6aa000: 
192.168.120.21 - - [11/Jan/2018:20:14:36 +] "HEAD 
/Jobimages/vendor05/10/3962896/3962896_cover.pdf HTTP/1.1" 1 0 - 
aws-sdk-dotnet-35/2

.0.2.2 .NET Runtime/4.0 .NET Framework/4.0 OS/6.2.9200.0 FileIO

2018-01-11 20:14:38.163617 7f5836ac3700  1 == starting new request 
req=0x7f5836abd1f0 =

2018-01-11 20:14:38.165352 7f5836ac3700  1 == req done req=0x7f5836abd1f0 
op status=0 http_status=404 ==

2018-01-11 20:14:38.165401 7f5836ac3700  1 civetweb: 0x56202c4e2000: 
192.168.120.21 - - [11/Jan/2018:20:14:38 +] "HEAD 
/Jobimages/vendor05/10/3445645/3445645_cover.pdf HTTP/1.1" 1 0 - 
aws-sdk-dotnet-35/2

.0.2.2 .NET Runtime/4.0 .NET Framework/4.0 OS/6.2.9200.0 FileIO

2018-01-11 20:14:38.170551 7f5807a65700  1 == starting new request 
req=0x7f5807a5f1f0 =

2018-01-11 20:14:40.322236 7f58352c0700  1 == starting new request 
req=0x7f58352ba1f0 =

2018-01-11 20:14:40.323468 7f5834abf700  1 == starting new request 
req=0x7f5834ab91f0 =

2018-01-11 20:14:41.643365 7f58342be700  1 == starting new request 
req=0x7f58342b81f0 =

2018-01-11 20:14:41.643358 7f58312b8700  1 == starting new request 
req=0x7f58312b21f0 =

2018-01-11 20:14:50.324196 7f5829aa97

[ceph-users] 4 incomplete PGs causing RGW to go offline?

[Brent Kennedy]

We have 3 RadosGW servers running behind HAProxy to enable clients to
connect to the ceph cluster like an amazon bucket.  After all the failures
and upgrade issues were resolved, I cannot get the RadosGW servers to stay
online.  They were upgraded to luminous, I even upgraded the OS to Ubuntu 16
on them ( before upgrading to Luminous ).  They used to have apache on them
as they ran Hammer and before that firefly.  I removed apache before
upgrading to Luminous.  The start up and run for about 4-6 hours before all
three start to go offline.  Client traffic is light right now as we are just
testing file read/write before we reactivate them ( they switched back to
amazon while we fix them ).  

 

Could the 4 incomplete PGs be causing them to go offline?  The last time I
saw an issue like this was when recovery wasn't working 100%, so it seems
related since they haven't been stable since we upgraded( but that was also
after the failures we had, which is why I am not trying to specifically
blame the upgrade ).

 

When I look at the radosgw log, this is what I see ( the first 2 lines show
up plenty before this, they are health checks by the haproxy server, the
next two are file requests that 404 fail I am guessing, then the last one is
me restarting the service ):

 

2018-01-11 20:14:36.640577 7f5826aa3700  1 == req done
req=0x7f5826a9d1f0 op status=0 http_status=200 ==

2018-01-11 20:14:36.640602 7f5826aa3700  1 civetweb: 0x56202c567000:
192.168.120.21 - - [11/Jan/2018:20:14:36 +] "HEAD / HTTP/1.0" 1 0 - -

2018-01-11 20:14:36.640835 7f5816282700  1 == req done
req=0x7f581627c1f0 op status=0 http_status=200 ==

2018-01-11 20:14:36.640859 7f5816282700  1 civetweb: 0x56202c61:
192.168.120.22 - - [11/Jan/2018:20:14:36 +] "HEAD / HTTP/1.0" 1 0 - -

2018-01-11 20:14:36.761917 7f5835ac1700  1 == starting new request
req=0x7f5835abb1f0 =

2018-01-11 20:14:36.763936 7f5835ac1700  1 == req done
req=0x7f5835abb1f0 op status=0 http_status=404 ==

2018-01-11 20:14:36.763983 7f5835ac1700  1 civetweb: 0x56202c4ce000:
192.168.120.21 - - [11/Jan/2018:20:14:36 +] "HEAD
/Jobimages/vendor05/10/3962896/3962896_cover.pdf HTTP/1.1" 1 0 -
aws-sdk-dotnet-35/2

.0.2.2 .NET Runtime/4.0 .NET Framework/4.0 OS/6.2.9200.0 FileIO

2018-01-11 20:14:36.772611 7f5808266700  1 == starting new request
req=0x7f58082601f0 =

2018-01-11 20:14:36.773733 7f5808266700  1 == req done
req=0x7f58082601f0 op status=0 http_status=404 ==

2018-01-11 20:14:36.773769 7f5808266700  1 civetweb: 0x56202c6aa000:
192.168.120.21 - - [11/Jan/2018:20:14:36 +] "HEAD
/Jobimages/vendor05/10/3962896/3962896_cover.pdf HTTP/1.1" 1 0 -
aws-sdk-dotnet-35/2

.0.2.2 .NET Runtime/4.0 .NET Framework/4.0 OS/6.2.9200.0 FileIO

2018-01-11 20:14:38.163617 7f5836ac3700  1 == starting new request
req=0x7f5836abd1f0 =

2018-01-11 20:14:38.165352 7f5836ac3700  1 == req done
req=0x7f5836abd1f0 op status=0 http_status=404 ==

2018-01-11 20:14:38.165401 7f5836ac3700  1 civetweb: 0x56202c4e2000:
192.168.120.21 - - [11/Jan/2018:20:14:38 +] "HEAD
/Jobimages/vendor05/10/3445645/3445645_cover.pdf HTTP/1.1" 1 0 -
aws-sdk-dotnet-35/2

.0.2.2 .NET Runtime/4.0 .NET Framework/4.0 OS/6.2.9200.0 FileIO

2018-01-11 20:14:38.170551 7f5807a65700  1 == starting new request
req=0x7f5807a5f1f0 =

2018-01-11 20:14:40.322236 7f58352c0700  1 == starting new request
req=0x7f58352ba1f0 =

2018-01-11 20:14:40.323468 7f5834abf700  1 == starting new request
req=0x7f5834ab91f0 =

2018-01-11 20:14:41.643365 7f58342be700  1 == starting new request
req=0x7f58342b81f0 =

2018-01-11 20:14:41.643358 7f58312b8700  1 == starting new request
req=0x7f58312b21f0 =

2018-01-11 20:14:50.324196 7f5829aa9700  1 == starting new request
req=0x7f5829aa31f0 =

2018-01-11 20:14:50.325622 7f58332bc700  1 == starting new request
req=0x7f58332b61f0 =

2018-01-11 20:14:51.645678 7f58362c2700  1 == starting new request
req=0x7f58362bc1f0 =

2018-01-11 20:14:51.645671 7f582e2b2700  1 == starting new request
req=0x7f582e2ac1f0 =

2018-01-11 20:15:00.326452 7f5815a81700  1 == starting new request
req=0x7f5815a7b1f0 =

2018-01-11 20:15:00.328787 7f5828aa7700  1 == starting new request
req=0x7f5828aa11f0 =

2018-01-11 20:15:01.648196 7f580ea73700  1 == starting new request
req=0x7f580ea6d1f0 =

2018-01-11 20:15:01.648698 7f5830ab7700  1 == starting new request
req=0x7f5830ab11f0 =

2018-01-11 20:15:10.328810 7f5832abb700  1 == starting new request
req=0x7f5832ab51f0 =

2018-01-11 20:15:10.329541 7f582f2b4700  1 == starting new request
req=0x7f582f2ae1f0 =

2018-01-11 20:15:11.650655 7f582d2b0700  1 == starting new request
req=0x7f582d2aa1f0 =

2018-01-11 20:15:11.651401 7f582aaab700  1 == starting new request
req=0x7f582aaa51f0 =

2018-01-11 20:15:20.332032 7f582c2ae700  1 == starting new request
req=0x7f582c2a81f0 

Re: [ceph-users] Incomplete pgs and no data movement ( cluster appears readonly )

[Brent Kennedy]

Ugh, that’s what I was hoping to avoid.  OSD 13 is still in the server, I 
wonder if I could somehow bring it back in as OSD 13 to see if it has the 
missing data.  

I was looking into using the ceph-objectstore tool, but the only instructions I 
can find online are sparse and mostly in this lists archive.  I am still trying 
to get clarification on the data itself as I was hoping to delete it, but even 
the deletion process doesn’t seem to exist.  All I can find is a force-create 
feature that seems to force recreate the pg, but again the documentation that 
is weak as well.

-Brent

  

-Original Message-
From: Gregory Farnum [mailto:gfar...@redhat.com] 
Sent: Wednesday, January 10, 2018 3:15 PM
To: Brent Kennedy <bkenn...@cfl.rr.com>
Cc: Janne Johansson <icepic...@gmail.com>; Ceph Users 
<ceph-users@lists.ceph.com>
Subject: Re: [ceph-users] Incomplete pgs and no data movement ( cluster appears 
readonly )

On Wed, Jan 10, 2018 at 11:14 AM, Brent Kennedy <bkenn...@cfl.rr.com> wrote:
> I adjusted “osd max pg per osd hard ratio ” to 50.0 and left “mon max 
> pg per osd” at 5000 just to see if things would allow data movement.  
> This worked, the new pool I created finished its creation and spread 
> out.  I was able to then copy the data from the existing pool into the 
> new pool and delete the old one.
>
>
>
> Used this process for copying the default pools:
>
> ceph osd pool create .users.email.new 16
>
> rados cppool .users.email .users.email.new
>
> ceph osd pool delete .users.email .users.email 
> --yes-i-really-really-mean-it
>
> ceph osd pool rename .users.email.new .users.email
>
> ceph osd pool application enable .users.email rgw
>
>
>
>
>
> So at this point, I have recreated all the .rgw and .user pools except 
> .rgw.buckets with a pg_num of 16, which significantly reduced the pgs, 
> unfortunately, the incompletes are still there:
>
>
>
>   cluster:
>
>health: HEALTH_WARN
>
> Reduced data availability: 4 pgs inactive, 4 pgs 
> incomplete
>
> Degraded data redundancy: 4 pgs unclean

There seems to have been some confusion here. From your prior thread:

On Thu, Jan 4, 2018 at 9:56 PM, Brent Kennedy <bkenn...@cfl.rr.com> wrote:
> We have upgraded from Hammer to Jewel and then Luminous 12.2.2 as of today.
> During the hammer upgrade to Jewel we lost two host servers

So, if you have size two, and you lose two servers before the data has finished 
recovering...you've lost data. And that is indeed what "incomplete" means: the 
PG thinks writes may have happened, but the OSDs which held the data at that 
time aren't available. You'll need to dive into doing PG recovery with the 
ceph-objectstore tool and things, or find one of the groups that does 
consulting around recovery.
-Greg

>
>
>
>   services:
>
> mon: 3 daemons, quorum mon1,mon2,mon3
>
> mgr: mon3(active), standbys: mon1, mon2
>
> osd: 43 osds: 43 up, 43 in
>
>
>
>   data:
>
> pools:   10 pools, 4240 pgs
>
> objects: 8148k objects, 10486 GB
>
> usage:   21536 GB used, 135 TB / 156 TB avail
>
> pgs: 0.094% pgs not active
>
>  4236 active+clean
>
>  4incomplete
>
>
>
> The health page is showing blue instead of read on the donut chart, at 
> one point it jumped to green but its back to blue currently.  There 
> are no more ops blocked/delayed either.
>
>
>
> Thanks for assistance, it seems the cluster will play nice now.  Any 
> thoughts on the stuck pgs?  I ran a query on 11.720 and it shows:
>
> "blocked_by": [
>
> 13,
>
> 27,
>
> 28
>
>
>
> OSD 13 was acting strange so I wiped it and removed it from the cluster.
> This was during the rebuild so I wasn’t aware of it blocking.  Now I 
> am trying to figure out how a removed OSD is blocking.  I went through 
> the process to remove it:
>
> ceph osd crush remove
>
> ceph auth del
>
> ceph osd rm
>
>
>
> I guess since the cluster was a hot mess at that point, its possible 
> it was borked and therefore the pg is borked.  I am trying to avoid 
> deleting the data as there is data in the OSDs that are online.
>
>
>
> -Brent
>
>
>
>
>
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf 
> Of Brent Kennedy
> Sent: Wednesday, January 10, 2018 12:20 PM
> To: 'Janne Johansson' <icepic...@gmail.com>
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Incomplete pgs and no data movement ( 
> cluster appears readonly )
>
>
>
> I change “mon max pg per osd” to 5000 because when I changed it to 
&g

Re: [ceph-users] Incomplete pgs and no data movement ( cluster appears readonly )

[Brent Kennedy]

I adjusted “osd max pg per osd hard ratio ” to 50.0 and left “mon max pg per 
osd” at 5000 just to see if things would allow data movement.  This worked, the 
new pool I created finished its creation and spread out.  I was able to then 
copy the data from the existing pool into the new pool and delete the old one.  

 

Used this process for copying the default pools:

ceph osd pool create .users.email.new 16

rados cppool .users.email .users.email.new

ceph osd pool delete .users.email .users.email --yes-i-really-really-mean-it

ceph osd pool rename .users.email.new .users.email

ceph osd pool application enable .users.email rgw

 

 

So at this point, I have recreated all the .rgw and .user pools except 
.rgw.buckets with a pg_num of 16, which significantly reduced the pgs, 
unfortunately, the incompletes are still there:

 

  cluster:

   health: HEALTH_WARN

Reduced data availability: 4 pgs inactive, 4 pgs incomplete

Degraded data redundancy: 4 pgs unclean

 

  services:

mon: 3 daemons, quorum mon1,mon2,mon3

mgr: mon3(active), standbys: mon1, mon2

osd: 43 osds: 43 up, 43 in

 

  data:

pools:   10 pools, 4240 pgs

objects: 8148k objects, 10486 GB

usage:   21536 GB used, 135 TB / 156 TB avail

pgs: 0.094% pgs not active

 4236 active+clean

 4incomplete

 

The health page is showing blue instead of read on the donut chart, at one 
point it jumped to green but its back to blue currently.  There are no more ops 
blocked/delayed either. 

 

Thanks for assistance, it seems the cluster will play nice now.  Any thoughts 
on the stuck pgs?  I ran a query on 11.720 and it shows:

"blocked_by": [

13,

27,

28

 

OSD 13 was acting strange so I wiped it and removed it from the cluster.  This 
was during the rebuild so I wasn’t aware of it blocking.  Now I am trying to 
figure out how a removed OSD is blocking.  I went through the process to remove 
it:

ceph osd crush remove

ceph auth del

ceph osd rm

 

I guess since the cluster was a hot mess at that point, its possible it was 
borked and therefore the pg is borked.  I am trying to avoid deleting the data 
as there is data in the OSDs that are online.

 

-Brent

 

 

From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Brent 
Kennedy
Sent: Wednesday, January 10, 2018 12:20 PM
To: 'Janne Johansson' <icepic...@gmail.com>
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Incomplete pgs and no data movement ( cluster appears 
readonly )

 

I change “mon max pg per osd” to 5000 because when I changed it to zero, which 
was supposed to disable it, it caused an issue where I couldn’t create any 
pools.  It would say 0 was larger than the minimum.  I imagine that’s a bug, if 
I wanted it disabled, then it shouldn’t use the calculation.  I then set “osd 
max pg per osd hard ratio ” to 5 after changing “mon max pg per osd” to 5000, 
figuring 5*5000 would cover it.  Perhaps not.  I will adjust it to 30 and 
restart the OSDs.

 

-Brent

 

 

 

From: Janne Johansson [mailto:icepic...@gmail.com] 
Sent: Wednesday, January 10, 2018 3:00 AM
To: Brent Kennedy <bkenn...@cfl.rr.com <mailto:bkenn...@cfl.rr.com> >
Cc: ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> 
Subject: Re: [ceph-users] Incomplete pgs and no data movement ( cluster appears 
readonly )

 

 

 

2018-01-10 8:51 GMT+01:00 Brent Kennedy <bkenn...@cfl.rr.com 
<mailto:bkenn...@cfl.rr.com> >:

As per a previous thread, my pgs are set too high.  I tried adjusting the “mon 
max pg per osd” up higher and higher, which did clear the error(restarted 
monitors and managers each time), but it seems that data simply wont move 
around the cluster.  If I stop the primary OSD of an incomplete pg, the cluster 
just shows those affected pages as active+undersized+degraded:

 

I also adjusted “osd max pg per osd hard ratio ” to 5, but that didn’t seem to 
trigger any data moved.  I did restart the OSDs each time I changed it.  The 
data just wont finish moving.  “ceph –w” shows this:

2018-01-10 07:49:27.715163 osd.20 [WRN] slow request 960.675164 seconds old, 
received at 2018-01-10 07:33:27.039907: osd_op(client.3542508.0:4097 14.0 
14.50e8d0b0 (undecoded) ondisk+write+known_if_redirected e125984) currently 
queued_for_pg

 

 

Did you bump the ratio so that the PGs per OSD max * hard ratio actually became 
more than the amount of PGs you had?

Last time you mailed the ratio was 25xx and the max was 200 which meant the 
ratio would have needed to be far more than 5.0.

 

 

-- 

May the most significant bit of your life be positive.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Incomplete pgs and no data movement ( cluster appears readonly )

[Brent Kennedy]

I change “mon max pg per osd” to 5000 because when I changed it to zero, which 
was supposed to disable it, it caused an issue where I couldn’t create any 
pools.  It would say 0 was larger than the minimum.  I imagine that’s a bug, if 
I wanted it disabled, then it shouldn’t use the calculation.  I then set “osd 
max pg per osd hard ratio ” to 5 after changing “mon max pg per osd” to 5000, 
figuring 5*5000 would cover it.  Perhaps not.  I will adjust it to 30 and 
restart the OSDs.

 

-Brent

 

 

 

From: Janne Johansson [mailto:icepic...@gmail.com] 
Sent: Wednesday, January 10, 2018 3:00 AM
To: Brent Kennedy <bkenn...@cfl.rr.com>
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Incomplete pgs and no data movement ( cluster appears 
readonly )

 

 

 

2018-01-10 8:51 GMT+01:00 Brent Kennedy <bkenn...@cfl.rr.com 
<mailto:bkenn...@cfl.rr.com> >:

As per a previous thread, my pgs are set too high.  I tried adjusting the “mon 
max pg per osd” up higher and higher, which did clear the error(restarted 
monitors and managers each time), but it seems that data simply wont move 
around the cluster.  If I stop the primary OSD of an incomplete pg, the cluster 
just shows those affected pages as active+undersized+degraded:

 

I also adjusted “osd max pg per osd hard ratio ” to 5, but that didn’t seem to 
trigger any data moved.  I did restart the OSDs each time I changed it.  The 
data just wont finish moving.  “ceph –w” shows this:

2018-01-10 07:49:27.715163 osd.20 [WRN] slow request 960.675164 seconds old, 
received at 2018-01-10 07:33:27.039907: osd_op(client.3542508.0:4097 14.0 
14.50e8d0b0 (undecoded) ondisk+write+known_if_redirected e125984) currently 
queued_for_pg

 

 

Did you bump the ratio so that the PGs per OSD max * hard ratio actually became 
more than the amount of PGs you had?

Last time you mailed the ratio was 25xx and the max was 200 which meant the 
ratio would have needed to be far more than 5.0.

 

 

-- 

May the most significant bit of your life be positive.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Incomplete pgs and no data movement ( cluster appears readonly )

[Brent Kennedy]

As per a previous thread, my pgs are set too high.  I tried adjusting the
"mon max pg per osd" up higher and higher, which did clear the
error(restarted monitors and managers each time), but it seems that data
simply wont move around the cluster.  If I stop the primary OSD of an
incomplete pg, the cluster just shows those affected pages as
active+undersized+degraded:

 

services:

mon: 3 daemons, quorum mon1,mon2,mon3

mgr: mon3(active), standbys: mon1, mon2

osd: 43 osds: 43 up, 43 in

 

data:

pools:   11 pools, 36896 pgs

objects: 8148k objects, 10486 GB

usage:   21532 GB used, 135 TB / 156 TB avail

pgs: 0.043% pgs unknown

 0.011% pgs not active

 362942/16689272 objects degraded (2.175%)

 34483 active+clean

 2393  active+undersized+degraded

 16unknown

 3 incomplete

1   down

 

The 16 unknown are from me trying to setup a new pool, which was successful,
but when I tried to copy an existing pool to it, the command just sat there.
I did this in the hopes of copying the existing oversized pg pools to new
pools and then deleting the old pools.  I really didn't want to move the
data, but the issue needs to be dealt with.

 

If I start the OSD back up, the cluster goes back to:

services:

mon: 3 daemons, quorum mon1,mon2,mon3

mgr: mon3(active), standbys: mon1, mon2

osd: 43 osds: 43 up, 43 in

 

  data:

pools:   11 pools, 36896 pgs

objects: 8148k objects, 10486 GB

usage:   21533 GB used, 135 TB / 156 TB avail

pgs: 0.041% pgs unknown

 0.014% pgs not active

 36876 active+clean

 16unknown

 4 incomplete

 

The cluster was upgraded from Hammer .94 without issues to Jewel and then
Luminous 12.2.2 last week using the latest ceph-deploy.

 

I guess the issue at the moment is that data is not moving either for
recovery or new data being added( basically the new data just times out ).  

 

I also adjusted "osd max pg per osd hard ratio " to 5, but that didn't seem
to trigger any data moved.  I did restart the OSDs each time I changed it.
The data just wont finish moving.  "ceph -w" shows this:

2018-01-10 07:49:27.715163 osd.20 [WRN] slow request 960.675164 seconds old,
received at 2018-01-10 07:33:27.039907: osd_op(client.3542508.0:4097 14.0
14.50e8d0b0 (undecoded) ondisk+write+known_if_redirected e125984) currently
queued_for_pg

 

Ceph health detail this:

HEALTH_ERR Reduced data availability: 20 pgs inactive, 4 pgs incomplete;
Degraded data redundancy: 20 pgs unclean; 2 slow requests are blocked > 32
sec; 66 stuck requests are blocked > 4096 sec

PG_AVAILABILITY Reduced data availability: 20 pgs inactive, 4 pgs incomplete

pg 11.720 is incomplete, acting [21,10]

pg 11.9ab is incomplete, acting [14,2]

pg 11.9fb is incomplete, acting [32,43]

pg 11.c13 is incomplete, acting [42,26]

pg 14.0 is stuck inactive for 1046.844458, current state unknown, last
acting []

pg 14.1 is stuck inactive for 1046.844458, current state unknown, last
acting []

pg 14.2 is stuck inactive for 1046.844458, current state unknown, last
acting []

pg 14.3 is stuck inactive for 1046.844458, current state unknown, last
acting []

pg 14.4 is stuck inactive for 1046.844458, current state unknown, last
acting []

pg 14.5 is stuck inactive for 1046.844458, current state unknown, last
acting []

pg 14.6 is stuck inactive for 1046.844458, current state unknown, last
acting []

pg 14.7 is stuck inactive for 1046.844458, current state
creating+activating, last acting [21,40,5]

pg 14.8 is stuck inactive for 1046.844458, current state unknown, last
acting []

pg 14.9 is stuck inactive for 1046.844458, current state unknown, last
acting []

pg 14.a is stuck inactive for 1046.844458, current state unknown, last
acting []

pg 14.b is stuck inactive for 1046.844458, current state unknown, last
acting []

pg 14.c is stuck inactive for 1046.844458, current state unknown, last
acting []

   pg 14.d is stuck inactive for 1046.844458, current state unknown, last
acting []

pg 14.e is stuck inactive for 1046.844458, current state unknown, last
acting []

pg 14.f is stuck inactive for 1046.844458, current state unknown, last
acting []

PG_DEGRADED Degraded data redundancy: 20 pgs unclean

pg 11.720 is stuck unclean since forever, current state incomplete, last
acting [21,10]

pg 11.9ab is stuck unclean since forever, current state incomplete, last
acting [14,2]

pg 11.9fb is stuck unclean since forever, current state incomplete, last
acting [32,43]

pg 11.c13 is stuck unclean since forever, current state incomplete, last
acting [42,26]

pg 14.0 is stuck unclean for 1046.844458, current state unknown, last
acting []

pg 14.1 is stuck unclean for 1046.844458, current state unknown, last
acting []

pg 14.2 is stuck unclean for 

Re: [ceph-users] Reduced data availability: 4 pgs inactive, 4 pgs incomplete

[Brent Kennedy]

Unfortunately, I don’t see that setting documented anywhere other than the 
release notes.  Its hard to find guidance for questions in that case, but 
luckily you noted it in your blog post.  I wish I knew what setting to put that 
at.  I did use the deprecated one after moving to hammer a while back due to 
the mis-calcuated PGs.  I have now that settings, but used 0 as the value, 
which cleared the error in the status, but the stuck incomplete pgs persist.  I 
restarted all daemons, so it should be in full effect.  Interestingly enough, 
it added the hdd in the ceph osd tree output...   anyhow, I know this is a 
dirty cluster due to this mis-calcuation, I would like to fix the cluster if 
possible ( both the stuck/incomplete and the underlying too many pgs issue )

Thanks for the information!

-Brent

-Original Message-
From: Jens-U. Mozdzen [mailto:jmozd...@nde.ag] 
Sent: Sunday, January 7, 2018 1:23 PM
To: bkenn...@cfl.rr.com
Cc: ste...@bit.nl
Subject: Fwd: [ceph-users] Reduced data availability: 4 pgs inactive, 4 pgs 
incomplete

Hi Brent,

sorry, the quoting style had me confused, this actually was targeted at your 
question, I believe.

@Stefan: Sorry for the noise

Regards,
Jens
- Weitergeleitete Nachricht von "Jens-U. Mozdzen" <jmozd...@nde.ag> -
   Datum: Sun, 07 Jan 2018 18:18:00 +
 Von: "Jens-U. Mozdzen" <jmozd...@nde.ag>
Betreff: Re: [ceph-users] Reduced data availability: 4 pgs inactive, 4 pgs 
incomplete
  An: ste...@bit.nl

Hi Stefan,

I'm in a bit of a hurry, so just a short offline note:

>>> Quoting Brent Kennedy (bkenn...@cfl.rr.com):
>>> Unfortunately, this cluster was setup before the calculator was in 
>>> place and when the equation was not well understood.  We have the 
>>> storage space to move the pools and recreate them, which was 
>>> apparently the only way to handle the issue( you are suggesting what 
>>> appears to be a different approach ).  I was hoping to avoid doing 
>>> all of this because the migration would be very time consuming.  
>>> There is no way to fix the stuck pg?s though?  If I were to expand 
>>> the replication to 3 instances, would that help with the PGs per OSD 
>>> issue any?
>> No! It will make the problem worse because you need PGs to host these 
>> copies. The more replicas, the more PGs you need.
> Guess I am confused here, wouldn?t it spread out the existing data to 
> more PGs?  Or are you saying that it couldn?t spread out because the 
> PGs are already in use?  Previously it was set to 3 and we reduced it 
> to 2 because of failures.

If you increase the replication size, you'll ask RADOS to create additional 
copies in additional PGs. So the number of PGs will increase...

>>> When you say enforce, do you mean it will block all access to the 
>>> cluster/OSDs?
>> No, [...]

My experience differs: If you cluster already has too many PGs per OSD  
(before upgrading to 12.2.2) and anything PG-per-OSD-related changes  
(i.e. re-distributing PGs when OSDs go down), any access to the  
over-sized OSDs *will block*. Cost me a number of days to figure out  
and was recently discussed by someone else on the ML. Increase the  
according parameter ("mon_max_pg_per_osd") in the global section and  
restart your MONs, MGRs and OSDs (OSDs one by one, if you don't know  
your layout, to avoid data loss). Made me even create a blog entry,  
for future reference:  
http://technik.blogs.nde.ag/2017/12/26/ceph-12-2-2-minor-update-major-trouble/

If this needs to go into more details, let's take it back to the  
mailing list, I'll be available again during the upcoming week.

Regards,
Jens

- Ende der weitergeleiteten Nachricht -


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Reduced data availability: 4 pgs inactive, 4 pgs incomplete

[Brent Kennedy]

-Original Message-
From: Stefan Kooman [mailto:ste...@bit.nl] 
Sent: Friday, January 5, 2018 2:58 PM
To: Brent Kennedy <bkenn...@cfl.rr.com>
Cc: 'Janne Johansson' <icepic...@gmail.com>; 'ceph-users' <ceph-us...@ceph.com>
Subject: Re: [ceph-users] Reduced data availability: 4 pgs inactive, 4 pgs 
incomplete

Quoting Brent Kennedy (bkenn...@cfl.rr.com):
> Unfortunately, this cluster was setup before the calculator was in 
> place and when the equation was not well understood.  We have the 
> storage space to move the pools and recreate them, which was 
> apparently the only way to handle the issue( you are suggesting what 
> appears to be a different approach ).  I was hoping to avoid doing all 
> of this because the migration would be very time consuming.  There is 
> no way to fix the stuck pg’s though?  If I were to expand the 
> replication to 3 instances, would that help with the PGs per OSD issue 
> any?
No! It will make the problem worse because you need PGs to host these copies. 
The more replicas, the more PGs you need.
Guess I am confused here, wouldn’t it spread out the existing data to more PGs? 
 Or are you saying that it couldn’t spread out because the PGs are already in 
use?  Previously it was set to 3 and we reduced it to 2 because of failures.

> The math was originally based on 3 not the current 2.  Sounds like it 
> may change to 300 max which may not be helpful…

> When you say enforce, do you mean it will block all access to the 
> cluster/OSDs?

No, you will not be able to increase the number of PGs on the pool.
I had no intention of doing so, but it sounds like you are saying is that 
"enforcing" means it wont allow additional PGs.   That makes sense.

> 
> We have upgraded from Hammer to Jewel and then Luminous 12.2.2 as of 
> today.  During the hammer upgrade to Jewel we lost two host servers 
> and let the cluster rebalance/recover, it ran out of space and 
> stalled.  We then added three new host servers and then let the 
> cluster rebalance/recover. During that process, at some point, we 
> ended up with 4 pgs not being able to be repaired using “ceph pg 
> repair xx.xx”.  I tried using ceph pg 11.720 query and from what I can 
> tell the missing information matches, but is being blocked from being 
> marked clean.  I keep seeing references to the ceph-object-store tool 
> to use as an export/restore method, but I cannot find details on a 
> step by step process given the current predicament.  It may also be 
> possible for us to just lose the data if it cant be extracted so we 
> can at least return the cluster to a healthy state.  Any thoughts?

What is the output of "ceph daemon osd.$ID config show | grep 
osd_allow_recovery_below_min_size

The output was "true" for all OSDs in the cluster, so should be in the clear 
there.

If you are below min_size recovery will not complete when that setting is not 
true. Maybe this thread is interesting:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-October/005613.html

Especially when a OSD is candidate for backfilling a target but does not 
contain any data.

Gr. Stefan

So perhaps the only way to deal with this would be creating a new pool and 
migrating all the data to the new pool?  We are using radosgw, so that 
complicates the process a bit as I believe it requires certain pools to be in 
place.  I was trying to find a list of the default pools that could be deleted, 
mainly because some of those pools are set for a high amount of PGs as well, 
deleting them should help.

Pools List:
NamePG status   Usage   Activity
data512 active+clean0 / 45.5T   0 rd, 0 
wr
rbd 512 active+clean0 / 45.5T   0 rd, 0 
wr
.rgw.root   4096 active+clean   1.01k / 45.5T   0 rd, 0 
wr
.rgw.control4096 active+clean   0 / 45.5T   0 rd, 0 
wr
.rgw4096 active+clean   9.48k / 45.5T   0 rd, 0 
wr
.rgw.gc 4096 active+clean   0 / 45.5T   0 rd, 0 
wr
.users.uid  4096 active+clean   961 / 45.5T 0 rd, 0 
wr
.users  4096 active+clean   33 / 45.5T  0 rd, 0 
wr
.rgw.buckets.index  4096 active+clean   0 / 45.5T   0 rd, 0 
wr
.rgw.buckets4092 active+clean, 4 incomplete 11.2T / 68.3T   0 rd, 0 
wr
.users.email4096 active+clean   0 / 45.5T   0 rd, 0 
wr
.log16 active+clean 0 / 45.5T   0 rd, 0 
wr


-Brent

-- 
| BIT BV  http://www.bit.nl/Kamer van Koophandel 09090351
| GPG: 0xD14839C6   +31 318 648 688 / i...@bit.nl

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Reduced data availability: 4 pgs inactive, 4 pgs incomplete

[Brent Kennedy]

Unfortunately, this cluster was setup before the calculator was in place and 
when the equation was not well understood.  We have the storage space to move 
the pools and recreate them, which was apparently the only way to handle the 
issue( you are suggesting what appears to be a different approach ).  I was 
hoping to avoid doing all of this because the migration would be very time 
consuming.  There is no way to fix the stuck pg’s though?  If I were to expand 
the replication to 3 instances, would that help with the PGs per OSD issue any? 
 The math was originally based on 3 not the current 2.  Sounds like it may 
change to 300 max which may not be helpful…

 

When you say enforce, do you mean it will block all access to the cluster/OSDs?

 

-Brent

 

From: Janne Johansson [mailto:icepic...@gmail.com] 
Sent: Friday, January 5, 2018 3:34 AM
To: Brent Kennedy <bkenn...@cfl.rr.com>; ceph-users <ceph-us...@ceph.com>
Subject: Re: [ceph-users] Reduced data availability: 4 pgs inactive, 4 pgs 
incomplete

 

 

 

2018-01-05 6:56 GMT+01:00 Brent Kennedy <bkenn...@cfl.rr.com 
<mailto:bkenn...@cfl.rr.com> >:

We have upgraded from Hammer to Jewel and then Luminous 12.2.2 as of today.  
During the hammer upgrade to Jewel we lost two host servers and let the cluster 
rebalance/recover, it ran out of space and stalled.  We then added three new 
host servers and then let the cluster rebalance/recover. During that process, 
at some point, we ended up with 4 pgs not being able to be repaired using “ceph 
pg repair xx.xx”.  I tried using ceph pg 11.720 query and from what I can tell 
the missing information matches, but is being blocked from being marked clean.  
I keep seeing references to the ceph-object-store tool to use as an 
export/restore method, but I cannot find details on a step by step process 
given the current predicament.  It may also be possible for us to just lose the 
data if it cant be extracted so we can at least return the cluster to a healthy 
state.  Any thoughts?

Ceph –s output:

 

cluster:

health: HEALTH_ERR

Reduced data availability: 4 pgs inactive, 4 pgs incomplete

Degraded data redundancy: 4 pgs unclean

4 stuck requests are blocked > 4096 sec

too many PGs per OSD (2549 > max 200)

^

 

This is the issue.

A temp workaround will be to bump the hard_ratio and perhaps restart the OSDs 
after (or add a ton of OSDs so the

PG/OSD gets below 200) 

In your case, the

osd max pg per osd hard ratio

needs to go from 2.0 to 26.0 or above, which probably is rather crazy.

 

The thing is that Luminous 12.2.2 starts enforcing this which previous versions 
didn't (at least not in the same way).

 

Even if it is rather weird to run into this, you should have seen the warning 
before (even if it was > 300 previously) which also means

you should perhaps have considered not upgrading when the cluster wasn't 
HEALTH_OK if it was warning about huge amount of PGs

before going to 12.2.2.

 

-- 

May the most significant bit of your life be positive.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Reduced data availability: 4 pgs inactive, 4 pgs incomplete

[Brent Kennedy]

We have upgraded from Hammer to Jewel and then Luminous 12.2.2 as of today.
During the hammer upgrade to Jewel we lost two host servers and let the
cluster rebalance/recover, it ran out of space and stalled.  We then added
three new host servers and then let the cluster rebalance/recover. During
that process, at some point, we ended up with 4 pgs not being able to be
repaired using "ceph pg repair xx.xx".  I tried using ceph pg 11.720 query
and from what I can tell the missing information matches, but is being
blocked from being marked clean.  I keep seeing references to the
ceph-object-store tool to use as an export/restore method, but I cannot find
details on a step by step process given the current predicament.  It may
also be possible for us to just lose the data if it cant be extracted so we
can at least return the cluster to a healthy state.  Any thoughts?

Ceph -s output:

 

cluster:

health: HEALTH_ERR

Reduced data availability: 4 pgs inactive, 4 pgs incomplete

Degraded data redundancy: 4 pgs unclean

4 stuck requests are blocked > 4096 sec

too many PGs per OSD (2549 > max 200)

 

  services:

mon: 3 daemons, quorum ukpixmon1,ukpixmon2,ukpixmon3

mgr: ukpixmon1(active), standbys: ukpixmon3, ukpixmon2

osd: 43 osds: 43 up, 43 in

rgw: 3 daemons active

 

  data:

pools:   12 pools, 37904 pgs

objects: 8148k objects, 10486 GB

usage:   21530 GB used, 135 TB / 156 TB avail

pgs: 0.011% pgs not active

 37900 active+clean

 4 incomplete

 

OSD TREE output:

 

ID CLASS WEIGHTTYPE NAME   STATUS REWEIGHT PRI-AFF

-1   156.10268 root default

-232.57996 host osdhost1

0 3.62000 osd.0   up  1.0 1.0

1 3.62000 osd.1   up  1.0 1.0

2 3.62000 osd.2   up  1.0 1.0

3 3.62000 osd.3   up  1.0 1.0

4 3.62000 osd.4   up  1.0 1.0

5 3.62000 osd.5   up  1.0 1.0

6 3.62000 osd.6   up  1.0 1.0

7 3.62000 osd.7   up  1.0 1.0

8 3.62000 osd.8   up  1.0 1.0

-325.33997 host osdhost2

9 3.62000 osd.9   up  1.0 1.0

10 3.62000 osd.10  up  1.0 1.0

11 3.62000 osd.11  up  1.0 1.0

12 3.62000 osd.12  up  1.0 1.0

15 3.62000 osd.15  up  1.0 1.0

16 3.62000 osd.16  up  1.0 1.0

17 3.62000 osd.17  up  1.0 1.0

-832.72758 host osdhost6

14 3.63640 osd.14  up  1.0 1.0

21 3.63640 osd.21  up  1.0 1.0

23 3.63640 osd.23  up  1.0 1.0

26 3.63640 osd.26  up  1.0 1.0

32 3.63640 osd.32  up  1.0 1.0

33 3.63640 osd.33  up  1.0 1.0

34 3.63640 osd.34  up  1.0 1.0

35 3.63640 osd.35  up  1.0 1.0

36 3.63640 osd.36  up  1.0 1.0

-932.72758 host osdhost7

19 3.63640 osd.19  up  1.0 1.0

37 3.63640 osd.37  up  1.0 1.0

38 3.63640 osd.38  up  1.0 1.0

39 3.63640 osd.39  up  1.0 1.0

40 3.63640 osd.40  up  1.0 1.0

41 3.63640 osd.41  up  1.0 1.0

42 3.63640 osd.42  up  1.0 1.0

43 3.63640 osd.43  up  1.0 1.0

44 3.63640 osd.44  up  1.0 1.0

-732.72758 host osdhost8

20 3.63640 osd.20  up  1.0 1.0

45 3.63640 osd.45  up  1.0 1.0

46 3.63640 osd.46  up  1.0 1.0

47 3.63640 osd.47  up  1.0 1.0

48 3.63640 osd.48  up  1.0 1.0

49 3.63640 osd.49  up  1.0 1.0

50 3.63640 osd.50  up  1.0 1.0

51 3.63640 osd.51  up  1.0 1.0

52 3.63640 osd.52  up  1.0 1.0

 

Ceph health detail output:

HEALTH_ERR Reduced data availability: 4 pgs inactive, 4 pgs incomplete;
Degraded data redundancy: 4 pgs unclean; 4 stuck requests are blocked > 4096
sec; too many PGs per OSD (2549 > max 200)

PG_AVAILABILITY Reduced data availability: 4 pgs inactive, 4 pgs incomplete

pg 11.720 is incomplete, acting [21,10]

pg 

[ceph-users] Removing an OSD host server

[Brent Kennedy]

Been looking around the web and I cant find a what seems to be "clean way"
to remove an OSD host from the "ceph osd tree" command output.  I am
therefore hesitant to add a server with the same name, but I still see the
removed/failed nodes from the list.  Anyone know how to do that?  I found an
article here, but it doesn't seem to be a clean way:
https://arvimal.blog/2015/05/07/how-to-remove-a-host-from-a-ceph-cluster/

 

Regards,

-Brent

 

Existing Clusters:

Test: Jewel with 3 osd servers, 1 mon, 1 gateway

US Production: Firefly with 4 osd servers, 3 mons, 3 gateways behind haproxy
LB

UK Production: Hammer with 5 osd servers, 3 mons, 3 gateways behind haproxy
LB

 

 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Gateway timeout

[Brent Kennedy]

I have noticed over the years ( been using ceph since 2013 ) that when an
OSD attached to a single physical drive ( JBOD setup ) that is failing, that
at times this will cause rados gateways to go offline.  I have two clusters
running ( one on firefly and one on hammer, both scheduled for upgrades next
year ) and it happens on both when a drive is not marked out but has many
blocked ops requests.  The drive is physically still functioning but is most
likely failing, just not failed yet.  The issue is that the gateways will
just stop responding to all requests.  Both of our clusters have 3 rados
gateways behind a haproxy load balancer, so we know immediately when they
drop.  This will occur continually until we out the failing OSD ( normally
we restart the gateways or the services on them first, then move to out the
drive ).  

 

Wonder if anyone runs into this, a quick search revealed one hit with no
actual resolution.  Also wondering if there is some way I could prevent the
gateways from falling over due to the unresponsive OSD?

 

I did setup a test Jewel install in our dev and semi-recreate the problem by
shutting down all the OSDs.  This resulted in the gateway going down
completely as well.  I imagine taking the OSDs offline like that wouldn't be
expected though.  It would be nice if the gateway would just throw a message
back, like service unavailable.  I suppose haproxy is doing this for it
though.

 

Regards,

Brent

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com