[ceph-users] luminous OSD_ORPHAN

[Vladimir Prokofev]

Hello.

After some furious "ceph-deploy osd prepare/osd zap" cycles to figure out a
correct command for ceph-deploy to create a bluestore HDD with wal/db SSD,
I now have orphant OSDs, which are nowhere to be found in CRUSH map!

$ ceph health detail
HEALTH_WARN 4 osds exist in the crush map but not in the osdmap

OSD_ORPHAN 4 osds exist in the crush map but not in the osdmap
osd.20 exists in crush map but not in osdmap
osd.30 exists in crush map but not in osdmap
osd.31 exists in crush map but not in osdmap
osd.32 exists in crush map but not in osdmap

$ ceph osd crush remove osd.30
device 'osd.30' does not appear in the crush map
$ ceph osd crush remove 30
device '30' does not appear in the crush map

If I get CRUSH map with
$ ceph osd getcrushmap -o crm
$ crushtool -d crm -o crm.d
I don't see any mentioning of those OSDs there either.

I don't see this affecting my cluster in any way(yet), so as for now this
is a cosmetic issue.
But I'm worried it may somehow affect it in the future(not too much, as I
don't really see this happening), and what's worse, that cluster will not
return to "healty" state after it completes remapping/fixing degraded PGs.

Any ideas how to fix this?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph over IP over Infiniband

[Phil Schwarz]

Hi,
I'm currently trying to set up a brand new home cluster :
- 5 nodes, with each :

- 1 HCA Mellanox ConnectX-2
- 1 GB Ethernet (Proxmox 5.1 Network Admin)
- 1 CX4 to CX4 cable

All together connected to a SDR Flextronics IB Switch.

This setup should back a Ceph Luminous (V12.2.2 included in proxmox
V5.1) On all nodes, I did:
- apt-get infiniband-diags
- modprobe mlx4_ib
- modprobe ib_ipoib
- modprobe ib_umad
- ifconfig ib0 IP/MASK

On two nodes (tried previously on a single on, same issue), i installed
opensm ( The switch doesn't have SM included) :
apt-get install opensm
/etc/init.d/opensm stop
/etc/init.d/opensm start
(Necessary to let the daemon create the logfiles)

I tailed the logfile and got a "Active" Setup, with "SUBNET UP"

Every node is OK regardless to IB Setup :
- All ib0 are UP, using ibstat
- ibhosts and ibswitches seem to be OK

On a node :
ibping -S

On every other node :
ibping -G GID_Of_Previous_Server_Port

I got a nice pong reply on every node. Should be happy, but...
But i never went further.. Tried to ping each other. No way to get into
this (mostly probably) simple issue...


Any hint to achieve this task ??


Thanks for all
Best regards

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] determining the source of io in the cluster

[Konstantin Shalygin]

I can see, that the io/read ops come from the pool where we
store VM volumes, but i can't source this issue to a particular volume.
You can use this script 
https://github.com/cernceph/ceph-scripts/blob/master/tools/rbd-io-stats.pl


This is for filestore only. I adapted it to use bluestore for myself, 
but fast and it's not looks good.


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] using different version of ceph on cluster and client?

[13605702...@163.com]

hi 

my ceph  cluster is using Jewel on centos 7.3, kernel 3.10;
while our business running on centos 6.8, kernel 2.6.32, want to use rbd; 

is it ok to use Hammer on client?
or which version of ceph should be installed on client?

thanks



13605702...@163.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Luminous on armhf

[Peter Woodman]

er, yeah, i didn't read before i replied. that's fair, though it is
only some of the integration test binaries that tax that limit in a
single compile step.

On Mon, Dec 18, 2017 at 4:52 PM, Peter Woodman  wrote:
> not the larger "intensive" instance types! they go up to 128gb ram.
>
> On Mon, Dec 18, 2017 at 4:46 PM, Ean Price  wrote:
>> The problem with the native build on armhf is the compilation exceeds the 2 
>> GB of memory that ARMv7 (armhf) supports. Scaleway is pretty awesome but 
>> their 32 bit ARM systems have the same 2 GB limit. I haven’t tried the 
>> cross-compile on the 64 bit ARMv8 they offer and that might be easier than 
>> trying to do it on x86_64.
>>
>>> On Dec 18, 2017, at 4:41 PM, Peter Woodman  wrote:
>>>
>>> https://www.scaleway.com/
>>>
>>> they rent access to arm servers with gobs of ram.
>>>
>>> i've been building my own, but with some patches (removal of some
>>> asserts that were unnecessarily causing crashes while i try and track
>>> down the bug) that make it unsuitable for public consumption
>>>
>>> On Mon, Dec 18, 2017 at 4:38 PM, Andrew Knapp  wrote:
 I have no idea what this response means.

 I have tried building the armhf and arm64 package on my raspberry pi 3 to
 no avail.  Would love to see someone post Debian packages for stretch on
 arm64 or armhf.

 On Dec 18, 2017 4:12 PM, "Peter Woodman"  wrote:
>
> YMMV, but I've been using Scaleway instances to build packages for
> arm64- AFAIK you should be able to run any armhf distro on those
> machines as well.
>
> On Mon, Dec 18, 2017 at 4:02 PM, Andrew Knapp  wrote:
>> I would also love to see these packages!!!
>>
>> On Dec 18, 2017 3:46 PM, "Ean Price"  wrote:
>>
>> Hi everyone,
>>
>> I have a test cluster of armhf arch SoC systems running Xenial and Jewel
>> (10.2). I’m looking to do a clean rebuild with Luminous (12.2) but there
>> are
>> no 32 bit armhf binaries available. This is just a toy cluster and not
>> in
>> production.
>>
>> I have tried, unsuccessfully, to compile from source but they only have
>> 2GB
>> of memory and the system runs out of memory even with tuning and dialing
>> back compile options. I have tinkered around with cross compiling but
>> that
>> seems to land me in dependency hell on Xenial and I am a cross compile
>> newbie at any rate.
>>
>> Does anyone know of a source for a packaged version of Luminous for the
>> armhf architecture? Like I said, it’s just a test cluster so I’m not
>> overly
>> concerned about stability.
>>
>> Thanks in advance,
>> Ean
>> --
>> __
>>
>> This message contains information which may be confidential.  Unless you
>> are the addressee (or authorized to receive for the addressee), you may
>> not
>> use, copy, or disclose to anyone the message or any information
>> contained
>> in the message.  If you have received the message in error, please
>> advise
>> the sender by reply e-mail or contact the sender at Price Paper & Twine
>> Company by phone at (516) 378-7842 and delete the message.  Thank you
>> very
>> much.
>>
>> __
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>> --
>> __
>>
>> This message contains information which may be confidential.  Unless you
>> are the addressee (or authorized to receive for the addressee), you may not
>> use, copy, or disclose to anyone the message or any information contained
>> in the message.  If you have received the message in error, please advise
>> the sender by reply e-mail or contact the sender at Price Paper & Twine
>> Company by phone at (516) 378-7842 and delete the message.  Thank you very
>> much.
>>
>> __
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Luminous on armhf

[Peter Woodman]

not the larger "intensive" instance types! they go up to 128gb ram.

On Mon, Dec 18, 2017 at 4:46 PM, Ean Price  wrote:
> The problem with the native build on armhf is the compilation exceeds the 2 
> GB of memory that ARMv7 (armhf) supports. Scaleway is pretty awesome but 
> their 32 bit ARM systems have the same 2 GB limit. I haven’t tried the 
> cross-compile on the 64 bit ARMv8 they offer and that might be easier than 
> trying to do it on x86_64.
>
>> On Dec 18, 2017, at 4:41 PM, Peter Woodman  wrote:
>>
>> https://www.scaleway.com/
>>
>> they rent access to arm servers with gobs of ram.
>>
>> i've been building my own, but with some patches (removal of some
>> asserts that were unnecessarily causing crashes while i try and track
>> down the bug) that make it unsuitable for public consumption
>>
>> On Mon, Dec 18, 2017 at 4:38 PM, Andrew Knapp  wrote:
>>> I have no idea what this response means.
>>>
>>> I have tried building the armhf and arm64 package on my raspberry pi 3 to
>>> no avail.  Would love to see someone post Debian packages for stretch on
>>> arm64 or armhf.
>>>
>>> On Dec 18, 2017 4:12 PM, "Peter Woodman"  wrote:

 YMMV, but I've been using Scaleway instances to build packages for
 arm64- AFAIK you should be able to run any armhf distro on those
 machines as well.

 On Mon, Dec 18, 2017 at 4:02 PM, Andrew Knapp  wrote:
> I would also love to see these packages!!!
>
> On Dec 18, 2017 3:46 PM, "Ean Price"  wrote:
>
> Hi everyone,
>
> I have a test cluster of armhf arch SoC systems running Xenial and Jewel
> (10.2). I’m looking to do a clean rebuild with Luminous (12.2) but there
> are
> no 32 bit armhf binaries available. This is just a toy cluster and not
> in
> production.
>
> I have tried, unsuccessfully, to compile from source but they only have
> 2GB
> of memory and the system runs out of memory even with tuning and dialing
> back compile options. I have tinkered around with cross compiling but
> that
> seems to land me in dependency hell on Xenial and I am a cross compile
> newbie at any rate.
>
> Does anyone know of a source for a packaged version of Luminous for the
> armhf architecture? Like I said, it’s just a test cluster so I’m not
> overly
> concerned about stability.
>
> Thanks in advance,
> Ean
> --
> __
>
> This message contains information which may be confidential.  Unless you
> are the addressee (or authorized to receive for the addressee), you may
> not
> use, copy, or disclose to anyone the message or any information
> contained
> in the message.  If you have received the message in error, please
> advise
> the sender by reply e-mail or contact the sender at Price Paper & Twine
> Company by phone at (516) 378-7842 and delete the message.  Thank you
> very
> much.
>
> __
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> --
> __
>
> This message contains information which may be confidential.  Unless you
> are the addressee (or authorized to receive for the addressee), you may not
> use, copy, or disclose to anyone the message or any information contained
> in the message.  If you have received the message in error, please advise
> the sender by reply e-mail or contact the sender at Price Paper & Twine
> Company by phone at (516) 378-7842 and delete the message.  Thank you very
> much.
>
> __
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Copy RBD image from replicated to erasure pool possible?

[Caspar Smit]

Hi all,

Allthough undocumented, i just tried:

"rbd -p rbd copy disk1 disk1ec --data-pool ecpool"

And it worked! :)

The copy is now on the erasure coded pool.

Kind regards,
Caspar

2017-12-18 22:32 GMT+01:00 Caspar Smit :

> Hi all,
>
> http://ceph.com/community/new-luminous-erasure-coding-rbd-cephfs/
>
> Since it is possible in Luminous to use RBD directly on erasure coded
> pools the question arises how i can migrate an RBD image from a replicated
> pool to an erasure coded pool.
>
> I've got two pools configured, one replicated pool named rbd and an
> erasure coded pool named ecpool
>
> The rbd pool stores a 100GB rbd image called 'disk1'
>
> i've created a 'destination' rbd image in the erasure coded pool called
> 'disk1ec' using:
>
> rbd -p rbd create --size 100G --data-pool ecpool disk1ec
>
> I've tried: rbd -p rbd copy disk1 disk1ec
>
> but that fails with:
>
> librbd: rbd image disk1ec already exists
>
> Which is obvious because the docs says the copy command will create the
> destination rbd image.
>
> When i don't create the disk1ec image first and do a rbd copy the result
> is just another rbd image in the replicated pool.
>
> So how would i copy (or move) the rbd image from replicated to erasure? Is
> there maybe a flag to the copy command to specify the data_pool for the
> destination?
>
> Kind regards and thanks in advance for any advice,
> Caspar
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Luminous on armhf

[Ean Price]

The problem with the native build on armhf is the compilation exceeds the 2 GB 
of memory that ARMv7 (armhf) supports. Scaleway is pretty awesome but their 32 
bit ARM systems have the same 2 GB limit. I haven’t tried the cross-compile on 
the 64 bit ARMv8 they offer and that might be easier than trying to do it on 
x86_64.

> On Dec 18, 2017, at 4:41 PM, Peter Woodman  wrote:
> 
> https://www.scaleway.com/
> 
> they rent access to arm servers with gobs of ram.
> 
> i've been building my own, but with some patches (removal of some
> asserts that were unnecessarily causing crashes while i try and track
> down the bug) that make it unsuitable for public consumption
> 
> On Mon, Dec 18, 2017 at 4:38 PM, Andrew Knapp  wrote:
>> I have no idea what this response means.
>> 
>> I have tried building the armhf and arm64 package on my raspberry pi 3 to
>> no avail.  Would love to see someone post Debian packages for stretch on
>> arm64 or armhf.
>> 
>> On Dec 18, 2017 4:12 PM, "Peter Woodman"  wrote:
>>> 
>>> YMMV, but I've been using Scaleway instances to build packages for
>>> arm64- AFAIK you should be able to run any armhf distro on those
>>> machines as well.
>>> 
>>> On Mon, Dec 18, 2017 at 4:02 PM, Andrew Knapp  wrote:
 I would also love to see these packages!!!
 
 On Dec 18, 2017 3:46 PM, "Ean Price"  wrote:
 
 Hi everyone,
 
 I have a test cluster of armhf arch SoC systems running Xenial and Jewel
 (10.2). I’m looking to do a clean rebuild with Luminous (12.2) but there
 are
 no 32 bit armhf binaries available. This is just a toy cluster and not
 in
 production.
 
 I have tried, unsuccessfully, to compile from source but they only have
 2GB
 of memory and the system runs out of memory even with tuning and dialing
 back compile options. I have tinkered around with cross compiling but
 that
 seems to land me in dependency hell on Xenial and I am a cross compile
 newbie at any rate.
 
 Does anyone know of a source for a packaged version of Luminous for the
 armhf architecture? Like I said, it’s just a test cluster so I’m not
 overly
 concerned about stability.
 
 Thanks in advance,
 Ean
 --
 __
 
 This message contains information which may be confidential.  Unless you
 are the addressee (or authorized to receive for the addressee), you may
 not
 use, copy, or disclose to anyone the message or any information
 contained
 in the message.  If you have received the message in error, please
 advise
 the sender by reply e-mail or contact the sender at Price Paper & Twine
 Company by phone at (516) 378-7842 and delete the message.  Thank you
 very
 much.
 
 __
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 
 
 
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 


-- 
__

This message contains information which may be confidential.  Unless you 
are the addressee (or authorized to receive for the addressee), you may not 
use, copy, or disclose to anyone the message or any information contained 
in the message.  If you have received the message in error, please advise 
the sender by reply e-mail or contact the sender at Price Paper & Twine 
Company by phone at (516) 378-7842 and delete the message.  Thank you very 
much.

__
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Luminous on armhf

[Peter Woodman]

https://www.scaleway.com/

they rent access to arm servers with gobs of ram.

i've been building my own, but with some patches (removal of some
asserts that were unnecessarily causing crashes while i try and track
down the bug) that make it unsuitable for public consumption

On Mon, Dec 18, 2017 at 4:38 PM, Andrew Knapp  wrote:
> I have no idea what this response means.
>
>  I have tried building the armhf and arm64 package on my raspberry pi 3 to
> no avail.  Would love to see someone post Debian packages for stretch on
> arm64 or armhf.
>
> On Dec 18, 2017 4:12 PM, "Peter Woodman"  wrote:
>>
>> YMMV, but I've been using Scaleway instances to build packages for
>> arm64- AFAIK you should be able to run any armhf distro on those
>> machines as well.
>>
>> On Mon, Dec 18, 2017 at 4:02 PM, Andrew Knapp  wrote:
>> > I would also love to see these packages!!!
>> >
>> > On Dec 18, 2017 3:46 PM, "Ean Price"  wrote:
>> >
>> > Hi everyone,
>> >
>> > I have a test cluster of armhf arch SoC systems running Xenial and Jewel
>> > (10.2). I’m looking to do a clean rebuild with Luminous (12.2) but there
>> > are
>> > no 32 bit armhf binaries available. This is just a toy cluster and not
>> > in
>> > production.
>> >
>> > I have tried, unsuccessfully, to compile from source but they only have
>> > 2GB
>> > of memory and the system runs out of memory even with tuning and dialing
>> > back compile options. I have tinkered around with cross compiling but
>> > that
>> > seems to land me in dependency hell on Xenial and I am a cross compile
>> > newbie at any rate.
>> >
>> > Does anyone know of a source for a packaged version of Luminous for the
>> > armhf architecture? Like I said, it’s just a test cluster so I’m not
>> > overly
>> > concerned about stability.
>> >
>> > Thanks in advance,
>> > Ean
>> > --
>> > __
>> >
>> > This message contains information which may be confidential.  Unless you
>> > are the addressee (or authorized to receive for the addressee), you may
>> > not
>> > use, copy, or disclose to anyone the message or any information
>> > contained
>> > in the message.  If you have received the message in error, please
>> > advise
>> > the sender by reply e-mail or contact the sender at Price Paper & Twine
>> > Company by phone at (516) 378-7842 and delete the message.  Thank you
>> > very
>> > much.
>> >
>> > __
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>> >
>> >
>> > ___
>> > ceph-users mailing list
>> > ceph-users@lists.ceph.com
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Luminous on armhf

[Andrew Knapp]

I have no idea what this response means.

 I have tried building the armhf and arm64 package on my raspberry pi 3 to
no avail.  Would love to see someone post Debian packages for stretch on
arm64 or armhf.

On Dec 18, 2017 4:12 PM, "Peter Woodman"  wrote:

> YMMV, but I've been using Scaleway instances to build packages for
> arm64- AFAIK you should be able to run any armhf distro on those
> machines as well.
>
> On Mon, Dec 18, 2017 at 4:02 PM, Andrew Knapp  wrote:
> > I would also love to see these packages!!!
> >
> > On Dec 18, 2017 3:46 PM, "Ean Price"  wrote:
> >
> > Hi everyone,
> >
> > I have a test cluster of armhf arch SoC systems running Xenial and Jewel
> > (10.2). I’m looking to do a clean rebuild with Luminous (12.2) but there
> are
> > no 32 bit armhf binaries available. This is just a toy cluster and not in
> > production.
> >
> > I have tried, unsuccessfully, to compile from source but they only have
> 2GB
> > of memory and the system runs out of memory even with tuning and dialing
> > back compile options. I have tinkered around with cross compiling but
> that
> > seems to land me in dependency hell on Xenial and I am a cross compile
> > newbie at any rate.
> >
> > Does anyone know of a source for a packaged version of Luminous for the
> > armhf architecture? Like I said, it’s just a test cluster so I’m not
> overly
> > concerned about stability.
> >
> > Thanks in advance,
> > Ean
> > --
> > __
> >
> > This message contains information which may be confidential.  Unless you
> > are the addressee (or authorized to receive for the addressee), you may
> not
> > use, copy, or disclose to anyone the message or any information contained
> > in the message.  If you have received the message in error, please advise
> > the sender by reply e-mail or contact the sender at Price Paper & Twine
> > Company by phone at (516) 378-7842 and delete the message.  Thank you
> very
> > much.
> >
> > __
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] radosgw: Couldn't init storage provider (RADOS)

[Youzhong Yang]

Hello,

I tried to install Ceph 12.2.2 (Luminous) on Ubuntu 16.04.3 LTS (kernel
4.4.0-104-generic), but I am having trouble starting radosgw service:

# systemctl status ceph-rado...@rgw.ceph-rgw1
â ceph-rado...@rgw.ceph-rgw1.service - Ceph rados gateway
   Loaded: loaded (/lib/systemd/system/ceph-radosgw@.service; enabled;
vendor preset: enabled)
   Active: inactive (dead) (Result: exit-code) since Mon 2017-12-18
16:10:18 EST; 15min ago
  Process: 4571 ExecStart=/usr/bin/radosgw -f --cluster ${CLUSTER} --name
client.%i --setuser ceph --setgroup ceph (code=exited, status=5)
 Main PID: 4571 (code=exited, status=5)

Dec 18 16:10:17 ceph-rgw1 systemd[1]: ceph-rado...@rgw.ceph-rgw1.service:
Unit entered failed state.
Dec 18 16:10:17 ceph-rgw1 systemd[1]: ceph-rado...@rgw.ceph-rgw1.service:
Failed with result 'exit-code'.
Dec 18 16:10:18 ceph-rgw1 systemd[1]: ceph-rado...@rgw.ceph-rgw1.service:
Service hold-off time over, scheduling restart.
Dec 18 16:10:18 ceph-rgw1 systemd[1]: Stopped Ceph rados gateway.
Dec 18 16:10:18 ceph-rgw1 systemd[1]: ceph-rado...@rgw.ceph-rgw1.service:
Start request repeated too quickly.
Dec 18 16:10:18 ceph-rgw1 systemd[1]: Failed to start Ceph rados gateway.

If I ran the following command directly, it failed immediately:

# /usr/bin/radosgw -f --cluster ceph --name client.rgw.ceph-rgw1 --setuser
ceph --setgroup ceph
2017-12-18 16:26:56.413135 7ff11b00fe80 -1 Couldn't init storage provider
(RADOS)

There's no issue when I installed Kraken (version 11.2.1). Did I miss
anything?

Your help would be very much appreciated.

Thanks,

--Youzhong
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Copy RBD image from replicated to erasure pool possible?

[Caspar Smit]

Hi all,

http://ceph.com/community/new-luminous-erasure-coding-rbd-cephfs/

Since it is possible in Luminous to use RBD directly on erasure coded pools
the question arises how i can migrate an RBD image from a replicated pool
to an erasure coded pool.

I've got two pools configured, one replicated pool named rbd and an erasure
coded pool named ecpool

The rbd pool stores a 100GB rbd image called 'disk1'

i've created a 'destination' rbd image in the erasure coded pool called
'disk1ec' using:

rbd -p rbd create --size 100G --data-pool ecpool disk1ec

I've tried: rbd -p rbd copy disk1 disk1ec

but that fails with:

librbd: rbd image disk1ec already exists

Which is obvious because the docs says the copy command will create the
destination rbd image.

When i don't create the disk1ec image first and do a rbd copy the result is
just another rbd image in the replicated pool.

So how would i copy (or move) the rbd image from replicated to erasure? Is
there maybe a flag to the copy command to specify the data_pool for the
destination?

Kind regards and thanks in advance for any advice,
Caspar
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Luminous on armhf

[Peter Woodman]

YMMV, but I've been using Scaleway instances to build packages for
arm64- AFAIK you should be able to run any armhf distro on those
machines as well.

On Mon, Dec 18, 2017 at 4:02 PM, Andrew Knapp  wrote:
> I would also love to see these packages!!!
>
> On Dec 18, 2017 3:46 PM, "Ean Price"  wrote:
>
> Hi everyone,
>
> I have a test cluster of armhf arch SoC systems running Xenial and Jewel
> (10.2). I’m looking to do a clean rebuild with Luminous (12.2) but there are
> no 32 bit armhf binaries available. This is just a toy cluster and not in
> production.
>
> I have tried, unsuccessfully, to compile from source but they only have 2GB
> of memory and the system runs out of memory even with tuning and dialing
> back compile options. I have tinkered around with cross compiling but that
> seems to land me in dependency hell on Xenial and I am a cross compile
> newbie at any rate.
>
> Does anyone know of a source for a packaged version of Luminous for the
> armhf architecture? Like I said, it’s just a test cluster so I’m not overly
> concerned about stability.
>
> Thanks in advance,
> Ean
> --
> __
>
> This message contains information which may be confidential.  Unless you
> are the addressee (or authorized to receive for the addressee), you may not
> use, copy, or disclose to anyone the message or any information contained
> in the message.  If you have received the message in error, please advise
> the sender by reply e-mail or contact the sender at Price Paper & Twine
> Company by phone at (516) 378-7842 and delete the message.  Thank you very
> much.
>
> __
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Luminous on armhf

[Andrew Knapp]

I would also love to see these packages!!!

On Dec 18, 2017 3:46 PM, "Ean Price"  wrote:

Hi everyone,

I have a test cluster of armhf arch SoC systems running Xenial and Jewel
(10.2). I’m looking to do a clean rebuild with Luminous (12.2) but there
are no 32 bit armhf binaries available. This is just a toy cluster and not
in production.

I have tried, unsuccessfully, to compile from source but they only have 2GB
of memory and the system runs out of memory even with tuning and dialing
back compile options. I have tinkered around with cross compiling but that
seems to land me in dependency hell on Xenial and I am a cross compile
newbie at any rate.

Does anyone know of a source for a packaged version of Luminous for the
armhf architecture? Like I said, it’s just a test cluster so I’m not overly
concerned about stability.

Thanks in advance,
Ean
--
__

This message contains information which may be confidential.  Unless you
are the addressee (or authorized to receive for the addressee), you may not
use, copy, or disclose to anyone the message or any information contained
in the message.  If you have received the message in error, please advise
the sender by reply e-mail or contact the sender at Price Paper & Twine
Company by phone at (516) 378-7842 and delete the message.  Thank you very
much.

__
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Luminous on armhf

[Ean Price]

Hi everyone,

I have a test cluster of armhf arch SoC systems running Xenial and Jewel 
(10.2). I’m looking to do a clean rebuild with Luminous (12.2) but there are no 
32 bit armhf binaries available. This is just a toy cluster and not in 
production.

I have tried, unsuccessfully, to compile from source but they only have 2GB of 
memory and the system runs out of memory even with tuning and dialing back 
compile options. I have tinkered around with cross compiling but that seems to 
land me in dependency hell on Xenial and I am a cross compile newbie at any 
rate.

Does anyone know of a source for a packaged version of Luminous for the armhf 
architecture? Like I said, it’s just a test cluster so I’m not overly concerned 
about stability. 

Thanks in advance,
Ean
-- 
__

This message contains information which may be confidential.  Unless you 
are the addressee (or authorized to receive for the addressee), you may not 
use, copy, or disclose to anyone the message or any information contained 
in the message.  If you have received the message in error, please advise 
the sender by reply e-mail or contact the sender at Price Paper & Twine 
Company by phone at (516) 378-7842 and delete the message.  Thank you very 
much.

__
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] add hard drives to 3 CEPH servers (3 server cluster)

[Cary]

James,

If your replication factor is 3, for every 1GB added, your GB avail
with decrease by 3GB.


Cary
-Dynamic

On Mon, Dec 18, 2017 at 6:18 PM, James Okken  wrote:
> Thanks David.
> Thanks again Cary.
>
> If I have
> 682 GB used, 12998 GB / 13680 GB avail,
> then I still need to divide 13680/3 (my replication setting) to get what my 
> total storage really is, right?
>
> Thanks!
>
>
> James Okken
> Lab Manager
> Dialogic Research Inc.
> 4 Gatehall Drive
> Parsippany
> NJ 07054
> USA
>
> Tel:   973 967 5179
> Email:   james.ok...@dialogic.com
> Web:www.dialogic.com – The Network Fuel Company
>
> This e-mail is intended only for the named recipient(s) and may contain 
> information that is privileged, confidential and/or exempt from disclosure 
> under applicable law. No waiver of privilege, confidence or otherwise is 
> intended by virtue of communication via the internet. Any unauthorized use, 
> dissemination or copying is strictly prohibited. If you have received this 
> e-mail in error, or are not named as a recipient, please immediately notify 
> the sender and destroy all copies of this e-mail.
>
>
> -Original Message-
> From: Cary [mailto:dynamic.c...@gmail.com]
> Sent: Friday, December 15, 2017 5:56 PM
> To: David Turner
> Cc: James Okken; ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] add hard drives to 3 CEPH servers (3 server cluster)
>
> James,
>
> You can set these values in ceph.conf.
>
> [global]
> ...
> osd pool default size = 3
> osd pool default min size  = 2
> ...
>
> New pools that are created will use those values.
>
> If you run a "ceph -s"  and look at the "usage" line, it shows how much space 
> is: 1 used, 2 available, 3 total. ie.
>
> usage:   19465 GB used, 60113 GB / 79578 GB avail
>
> We choose to use Openstack with Ceph in this decade and do the other things, 
> not because they are easy, but because they are hard...;-p
>
>
> Cary
> -Dynamic
>
> On Fri, Dec 15, 2017 at 10:12 PM, David Turner  wrote:
>> In conjunction with increasing the pool size to 3, also increase the
>> pool min_size to 2.  `ceph df` and `ceph osd df` will eventually show
>> the full size in use in your cluster.  In particular the output of
>> `ceph df` with available size in a pool takes into account the pools 
>> replication size.
>> Continue watching ceph -s or ceph -w to see when the backfilling for
>> your change to replication size finishes.
>>
>> On Fri, Dec 15, 2017 at 5:06 PM James Okken 
>> wrote:
>>>
>>> This whole effort went extremely well, thanks to Cary, and Im not
>>> used to that with CEPH so far. (And openstack ever) Thank you
>>> Cary.
>>>
>>> Ive upped the replication factor and now I see "replicated size 3" in
>>> each of my pools. Is this the only place to check replication level?
>>> Is there a Global setting or only a setting per Pool?
>>>
>>> ceph osd pool ls detail
>>> pool 0 'rbd' replicated size 3..
>>> pool 1 'images' replicated size 3...
>>> ...
>>>
>>> One last question!
>>> At this replication level how can I tell how much total space I
>>> actually have now?
>>> Do I just 1/3 the Global size?
>>>
>>> ceph df
>>> GLOBAL:
>>> SIZE   AVAIL  RAW USED %RAW USED
>>> 13680G 12998G 682G  4.99
>>> POOLS:
>>> NAMEID USED %USED MAX AVAIL OBJECTS
>>> rbd 0 0 0 6448G   0
>>> images  1  216G  3.24 6448G   27745
>>> backups 2 0 0 6448G   0
>>> volumes 3  117G  1.79 6448G   30441
>>> compute 4 0 0 6448G   0
>>>
>>> ceph osd df
>>> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE VAR  PGS
>>>  0 0.81689  1.0   836G 36549M   800G 4.27 0.86  67
>>>  4 3.7  1.0  3723G   170G  3553G 4.58 0.92 270
>>>  1 0.81689  1.0   836G 49612M   788G 5.79 1.16  56
>>>  5 3.7  1.0  3723G   192G  3531G 5.17 1.04 282
>>>  2 0.81689  1.0   836G 33639M   803G 3.93 0.79  58
>>>  3 3.7  1.0  3723G   202G  3521G 5.43 1.09 291
>>>   TOTAL 13680G   682G 12998G 4.99
>>> MIN/MAX VAR: 0.79/1.16  STDDEV: 0.67
>>>
>>> Thanks!
>>>
>>> -Original Message-
>>> From: Cary [mailto:dynamic.c...@gmail.com]
>>> Sent: Friday, December 15, 2017 4:05 PM
>>> To: James Okken
>>> Cc: ceph-users@lists.ceph.com
>>> Subject: Re: [ceph-users] add hard drives to 3 CEPH servers (3 server
>>> cluster)
>>>
>>> James,
>>>
>>>  Those errors are normal. Ceph creates the missing files. You can
>>> check "/var/lib/ceph/osd/ceph-6", before and after you run those
>>> commands to see what files are added there.
>>>
>>>  Make sure you get the replication factor set.
>>>
>>>
>>> Cary
>>> -Dynamic
>>>
>>> On Fri, Dec 15, 2017 at 6:11 PM, James Okken
>>> 
>>> wrote:
>>> > Thanks again Cary,
>>> >
>>> > Yes, once all the 

Re: [ceph-users] PG active+clean+remapped status

[Karun Josy]

I think what happened is this :

http://docs.ceph.com/docs/master/rados/operations/add-or-rm-osds/


Note


Sometimes, typically in a “small” cluster with few hosts (for instance with
a small testing cluster), the fact to take out the OSD can spawn a CRUSH
corner case where some PGs remain stuck in the active+remapped state

Its a small cluster with unequal number of osds and one of the OSD disk
failed and I had taken it out.
I have already purged it, so I cannot use the reweight option mentioned in
that link.


So any other workarounds ?
Will adding more disks will clear it ?

Karun Josy

On Mon, Dec 18, 2017 at 9:06 AM, David Turner  wrote:

> Maybe try outing the disk that should have a copy of the PG, but doesn't.
> Then mark it back in. It might check that it has everything properly and
> pull a copy of the data it's missing. I dunno.
>
> On Sun, Dec 17, 2017, 10:00 PM Karun Josy  wrote:
>
>> Tried restarting all osds. Still no luck.
>>
>> Will adding a new disk to any of the server forces a rebalance and fix it?
>>
>> Karun Josy
>>
>> On Sun, Dec 17, 2017 at 12:22 PM, Cary  wrote:
>>
>>> Karun,
>>>
>>>  Could you paste in the output from "ceph health detail"? Which OSD
>>> was just added?
>>>
>>> Cary
>>> -Dynamic
>>>
>>> On Sun, Dec 17, 2017 at 4:59 AM, Karun Josy 
>>> wrote:
>>> > Any help would be appreciated!
>>> >
>>> > Karun Josy
>>> >
>>> > On Sat, Dec 16, 2017 at 11:04 PM, Karun Josy 
>>> wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> Repair didnt fix the issue.
>>> >>
>>> >> In the pg dump details, I notice this None. Seems pg is missing from
>>> one
>>> >> of the OSD
>>> >>
>>> >> [0,2,NONE,4,12,10,5,1]
>>> >> [0,2,1,4,12,10,5,1]
>>> >>
>>> >> There is no way Ceph corrects this automatically ? I have to edit/
>>> >> troubleshoot it manually ?
>>> >>
>>> >> Karun
>>> >>
>>> >> On Sat, Dec 16, 2017 at 10:44 PM, Cary 
>>> wrote:
>>> >>>
>>> >>> Karun,
>>> >>>
>>> >>>  Running ceph pg repair should not cause any problems. It may not fix
>>> >>> the issue though. If that does not help, there is more information at
>>> >>> the link below.
>>> >>> http://ceph.com/geen-categorie/ceph-manually-repair-object/
>>> >>>
>>> >>> I recommend not rebooting, or restarting while Ceph is repairing or
>>> >>> recovering. If possible, wait until the cluster is in a healthy state
>>> >>> first.
>>> >>>
>>> >>> Cary
>>> >>> -Dynamic
>>> >>>
>>> >>> On Sat, Dec 16, 2017 at 2:05 PM, Karun Josy 
>>> wrote:
>>> >>> > Hi Cary,
>>> >>> >
>>> >>> > No, I didnt try to repair it.
>>> >>> > I am comparatively new in ceph. Is it okay to try to repair it ?
>>> >>> > Or should I take any precautions while doing it ?
>>> >>> >
>>> >>> > Karun Josy
>>> >>> >
>>> >>> > On Sat, Dec 16, 2017 at 2:08 PM, Cary 
>>> wrote:
>>> >>> >>
>>> >>> >> Karun,
>>> >>> >>
>>> >>> >>  Did you attempt a "ceph pg repair "? Replace  with
>>> the pg
>>> >>> >> ID that needs repaired, 3.4.
>>> >>> >>
>>> >>> >> Cary
>>> >>> >> -D123
>>> >>> >>
>>> >>> >> On Sat, Dec 16, 2017 at 8:24 AM, Karun Josy >> >
>>> >>> >> wrote:
>>> >>> >> > Hello,
>>> >>> >> >
>>> >>> >> > I added 1 disk to the cluster and after rebalancing, it shows 1
>>> PG
>>> >>> >> > is in
>>> >>> >> > remapped state. How can I correct it ?
>>> >>> >> >
>>> >>> >> > (I had to restart some osds during the rebalancing as there were
>>> >>> >> > some
>>> >>> >> > slow
>>> >>> >> > requests)
>>> >>> >> >
>>> >>> >> > $ ceph pg dump | grep remapped
>>> >>> >> > dumped all
>>> >>> >> > 3.4 981  00 0   0
>>> >>> >> > 2655009792
>>> >>> >> > 1535 1535 active+clean+remapped 2017-12-15 22:07:21.663964
>>> >>> >> > 2824'785115
>>> >>> >> > 2824:2297888 [0,2,NONE,4,12,10,5,1]  0
>>>  [0,2,1,4,12,10,5,1]
>>> >>> >> > 0  2288'767367 2017-12-14 11:00:15.576741  417'518549
>>> 2017-12-08
>>> >>> >> > 03:56:14.006982
>>> >>> >> >
>>> >>> >> > That PG belongs to an erasure pool with k=5, m =3 profile,
>>> failure
>>> >>> >> > domain is
>>> >>> >> > host.
>>> >>> >> >
>>> >>> >> > ===
>>> >>> >> >
>>> >>> >> > $ ceph osd tree
>>> >>> >> > ID  CLASS WEIGHT   TYPE NAMESTATUS REWEIGHT
>>> PRI-AFF
>>> >>> >> >  -1   16.94565 root default
>>> >>> >> >  -32.73788 host ceph-a1
>>> >>> >> >   0   ssd  1.86469 osd.0up  1.0
>>> 1.0
>>> >>> >> >  14   ssd  0.87320 osd.14   up  1.0
>>> 1.0
>>> >>> >> >  -52.73788 host ceph-a2
>>> >>> >> >   1   ssd  1.86469 osd.1up  1.0
>>> 1.0
>>> >>> >> >  15   ssd  0.87320 osd.15   up  1.0
>>> 1.0
>>> >>> >> >  -71.86469 host ceph-a3
>>> >>> >> >   2   ssd  1.86469 osd.2up  1.0
>>> 1.0
>>> >>> >> >  -9

Re: [ceph-users] High Load and High Apply Latency

[John Petrini]

Another strange thing I'm seeing is that two of the nodes in the cluster
have some OSD's with almost no activity. If I watch top long enough I'll
eventually see cpu utilization on these osds but for the most part they sit
a 0% cpu utilization. I'm not sure if this is expected behavior or not
though. I have another cluster running the same version of ceph that has
the same symptom but the osds in our jewel cluster always show activity.


John Petrini
Platforms Engineer

[image: Call CoreDial] 215.297.4400 x 232 <215-297-4400>
[image: Call CoreDial] www.coredial.com 
[image: CoreDial] 751 Arbor Way, Hillcrest I, Suite 150 Blue Bell, PA 19422

The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.

On Mon, Dec 18, 2017 at 11:51 AM, John Petrini 
wrote:

> Hi David,
>
> Thanks for the info. The controller in the server (perc h730) was just
> replaced and the battery is at full health. Prior to replacing the
> controller I was seeing very high iowait when running iostat but I no
> longer see that behavior - just apply latency when running ceph osd perf.
> Since there's no iowait it makes me believe that the latency is not being
> introduced by the hardware; though I'm not ruling it out completely. I'd
> like to know what I can do to get a better understanding of what the OSD
> processes are so busy doing because they are working much harder on this
> server than the others.
>
>
>
>
>
> On Thu, Dec 14, 2017 at 11:33 AM, David Turner 
> wrote:
>
>> We show high disk latencies on a node when the controller's cache battery
>> dies.  This is assuming that you're using a controller with cache enabled
>> for your disks.  In any case, I would look at the hardware on the server.
>>
>> On Thu, Dec 14, 2017 at 10:15 AM John Petrini 
>> wrote:
>>
>>> Anyone have any ideas on this?
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Migrating to new pools (RBD, CephFS)

[Denes Dolhay]

Hi,


If the problem is not severe and you can wait, then according to this:

http://ceph.com/community/new-luminous-pg-overdose-protection/

there is a pg merge feature coming.


Regards,

Denes.


On 12/18/2017 02:18 PM, Jens-U. Mozdzen wrote:

Hi *,

facing the problem to reduce the number of PGs for a pool, I've found 
various information and suggestions, but no "definite guide" to handle 
pool migration with Ceph 12.2.x. This seems to be a fairly common 
problem when having to deal with "teen-age clusters", so consolidated 
information would be a real help. I'm willing to start writing things 
up, but don't want to duplicate information. So:


Are there any documented "operational procedures" on how to migrate

- an RBD pool (with snapshots created by Openstack)

- a CephFS data pool

- a CephFS metadata pool

to a different volume, in order to be able to utilize pool settings 
that cannot be changed on an existing pool?


---

RBD pools: From what I've read, RBD snapshots are "broken" after using 
"rados cppool" to move the content of an "RBD pool" to a new pool.


---

CephFS data pool: I know I can add additional pools to a CephFS 
instance ("ceph fs add_data_pool"), and have newly created files to be 
placed in the new pool ("file layouts"). But according to the docs, a 
small amount of metadata is kept in the primary data pool for all 
files, so I cannot remove the original pool.


I couldn't identify how CephFS (MDS) identifies it's current data pool 
(or "default data pool" in case of multiple pools - the one named in 
"ceph fs new"), so "rados cppool"-moving the data to a new pool and 
then reconfiguring CephFS to use the new pool (while MDS are stopped, 
of course) is not yet an option? And there might be references to the 
pool id hiding in CephFS metadata, too, invalidating this approach 
altogether.


Of course, dumping the current content of the CephFS to external 
storage and recreating the CephFS instance with new pools is a 
potential option, but may required a substantial amount of extra 
storage ;)


---

CephFS metadata pool: I've not seen any indication of a procedure to 
swap metadata pools.



I couldn't identify how CephFS (MDS) identifies it's current metadata 
pool, so "rados cppool"-moving the metadata to a new pool and then 
reconfiguring CephFS to use the new pool (while MDS are stopped, of 
course) is not yet an option?


Of course, dumping the current content of the CephFS to external 
storage and recreating the CephFS instance with new pools is a 
potential option, but may required a substantial amount of extra 
storage ;)


---

http://cephnotes.ksperis.com/blog/2015/04/15/ceph-pool-migration 
describes an interesting approach to migrate all pool contents by 
making the current pool a cache tier to the new pool and then migrate 
the "cache tier content" to the (new) base pool. But I'm not yet able 
to judge the approach and will have to conduct tests. Can anyone 
already make an educated guess if especially the "snapshot" problem 
for RBD pools will be circumvented this way and how CephFS will react 
to this approach? This "cache tier" approach, if feasible, would be a 
nice way to circumvent downtime and extra space requirements.


Thank you for any ideas, insight and experience you can share!

Regards,
J

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] add hard drives to 3 CEPH servers (3 server cluster)

[James Okken]

Thanks David.
Thanks again Cary.

If I have 
682 GB used, 12998 GB / 13680 GB avail,
then I still need to divide 13680/3 (my replication setting) to get what my 
total storage really is, right?

Thanks!


James Okken
Lab Manager
Dialogic Research Inc.
4 Gatehall Drive
Parsippany
NJ 07054
USA

Tel:   973 967 5179
Email:   james.ok...@dialogic.com
Web:    www.dialogic.com – The Network Fuel Company

This e-mail is intended only for the named recipient(s) and may contain 
information that is privileged, confidential and/or exempt from disclosure 
under applicable law. No waiver of privilege, confidence or otherwise is 
intended by virtue of communication via the internet. Any unauthorized use, 
dissemination or copying is strictly prohibited. If you have received this 
e-mail in error, or are not named as a recipient, please immediately notify the 
sender and destroy all copies of this e-mail.


-Original Message-
From: Cary [mailto:dynamic.c...@gmail.com] 
Sent: Friday, December 15, 2017 5:56 PM
To: David Turner
Cc: James Okken; ceph-users@lists.ceph.com
Subject: Re: [ceph-users] add hard drives to 3 CEPH servers (3 server cluster)

James,

You can set these values in ceph.conf.

[global]
...
osd pool default size = 3
osd pool default min size  = 2
...

New pools that are created will use those values.

If you run a "ceph -s"  and look at the "usage" line, it shows how much space 
is: 1 used, 2 available, 3 total. ie.

usage:   19465 GB used, 60113 GB / 79578 GB avail

We choose to use Openstack with Ceph in this decade and do the other things, 
not because they are easy, but because they are hard...;-p


Cary
-Dynamic

On Fri, Dec 15, 2017 at 10:12 PM, David Turner  wrote:
> In conjunction with increasing the pool size to 3, also increase the 
> pool min_size to 2.  `ceph df` and `ceph osd df` will eventually show 
> the full size in use in your cluster.  In particular the output of 
> `ceph df` with available size in a pool takes into account the pools 
> replication size.
> Continue watching ceph -s or ceph -w to see when the backfilling for 
> your change to replication size finishes.
>
> On Fri, Dec 15, 2017 at 5:06 PM James Okken 
> wrote:
>>
>> This whole effort went extremely well, thanks to Cary, and Im not 
>> used to that with CEPH so far. (And openstack ever) Thank you 
>> Cary.
>>
>> Ive upped the replication factor and now I see "replicated size 3" in 
>> each of my pools. Is this the only place to check replication level? 
>> Is there a Global setting or only a setting per Pool?
>>
>> ceph osd pool ls detail
>> pool 0 'rbd' replicated size 3..
>> pool 1 'images' replicated size 3...
>> ...
>>
>> One last question!
>> At this replication level how can I tell how much total space I 
>> actually have now?
>> Do I just 1/3 the Global size?
>>
>> ceph df
>> GLOBAL:
>> SIZE   AVAIL  RAW USED %RAW USED
>> 13680G 12998G 682G  4.99
>> POOLS:
>> NAMEID USED %USED MAX AVAIL OBJECTS
>> rbd 0 0 0 6448G   0
>> images  1  216G  3.24 6448G   27745
>> backups 2 0 0 6448G   0
>> volumes 3  117G  1.79 6448G   30441
>> compute 4 0 0 6448G   0
>>
>> ceph osd df
>> ID WEIGHT  REWEIGHT SIZE   USEAVAIL  %USE VAR  PGS
>>  0 0.81689  1.0   836G 36549M   800G 4.27 0.86  67
>>  4 3.7  1.0  3723G   170G  3553G 4.58 0.92 270
>>  1 0.81689  1.0   836G 49612M   788G 5.79 1.16  56
>>  5 3.7  1.0  3723G   192G  3531G 5.17 1.04 282
>>  2 0.81689  1.0   836G 33639M   803G 3.93 0.79  58
>>  3 3.7  1.0  3723G   202G  3521G 5.43 1.09 291
>>   TOTAL 13680G   682G 12998G 4.99
>> MIN/MAX VAR: 0.79/1.16  STDDEV: 0.67
>>
>> Thanks!
>>
>> -Original Message-
>> From: Cary [mailto:dynamic.c...@gmail.com]
>> Sent: Friday, December 15, 2017 4:05 PM
>> To: James Okken
>> Cc: ceph-users@lists.ceph.com
>> Subject: Re: [ceph-users] add hard drives to 3 CEPH servers (3 server
>> cluster)
>>
>> James,
>>
>>  Those errors are normal. Ceph creates the missing files. You can 
>> check "/var/lib/ceph/osd/ceph-6", before and after you run those 
>> commands to see what files are added there.
>>
>>  Make sure you get the replication factor set.
>>
>>
>> Cary
>> -Dynamic
>>
>> On Fri, Dec 15, 2017 at 6:11 PM, James Okken 
>> 
>> wrote:
>> > Thanks again Cary,
>> >
>> > Yes, once all the backfilling was done I was back to a Healthy cluster.
>> > I moved on to the same steps for the next server in the cluster, it 
>> > is backfilling now.
>> > Once that is done I will do the last server in the cluster, and 
>> > then I think I am done!
>> >
>> > Just checking on one thing. I get these messages when running this 
>> > command. I assume this is OK, 

Re: [ceph-users] Migrating to new pools (RBD, CephFS)

[Cary]

A possible option. They do not recommend using cppool.

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-July/011460.html

**COMPLETELY UNTESTED AND DANGEROUS**

stop all MDS daemons
delete your filesystem (but leave the pools)
use "rados export" and "rados import" to do a full copy of the
metadata to a new pool (*not* cppool, it doesn't copy OMAP data)
use "ceph fs new" to create a new filesystem that uses your new metadata pool
use "ceph fs reset" to skip the creating phase of the new filesystem
start MDS daemons

**COMPLETELY UNTESTED AND DANGEROUS**


On Mon, Dec 18, 2017 at 1:18 PM, Jens-U. Mozdzen  wrote:
> Hi *,
>
> facing the problem to reduce the number of PGs for a pool, I've found
> various information and suggestions, but no "definite guide" to handle pool
> migration with Ceph 12.2.x. This seems to be a fairly common problem when
> having to deal with "teen-age clusters", so consolidated information would
> be a real help. I'm willing to start writing things up, but don't want to
> duplicate information. So:
>
> Are there any documented "operational procedures" on how to migrate
>
> - an RBD pool (with snapshots created by Openstack)
>
> - a CephFS data pool
>
> - a CephFS metadata pool
>
> to a different volume, in order to be able to utilize pool settings that
> cannot be changed on an existing pool?
>
> ---
>
> RBD pools: From what I've read, RBD snapshots are "broken" after using
> "rados cppool" to move the content of an "RBD pool" to a new pool.
>
> ---
>
> CephFS data pool: I know I can add additional pools to a CephFS instance
> ("ceph fs add_data_pool"), and have newly created files to be placed in the
> new pool ("file layouts"). But according to the docs, a small amount of
> metadata is kept in the primary data pool for all files, so I cannot remove
> the original pool.
>
> I couldn't identify how CephFS (MDS) identifies it's current data pool (or
> "default data pool" in case of multiple pools - the one named in "ceph fs
> new"), so "rados cppool"-moving the data to a new pool and then
> reconfiguring CephFS to use the new pool (while MDS are stopped, of course)
> is not yet an option? And there might be references to the pool id hiding in
> CephFS metadata, too, invalidating this approach altogether.
>
> Of course, dumping the current content of the CephFS to external storage and
> recreating the CephFS instance with new pools is a potential option, but may
> required a substantial amount of extra storage ;)
>
> ---
>
> CephFS metadata pool: I've not seen any indication of a procedure to swap
> metadata pools.
>
>
> I couldn't identify how CephFS (MDS) identifies it's current metadata pool,
> so "rados cppool"-moving the metadata to a new pool and then reconfiguring
> CephFS to use the new pool (while MDS are stopped, of course) is not yet an
> option?
>
> Of course, dumping the current content of the CephFS to external storage and
> recreating the CephFS instance with new pools is a potential option, but may
> required a substantial amount of extra storage ;)
>
> ---
>
> http://cephnotes.ksperis.com/blog/2015/04/15/ceph-pool-migration describes
> an interesting approach to migrate all pool contents by making the current
> pool a cache tier to the new pool and then migrate the "cache tier content"
> to the (new) base pool. But I'm not yet able to judge the approach and will
> have to conduct tests. Can anyone already make an educated guess if
> especially the "snapshot" problem for RBD pools will be circumvented this
> way and how CephFS will react to this approach? This "cache tier" approach,
> if feasible, would be a nice way to circumvent downtime and extra space
> requirements.
>
> Thank you for any ideas, insight and experience you can share!
>
> Regards,
> J
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] determining the source of io in the cluster

[Luis Periquito]

As that is a small cluster I hope you still don't have a lot of
instances running...

You can add "admin socket" to the client configuration part and then
read performance information via that. IIRC that prints total bytes
and IOPS, but it should be simple to read/calculate difference. This
will generate one socket per volume mounted (thus the I hope you don't
have many).

On Mon, Dec 18, 2017 at 4:36 PM, Josef Zelenka
 wrote:
> Hi everyone,
>
> we have recently deployed a Luminous(12.2.1) cluster on Ubuntu - three osd
> nodes and three monitors, every osd has 3x 2TB SSD + an NVMe drive for a
> blockdb. We use it as a backend for our Openstack cluster, so we store
> volumes there. IN the last few days, the read op/s rose to around 10k-25k
> constantly(it fluctuates between those two) and it doesn't seem to go down.
> I can see, that the io/read ops come from the pool where we store VM
> volumes, but i can't source this issue to a particular volume. Is that even
> possible? Any experiences with debugging this? Any info or advice is greatly
> appreciated.
>
> Thanks
>
> Josef Zelenka
>
> Cloudevelops
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] determining the source of io in the cluster

[Stefan Kooman]

Quoting Josef Zelenka (josef.zele...@cloudevelops.com):
> Hi everyone,
> 
> we have recently deployed a Luminous(12.2.1) cluster on Ubuntu - three osd
> nodes and three monitors, every osd has 3x 2TB SSD + an NVMe drive for a
> blockdb. We use it as a backend for our Openstack cluster, so we store
> volumes there. IN the last few days, the read op/s rose to around 10k-25k
> constantly(it fluctuates between those two) and it doesn't seem to go down.
> I can see, that the io/read ops come from the pool where we store VM
> volumes, but i can't source this issue to a particular volume. Is that even
> possible? Any experiences with debugging this? Any info or advice is greatly
> appreciated.

Ceph has no "QoS" as of yet. You might want to collect the libvirt data
from your domains (assuming you are using libvirt / kvm) with:

virsh domblkstat domain-id device

And see how it changes over time. You might then get an idea what VM
uses the most IO. Maybe OpenStack has metrics about the amount of IOPS VM's
are doing

Gr. Stefan

-- 
| BIT BV  http://www.bit.nl/Kamer van Koophandel 09090351
| GPG: 0xD14839C6   +31 318 648 688 / i...@bit.nl
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] High Load and High Apply Latency

[John Petrini]

Hi David,

Thanks for the info. The controller in the server (perc h730) was just
replaced and the battery is at full health. Prior to replacing the
controller I was seeing very high iowait when running iostat but I no
longer see that behavior - just apply latency when running ceph osd perf.
Since there's no iowait it makes me believe that the latency is not being
introduced by the hardware; though I'm not ruling it out completely. I'd
like to know what I can do to get a better understanding of what the OSD
processes are so busy doing because they are working much harder on this
server than the others.




On Thu, Dec 14, 2017 at 11:33 AM, David Turner 
wrote:

> We show high disk latencies on a node when the controller's cache battery
> dies.  This is assuming that you're using a controller with cache enabled
> for your disks.  In any case, I would look at the hardware on the server.
>
> On Thu, Dec 14, 2017 at 10:15 AM John Petrini 
> wrote:
>
>> Anyone have any ideas on this?
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] determining the source of io in the cluster

[Josef Zelenka]

Hi everyone,

we have recently deployed a Luminous(12.2.1) cluster on Ubuntu - three 
osd nodes and three monitors, every osd has 3x 2TB SSD + an NVMe drive 
for a blockdb. We use it as a backend for our Openstack cluster, so we 
store volumes there. IN the last few days, the read op/s rose to around 
10k-25k constantly(it fluctuates between those two) and it doesn't seem 
to go down. I can see, that the io/read ops come from the pool where we 
store VM volumes, but i can't source this issue to a particular volume. 
Is that even possible? Any experiences with debugging this? Any info or 
advice is greatly appreciated.


Thanks

Josef Zelenka

Cloudevelops

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Unable to ceph-deploy luminos

[Burkhard Linke]

Hi,


On 12/18/2017 05:28 PM, Andre Goree wrote:
I'm working on setting up a cluster for testing purposes and I can't 
see to install luminos.  All nodes are runnind Ubuntu 16.04.


[cephadmin][DEBUG ] Err:7 https://download.ceph.com/debian-luminos 
xenial/main amd64 Packages

[cephadmin][DEBUG ]   404  Not Found
[cephadmin][DEBUG ] Ign:8 https://download.ceph.com/debian-luminos 
xenial/main i386 Packages
[cephadmin][DEBUG ] Ign:9 https://download.ceph.com/debian-luminos 
xenial/main all Packages
[cephadmin][DEBUG ] Ign:10 https://download.ceph.com/debian-luminos 
xenial/main Translation-en_US
[cephadmin][DEBUG ] Ign:11 https://download.ceph.com/debian-luminos 
xenial/main Translation-en

[cephadmin][DEBUG ] Fetched 306 kB in 1s (178 kB/s)
[cephadmin][DEBUG ] Reading package lists...
[cephadmin][WARNIN] W: The repository 
'https://download.ceph.com/debian-luminos xenial Release' does not 
have a Release file.
[cephadmin][WARNIN] E: Failed to fetch 
https://download.ceph.com/debian-luminos/dists/xenial/main/binary-amd64/Packages 
 404  Not Found
[cephadmin][WARNIN] E: Some index files failed to download. They have 
been ignored, or old ones used instead.
[cephadmin][ERROR ] RuntimeError: command returned non-zero exit 
status: 100
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env 
DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get 
--assume-yes -q update



What's weird is that the Release file and 'Packages" does appear to be 
available when I visit download.ceph.com in my web browser. Any ideas?
Did you specify the release on the command line and made a typo? luminos 
vs. luminous


Regards,
Burkhard
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Unable to ceph-deploy luminos

[Behnam Loghmani]

you have typo in apt source
it must be

https://download.ceph.com/debian-luminous/

not

https://download.ceph.com/debian-luminos/


On Mon, Dec 18, 2017 at 7:58 PM, Andre Goree  wrote:

> I'm working on setting up a cluster for testing purposes and I can't see
> to install luminos.  All nodes are runnind Ubuntu 16.04.
>
> [cephadmin][DEBUG ] Err:7 https://download.ceph.com/debian-luminos
> xenial/main amd64 Packages
> [cephadmin][DEBUG ]   404  Not Found
> [cephadmin][DEBUG ] Ign:8 https://download.ceph.com/debian-luminos
> xenial/main i386 Packages
> [cephadmin][DEBUG ] Ign:9 https://download.ceph.com/debian-luminos
> xenial/main all Packages
> [cephadmin][DEBUG ] Ign:10 https://download.ceph.com/debian-luminos
> xenial/main Translation-en_US
> [cephadmin][DEBUG ] Ign:11 https://download.ceph.com/debian-luminos
> xenial/main Translation-en
> [cephadmin][DEBUG ] Fetched 306 kB in 1s (178 kB/s)
> [cephadmin][DEBUG ] Reading package lists...
> [cephadmin][WARNIN] W: The repository 'https://download.ceph.com/deb
> ian-luminos xenial Release' does not have a Release file.
> [cephadmin][WARNIN] E: Failed to fetch https://download.ceph.com/debi
> an-luminos/dists/xenial/main/binary-amd64/Packages  404  Not Found
> [cephadmin][WARNIN] E: Some index files failed to download. They have been
> ignored, or old ones used instead.
> [cephadmin][ERROR ] RuntimeError: command returned non-zero exit status:
> 100
> [ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env
> DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get
> --assume-yes -q update
>
>
> What's weird is that the Release file and 'Packages" does appear to be
> available when I visit download.ceph.com in my web browser.  Any ideas?
>
>
> --
> Andre Goree
> -=-=-=-=-=-
> Email - andre at drenet.net
> Website   - http://blog.drenet.net
> PGP key   - http://www.drenet.net/pubkey.html
> -=-=-=-=-=-
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Unable to ceph-deploy luminos

[Andre Goree]

On 2017/12/18 11:28 am, Andre Goree wrote:

I'm working on setting up a cluster for testing purposes and I can't
see to install luminos.  All nodes are runnind Ubuntu 16.04.

[cephadmin][DEBUG ] Err:7 https://download.ceph.com/debian-luminos
xenial/main amd64 Packages
[cephadmin][DEBUG ]   404  Not Found
[cephadmin][DEBUG ] Ign:8 https://download.ceph.com/debian-luminos
xenial/main i386 Packages
[cephadmin][DEBUG ] Ign:9 https://download.ceph.com/debian-luminos
xenial/main all Packages
[cephadmin][DEBUG ] Ign:10 https://download.ceph.com/debian-luminos
xenial/main Translation-en_US
[cephadmin][DEBUG ] Ign:11 https://download.ceph.com/debian-luminos
xenial/main Translation-en
[cephadmin][DEBUG ] Fetched 306 kB in 1s (178 kB/s)
[cephadmin][DEBUG ] Reading package lists...
[cephadmin][WARNIN] W: The repository
'https://download.ceph.com/debian-luminos xenial Release' does not
have a Release file.
[cephadmin][WARNIN] E: Failed to fetch
https://download.ceph.com/debian-luminos/dists/xenial/main/binary-amd64/Packages
 404  Not Found
[cephadmin][WARNIN] E: Some index files failed to download. They have
been ignored, or old ones used instead.
[cephadmin][ERROR ] RuntimeError: command returned non-zero exit 
status: 100

[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env
DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get
--assume-yes -q update


What's weird is that the Release file and 'Packages" does appear to be
available when I visit download.ceph.com in my web browser.  Any
ideas?




I see my error, it was a typo, sorry!


--
Andre Goree
-=-=-=-=-=-
Email - andre at drenet.net
Website   - http://blog.drenet.net
PGP key   - http://www.drenet.net/pubkey.html
-=-=-=-=-=-
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Unable to ceph-deploy luminos

[Andre Goree]

I'm working on setting up a cluster for testing purposes and I can't see 
to install luminos.  All nodes are runnind Ubuntu 16.04.


[cephadmin][DEBUG ] Err:7 https://download.ceph.com/debian-luminos 
xenial/main amd64 Packages

[cephadmin][DEBUG ]   404  Not Found
[cephadmin][DEBUG ] Ign:8 https://download.ceph.com/debian-luminos 
xenial/main i386 Packages
[cephadmin][DEBUG ] Ign:9 https://download.ceph.com/debian-luminos 
xenial/main all Packages
[cephadmin][DEBUG ] Ign:10 https://download.ceph.com/debian-luminos 
xenial/main Translation-en_US
[cephadmin][DEBUG ] Ign:11 https://download.ceph.com/debian-luminos 
xenial/main Translation-en

[cephadmin][DEBUG ] Fetched 306 kB in 1s (178 kB/s)
[cephadmin][DEBUG ] Reading package lists...
[cephadmin][WARNIN] W: The repository 
'https://download.ceph.com/debian-luminos xenial Release' does not have 
a Release file.
[cephadmin][WARNIN] E: Failed to fetch 
https://download.ceph.com/debian-luminos/dists/xenial/main/binary-amd64/Packages 
 404  Not Found
[cephadmin][WARNIN] E: Some index files failed to download. They have 
been ignored, or old ones used instead.
[cephadmin][ERROR ] RuntimeError: command returned non-zero exit status: 
100
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: env 
DEBIAN_FRONTEND=noninteractive DEBIAN_PRIORITY=critical apt-get 
--assume-yes -q update



What's weird is that the Release file and 'Packages" does appear to be 
available when I visit download.ceph.com in my web browser.  Any ideas?



--
Andre Goree
-=-=-=-=-=-
Email - andre at drenet.net
Website   - http://blog.drenet.net
PGP key   - http://www.drenet.net/pubkey.html
-=-=-=-=-=-
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Integrating Ceph RGW 12.2.2 with OpenStack

[Rafał Wądołowski]

Hi ceph-users!

I'm trying to integrate Swift in OpenStack with ceph rgw(12.2.2) as a 
backend.


I'm facing problem with creating bucket. I see return code -34. Has 
anybody similiar issue? My config and log below.


ceph.conf

rgw keystone verify ssl = false
rgw keystone accepted roles = Member, _member_, admin
rgw keystone admin domain = default
rgw keystone admin project = service
rgw keystone admin user = swift
rgw keystone admin password = password
rgw keystone api version = 3
rgw keystone revocation interval = 0
rgw keystone token cache size = 1
rgw keystone url = https://ip_address:5000
rgw s3 auth use keystone = true

Log from rgw

2017-12-18 15:44:41.627043 7f065499c700  0 validated token: admin:admin 
expires: 1513615480
2017-12-18 15:44:41.627070 7f065499c700 20 
rgw::auth::keystone::TokenEngine granted access
2017-12-18 15:44:41.627087 7f065499c700 20 get_system_obj_state: 
rctx=0x7f0654994140 
obj=default.rgw.meta:users.uid:06b25e10c29c4e5aa725f545df5e2dd8$06b25e10c29c4e5aa725f545df5e2dd8 
state=0x56175a9aeba0 s->prefetch_data=0
2017-12-18 15:44:41.627104 7f065499c700 10 cache get: 
name=default.rgw.meta+users.uid+06b25e10c29c4e5aa725f545df5e2dd8$06b25e10c29c4e5aa725f545df5e2dd8 
: miss
2017-12-18 15:44:41.627144 7f065499c700 10 librados: getxattrs 
oid=06b25e10c29c4e5aa725f545df5e2dd8$06b25e10c29c4e5aa725f545df5e2dd8 
nspace=users.uid
2017-12-18 15:44:41.628089 7f065499c700 10 librados: Objecter returned 
from getxattrs r=-2
2017-12-18 15:44:41.628112 7f065499c700 10 cache put: 
name=default.rgw.meta+users.uid+06b25e10c29c4e5aa725f545df5e2dd8$06b25e10c29c4e5aa725f545df5e2dd8 
info.flags=0x0
2017-12-18 15:44:41.628119 7f065499c700 10 adding 
default.rgw.meta+users.uid+06b25e10c29c4e5aa725f545df5e2dd8$06b25e10c29c4e5aa725f545df5e2dd8 
to cache LRU end
2017-12-18 15:44:41.628130 7f065499c700 20 get_system_obj_state: 
rctx=0x7f0654994140 
obj=default.rgw.meta:users.uid:06b25e10c29c4e5aa725f545df5e2dd8 
state=0x56175a9aeba0 s->prefetch_data=0
2017-12-18 15:44:41.628135 7f065499c700 10 cache get: 
name=default.rgw.meta+users.uid+06b25e10c29c4e5aa725f545df5e2dd8 : miss
2017-12-18 15:44:41.628146 7f065499c700 10 librados: getxattrs 
oid=06b25e10c29c4e5aa725f545df5e2dd8 nspace=users.uid
2017-12-18 15:44:41.628753 7f065499c700 10 librados: Objecter returned 
from getxattrs r=0
2017-12-18 15:44:41.628764 7f065499c700 10 cache put: 
name=default.rgw.meta+users.uid+06b25e10c29c4e5aa725f545df5e2dd8 
info.flags=0x6
2017-12-18 15:44:41.628779 7f065499c700 10 adding 
default.rgw.meta+users.uid+06b25e10c29c4e5aa725f545df5e2dd8 to cache LRU end
2017-12-18 15:44:41.628784 7f065499c700 20 get_system_obj_state: 
s->obj_tag was set empty
2017-12-18 15:44:41.628787 7f065499c700 10 cache get: 
name=default.rgw.meta+users.uid+06b25e10c29c4e5aa725f545df5e2dd8 : type 
miss (requested=0x1, cached=0x6)

2017-12-18 15:44:41.628791 7f065499c700 20 rados->read ofs=0 len=524288
2017-12-18 15:44:41.628800 7f065499c700 10 librados: read 
oid=06b25e10c29c4e5aa725f545df5e2dd8 nspace=users.uid
2017-12-18 15:44:41.629289 7f065499c700 10 librados: Objecter returned 
from read r=0

2017-12-18 15:44:41.629296 7f065499c700 20 rados->read r=0 bl.length=232
2017-12-18 15:44:41.629312 7f065499c700 10 cache put: 
name=default.rgw.meta+users.uid+06b25e10c29c4e5aa725f545df5e2dd8 
info.flags=0x1
2017-12-18 15:44:41.629314 7f065499c700 10 moving 
default.rgw.meta+users.uid+06b25e10c29c4e5aa725f545df5e2dd8 to cache LRU end
2017-12-18 15:44:41.629336 7f065499c700  2 req 1:0.854982:swift:PUT 
/swift/v1/asd:create_bucket:normalizing buckets and tenants

2017-12-18 15:44:41.629342 7f065499c700 10 s->object= s->bucket=asd
2017-12-18 15:44:41.629348 7f065499c700  2 req 1:0.854994:swift:PUT 
/swift/v1/asd:create_bucket:init permissions
2017-12-18 15:44:41.629351 7f065499c700  2 req 1:0.854997:swift:PUT 
/swift/v1/asd:create_bucket:recalculating target

2017-12-18 15:44:41.629353 7f065499c700 10 Starting retarget
2017-12-18 15:44:41.629361 7f065499c700  2 req 1:0.855008:swift:PUT 
/swift/v1/asd:create_bucket:reading permissions
2017-12-18 15:44:41.629366 7f065499c700  2 req 1:0.855010:swift:PUT 
/swift/v1/asd:create_bucket:init op
2017-12-18 15:44:41.629368 7f065499c700  2 req 1:0.855014:swift:PUT 
/swift/v1/asd:create_bucket:verifying op mask

2017-12-18 15:44:41.629370 7f065499c700 20 required_mask= 2 user.op_mask=7
2017-12-18 15:44:41.629371 7f065499c700  2 req 1:0.855017:swift:PUT 
/swift/v1/asd:create_bucket:verifying op permissions
2017-12-18 15:44:41.629388 7f065499c700 10 librados: call 
oid=06b25e10c29c4e5aa725f545df5e2dd8.buckets nspace=users.uid
2017-12-18 15:44:41.629913 7f065499c700 10 librados: Objecter returned 
from call r=-2
2017-12-18 15:44:41.629924 7f065499c700  2 req 1:0.855570:swift:PUT 
/swift/v1/asd:create_bucket:verifying op params
2017-12-18 15:44:41.629927 7f065499c700  2 req 1:0.855573:swift:PUT 
/swift/v1/asd:create_bucket:pre-executing
2017-12-18 15:44:41.629929 7f065499c700  2 req 1:0.855575:swift:PUT 

[ceph-users] RGW default quotas, Luminous

[Andreas Calminder]

Hello!
According to the documentation at
http://docs.ceph.com/docs/master/radosgw/admin/#quota-management
there's a way to set the default quota for all RGW users, if I
understand it correctly it'll apply the quota for all users created
after the default quota is set. For instance, I want to all buckets to
have a limit of 70 objects to avoid slow index operations since
dynamic resharding doesn't work in multi-site yet. I would set

rgw_bucket_default_quota_max_objects = 70

under [global] in ceph.conf.

Is there any way to set this via command line instead? A couple of
headings down in the same document I see "Reading / Writing Global
Quotas" which I can set via command line, but it's not clear if these
are the same as defaults or if it's something else.

Regards,
Andreas
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] [Luminous 12.2.2] Cluster peformance drops after certain point of time

[shadow_lin]

Thanks for the information, but I think that is not my case because I am using 
only hdd in my cluster.

From the command you provide I found the db_used_bytes is quite large, but I am 
not sure how the db used bytes is related to the amount of stored data and the 
performance.

ceph daemon osd.0 perf dump | jq '.bluefs' | grep -E '(db|slow)'
  "db_total_bytes": 400029646848,
  "db_used_bytes": 9347006464,
  "slow_total_bytes": 0,
  "slow_used_bytes": 0


2017-12-18 

shadow_lin 



发件人：Konstantin Shalygin 
发送时间：2017-12-18 13:52
主题：Re: [ceph-users] [Luminous 12.2.2] Cluster peformance drops after certain 
point of time
收件人："ceph-users"
抄送："shadow_lin"

I am testing luminous 12.2.2 and find a strange behavior of my cluster.
Check your block.db usage. Luminous 12.2.2 is affected 
http://tracker.ceph.com/issues/22264


[root@ceph-osd0]# ceph daemon osd.46 perf dump | jq '.bluefs' | grep -E 
'(db|slow)'
  "db_total_bytes": 30064762880,
  "db_used_bytes": 16777216,
  "slow_total_bytes": 240043163648,
  "slow_used_bytes": 659554304,___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Migrating to new pools (RBD, CephFS)

[Jens-U. Mozdzen]

Hi *,

facing the problem to reduce the number of PGs for a pool, I've found  
various information and suggestions, but no "definite guide" to handle  
pool migration with Ceph 12.2.x. This seems to be a fairly common  
problem when having to deal with "teen-age clusters", so consolidated  
information would be a real help. I'm willing to start writing things  
up, but don't want to duplicate information. So:


Are there any documented "operational procedures" on how to migrate

- an RBD pool (with snapshots created by Openstack)

- a CephFS data pool

- a CephFS metadata pool

to a different volume, in order to be able to utilize pool settings  
that cannot be changed on an existing pool?


---

RBD pools: From what I've read, RBD snapshots are "broken" after using  
"rados cppool" to move the content of an "RBD pool" to a new pool.


---

CephFS data pool: I know I can add additional pools to a CephFS  
instance ("ceph fs add_data_pool"), and have newly created files to be  
placed in the new pool ("file layouts"). But according to the docs, a  
small amount of metadata is kept in the primary data pool for all  
files, so I cannot remove the original pool.


I couldn't identify how CephFS (MDS) identifies it's current data pool  
(or "default data pool" in case of multiple pools - the one named in  
"ceph fs new"), so "rados cppool"-moving the data to a new pool and  
then reconfiguring CephFS to use the new pool (while MDS are stopped,  
of course) is not yet an option? And there might be references to the  
pool id hiding in CephFS metadata, too, invalidating this approach  
altogether.


Of course, dumping the current content of the CephFS to external  
storage and recreating the CephFS instance with new pools is a  
potential option, but may required a substantial amount of extra  
storage ;)


---

CephFS metadata pool: I've not seen any indication of a procedure to  
swap metadata pools.



I couldn't identify how CephFS (MDS) identifies it's current metadata  
pool, so "rados cppool"-moving the metadata to a new pool and then  
reconfiguring CephFS to use the new pool (while MDS are stopped, of  
course) is not yet an option?


Of course, dumping the current content of the CephFS to external  
storage and recreating the CephFS instance with new pools is a  
potential option, but may required a substantial amount of extra  
storage ;)


---

http://cephnotes.ksperis.com/blog/2015/04/15/ceph-pool-migration  
describes an interesting approach to migrate all pool contents by  
making the current pool a cache tier to the new pool and then migrate  
the "cache tier content" to the (new) base pool. But I'm not yet able  
to judge the approach and will have to conduct tests. Can anyone  
already make an educated guess if especially the "snapshot" problem  
for RBD pools will be circumvented this way and how CephFS will react  
to this approach? This "cache tier" approach, if feasible, would be a  
nice way to circumvent downtime and extra space requirements.


Thank you for any ideas, insight and experience you can share!

Regards,
J

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph with multiple public networks

[Jakub Jaszewski]

Hi,

We have ceph cluster in version luminous 12.2.2. It has public network and
cluster network configured.

Cluster provides services for two big groups of clients and some individual
clients
One group uses RGW and another uses RBD.
Ceph's public network and two mentioned groups are located in three
different vlans. Each clients group generates traffic above limit of
routing devices.

Right now RGW and MON roles are served by the same hosts.

I'd like to add additional vlan tagged interface to all MON and OSD ceph
nodes to streamline communication with big group of clients using RBD and
keep current public network for individual requests.
>From what I can find it is supported to have more than one public network,
according to
http://docs.ceph.com/docs/master/rados/configuration/network-config-ref/#id1
Is it possible to have MON host with two pulblic addresses assigned ? or I
need to desginate another hosts to handel MON roles with different public
IP addresses?

How should I approach with RGW service? In this case I also need to provide
RGW for big group of clients in dedicated vlan and keep access for
individual requests coming to IP in currently set public network.
Is it possible to bind one civetweb instance to two ip addresses or need
separete instances per network address ?

Current ceph.conf is

[global]
  fsid = 1023c49f-3a10-42de-9f62-9b122db32f1f
  mon_initial_members = host01,host02,host03
  mon_host = 10.212.32.18,10.212.32.19,10.212.32.20
  auth_supported = cephx
  public_network = 10.212.32.0/24
  cluster_network = 10.212.14.0/24
[client.rgw.host01]
  rgw host = host01
  rgw enable usage log = true
#  debug_rgw = 20
[client.rgw.host02]
  rgw host = host02
  rgw enable usage log = true
[client.rgw.host03]
  rgw host = host03
  rgw enable usage log = true
[osd]
  filestore xattr use omap = true
  osd journal size = 10240
  osd mount options xfs = noatime,inode64,logbsize=256k,logbufs=8
  osd crush location hook = /usr/bin/opera-ceph-crush-location.sh
  osd pool default size = 3
[mon]
  mon compact on start = true
  mon compact on trim = true


Thanks
Jakub
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Random checksum errors (bluestore on Luminous)

[Martin Preuss]

Hi,
Am 17.12.2017 um 10:40 schrieb Martin Preuss:
[...]
> is there a way to find out which files on CephFS is are using a given
> pg? I'd like to check whether those files are corrupted...
[...]

Nobody? Any hint, maybe?

Failing checksums for no apparent reason seem to me like quite a serious
problem for a stable storage system...


BTW: I added a bug report ticket to the tracker
(http://tracker.ceph.com/issues/22464).


Regards
Martin



-- 
"Things are only impossible until they're not"
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Snap trim queue length issues

[Piotr Dałek]

On 17-12-15 03:58 PM, Sage Weil wrote:

On Fri, 15 Dec 2017, Piotr Dałek wrote:

On 17-12-14 05:31 PM, David Turner wrote:

I've tracked this in a much more manual way.  I would grab a random subset
[..]

This was all on a Hammer cluster.  The changes to the snap trimming queues
going into the main osd thread made it so that our use case was not viable
on Jewel until changes to Jewel that happened after I left.  It's exciting
that this will actually be a reportable value from the cluster.

Sorry that this story doesn't really answer your question, except to say
that people aware of this problem likely have a work around for it.  However
I'm certain that a lot more clusters are impacted by this than are aware of
it and being able to quickly see that would be beneficial to troubleshooting
problems.  Backporting would be nice.  I run a few Jewel clusters that have
some VM's and it would be nice to see how well the cluster handle snap
trimming.  But they are much less critical on how much snapshots they do.


Thanks for your response, it pretty much confirms what I though:
- users aware of issue have their own hacks that don't need to be efficient or
convenient.
- users unaware of issue are, well, unaware and at risk of serious service
disruption once disk space is all used up.

Hopefully it'll be convincing enough for devs. ;)


Your PR looks great!  I commented with a nit on the format of the warning
itself.


I just adressed the comments.


I expect this is trivial to backport to luminous; it will need to be
partially reimplemented for jewel (with some care around the pg_stat_t and
a different check for the jewel-style health checks).


Yeah, that's why I expected some resistance here and asked for comments. I 
really don't mind reimplementing this, it's not a big deal.


--
Piotr Dałek
piotr.da...@corp.ovh.com
https://www.ovh.com/us/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com