[ceph-users] Accessing krbd client metrics

[Mingliang LIU]

Hi all,

I have a quick question about the RBD kernel module - how to best collect
the metrics or perf numbers? The command 'ceph -w' does print some useful
event logs of cluster-wide while I'm interested in
per-client/per-image/per-volume read/write bytes and latency etc.

For the *librbd*, I found the admin socket is very handy. However, this
seems not an option for *krbd* kernel module (correct me if I'm wrong).

I found when loading one kernel module or mapping an image, there are some
related system message readable via *dmesg*. But I did not find any logs
about read/write.

Any pointers?

Thanks,
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Luminous radosgw hangs after a few hours

[Kamble, Nitin A]

I see the same issue with ceph v12.1.4 as well. We are not using openstack or 
keystone, and see these errors in the rgw log. RGW is not hanging though.

Thanks,
Nitin


From: ceph-users  on behalf of Martin Emrich 

Date: Monday, July 24, 2017 at 10:08 PM
To: Vasu Kulkarni , Vaibhav Bhembre 

Cc: "ceph-users@lists.ceph.com" 
Subject: Re: [ceph-users] Luminous radosgw hangs after a few hours

I created an issue: http://tracker.ceph.com/issues/20763
 
Regards,
 
Martin
 
Von: Vasu Kulkarni 
Datum: Montag, 24. Juli 2017 um 19:26
An: Vaibhav Bhembre 
Cc: Martin Emrich , "ceph-users@lists.ceph.com" 

Betreff: Re: [ceph-users] Luminous radosgw hangs after a few hours
 
Please raise a tracker for rgw and also provide some additional journalctl logs 
and info(ceph version, os version etc): http://tracker.ceph.com/projects/rgw
 
On Mon, Jul 24, 2017 at 9:03 AM, Vaibhav Bhembre  
wrote:
I am seeing the same issue on upgrade to Luminous v12.1.0 from Jewel.
I am not using Keystone or OpenStack either and my radosgw daemon
hangs as well. I have to restart it to resume processing.

2017-07-24 00:23:33.057401 7f196096a700  0 ERROR: keystone revocation
processing returned error r=-22
2017-07-24 00:38:33.057524 7f196096a700  0 ERROR: keystone revocation
processing returned error r=-22
2017-07-24 00:53:33.057648 7f196096a700  0 ERROR: keystone revocation
processing returned error r=-22
2017-07-24 01:08:33.057749 7f196096a700  0 ERROR: keystone revocation
processing returned error r=-22
2017-07-24 01:23:33.057878 7f196096a700  0 ERROR: keystone revocation
processing returned error r=-22
2017-07-24 01:38:33.057964 7f196096a700  0 ERROR: keystone revocation
processing returned error r=-22
2017-07-24 01:53:33.058098 7f196096a700  0 ERROR: keystone revocation
processing returned error r=-22
2017-07-24 02:08:33.058225 7f196096a700  0 ERROR: keystone revocation
processing returned error r=-22

The following are my keystone config options:

"rgw_keystone_url": ""
"rgw_keystone_admin_token": ""
"rgw_keystone_admin_user": ""
"rgw_keystone_admin_password": ""
"rgw_keystone_admin_tenant": ""
"rgw_keystone_admin_project": ""
"rgw_keystone_admin_domain": ""
"rgw_keystone_barbican_user": ""
"rgw_keystone_barbican_password": ""
"rgw_keystone_barbican_tenant": ""
"rgw_keystone_barbican_project": ""
"rgw_keystone_barbican_domain": ""
"rgw_keystone_api_version": "2"
"rgw_keystone_accepted_roles": "Member
"rgw_keystone_accepted_admin_roles": ""
"rgw_keystone_token_cache_size": "1"
"rgw_keystone_revocation_interval": "900"
"rgw_keystone_verify_ssl": "true"
"rgw_keystone_implicit_tenants": "false"
"rgw_s3_auth_use_keystone": "false"

Is this fixed in RC2 by any chance?

On Thu, Jun 29, 2017 at 3:11 AM, Martin Emrich
 wrote:
> Since upgrading to 12.1, our Object Gateways hang after a few hours, I only
> see these messages in the log file:
>
>
>
> 2017-06-29 07:52:20.877587 7fa8e01e5700  0 ERROR: keystone revocation
> processing returned error r=-22
>
> 2017-06-29 08:07:20.877761 7fa8e01e5700  0 ERROR: keystone revocation
> processing returned error r=-22
>
> 2017-06-29 08:07:29.994979 7fa8e11e7700  0 process_single_logshard: Error in
> get_bucket_info: (2) No such file or directory
>
> 2017-06-29 08:22:20.877911 7fa8e01e5700  0 ERROR: keystone revocation
> processing returned error r=-22
>
> 2017-06-29 08:27:30.086119 7fa8e11e7700  0 process_single_logshard: Error in
> get_bucket_info: (2) No such file or directory
>
> 2017-06-29 08:37:20.878108 7fa8e01e5700  0 ERROR: keystone revocation
> processing returned error r=-22
>
> 2017-06-29 08:37:30.187696 7fa8e11e7700  0 process_single_logshard: Error in
> get_bucket_info: (2) No such file or directory
>
> 2017-06-29 08:52:20.878283 7fa8e01e5700  0 ERROR: keystone revocation
> processing returned error r=-22
>
> 2017-06-29 08:57:30.280881 7fa8e11e7700  0 process_single_logshard: Error in
> get_bucket_info: (2) No such file or directory
>
> 2017-06-29 09:07:20.878451 7fa8e01e5700  0 ERROR: keystone revocation
> processing returned error r=-22
>
>
>
> FYI: we do not use Keystone or Openstack.
>
>
>
> This started after upgrading from jewel (via kraken) to luminous.
>
>
>
> What could I do to fix this?
>
> Is there some “fsck” like consistency check + repair for the radosgw
> buckets?
>
>
>
> Thanks,
>
>
>
> Martin
>
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 


___
ceph-users mailing list

Re: [ceph-users] Fwd: Can't get fullpartition space

[David Turner]

Have you checked that zapping your disks to remove any and all partitions
didn't work?  sgdisk -Z /dev/sda3

On Fri, Aug 18, 2017 at 12:48 PM Maiko de Andrade 
wrote:

> Hi,
>
> I try use bluestore_block_size but I recive this error (I used values in
> byte, kb, mb, gb and 1) :
>
> [ceph][WARNIN] /build/ceph-12.1.4/src/os/bluestore/BlueFS.cc: 172: FAILED
> assert(bdev[id]->get_size() >= offset + length)
>
>
>
> ALL LOG
> $ ceph-deploy osd activate ceph:/dev/sda3
> [ceph_deploy.conf][DEBUG ] found configuration file at:
> /home/cephadmin/.cephdeploy.conf
> [ceph_deploy.cli][INFO  ] Invoked (1.5.38): /usr/bin/ceph-deploy osd
> activate ceph:/dev/sda3
> [ceph_deploy.cli][INFO  ] ceph-deploy options:
> [ceph_deploy.cli][INFO  ]  username  : None
> [ceph_deploy.cli][INFO  ]  verbose   : False
> [ceph_deploy.cli][INFO  ]  overwrite_conf: False
> [ceph_deploy.cli][INFO  ]  subcommand: activate
> [ceph_deploy.cli][INFO  ]  quiet : False
> [ceph_deploy.cli][INFO  ]  cd_conf   :
> 
> [ceph_deploy.cli][INFO  ]  cluster   : ceph
> [ceph_deploy.cli][INFO  ]  func  :  at 0x7f0da96905f0>
> [ceph_deploy.cli][INFO  ]  ceph_conf : None
> [ceph_deploy.cli][INFO  ]  default_release   : False
> [ceph_deploy.cli][INFO  ]  disk  : [('ceph',
> '/dev/sda3', None)]
> [ceph_deploy.osd][DEBUG ] Activating cluster ceph disks ceph:/dev/sda3:
> [ceph][DEBUG ] connection detected need for sudo
> [ceph][DEBUG ] connected to host: ceph
> [ceph][DEBUG ] detect platform information from remote host
> [ceph][DEBUG ] detect machine type
> [ceph][DEBUG ] find the location of an executable
> [ceph_deploy.osd][INFO  ] Distro info: Ubuntu 16.04 xenial
> [ceph_deploy.osd][DEBUG ] activating host ceph disk /dev/sda3
> [ceph_deploy.osd][DEBUG ] will use init type: systemd
> [ceph][DEBUG ] find the location of an executable
> [ceph][INFO  ] Running command: sudo /usr/sbin/ceph-disk -v activate
> --mark-init systemd --mount /dev/sda3
> [ceph][WARNIN] main_activate: path = /dev/sda3
> [ceph][WARNIN] get_dm_uuid: get_dm_uuid /dev/sda3 uuid path is
> /sys/dev/block/8:3/dm/uuid
> [ceph][WARNIN] command: Running command: /sbin/blkid -o udev -p /dev/sda3
> [ceph][WARNIN] command: Running command: /sbin/blkid -p -s TYPE -o value
> -- /dev/sda3
> [ceph][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph
> --name=osd. --lookup osd_mount_options_xfs
> [ceph][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph
> --name=osd. --lookup osd_fs_mount_options_xfs
> [ceph][WARNIN] mount: Mounting /dev/sda3 on /var/lib/ceph/tmp/mnt.VA1j0C
> with options noatime,inode64
> [ceph][WARNIN] command_check_call: Running command: /bin/mount -t xfs -o
> noatime,inode64 -- /dev/sda3 /var/lib/ceph/tmp/mnt.VA1j0C
> [ceph][WARNIN] activate: Cluster uuid is
> 2d3a3e20-84e9-499d-a604-ab6fa8643387
> [ceph][WARNIN] command: Running command: /usr/bin/ceph-osd --cluster=ceph
> --show-config-value=fsid
> [ceph][WARNIN] activate: Cluster name is ceph
> [ceph][WARNIN] activate: OSD uuid is f48afefb-e146-4fdd-8e8f-1d7dba08ec75
> [ceph][WARNIN] activate: OSD id is 0
> [ceph][WARNIN] activate: Initializing OSD...
> [ceph][WARNIN] command_check_call: Running command: /usr/bin/ceph
> --cluster ceph --name client.bootstrap-osd --keyring
> /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
> /var/lib/ceph/tmp/mnt.VA1j0C/activate.monmap
> [ceph][WARNIN] got monmap epoch 1
> [ceph][WARNIN] command_check_call: Running command: /usr/bin/ceph-osd
> --cluster ceph --mkfs -i 0 --monmap
> /var/lib/ceph/tmp/mnt.VA1j0C/activate.monmap --osd-data
> /var/lib/ceph/tmp/mnt.VA1j0C --osd-uuid
> f48afefb-e146-4fdd-8e8f-1d7dba08ec75 --setuser ceph --setgroup ceph
> [ceph][WARNIN] /build/ceph-12.1.4/src/os/bluestore/BlueFS.cc: In function
> 'void BlueFS::add_block_extent(unsigned int, uint64_t, uint64_t)' thread
> 7fe2474f0e00 time 2017-08-18 13:36:45.941833
> [ceph][WARNIN] /build/ceph-12.1.4/src/os/bluestore/BlueFS.cc: 172: FAILED
> assert(bdev[id]->get_size() >= offset + length)
> [ceph][WARNIN]  ceph version 12.1.4
> (a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)
> [ceph][WARNIN]  1: (ceph::__ceph_assert_fail(char const*, char const*,
> int, char const*)+0x102) [0x563318c61042]
> [ceph][WARNIN]  2: (BlueFS::add_block_extent(unsigned int, unsigned long,
> unsigned long)+0x4da) [0x563318be753a]
> [ceph][WARNIN]  3: (BlueStore::_open_db(bool)+0x964) [0x563318af1de4]
> [ceph][WARNIN]  4: (BlueStore::mkfs()+0xcc5) [0x563318b20f65]
> [ceph][WARNIN]  5: (OSD::mkfs(CephContext*, ObjectStore*,
> std::__cxx11::basic_string std::allocator > const&, uuid_d, int)+0x164) [0x563318661164]
> [ceph][WARNIN]  6: (main()+0xe3c) [0x5633185ae0bc]
> [ceph][WARNIN]  7: (__libc_start_main()+0xf0) [0x7fe244959830]
> 

Re: [ceph-users] Fwd: Can't get fullpartition space

[Maiko de Andrade]

Hi,

I try use bluestore_block_size but I recive this error (I used values in
byte, kb, mb, gb and 1) :

[ceph][WARNIN] /build/ceph-12.1.4/src/os/bluestore/BlueFS.cc: 172: FAILED
assert(bdev[id]->get_size() >= offset + length)



ALL LOG
$ ceph-deploy osd activate ceph:/dev/sda3
[ceph_deploy.conf][DEBUG ] found configuration file at:
/home/cephadmin/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.38): /usr/bin/ceph-deploy osd
activate ceph:/dev/sda3
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username  : None
[ceph_deploy.cli][INFO  ]  verbose   : False
[ceph_deploy.cli][INFO  ]  overwrite_conf: False
[ceph_deploy.cli][INFO  ]  subcommand: activate
[ceph_deploy.cli][INFO  ]  quiet : False
[ceph_deploy.cli][INFO  ]  cd_conf   :

[ceph_deploy.cli][INFO  ]  cluster   : ceph
[ceph_deploy.cli][INFO  ]  func  : 
[ceph_deploy.cli][INFO  ]  ceph_conf : None
[ceph_deploy.cli][INFO  ]  default_release   : False
[ceph_deploy.cli][INFO  ]  disk  : [('ceph',
'/dev/sda3', None)]
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks ceph:/dev/sda3:
[ceph][DEBUG ] connection detected need for sudo
[ceph][DEBUG ] connected to host: ceph
[ceph][DEBUG ] detect platform information from remote host
[ceph][DEBUG ] detect machine type
[ceph][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: Ubuntu 16.04 xenial
[ceph_deploy.osd][DEBUG ] activating host ceph disk /dev/sda3
[ceph_deploy.osd][DEBUG ] will use init type: systemd
[ceph][DEBUG ] find the location of an executable
[ceph][INFO  ] Running command: sudo /usr/sbin/ceph-disk -v activate
--mark-init systemd --mount /dev/sda3
[ceph][WARNIN] main_activate: path = /dev/sda3
[ceph][WARNIN] get_dm_uuid: get_dm_uuid /dev/sda3 uuid path is
/sys/dev/block/8:3/dm/uuid
[ceph][WARNIN] command: Running command: /sbin/blkid -o udev -p /dev/sda3
[ceph][WARNIN] command: Running command: /sbin/blkid -p -s TYPE -o value --
/dev/sda3
[ceph][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph
--name=osd. --lookup osd_mount_options_xfs
[ceph][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph
--name=osd. --lookup osd_fs_mount_options_xfs
[ceph][WARNIN] mount: Mounting /dev/sda3 on /var/lib/ceph/tmp/mnt.VA1j0C
with options noatime,inode64
[ceph][WARNIN] command_check_call: Running command: /bin/mount -t xfs -o
noatime,inode64 -- /dev/sda3 /var/lib/ceph/tmp/mnt.VA1j0C
[ceph][WARNIN] activate: Cluster uuid is
2d3a3e20-84e9-499d-a604-ab6fa8643387
[ceph][WARNIN] command: Running command: /usr/bin/ceph-osd --cluster=ceph
--show-config-value=fsid
[ceph][WARNIN] activate: Cluster name is ceph
[ceph][WARNIN] activate: OSD uuid is f48afefb-e146-4fdd-8e8f-1d7dba08ec75
[ceph][WARNIN] activate: OSD id is 0
[ceph][WARNIN] activate: Initializing OSD...
[ceph][WARNIN] command_check_call: Running command: /usr/bin/ceph --cluster
ceph --name client.bootstrap-osd --keyring
/var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o
/var/lib/ceph/tmp/mnt.VA1j0C/activate.monmap
[ceph][WARNIN] got monmap epoch 1
[ceph][WARNIN] command_check_call: Running command: /usr/bin/ceph-osd
--cluster ceph --mkfs -i 0 --monmap
/var/lib/ceph/tmp/mnt.VA1j0C/activate.monmap --osd-data
/var/lib/ceph/tmp/mnt.VA1j0C --osd-uuid
f48afefb-e146-4fdd-8e8f-1d7dba08ec75 --setuser ceph --setgroup ceph
[ceph][WARNIN] /build/ceph-12.1.4/src/os/bluestore/BlueFS.cc: In function
'void BlueFS::add_block_extent(unsigned int, uint64_t, uint64_t)' thread
7fe2474f0e00 time 2017-08-18 13:36:45.941833
[ceph][WARNIN] /build/ceph-12.1.4/src/os/bluestore/BlueFS.cc: 172: FAILED
assert(bdev[id]->get_size() >= offset + length)
[ceph][WARNIN]  ceph version 12.1.4
(a5f84b37668fc8e03165aaf5cbb380c78e4deba4) luminous (rc)
[ceph][WARNIN]  1: (ceph::__ceph_assert_fail(char const*, char const*, int,
char const*)+0x102) [0x563318c61042]
[ceph][WARNIN]  2: (BlueFS::add_block_extent(unsigned int, unsigned long,
unsigned long)+0x4da) [0x563318be753a]
[ceph][WARNIN]  3: (BlueStore::_open_db(bool)+0x964) [0x563318af1de4]
[ceph][WARNIN]  4: (BlueStore::mkfs()+0xcc5) [0x563318b20f65]
[ceph][WARNIN]  5: (OSD::mkfs(CephContext*, ObjectStore*,
std::__cxx11::basic_string const&, uuid_d, int)+0x164) [0x563318661164]
[ceph][WARNIN]  6: (main()+0xe3c) [0x5633185ae0bc]
[ceph][WARNIN]  7: (__libc_start_main()+0xf0) [0x7fe244959830]
[ceph][WARNIN]  8: (_start()+0x29) [0x56331863ba69]
[ceph][WARNIN]  NOTE: a copy of the executable, or `objdump -rdS
` is needed to interpret this.
[ceph][WARNIN] 2017-08-18 13:36:45.943817 7fe2474f0e00 -1
/build/ceph-12.1.4/src/os/bluestore/BlueFS.cc: In function 'void
BlueFS::add_block_extent(unsigned int, uint64_t, uint64_t)' thread
7fe2474f0e00 time 2017-08-18 13:36:45.941833
[ceph][WARNIN] 

Re: [ceph-users] RBD only keyring for client

[David Turner]

That's exactly what I was missing.  Thank you.

On Thu, Aug 17, 2017 at 3:15 PM Jason Dillaman  wrote:

> You should be able to set a CEPH_ARGS='--id rbd' environment variable.
>
> On Thu, Aug 17, 2017 at 2:25 PM, David Turner 
> wrote:
> > I already tested putting name, user, and id in the global section with
> > client.rbd and rbd as the value (one at a time, testing in between).
> None of
> > them had any affect. This is on a 10.2.7 cluster.
> >
> >
> > On Thu, Aug 17, 2017, 2:06 PM Gregory Farnum  wrote:
> >>
> >> I think you just specify "name = client.rbd" as a config in the global
> >> section of the machine's ceph.conf and it will use that automatically.
> >> -Greg
> >>
> >> On Thu, Aug 17, 2017 at 10:34 AM, David Turner 
> >> wrote:
> >> > I created a user/keyring to be able to access RBDs, but I'm trying to
> >> > find a
> >> > way to set the config file on the client machine such that I don't
> need
> >> > to
> >> > use -n client.rbd in my commands when I'm on that host.  Currently I'm
> >> > testing rbd-fuse vs rbd-nbd for our use case, but I'm having a hard
> time
> >> > with the authentication because of the cephx user name.
> >> >
> >> > Does anyone have some experience they'd like to shine on this
> situation?
> >> > Thank you.
> >> >
> >> > David Turner
> >> >
> >> > ___
> >> > ceph-users mailing list
> >> > ceph-users@lists.ceph.com
> >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >> >
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
>
> --
> Jason
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] BlueStore WAL or DB devices on a distant SSD ?

[David Turner]

Specifying them to be the same device is redundant and not necessary.  They
will be put on the bluestore device by default unless you specify them to
go to another device.

On Fri, Aug 18, 2017 at 9:17 AM Hervé Ballans 
wrote:

> Le 16/08/2017 à 16:19, David Turner a écrit :
>
> Would reads and writes to the SSD on another server be faster than reads
> and writes to HDD on the local server? If the answer is no, then even if
> this was possible it would be worse than just putting your WAL and DB on
> the same HDD locally.  I don't think this is a use case the devs planned
> for.
>
> You can definitely put multiple WAL land DB partitions on a single SSD.
>
>
> Thanks David for your reply.
> Indeed, I have no idea if I/O operations on a distant SSD are faster than
> on a local HDD ?..I thought naively that in the case of a 10Gbs network,
> the speed of the bandwidth was always higher than that of the disk,
> whatever its technology, but it's true that many other factors come into
> play (in any case, it's an interesting problem, if I have time, I will try
> to do some benchmarks...)
>
> In relation to your answer, I have another question please, in the case of
> an OSDs server composed only with HDD disks, what is the best practice :
> just configure each OSD as a single BlueStore storage device or add on each
> disk both WAL and DB devices (e.g. ceph-disk prepare --bluestore /dev/sdd
> --block.wal /dev/sdd --block.db /dev/sdd) ?
>
> I ask this because the online documentation on "BlueStore Config
> Reference" states that adding these devices are useful only if the device
> used is faster than the primary device ?
>
> Thanks again,
> Hervé
>
>
> On Wed, Aug 16, 2017, 6:04 AM Hervé Ballans 
> wrote:
>
>> Hi,
>>
>> We are currently running two Proxmox/ceph clusters that work perfectly
>> (since 2014) and thank to this succesful experience, we plan to install
>> a new Ceph cluster for storage of our computing cluster.
>>
>> Until now, we only used RBD (virtualization context) but now we want to
>> use CephFS for this new cluster (separated from the other two, hardware
>> is different and dedicated for this new clusters).
>>
>> I'm interested in testing a CephFS cluster with BlueStore as a backend
>> storage.
>>
>> I have several OSDs servers (with a dozen SATA HDDs on each) but some do
>> not have an additional SSD disk (only 3 of the servers have an
>> additional SSD).
>>
>> My question is about BlueStore WAL/DB devices. When I read the
>> documentation, it seems that adding both WAL and DB devices improve
>> BlueStore performances.
>>
>> But, can we configure these devices on a distant SSD (I mean on a SSD
>> which is not on the local OSDs server but on an another machine which is
>> on the same Ceph cluster) ?
>>
>> If yes, can I configure mulitple WAL or DB devices on the same SSD ?
>>
>> And finally, is it relevant to do that (I mean in term of performance) ?
>>
>> Hoping to have been clear on my context, thanks in advance for your
>> reply  or your reflection.
>>
>> Regards,
>>
>> Hervé
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How to distribute data

[Oscar Segarra]

Hi,

Yes, you are right, the idea is cloning a snapshot taken from the base
image...

And yes, I'm working with the current RC of luminous.

In this scenario: base image (raw format)  + snapshot + snapshot clones
(for end user Windows 10 vdi). Does tiering ssd+hdd may help?

Thanks a lot


El 18 ago. 2017 4:05, "David Turner"  escribió:

Do you mean a lot of snapshots or creating a lot of clones from a snapshot?
I can agree to the pain of crating a lot of snapshots of rbds in ceph. I'm
assuming that you mean to say that you will have a template rbd with a
version snapshot that you clone each time you need to let someone log in.
Is that what you're planning?

On Thu, Aug 17, 2017, 9:51 PM Christian Balzer  wrote:

>
> Hello,
>
> On Fri, 18 Aug 2017 03:31:56 +0200 Oscar Segarra wrote:
>
> > Hi Christian,
> >
> > Thanks a lot for helping...
> >
> > Have you read:
> > http://docs.ceph.com/docs/master/rbd/rbd-openstack/
> >
> > So just from the perspective of qcow2, you seem to be doomed.
> > --> Sorry, I've talking about RAW + QCOW2 when I meant RBD images and RBD
> > snapshots...
> >
> I tested Snapshots with Hammer and the release before it, found them
> immensely painful (resource intensive) and avoided them since.
> That said, there are supposedly quite some improvements in recent versions
> (I suppose you'll deploy with Luminous), as well as more (and
> working) control knobs to reduce the impact of snapshot operations.
>
> > A sufficiently large cache tier should help there immensely and the base
> image
> > should be in cache (RAM, pagecache on the OSD servers really) all the
> time
> > anyway.
> > --> If talking about RBD images and RBD snapshots... it helps immensely
> as
> > well?
> >
> No experience, so nothing conclusive and authoritative from my end.
> If the VMs write/read alot of the same data (as in 4MB RBD objects),
> cache-tiering should help again.
> But promoting and demoting things through it when dealing with snapshots
> and deletions of them might be a pain.
>
> Christian
>
> > Sizing this and specifying the correct type of SSDs/NVMes for the
> cache-tier
> > is something that only you can answer based on existing data or
> sufficiently
> > detailed and realistic tests.
> > --> Yes, the problem is that I have to buy a HW and for Windows 10 VDI...
> > and I cannot make realistic tests previously :( but I will work on this
> > line...
> >
> > Thanks a lot again!
> >
> >
> >
> > 2017-08-18 3:14 GMT+02:00 Christian Balzer :
> >
> > >
> > > Hello,
> > >
> > > On Thu, 17 Aug 2017 23:56:49 +0200 Oscar Segarra wrote:
> > >
> > > > Hi David,
> > > >
> > > > Thanks a lot again for your quick answer...
> > > >
> > > > *The rules in the CRUSH map will always be followed.  It is not
> possible
> > > > for Ceph to go against that and put data into a root that shouldn't
> have
> > > > it.*
> > > > --> I will work on your proposal of creating two roots in the CRUSH
> > > map...
> > > > just one question more:
> > > > --> Regarding to space consumption, with this proposal, is it
> possible to
> > > > know how many disk space is it free in each pool?
> > > >
> > > > *The problem with a cache tier is that Ceph is going to need to
> promote
> > > and
> > > > evict stuff all the time (not free).  A lot of people that want to
> use
> > > SSD
> > > > cache tiering for RBDs end up with slower performance because of
> this.
> > > > Christian Balzer is the expert on Cache Tiers for RBD usage.  His
> primary
> > > > stance is that it's most likely a bad idea, but there are definite
> cases
> > > > where it's perfect.*
> > > > --> Christian, is there any advice for VDI --> BASE IMAGE (raw) +
> 1000
> > > > linked clones (qcow2)
> > > >
> > > Have you read:
> > > http://docs.ceph.com/docs/master/rbd/rbd-openstack/
> > >
> > > So just from the perspective of qcow2, you seem to be doomed.
> > >
> > > Windows always appears to be very chatty when it comes to I/O and also
> > > paging/swapping seemingly w/o need, rhyme or reason.
> > > A sufficiently large cache tier should help there immensely and the
> base
> > > image should be in cache (RAM, pagecache on the OSD servers really)
> all the
> > > time anyway.
> > > Sizing this and specifying the correct type of SSDs/NVMes for the
> > > cache-tier is something that only you can answer based on existing
> data or
> > > sufficiently detailed and realistic tests.
> > >
> > > Christian
> > >
> > > > Thanks a lot.
> > > >
> > > >
> > > > 2017-08-17 22:42 GMT+02:00 David Turner :
> > > >
> > > > > The rules in the CRUSH map will always be followed.  It is not
> possible
> > > > > for Ceph to go against that and put data into a root that shouldn't
> > > have it.
> > > > >
> > > > > The problem with a cache tier is that Ceph is going to need to
> promote
> > > and
> > > > > evict stuff all the time (not free).  A lot of people that want to
> use
> > > SSD
> > > > > cache tiering for RBDs end up with 

Re: [ceph-users] BlueStore WAL or DB devices on a distant SSD ?

[Hervé Ballans]

Le 16/08/2017 à 16:19, David Turner a écrit :


Would reads and writes to the SSD on another server be faster than 
reads and writes to HDD on the local server? If the answer is no, then 
even if this was possible it would be worse than just putting your WAL 
and DB on the same HDD locally.  I don't think this is a use case the 
devs planned for.


You can definitely put multiple WAL land DB partitions on a single SSD.



Thanks David for your reply.
Indeed, I have no idea if I/O operations on a distant SSD are faster 
than on a local HDD ?..I thought naively that in the case of a 10Gbs 
network, the speed of the bandwidth was always higher than that of the 
disk, whatever its technology, but it's true that many other factors 
come into play (in any case, it's an interesting problem, if I have 
time, I will try to do some benchmarks...)


In relation to your answer, I have another question please, in the case 
of an OSDs server composed only with HDD disks, what is the best 
practice : just configure each OSD as a single BlueStore storage device 
or add on each disk both WAL and DB devices (e.g. ceph-disk prepare 
--bluestore /dev/sdd --block.wal /dev/sdd --block.db /dev/sdd) ?


I ask this because the online documentation on "BlueStore Config 
Reference" states that adding these devices are useful only if the 
device used is faster than the primary device ?


Thanks again,
Hervé



On Wed, Aug 16, 2017, 6:04 AM Hervé Ballans 
> wrote:


Hi,

We are currently running two Proxmox/ceph clusters that work perfectly
(since 2014) and thank to this succesful experience, we plan to
install
a new Ceph cluster for storage of our computing cluster.

Until now, we only used RBD (virtualization context) but now we
want to
use CephFS for this new cluster (separated from the other two,
hardware
is different and dedicated for this new clusters).

I'm interested in testing a CephFS cluster with BlueStore as a backend
storage.

I have several OSDs servers (with a dozen SATA HDDs on each) but
some do
not have an additional SSD disk (only 3 of the servers have an
additional SSD).

My question is about BlueStore WAL/DB devices. When I read the
documentation, it seems that adding both WAL and DB devices improve
BlueStore performances.

But, can we configure these devices on a distant SSD (I mean on a SSD
which is not on the local OSDs server but on an another machine
which is
on the same Ceph cluster) ?

If yes, can I configure mulitple WAL or DB devices on the same SSD ?

And finally, is it relevant to do that (I mean in term of
performance) ?

Hoping to have been clear on my context, thanks in advance for your
reply  or your reflection.

Regards,

Hervé


___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Modify user metadata in RGW multi-tenant setup

[Sander van Schie]

I tried using quotes before, which doesn't suffice. Turns out you just
need to escape the dollar-sign:

radosgw-admin metadata get user:\$


On Thu, Aug 17, 2017 at 10:38 PM, Sander van Schie
 wrote:
> Hello,
>
> I'm trying to modify the metadata of a RGW user in a multi-tenant
> setup. For a regular user with the default implicit tenant it works
> fine using the following to get metadata:
>
> # radosgw-admin metadata get user:
>
> I however can't figure out how to do the same for a user with an
> explicit tenant. How would I go about doing this, if this is possible
> at all?
>
> What I'm trying to do is change the default_placement and
> placement_tag options. If there's other ways to do this that'd be
> great too.
>
> Running the latest version of Jewel.
>
> Thanks!
>
> - Sander
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph pgs state forever stale+active+clean

[David Turner]

What were the settings for your pool? What was the size?  It looks like the
size was 2 and that the PGs only existed on osds 2 and 6. If that's the
case, it's like having a 4 disk raid 1+0, removing 2 disks of the same
mirror, and complaining that the other mirror didn't pick up the data...
Don't delete all copies of your data.  If your replica size is 2, you
cannot loose 2 disks at the same time.

On Fri, Aug 18, 2017, 1:28 AM Hyun Ha  wrote:

> Hi, Cephers!
>
> I'm currently testing the situation of double failure for ceph cluster.
> But, I faced that pgs are in stale state forever.
>
> reproduce steps)
> 0. ceph version : jewel 10.2.3 (ecc23778eb545d8dd55e2e4735b53cc93f92e65b)
> 1. Pool create : exp-volumes (size = 2, min_size = 1)
> 2. rbd create : testvol01
> 3. rbd map and create mkfs.xfs
> 4. mount and create file
> 5. list rados object
> 6. check osd map of each object
>  # ceph osd map exp-volumes rbd_data.4a41f238e1f29.017a
>osdmap e199 pool 'exp-volumes' (2) object
> 'rbd_data.4a41f238e1f29.017a' -> pg 2.3f04d6e2 (2.62) -> up
> ([2,6], p2) acting ([2,6], p2)
> 7. stop primary osd.2 and secondary osd.6 of above object at the same time
> 8. check ceph status
> health HEALTH_ERR
> 16 pgs are stuck inactive for more than 300 seconds
> 16 pgs stale
> 16 pgs stuck stale
>  monmap e11: 3 mons at {10.105.176.85=
> 10.105.176.85:6789/0,10.110.248.154=10.110.248.154:6789/0,10.110.249.153=10.110.249.153:6789/0
> }
> election epoch 84, quorum 0,1,2
> 10.105.176.85,10.110.248.154,10.110.249.153
>  osdmap e248: 6 osds: 4 up, 4 in; 16 remapped pgs
> flags sortbitwise,require_jewel_osds
>   pgmap v112095: 128 pgs, 1 pools, 14659 kB data, 17 objects
> 165 MB used, 159 GB / 160 GB avail
>  112 active+clean
>   16 stale+active+clean
>
> # ceph health detail
> HEALTH_ERR 16 pgs are stuck inactive for more than 300 seconds; 16 pgs
> stale; 16 pgs stuck stale
> pg 2.67 is stuck stale for 689.171742, current state stale+active+clean,
> last acting [2,6]
> pg 2.5a is stuck stale for 689.171748, current state stale+active+clean,
> last acting [6,2]
> pg 2.52 is stuck stale for 689.171753, current state stale+active+clean,
> last acting [2,6]
> pg 2.4d is stuck stale for 689.171757, current state stale+active+clean,
> last acting [2,6]
> pg 2.56 is stuck stale for 689.171755, current state stale+active+clean,
> last acting [6,2]
> pg 2.d is stuck stale for 689.171811, current state stale+active+clean,
> last acting [6,2]
> pg 2.79 is stuck stale for 689.171808, current state stale+active+clean,
> last acting [2,6]
> pg 2.1f is stuck stale for 689.171782, current state stale+active+clean,
> last acting [6,2]
> pg 2.76 is stuck stale for 689.171809, current state stale+active+clean,
> last acting [6,2]
> pg 2.17 is stuck stale for 689.171794, current state stale+active+clean,
> last acting [6,2]
> pg 2.63 is stuck stale for 689.171794, current state stale+active+clean,
> last acting [2,6]
> pg 2.77 is stuck stale for 689.171816, current state stale+active+clean,
> last acting [2,6]
> pg 2.1b is stuck stale for 689.171793, current state stale+active+clean,
> last acting [6,2]
> pg 2.62 is stuck stale for 689.171765, current state stale+active+clean,
> last acting [2,6]
> pg 2.30 is stuck stale for 689.171799, current state stale+active+clean,
> last acting [2,6]
> pg 2.19 is stuck stale for 689.171798, current state stale+active+clean,
> last acting [6,2]
>
>  # ceph pg dump_stuck stale
> ok
> pg_stat state   up  up_primary  acting  acting_primary
> 2.67stale+active+clean  [2,6]   2   [2,6]   2
> 2.5astale+active+clean  [6,2]   6   [6,2]   6
> 2.52stale+active+clean  [2,6]   2   [2,6]   2
> 2.4dstale+active+clean  [2,6]   2   [2,6]   2
> 2.56stale+active+clean  [6,2]   6   [6,2]   6
> 2.d stale+active+clean  [6,2]   6   [6,2]   6
> 2.79stale+active+clean  [2,6]   2   [2,6]   2
> 2.1fstale+active+clean  [6,2]   6   [6,2]   6
> 2.76stale+active+clean  [6,2]   6   [6,2]   6
> 2.17stale+active+clean  [6,2]   6   [6,2]   6
> 2.63stale+active+clean  [2,6]   2   [2,6]   2
> 2.77stale+active+clean  [2,6]   2   [2,6]   2
> 2.1bstale+active+clean  [6,2]   6   [6,2]   6
> 2.62stale+active+clean  [2,6]   2   [2,6]   2
> 2.30stale+active+clean  [2,6]   2   [2,6]   2
> 2.19stale+active+clean  [6,2]   6   [6,2]   6
>
> # ceph pg 2.62 query
> Error ENOENT: i don't have pgid 2.62
>
>  # rados ls -p exp-volumes
> rbd_data.4a41f238e1f29.003f
> ^C --> hang
>
> I understand that this is a natural result becasue above pgs have no
> primary and seconary osd. But this situation can be occurred so, I want to
> recover ceph cluster and rbd images.
>
> Firstly I want to know how