Re: [ceph-users] radosgw multi site different period

both cluster are in the same epoch and period:   root@ceph-a-1:~# radosgw-admin period get-current  { "current_period": "b7392c41-9cbe-4d92-ad03-db607dd7d569" } root@ceph-b-1:~# radosgw-admin period get-current  { "current_period": "b7392c41-9cbe-4d92-ad03-db607dd7d569" } but the sync

[ceph-users] CephFS | Mounting Second CephFS

Hi, When running more than one cephfs how would I specify which file system I want to mount in ceph-fuse or the kernel client? OS: Ubuntu 16.04.3 LTS Ceph version: 12.2.1 - Luminous Kind regards Geoffrey Rhodes ___ ceph-users mailing list

[ceph-users] 10.2.10: "default" zonegroup in custom root pool not found

After creating a non-default root pool rgw_realm_root_pool = gold.rgw.root rgw_zonegroup_root_pool = gold.rgw.root rgw_period_root_pool = gold.rgw.root rgw_zone_root_pool = gold.rgw.root rgw_region = gold.rgw.root radosgw-admin realm create --rgw-realm gold --default radosgw-admin zonegroup

[ceph-users] Why keep old epochs?

Some questions about maps and epochs: I see that I can control the minimum number of osdmap epochs to keep with "mon min osdmap epoch". Why do I care? Why would I want any but the current osdmap, and why would the system keep more than my minimum? Similarly, "mon max pgmap epoch" controls the

Re: [ceph-users] Bluestore performance 50% of filestore

I'd probably say 50GB to leave some extra space over-provisioned. 50GB should definitely prevent any DB operations from spilling over to the HDD. On Tue, Nov 14, 2017, 5:43 PM Milanov, Radoslav Nikiforov wrote: > Thank you, > > It is 4TB OSDs and they might become full someday,

Re: [ceph-users] Bluestore performance 50% of filestore

Thank you, It is 4TB OSDs and they might become full someday, I’ll try 60GB db partition – this is the max OSD capacity. - Rado From: David Turner [mailto:drakonst...@gmail.com] Sent: Tuesday, November 14, 2017 5:38 PM To: Milanov, Radoslav Nikiforov Cc: Mark Nelson

Re: [ceph-users] Bluestore performance 50% of filestore

You have to configure the size of the db partition in the config file for the cluster. If you're db partition is 1GB, then I can all but guarantee that you're using your HDD for your blocks.db very quickly into your testing. There have been multiple threads recently about what size the db

Re: [ceph-users] S3/Swift :: Pools Ceph

While you can configure 1 pool to be used for RBD and Object storage, I believe that is being deprecated and can cause unforeseen problems in the future. It is definitely not a recommended or common use case. On Tue, Nov 14, 2017 at 4:51 PM Christian Wuerdig < christian.wuer...@gmail.com> wrote:

Re: [ceph-users] S3/Swift :: Pools Ceph

As per documentation: http://docs.ceph.com/docs/luminous/radosgw/ "The S3 and Swift APIs share a common namespace, so you may write data with one API and retrieve it with the other." So you can access one pool through both APIs and the data will be available via both. On Wed, Nov 15, 2017 at

Re: [ceph-users] Incorrect pool usage statistics

If you know that the pool should be empty, there wouldn't be a problem with piping the ouput of `rados ls` to `rados rm`. By the same notion, if nothing in the pool is needed you can delete the pool and create a new one that will be perfectly empty. On Tue, Nov 14, 2017 at 3:23 PM Karun Josy

Re: [ceph-users] Bluestore performance 50% of filestore

Block-db partition is the default 1GB (is there a way to modify this? journals are 5GB in filestore case) and usage is low: [root@kumo-ceph02 ~]# ceph df GLOBAL: SIZEAVAIL RAW USED %RAW USED 100602G 99146G1455G 1.45 POOLS: NAME ID

Re: [ceph-users] Bluestore performance 50% of filestore

16 MB block, single thread, sequential writes, this is [cid:image001.emz@01D35D67.61AF9D30] - Rado -Original Message- From: Mark Nelson [mailto:mnel...@redhat.com] Sent: Tuesday, November 14, 2017 4:36 PM To: Milanov, Radoslav Nikiforov ; ceph-users@lists.ceph.com

Re: [ceph-users] Bluestore performance 50% of filestore

How big was your blocks.db partition for each OSD and what size are your HDDs? Also how full is your cluster? It's possible that your blocks.db partition wasn't large enough to hold the entire db and it had to spill over onto the HDD which would definitely impact performance. On Tue, Nov 14,

Re: [ceph-users] Bluestore performance 50% of filestore

How big were the writes in the windows test and how much concurrency was there? Historically bluestore does pretty well for us with small random writes so your write results surprise me a bit. I suspect it's the low queue depth. Sometimes bluestore does worse with reads, especially if

Re: [ceph-users] Bluestore performance 50% of filestore

Hi Mark, Yes RBD is in write back, and the only thing that changed was converting OSDs to bluestore. It is 7200 rpm drives and triple replication. I also get same results (bluestore 2 times slower) testing continuous writes on a 40GB partition on a Windows VM, completely different tool. Right

Re: [ceph-users] Bluestore performance 50% of filestore

Hi Radoslav, Is RBD cache enabled and in writeback mode? Do you have client side readahead? Both are doing better for writes than you'd expect from the native performance of the disks assuming they are typical 7200RPM drives and you are using 3X replication (~150IOPS * 27 / 3 = ~1350

Re: [ceph-users] Incorrect pool usage statistics

Help?! There seems to be many objects still present in the pool : - $ rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPSRDWR_OPSWR vm 886 105 0 315 0 00

[ceph-users] Bluestore performance 50% of filestore

Hi We have 3 node, 27 OSDs cluster running Luminous 12.2.1 In filestore configuration there are 3 SSDs used for journals of 9 OSDs on each hosts (1 SSD has 3 journal paritions for 3 OSDs). I've converted filestore to bluestore by wiping 1 host a time and waiting for recovery. SSDs now contain

Re: [ceph-users] Deleting large pools

2 weeks later and things are still deleting, but getting really close to being done. I tried to use ceph-objectstore-tool to remove one of the PGs. I only tested on 1 PG on 1 OSD, but it's doing something really weird. While it was running, my connection to the DC reset and the command died.

[ceph-users] S3/Swift :: Pools Ceph

Hi Everyone, I was wondering, has anyone tried in a Test/Production environment, to have 1 pool, to which you can input/output data using S3 and Swift, or would each need a separate pool, one to serve via S3 and one to serve via Swift ? Also, I believe you can use 1 pool for RBD and Object

Re: [ceph-users] radosgw multi site different period

both cluster are in the same epoch and period: root@ceph-a-1:~# radosgw-admin period get-current  { "current_period": "b7392c41-9cbe-4d92-ad03-db607dd7d569" } root@ceph-b-1:~# radosgw-admin period get-current  { "current_period": "b7392c41-9cbe-4d92-ad03-db607dd7d569" } Am Dienstag, den

Re: [ceph-users] radosgw multi site different period

I'm assuming you've looked at the period in both places `radosgw-admin period get` and confirmed that the second site is behind the master site (based on epochs). I'm also assuming (since you linked the instructions) that you've done `radosgw-admin period pull` on the second site to get any

[ceph-users] radosgw multi site different period

hi, i've installed a ceph multi site setup with two ceph clusters and each one radosgw. the multi site setup was in sync, so i tried a failover. cluster A is going down and i've changed the zone (b) on cluster b to the new master zone. it's working fine. now i start the cluster A and try to

Re: [ceph-users] features required for live migration

Thanks a lot for all your comments, If you don't see any problem... I will enable the following features that might fit my requirements: Layering Striping Exclusive locking Object map Fast-diff Thanks a lot Óscar Segarra 2017-11-14 16:56 GMT+01:00 Jason Dillaman : >

Re: [ceph-users] features required for live migration

>From the documentation [1]: shareable If present, this indicates the device is expected to be shared between domains (assuming the hypervisor and OS support this), which means that caching should be deactivated for that device. Basically, it's the use-case for putting a clustered file system

Re: [ceph-users] features required for live migration

>From the documentation [1]: shareable If present, this indicates the device is expected to be shared between domains (assuming the hypervisor and OS support this), which means that caching should be deactivated for that device. Basically, it's the use-case for putting a clustered file system

Re: [ceph-users] features required for live migration

Hi Jason, The big use-case for sharing a block device is if you set up a clustered file system on top of it, and I'd argue that you'd probably be better off using CephFS. --> Nice to know! Thanks a lot for your clarifications, in this case I referenced the shareable flag that one can see in the

Re: [ceph-users] features required for live migration

On Tue, Nov 14, 2017 at 10:25 AM, Oscar Segarra wrote: > In my environment, I have a Centos7 updated todate therefore, all > features might work as expected to do... > > Regarding the other question, do you suggest making the virtual disk > "shareable" in rbd?

Re: [ceph-users] features required for live migration

In my environment, I have a Centos7 updated todate therefore, all features might work as expected to do... Regarding the other question, do you suggest making the virtual disk "shareable" in rbd? Thanks a lot 2017-11-14 15:58 GMT+01:00 Jason Dillaman : > Concur --

Re: [ceph-users] features required for live migration

Concur -- there aren't any RBD image features that should prevent live migration when using a compatible version of librbd. If, however, you had two hosts where librbd versions were out-of-sync and they didn't support the same features, you could hit an issue if a VM with fancy new features was

Re: [ceph-users] features required for live migration

Hi Oscar, exclusive-locking should not interfere with live-migration. I have a small virtualization cluster backed by ceph/rbd and I can migrate all the VMs which RBD image have exclusive-lock enabled without any issue. Em 11/14/2017 9:47 AM, Oscar Segarra escreveu: Hi Konstantin, Thanks a

Re: [ceph-users] Incomplete pgs on ceph which is partly on Bluestore

Sorry, I've not mentioned, the ceph version is Luminous 12.2.1 С уважением, Ухина Ольга Моб. тел.: 8(905)-566-46-62 2017-11-14 15:30 GMT+03:00 Ольга Ухина : > Hi! I've a ceph installation where one host with OSDs are on Blustore and > three other are on FileStore, it

[ceph-users] Incomplete pgs on ceph which is partly on Bluestore

Hi! I've a ceph installation where one host with OSDs are on Blustore and three other are on FileStore, it worked till deleting this first host with all Bluestore OSDs and then these OSDs were back completely clean. Ceph remapped and I ended up with 19 pgs inactive and 19 incomplete. Primary OSDs

Re: [ceph-users] features required for live migration

Hi, I include Jason Dillaman, the creator of this post http://tracker.ceph.com/issues/15000 in this thread Thanks a lot 2017-11-14 12:47 GMT+01:00 Oscar Segarra : > Hi Konstantin, > > Thanks a lot for your advice... > > I'm specially interested in feature "Exclusive

Re: [ceph-users] features required for live migration

Hi Konstantin, Thanks a lot for your advice... I'm specially interested in feature "Exclusive locking". Enabling this feature can affect live/offline migration? In this scenario (online/offline migration) I don't know if two hosts (source and destination) need access to the same rbd image at

Re: [ceph-users] features required for live migration

On 11/14/2017 06:19 PM, Oscar Segarra wrote: What I'm trying to do is reading documentation in order to understand how features work and what are they for. http://tracker.ceph.com/issues/15000 I would also be happy to read what features have negative sides. The problem is that

Re: [ceph-users] features required for live migration

Hi Konstantin, What I'm trying to do is reading documentation in order to understand how features work and what are they for. http://tracker.ceph.com/issues/15000 The problem is that documentation is not detailed enough. The proof-test method you suggest I think is not a good procedure because

Re: [ceph-users] features required for live migration

I misunderstand you. If you at the testing/deploy stage - why you can't test what features you need and what supported by your librbd? On 11/14/2017 05:39 PM, Oscar Segarra wrote: In this moment, I'm deploying and therefore I can upgrade every component... I have recently executed "yum

Re: [ceph-users] HW Raid vs. Multiple OSD

Hi Anthony, o I think you might have some misunderstandings about how Ceph works. Ceph is best deployed as a single cluster spanning multiple servers, generally at least 3. Is that your plan? I want to deply servers for 100VDI Windows 10 each (at least 3 servers). I plan to sell servers

Re: [ceph-users] features required for live migration

For understanding: live migration is just the same run like clean run from powered off state, exception only the copying memory from one host to another, i.e. if your VM start from powered off state, than live migration should works without any issues. Also, client must be compatible with the

Re: [ceph-users] mount failed since failed to load ceph kernel module

HI, You should do something like #ceph osd in osd.${num}: But If this is your tree, I do not see any osd available at this moment in your cluster, should be something similar to this xesample: ID CLASS WEIGHT TYPE NAMESTATUS REWEIGHT PRI-AFF -1 58.21509 root default

Re: [ceph-users] mount failed since failed to load ceph kernel module

On Tue, Nov 14, 2017 at 10:52:00AM +0100, Iban Cabrillo wrote: > Hi Dai Xiang, > There is no OSD available at this moment in your cluste, then you can't > read/write or mount anything, maybe the osds are configured but they are > out, please could you paste the "#ceph osd tree " command > to see

Re: [ceph-users] mount failed since failed to load ceph kernel module

Hi Dai Xiang, There is no OSD available at this moment in your cluste, then you can't read/write or mount anything, maybe the osds are configured but they are out, please could you paste the "#ceph osd tree " command to see your osd status ? Regards, I 2017-11-14 10:39 GMT+01:00 Dai Xiang

Re: [ceph-users] Cephalocon 2018?

In Sydney at the OpenStack Summit Sage announced a Cephalocon for 2018.03.22-23 in Beijing (China). Danny Am 12.10.2017 um 13:02 schrieb Matthew Vernon: > Hi, > > The recent FOSDEM CFP reminded me to wonder if there's likely to be a > Cephalocon in 2018? It was mentioned as a possibility when

Re: [ceph-users] features required for live migration

Hi, Yes, but looks lots of features like snapshot, fast-diff require some other features... If I enable exclusive-locking or journaling, live migration will be possible too? Is it recommended to set KVM disk "shareable" depending on the activated features? Thanks a lot! 2017-11-14 4:52

Re: [ceph-users] mount failed since failed to load ceph kernel module

On Tue, Nov 14, 2017 at 09:21:56AM +, Linh Vu wrote: > Odd, you only got 2 mons and 0 osds? Your cluster build looks incomplete. But from the log, osd seems normal: [172.17.0.4][INFO ] checking OSD status... [172.17.0.4][DEBUG ] find the location of an executable [172.17.0.4][INFO ] Running

Re: [ceph-users] mount failed since failed to load ceph kernel module

Odd, you only got 2 mons and 0 osds? Your cluster build looks incomplete. Get Outlook for Android From: Dai Xiang Sent: Tuesday, November 14, 2017 6:12:27 PM To: Linh Vu Cc: ceph-users@lists.ceph.com Subject: Re:

Re: [ceph-users] Incorrect pool usage statistics

Hello Karun, On Tue, Nov 14, 2017 at 04:16:51AM +0530, Karun Josy wrote: > Hello, > > Recently, I deleted all the disks from an erasure pool 'ecpool'. > The pool is empty. However the space usage shows around 400GB. > What might be wrong? > > > $ rbd ls -l ecpool > $ $ ceph df > > GLOBAL: >