Re: [ceph-users] Snapshot trimming

Is scrubbing and deep scrubbing necessary for Snaptrim operation to happen ? Karun Josy On Fri, Jan 26, 2018 at 9:29 PM, Karun Josy wrote: > Thank you for your quick response! > > I used the command to fetch the snap_trimq from many pgs, however it seems > they don't have

Re: [ceph-users] Migrating filestore to bluestore using ceph-volume

Bit late for this to be helpful, but instead of zapping the lvm labels, you could alternatively destroy the lvm volume by hand. > lvremove -f / > vgremove > pvremove /dev/ceph-device (should wipe labels) Then you should be able to run ‘ceph-volume lvm zap /dev/sdX’ and retry the 'ceph-volume

[ceph-users] Ceph OSDs fail to start with RDMA

Hi all, I am trying to bring up a Ceph cluster where the private network is communicating via RoCEv2. The storage nodes have 2 dual-port 25Gb Mellanox ConnectX-4 NICs, with each NIC's ports bonded (2x25Gb mode 4). I have set memory limits to unlimited, can rping to each node, and

Re: [ceph-users] Migrating filestore to bluestore using ceph-volume

Yeah, next one will be without double rebalance, I just had alot of time on my hands. Never did use kill before, however I followed the docs here. Should probably be updated. http://docs.ceph.com/docs/master/rados/operations/bluestore-migration/#convert-existing-osds

Re: [ceph-users] How ceph client read data from ceph cluster

Hi Lin, Yes it will read from the primary osd, but for the reasons stated this should not impact performance. Maged On 2018-01-26 19:52, shadow_lin wrote: > Hi Maged, > I just want to make sure if I understand how ceph client read from cluster.So > with current version of ceph(12.2.2) the

Re: [ceph-users] Migrating filestore to bluestore using ceph-volume

On 01/26/2018 07:09 PM, David Majchrzak wrote: destroy did remove the auth key, however create didnt add the auth, I had to do it manually. Then I tried to start the osd.0 again and it failed because osdmap said it was destroyed. That seems like this bug:

[ceph-users] Signature check failures.

Hello, We are running Luminous 12.2.2. 6 OSD hosts with 12 1TB OSDs, and 64GB RAM. Each host has a SSD for Bluestore's block.wal and block.db. There are 5 monitor nodes as well with 32GB RAM. All servers have Gentoo with kernel, 4.12.12-gentoo. When I export an image using: rbd export

Re: [ceph-users] Migrating filestore to bluestore using ceph-volume

destroy did remove the auth key, however create didnt add the auth, I had to do it manually. Then I tried to start the osd.0 again and it failed because osdmap said it was destroyed. I've summed my steps below: Here are my commands prior to create: root@int1:~# ceph osd out 0 <-- wait for

Re: [ceph-users] Migrating filestore to bluestore using ceph-volume

On 01/26/2018 06:53 PM, David Majchrzak wrote: I did do that. It didn't add the auth key to ceph, so I had to do that manually. Then it said that osd.0 was set as destroyed, which yes, it was still in crushmap. I followed the docs to a point. Odd, the 'destroy' command should remove the

Re: [ceph-users] Migrating filestore to bluestore using ceph-volume

I did do that. It didn't add the auth key to ceph, so I had to do that manually. Then it said that osd.0 was set as destroyed, which yes, it was still in crushmap. I followed the docs to a point. > 26 jan. 2018 kl. 18:50 skrev Wido den Hollander : > > > > On 01/26/2018 06:37

Re: [ceph-users] How ceph client read data from ceph cluster

Hi Maged, I just want to make sure if I understand how ceph client read from cluster.So with current version of ceph(12.2.2) the client only read from the primary osd(one copy),is that true? 2018-01-27 lin.yunfan 发件人：Maged Mokhtar 发送时间：2018-01-26 20:27 主题：Re:

Re: [ceph-users] Migrating filestore to bluestore using ceph-volume

On 01/26/2018 06:37 PM, David Majchrzak wrote: Ran: ceph auth del osd.0 ceph auth del osd.6 ceph auth del osd.7 ceph osd rm osd.0 ceph osd rm osd.6 ceph osd rm osd.7 which seems to have removed them. Did you destroy the OSD prior to running ceph-volume? $ ceph osd destroy 6 After you've

Re: [ceph-users] Importance of Stable Mon and OSD IPs

Resending in case this email was lost On Tue, Jan 23, 2018 at 10:50 PM Mayank Kumar wrote: > Thanks Burkhard for the detailed explanation. Regarding the following:- > > >>>The ceph client (librbd accessing a volume in this case) gets > asynchronous notification from the

Re: [ceph-users] Migrating filestore to bluestore using ceph-volume

Ran: ceph auth del osd.0 ceph auth del osd.6 ceph auth del osd.7 ceph osd rm osd.0 ceph osd rm osd.6 ceph osd rm osd.7 which seems to have removed them. Thanks for the help Reed! Kind Regards, David Majchrzak > 26 jan. 2018 kl. 18:32 skrev David Majchrzak : > > Thanks that

Re: [ceph-users] Migrating filestore to bluestore using ceph-volume

Thanks that helped! Since I had already "halfway" created a lvm volume I wanted to start from the beginning and zap it. Tried to zap the raw device but failed since --destroy doesn't seem to be in 12.2.2 http://docs.ceph.com/docs/master/ceph-volume/lvm/zap/

Re: [ceph-users] Migrating filestore to bluestore using ceph-volume

This is the exact issue that I ran into when starting my bluestore conversion journey. See my thread here: https://www.spinics.net/lists/ceph-users/msg41802.html Specifying --osd-id causes it to fail. Below are my steps for OSD

Re: [ceph-users] Weird issues related to (large/small) weights in mixed nvme/hdd pool

Ok, by randomly toggling settings *MOST* of the PGs in the test cluster is online, but a few are not. No matter how much I change, a few of them seem to not activate. They are running bluestore with version 12.2.2, i think created with ceph-volume. Here is the output from ceph pg X query of

[ceph-users] Bluefs WAL : bluefs _allocate failed to allocate on bdev 0

Hi all, I've a question regarding bluestore wal.db: We are running a 10 OSD node + 3 MON/MDS node cluster (luminous 12.2.2). Each OSD node has 22xHDD (8TB) OSDs, 2xSSD (1.6TB) OSDs and 2xNVME (800 GB) for bluestore wal and db. We have separated wal and db partitions wal partitions are 1GB db

[ceph-users] Migrating filestore to bluestore using ceph-volume

Hi! On luminous 12.2.2 I'm migrating some OSDs from filestore to bluestore using the "simple" method as described in docs: http://docs.ceph.com/docs/master/rados/operations/bluestore-migration/#convert-existing-osds

Re: [ceph-users] swift capabilities support in radosgw

Hi Syed, RGW supports Swift /info in Luminous. By default iirc those aren't at the root of the URL hierarchy, but there's an option to change that, since last year, see https://github.com/ceph/ceph/pull/10280. Matt On Fri, Jan 26, 2018 at 5:10 AM, Syed Armani wrote: >

Re: [ceph-users] Can't make LDAP work

Hi Theofilos, I'm not sure what's going wrong offhand, I see all the pieces in your writeup. The first thing I would verify is that "CN=cephs3,OU=Users,OU=Organic Units,DC=example,DC=com" see the users in ldaps://ldap.example.com:636, and that "cn=myuser..." can itself simple bind using standard

Re: [ceph-users] ceph-volume raw disks

On Fri, Jan 26, 2018 at 8:07 AM, David Turner wrote: > I didn't test those commands on 12.2.1 either. I've only used 12.2.2. > > On Fri, Jan 26, 2018 at 6:35 AM Alfredo Deza wrote: >> >> That looks like Luminous, but not 12.2.2 >> >> The 'raw' device

Re: [ceph-users] BlueStore.cc: 9363: FAILED assert(0 == "unexpected error")

I also just got my new SSDs that are 480GB if they could be used to move the PGs to. Thank you for your help. On Fri, Jan 26, 2018 at 8:33 AM David Turner wrote: > If I could get it started, I could flush-evict the cache, but that's not > seeming likely. > > On Fri, Jan

Re: [ceph-users] BlueStore.cc: 9363: FAILED assert(0 == "unexpected error")

If I could get it started, I could flush-evict the cache, but that's not seeming likely. On Fri, Jan 26, 2018 at 8:33 AM David Turner wrote: > I wouldn't be shocked if they were out of space, but `ceph osd df` only > showed them as 45% full when I was first diagnosing

Re: [ceph-users] Weird issues related to (large/small) weights in mixed nvme/hdd pool

Ok, so creating our setup in the lab and adding the pools, our hybrid pool cannot even be properly created with around 1/3 of the PGs stuck in various states:   cluster:     id: e07f568d-056c-4e01-9292-732c64ab4f8e     health: HEALTH_WARN     Reduced data availability: 1070 pgs

Re: [ceph-users] BlueStore.cc: 9363: FAILED assert(0 == "unexpected error")

I can see this in the logs: 2018-01-25 06:05:56.292124 7f37fa6ea700 -1 log_channel(cluster) log [ERR] : full status failsafe engaged, dropping updates, now 101% full 2018-01-25 06:05:56.325404 7f3803f9c700 -1 bluestore(/var/lib/ceph/osd/ceph-9) _do_alloc_write failed to reserve 0x4000

Re: [ceph-users] ceph-volume raw disks

I didn't test those commands on 12.2.1 either. I've only used 12.2.2. On Fri, Jan 26, 2018 at 6:35 AM Alfredo Deza wrote: > That looks like Luminous, but not 12.2.2 > > The 'raw' device handling is supported in 12.2.2 for sure. > > On Thu, Jan 25, 2018 at 10:42 PM, David

Re: [ceph-users] Snapshot trimming

You may find the information in this ML thread useful. https://www.spinics.net/lists/ceph-users/msg41279.html It talks about a couple ways to track your snaptrim queue. On Fri, Jan 26, 2018 at 2:09 AM Karun Josy wrote: > Hi, > > We have set no scrub , no deep scrub flag

[ceph-users] BlueStore.cc: 9363: FAILED assert(0 == "unexpected error")

http://tracker.ceph.com/issues/22796 I was curious if anyone here had any ideas or experience with this problem. I created the tracker for this yesterday when I woke up to find all 3 of my SSD OSDs not running and unable to start due to this segfault. These OSDs are in my small home cluster and

Re: [ceph-users] RGW Upgrade to Luminous Inconsistent PGs in index pools

That last part with the scrubs, when manually run returning clean may not be accurate. Doing more testing, but the problem is definitely persistent even after a repair returns to show the PG as clean. On Fri, Jan 26, 2018 at 7:41 AM David Turner wrote: > I just upgraded

[ceph-users] RGW Upgrade to Luminous Inconsistent PGs in index pools

I just upgraded to Luminous yesterday and before the upgrade was complete, we had SSD OSDs flapping up and down and scrub errors in the RGW index pools. I consistently made sure that we had all OSDs back up and the cluster healthy before continuing and never reduced the min_size below 2 for the

Re: [ceph-users] How ceph client read data from ceph cluster

On 2018-01-26 09:09, shadow_lin wrote: > Hi List, > I read a old article about how ceph client read from ceph cluster.It said the > client only read from the primary osd. Since ceph cluster in replicate mode > have serveral copys of data only read from one copy seems waste the > performance

Re: [ceph-users] Weird issues related to (large/small) weights in mixed nvme/hdd pool

Well, we do, but our problem is with our hybrid setup (1 nvme and 2 hdds). The other two (that we rarely use) are nvme only and hdd only, as far as I can tell they work and "take" command uses class to select only the relevant OSDs. I'll just paste our entire crushmap dump here. This one

Re: [ceph-users] ceph-volume raw disks

That looks like Luminous, but not 12.2.2 The 'raw' device handling is supported in 12.2.2 for sure. On Thu, Jan 25, 2018 at 10:42 PM, David Turner wrote: > Did you wipe all of the existing partitions and such first? Which version > of ceph? The below commands are what I

[ceph-users] Can't make LDAP work

They gave me a ldap server working with users inside, and I want to create tokens for these users to use s3 from their ldap credentials. I tried using the sanity check and I got this one working: ldapsearch -x -D "CN=cephs3,OU=Users,OU=Organic Units,DC=example,DC=com" -W -H

Re: [ceph-users] Weird issues related to (large/small) weights in mixed nvme/hdd pool

Hi Peter, Just to check if your problem is similar to mine: - Do you have any pools that follow a crush rule to only use osds that are backed by hdds (i.e not nvmes)? - Do these pools obey that rule? i.e do they maybe have pgs that are on nvmes? Regards, Tom On Fri, Jan 26, 2018 at

[ceph-users] swift capabilities support in radosgw

Hello folks, I am getting this error "Capabilities GET failed: https://SWIFT:8080/info 404 Not Found", when executing a "$ swift capabilities" command against a radosgw cluster. I was wondering whether radosgw supports the listing of activated capabilities[0] via Swift API? Something a

Re: [ceph-users] Weird issues related to (large/small) weights in mixed nvme/hdd pool

Hi Thomas, No, we haven't gotten any closer to resolving this, in fact we had another issue again when we added a new nvme drive to our nvme servers (storage11, storage12 and storage13) that had weight 1.7 instead of the usual 0.728 size. This (see below) is what a nvme and hdd server pair at