Re: [ceph-users] radosgw: can't delete bucket

Hi, I finaly managed to delete the bucket, I wrote a script that reads the omap keys from the bucket index and deletes every key without a matching object on the data pool. Not sure if this has any negative repercussions, but after the script deleted thousands of keys from the index, i was able

Re: [ceph-users] Purged a pool, buckets remain

On Tue, Apr 10, 2018 at 10:06:57PM -0500, Robert Stanford wrote: > I used this command to purge my rgw data: > > rados purge default.rgw.buckets.data --yes-i-really-really-mean-it > > Now, when I list the buckets with s3cmd, I still see the buckets (s3cmd ls > shows a listing of them.) When I

Re: [ceph-users] rbd-nbd not resizing even after kernel tweaks

On Tue, Apr 10, 2018 at 11:14:58PM -0400, Alex Gorbachev wrote: > So Josef fixed the one issue that enables e.g. lsblk and sysfs size to > reflect the correct siz on change. However, partptobe and parted > still do not detect the change, complete unmap and remap of rbd-nbd > device and remount of

Re: [ceph-users] ceph-fuse CPU and Memory usage vs CephFS kclient

On 04/10/2018 09:45 PM, Gregory Farnum wrote: > On Tue, Apr 10, 2018 at 12:36 PM, Wido den Hollander wrote: >> >> >> On 04/10/2018 09:22 PM, Gregory Farnum wrote: >>> On Tue, Apr 10, 2018 at 6:32 AM Wido den Hollander >> > wrote: >>> >>> Hi, >>> >>> There have been

Re: [ceph-users] Purged a pool, buckets remain

Now, when I list the buckets with s3cmd, I still see the buckets (s3cmd ls shows a listing of them.) When I try to delete one (s3cmd rb) I get this: ERROR: S3 error: 404 (NoSuchKey) Because you are drop all your data. But all your buckets is indexed. You shouldn't work with S3 like this

Re: [ceph-users] Ceph luminous - troubleshooting performance issues overall DSK 100%, busy 1%

ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE USEAVAIL %USE VAR PGS TYPE NAME -1 3.44714- 588G 80693M 509G 00 - root default -9 0.57458- 588G 80693M 509G 13.39 1.13 - host osd01 5 hdd 0.57458 1.0 588G 80693M 509G 13.39 1.13 64

Re: [ceph-users] cephfs snapshot format upgrade

On Wed, Apr 11, 2018 at 3:34 AM, Gregory Farnum wrote: > On Tue, Apr 10, 2018 at 5:54 AM, John Spray wrote: >> On Tue, Apr 10, 2018 at 1:44 PM, Yan, Zheng wrote: >>> Hello >>> >>> To simplify snapshot handling in multiple active mds setup, we changed >>> format of snaprealm in mimic dev. >>> htt

Re: [ceph-users] cephfs snapshot format upgrade

On Wed, Apr 11, 2018 at 10:10 AM, Sage Weil wrote: > On Tue, 10 Apr 2018, Patrick Donnelly wrote: >> On Tue, Apr 10, 2018 at 5:54 AM, John Spray wrote: >> > On Tue, Apr 10, 2018 at 1:44 PM, Yan, Zheng wrote: >> >> Hello >> >> >> >> To simplify snapshot handling in multiple active mds setup, we c

Re: [ceph-users] rbd-nbd not resizing even after kernel tweaks

On Sun, Mar 11, 2018 at 3:50 PM, Alex Gorbachev wrote: > On Sun, Mar 11, 2018 at 4:23 AM, Mykola Golub wrote: >> On Sat, Mar 10, 2018 at 08:25:15PM -0500, Alex Gorbachev wrote: >>> I am running into the problem described in >>> https://lkml.org/lkml/2018/2/19/565 and >>> https://tracker.ceph.com

[ceph-users] Purged a pool, buckets remain

I used this command to purge my rgw data: rados purge default.rgw.buckets.data --yes-i-really-really-mean-it Now, when I list the buckets with s3cmd, I still see the buckets (s3cmd ls shows a listing of them.) When I try to delete one (s3cmd rb) I get this: ERROR: S3 error: 404 (NoSuchKey)

Re: [ceph-users] cephfs snapshot format upgrade

On Tue, 10 Apr 2018, Patrick Donnelly wrote: > On Tue, Apr 10, 2018 at 5:54 AM, John Spray wrote: > > On Tue, Apr 10, 2018 at 1:44 PM, Yan, Zheng wrote: > >> Hello > >> > >> To simplify snapshot handling in multiple active mds setup, we changed > >> format of snaprealm in mimic dev. > >> https://

Re: [ceph-users] cephfs snapshot format upgrade

On Tue, Apr 10, 2018 at 5:54 AM, John Spray wrote: > On Tue, Apr 10, 2018 at 1:44 PM, Yan, Zheng wrote: >> Hello >> >> To simplify snapshot handling in multiple active mds setup, we changed >> format of snaprealm in mimic dev. >> https://github.com/ceph/ceph/pull/16779. >> >> The new version mds

Re: [ceph-users] Ceph luminous - troubleshooting performance issues overall DSK 100%, busy 1%

Hi With osd_debug increased to 5/5 I am seeing lots of these in the ceph-osd.5.log ( newly added OSD) Anyone know what it means ? 2018-04-10 16:05:33.317451 7f33610be700 5 osd.5 300 heartbeat: osd_stat(43897 MB used, 545 GB avail, 588 GB total, peers [0,1,2,3,4] op hist [0,0,0,0,0,1,0,3,0,0,2])

Re: [ceph-users] Dying OSDs

Am 10.04.18 um 20:22 schrieb Paul Emmerich: > Hi, > > I encountered the same crash a few months ago, see > https://tracker.ceph.com/issues/23030 > > Can you post the output of > >    ceph osd pool ls detail -f json-pretty > > > Paul Yes, of course. # ceph osd pool ls detail -f json-pretty [

Re: [ceph-users] ceph-fuse CPU and Memory usage vs CephFS kclient

On Tue, Apr 10, 2018 at 12:36 PM, Wido den Hollander wrote: > > > On 04/10/2018 09:22 PM, Gregory Farnum wrote: >> On Tue, Apr 10, 2018 at 6:32 AM Wido den Hollander > > wrote: >> >> Hi, >> >> There have been numerous threads about this in the past, but I wanted to >>

Re: [ceph-users] ceph-fuse CPU and Memory usage vs CephFS kclient

On 04/10/2018 09:22 PM, Gregory Farnum wrote: > On Tue, Apr 10, 2018 at 6:32 AM Wido den Hollander > wrote: > > Hi, > > There have been numerous threads about this in the past, but I wanted to > bring this up again in a new situation. > > Running with Lum

Re: [ceph-users] cephfs snapshot format upgrade

On Tue, Apr 10, 2018 at 5:54 AM, John Spray wrote: > On Tue, Apr 10, 2018 at 1:44 PM, Yan, Zheng wrote: >> Hello >> >> To simplify snapshot handling in multiple active mds setup, we changed >> format of snaprealm in mimic dev. >> https://github.com/ceph/ceph/pull/16779. >> >> The new version mds

Re: [ceph-users] Ceph luminous - troubleshooting performance issues overall DSK 100%, busy 1%

I've just added another server ( same specs) with one osd and the behavior is the same - bad performance ..cur MB/s 0 Check network with iperf3 ..no issues So it is not a server issue since I am getting same behavior with 2 different servers ... but I checked network with iperf3 ..no issues What

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

Hi Paul, This is a small test cluster, and the rbd pool is replicated. I am hardly using any clients on the cluster. Furthermore I have been the only one creating the snapshots and I know for sure that I was not trying to delete them. If so I have been doing this on one of the ceph nodes. I

Re: [ceph-users] ceph-fuse CPU and Memory usage vs CephFS kclient

On Tue, Apr 10, 2018 at 6:32 AM Wido den Hollander wrote: > Hi, > > There have been numerous threads about this in the past, but I wanted to > bring this up again in a new situation. > > Running with Luminous v12.2.4 I'm seeing some odd Memory and CPU usage > when using the ceph-fuse client to mo

Re: [ceph-users] Move ceph admin node to new other server

I appreciate for your kind Paul. On Wed, Apr 11, 2018 at 1:47 AM, Paul Emmerich wrote: > http://docs.ceph.com/ceph-deploy/docs/gatherkeys.html > > 2018-04-10 20:39 GMT+02:00 Nghia Than : > >> Hi Paul, >> >> Thanks for your information. >> >> May i know if i destroy this node so how they can gath

Re: [ceph-users] Move ceph admin node to new other server

http://docs.ceph.com/ceph-deploy/docs/gatherkeys.html 2018-04-10 20:39 GMT+02:00 Nghia Than : > Hi Paul, > > Thanks for your information. > > May i know if i destroy this node so how they can gatherkeys as this node > is already terminated and no data available. As you said it will get from > clu

Re: [ceph-users] Move ceph admin node to new other server

Hi Paul, Thanks for your information. May i know if i destroy this node so how they can gatherkeys as this node is already terminated and no data available. As you said it will get from cluster so will they get them all or i have to manual backup them (surely i will do this task before terminate

Re: [ceph-users] Move ceph admin node to new other server

Hi, yes, that folder contains everything you need. You can also use ceph-deploy gatherkeys to get them from your cluster. Paul 2018-04-09 10:04 GMT+02:00 Nghia Than : > Hello, > > We have use 1 server for deploy (called ceph-admin-node) for 3 mon and 4 > OSD node. > > We have created a folder

Re: [ceph-users] Dying OSDs

Hi, I encountered the same crash a few months ago, see https://tracker.ceph.com/issues/23030 Can you post the output of ceph osd pool ls detail -f json-pretty Paul 2018-04-10 18:09 GMT+02:00 Jan Marquardt : > Am 10.04.18 um 15:29 schrieb Brady Deetz: > > What distribution and kernel are

Re: [ceph-users] Ceph scrub logs: _scan_snaps no head for $object?

Hi, you'll usually see this if there are "orphaned" snapshot objects. One common cause for this are pre-12.2.2 clients trying to delete RBD snapshots with a data pool (i.e., erasure coded pools) They send the snapshot requests to the wrong pool and you end up with lots of problems. Paul 2018-04

Re: [ceph-users] Dying OSDs

Am 10.04.18 um 15:29 schrieb Brady Deetz: > What distribution and kernel are you running? > > I recently found my cluster running the 3.10 centos kernel when I > thought it was running the elrepo kernel. After forcing it to boot > correctly, my flapping osd issue went away.  We are running on Ubu

[ceph-users] ceph-fuse CPU and Memory usage vs CephFS kclient

Hi, There have been numerous threads about this in the past, but I wanted to bring this up again in a new situation. Running with Luminous v12.2.4 I'm seeing some odd Memory and CPU usage when using the ceph-fuse client to mount a multi-MDS CephFS filesystem. health: HEALTH_OK services:

Re: [ceph-users] Dying OSDs

What distribution and kernel are you running? I recently found my cluster running the 3.10 centos kernel when I thought it was running the elrepo kernel. After forcing it to boot correctly, my flapping osd issue went away. On Tue, Apr 10, 2018, 2:18 AM Jan Marquardt wrote: > Hi, > > we are expe

Re: [ceph-users] cephfs snapshot format upgrade

On Tue, Apr 10, 2018 at 1:44 PM, Yan, Zheng wrote: > Hello > > To simplify snapshot handling in multiple active mds setup, we changed > format of snaprealm in mimic dev. > https://github.com/ceph/ceph/pull/16779. > > The new version mds can handle old format snaprealm in single active > setup. It

Re: [ceph-users] Ceph luminous - troubleshooting performance issues overall DSK 100%, busy 1%

Hi, Thanks for providing guidance VD0 is the SSD drive Many people suggested to not enable WB for SSD so that cache can be used for HDD where is needed more Setup is 3 identical DELL R620 server OSD01, OSD02, OSD04 10 GB separate networks, 600 GB Entreprise HDD , 320 GB Entreprise SSD Blustore,

[ceph-users] cephfs snapshot format upgrade

Hello To simplify snapshot handling in multiple active mds setup, we changed format of snaprealm in mimic dev. https://github.com/ceph/ceph/pull/16779. The new version mds can handle old format snaprealm in single active setup. It also can convert old format snaprealm to the new format when snapr

Re: [ceph-users] Moving bluestore WAL and DB after bluestore creation

Hi, I just wanted to make sure that our latest findings reach the OP of this thread. We posted it in a different thread [1] and hope this helps some of you. It is possible to migrate a journal from one partition to another almost without downtime of the OSD. But it's *not* sufficient to dd

Re: [ceph-users] Ceph luminous - troubleshooting performance issues overall DSK 100%, busy 1%

Is this just from one server or from all servers? Just wondering why VD 0 is using WriteThrough compared to the others. If that's the setup for the OSD's you already have a cache setup problem. On 10.04.2018 13:44, Mohamad Gebai wrote: > megacli -LDGetProp -cache -Lall -a0 > > Adapter 0-VD 0(targ

Re: [ceph-users] Ceph luminous - troubleshooting performance issues overall DSK 100%, busy 1%

Just to be clear about the issue: You have a 3 servers setup, performance is good. You add a server (with 1 OSD?) and performance goes down, is that right? Can you give us more details? What's your complete setup? How many OSDs per node, bluestore/filestore, WAL/DB setup, etc. You're talking abou

Re: [ceph-users] Fwd: Separate --block.wal --block.db bluestore not working as expected.

No, you shouldn't invoke it that way, you should just not specify a WAL device at all if you want it to be stored with the DB - if not otherwise specified the WAL is automatically stored with the other metadata on the DB device. You should do something like: ceph-volume lvm prepare --bluestore

[ceph-users] Dying OSDs

Hi, we are experiencing massive problems with our Ceph setup. After starting a "repair pg" because of scrub errors OSDs started to crash, which we could not stop so far. We are running Ceph 12.2.4. Crashed OSDs are both bluestore and filestore. Our cluster currently looks like this: # ceph -s