[ceph-users] multi-site between luminous and mimic broke etag
Hi Ceph Users, In our lab on VirtualBox we have installed two Centos7 VMs. One with ceph v12.2.11 and the second one with ceph v13.2.5. We connect them using multi-site (it does not matter witch one is hosting the master zone). On master zone we: created a user, made a bucket, uploaded a file, listed the bucket. Then on the slave zone we: listed the bucket, uploaded a file. After that, listing the bucket on mimic works fine, but on luminous s3cmd reports: "ERROR: Error parsing xml: not well-formed (invalid token): line 1, column 574" which comes to this part of xml: 71c416d096979c7f3657d2927e29dfd3iU Both cluster in their ceph.conf has those added lines: osd pool default size = 1 osd pool default min size = 1 osd crush chooseleaf type = 0 mon allow pool delete = true rgw zone = test-dc1 # second has test-dc2 Has anyone had a similar issue before? Thanks. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] problem w libvirt version 4.5 and 12.2.7
Konstantin, Thanks for reply. I've managed to unravel it partially. Somehow (did not look into srpm) starting from this version libvirt started to calculate real allocation if fastdiff feature is present on image. Doing "rbd object-map rebuild" on every image helped (do not know why it was needed - it is a new cluster with ceph version 12.2.7). Now the only problem is 25T image on which "virsh vol-info" takes 13s (rbd du takes 1s) compared to few minutes before, so the questions remains: - why it happened, - how to monitor/foresee this, - how to improve virsh vol-info if rbd du take less time to execute? On 03.01.2019 at 13:51, Konstantin Shalygin wrote: After update to CentOS 7.6, libvirt was updated from 3.9 to 4.5. Executing: "virsh vol-list ceph --details" makes libvirtd using 300% CPU for 2 minutes to show volumes on rbd. Quick pick at tcpdump shows accessing rbd_data.* which previous version of libvirtd did not need. Ceph version is 12.2.7. Any help will be appreciated There is nothing special in libvirt 4.5, I was upgraded hypervisors to this version, still works flawless. k ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] problem w libvirt version 4.5 and 12.2.7
Hi Cephers, After update to CentOS 7.6, libvirt was updated from 3.9 to 4.5. Executing: "virsh vol-list ceph --details" makes libvirtd using 300% CPU for 2 minutes to show volumes on rbd. Quick pick at tcpdump shows accessing rbd_data.* which previous version of libvirtd did not need. Ceph version is 12.2.7. Any help will be appreciated Thanks Tom ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] compacting omap doubles its size
Hi, I have a ceph 12.2.8 cluster on filestore with rather large omap dirs (avg size is about 150G). Recently slow requests became a problem, so after some digging I decided to convert omap from leveldb to rocksdb. Conversion went fine and slow requests rate went down to acceptable level. Unfortunately conversion did not shrink most of omap dirs, so I tried online compaction: Before compaction: 50G /var/lib/ceph/osd/ceph-0/current/omap/ After compaction: 100G /var/lib/ceph/osd/ceph-0/current/omap/ Purge and recreate: 1.5G /var/lib/ceph/osd/ceph-0/current/omap/ Before compaction: 135G /var/lib/ceph/osd/ceph-5/current/omap/ After compaction: 260G /var/lib/ceph/osd/ceph-5/current/omap/ Purge and recreate: 2.5G /var/lib/ceph/osd/ceph-5/current/omap/ For me compaction which makes omap bigger is quite weird and frustrating. Please help. P.S. My cluster suffered from ongoing index reshards (it is disabled now) and on many buckets with 4m+ objects I have a lot of old indexes: 634 bucket1 651 bucket2 ... 1231 bucket17 1363 bucket18 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Large omap objects - how to fix ?
Hi hijackers,
Please read:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-October/030317.html
TL;DR: Ceph should reshard big indexes, but after that it leaves them to
be removed manually. Starting from some version, deep-scrub reports
indexes above some threshold as HALTH_WARN. You should find it in osd
logs. If You do not have logs, just listomapkeys on every object in
default.rgw.buckets.index and find the biggest ones... it should be safe
to remove those (radosgw-admin bi purge) but I can not guarantee it.
On 26.10.2018 at 17:18, Florian Engelmann wrote:
Hi,
hijacking the hijacker! Sorry!
radosgw-admin bucket reshard --bucket somebucket --num-shards 8
*** NOTICE: operation will not remove old bucket index objects ***
*** these will need to be removed manually ***
tenant:
bucket name: somebucket
old bucket instance id: cb1594b3-a782-49d0-a19f-68cd48870a63.1923153.1
new bucket instance id: cb1594b3-a782-49d0-a19f-68cd48870a63.3119759.1
total entries: 1000 2000 3000 4000 5000 6000 7000 8000 9000 1
11000 12000 13000 14000 15000 16000 17000 18000 19000 2 21000
22000 23000 24000 25000 26000 27000 28000 29000 3 31000 32000
33000 34000 35000 36000 37000 38000 39000 4 41000 42000 43000
44000 45000 46000 47000 48000 49000 5 51000 52000 53000 54000
55000 56000 57000 58000 59000 6 61000 62000 63000 64000 65000
66000 67000 68000 69000 7 71000 72000 73000 74000 75000 76000
77000 78000 79000 8 81000 82000 83000 84000 85000 86000 87000
88000 89000 9 91000 92000 93000 94000 95000 96000 97000 98000
99000 10 101000 102000 103000 104000 105000 106000 107000 108000
109000 11 111000 112000 113000 114000 115000 116000 117000 118000
119000 12 121000 122000 123000 124000 125000 126000 127000 128000
129000 13 131000 132000 133000 134000 135000 136000 137000 138000
139000 14 141000 142000 143000 144000 145000 146000 147000 148000
149000 15 151000 152000 153000 154000 155000 156000 157000 158000
159000 16 161000 162000 163000 164000 165000 166000 167000 168000
169000 17 171000 172000 173000 174000 175000 176000 177000 178000
179000 18 181000 182000 183000 184000 185000 186000 187000 188000
189000 19 191000 192000 193000 194000 195000 196000 197000 198000
199000 20 201000 202000 203000 204000 205000 206000 207000 207660
What to do now?
ceph -s is still:
health: HEALTH_WARN
1 large omap objects
But I have no idea how to:
*** NOTICE: operation will not remove old bucket index objects ***
*** these will need to be removed manually ***
All the best,
Flo
Am 10/26/18 um 3:56 PM schrieb Alexandru Cucu:
Hi,
Sorry to hijack this thread. I have a similar issue also with 12.2.8
recently upgraded from Jewel.
I my case all buckets are within limits:
# radosgw-admin bucket limit check | jq
'.[].buckets[].fill_status' | uniq
"OK"
# radosgw-admin bucket limit check | jq
'.[].buckets[].objects_per_shard' | sort -n | uniq
0
1
30
109
516
5174
50081
50088
50285
50323
50336
51826
rgw_max_objs_per_shard is set to the default of 100k
---
Alex Cucu
On Fri, Oct 26, 2018 at 4:09 PM Ben Morrice wrote:
Hello all,
After a recent Luminous upgrade (now running 12.2.8 with all OSDs
migrated to bluestore, upgraded from 11.2.0 and running filestore) I am
currently experiencing the warning 'large omap objects'.
I know this is related to large buckets in radosgw, and luminous
supports 'dynamic sharding' - however I feel that something is missing
from our configuration and i'm a bit confused on what the right
approach
is to fix it.
First a bit of background info:
We previously had a multi site radosgw installation, however
recently we
decommissioned the second site. With the radosgw multi-site
configuration we had 'bucket_index_max_shards = 0'. Since
decommissioning the second site, I have removed the secondary zonegroup
and changed 'bucket_index_max_shards' to be 16 for the single
primary zone.
All our buckets do not have a 'num_shards' field when running
'radosgw-admin bucket stats --bucket '
Is this normal ?
Also - I'm finding it difficult to find out exactly what to do with the
buckets that are affected with 'large omap' (see commands below).
My interpretation of 'search the cluster log' is also listed below.
What do I need to do to with the below buckets get back to an overall
ceph HEALTH OK state ? :)
# ceph health detail
HEALTH_WARN 2 large omap objects
2 large objects found in pool '.bbp-gva-master.rgw.buckets.index'
Search the cluster log for 'Large omap object found' for more details.
# ceph osd pool get .bbp-gva-master.rgw.buckets.index pg_num
pg_num: 64
# for i in `ceph pg ls-by-pool .bbp-gva-master.rgw.buckets.index | tail
-n +2 | awk '{print $1}'`; do echo -n "$i: "; ceph pg $i query |grep
num_large_omap_objects | head -1 | awk '{print $2}'; done | grep ": 1"
Re: [ceph-users] What is rgw.none
Hi,
I have heard rumors it is a know bug an I should not bother (but I have
not found bugtrack for it).
Best regards
Tomasz Płaza
On 19.10.2018 at 08:47, Arvydas Opulskis wrote:
Hi,
we have same question when trying to understand output of bucket
stats. Maybe you had found explanation somewhere else?
Thanks,
Arvydas
On Mon, Aug 6, 2018 at 10:28 AM Tomasz Płaza <mailto:tomasz.pl...@grupawp.pl>> wrote:
Hi all,
I have a bucket with a vary big num_objects in rgw.none:
{
"bucket": "dyna",
"zonegroup": "84d584b4-3e95-49f8-8285-4a704f8252e3",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "default.rgw.buckets.data",
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_pool": "default.rgw.buckets.index"
},
"id": "99a07ed8-2112-429b-9f94-81383220a95b.3676457.19",
"marker": "99a07ed8-2112-429b-9f94-81383220a95b.3676457.19",
"index_type": "Normal",
"owner": "dyna",
"ver":
"0#106001,1#106724,2#105646,3#105913,4#106485,5#105165,6#105712,7#107249",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0",
"mtime": "2017-09-25 11:24:25.568108",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#",
"usage": {
"rgw.none": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 18446744073709551615
},
"rgw.main": {
"size": 28380702832,
"size_actual": 28810874880,
"size_utilized": 28380702832,
"size_kb": 27715531,
"size_kb_actual": 28135620,
"size_kb_utilized": 27715531,
"num_objects": 210206
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 0
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}
What is rgw.none and is this big number OK?
--
Best regards
Tomasz Płaza
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Resolving Large omap objects in RGW index pool
Hi,
I have a similar issue, and created a simple bash file to delete old
indexes (it is PoC and have not been tested on production):
for bucket in `radosgw-admin metadata list bucket | jq -r '.[]' | sort`
do
actual_id=`radosgw-admin bucket stats --bucket=${bucket} | jq -r '.id'`
for instance in `radosgw-admin metadata list bucket.instance | jq -r
'.[]' | grep ${bucket}: | cut -d ':' -f 2`
do
if [ "$actual_id" != "$instance" ]
then
radosgw-admin bi purge --bucket=${bucket} --bucket-id=${instance}
radosgw-admin metadata rm bucket.instance:${bucket}:${instance}
fi
done
done
I find it more readable than mentioned one liner. Any sugestions on this
topic are greatly appreciated.
Tom
Hi,
Having spent some time on the below issue, here are the steps I took
to resolve the "Large omap objects" warning. Hopefully this will help
others who find themselves in this situation.
I got the object ID and OSD ID implicated from the ceph cluster
logfile on the mon. I then proceeded to the implicated host
containing the OSD, and extracted the implicated PG by running the
following, and looking at which PG had started and completed a
deep-scrub around the warning being logged:
grep -C 200 Large /var/log/ceph/ceph-osd.*.log | egrep '(Large
omap|deep-scrub)'
If the bucket had not been sharded sufficiently (IE the cluster log
showed a "Key Count" or "Size" over the thresholds), I ran through the
manual sharding procedure (shown here:
https://tracker.ceph.com/issues/24457#note-5)
Once this was successfully sharded, or if the bucket was previously
sufficiently sharded by Ceph prior to disabling the functionality I
was able to use the following command (seemingly undocumented for
Luminous http://docs.ceph.com/docs/mimic/man/8/radosgw-admin/#commands):
radosgw-admin bi purge --bucket ${bucketname} --bucket-id ${old_bucket_id}
I then issued a ceph pg deep-scrub against the PG that had contained
the Large omap object.
Once I had completed this procedure, my Large omap object warnings
went away and the cluster returned to HEALTH_OK.
However our radosgw bucket indexes pool now seems to be using
substantially more space than previously. Having looked initially at
this bug, and in particular the first comment:
http://tracker.ceph.com/issues/34307#note-1
I was able to extract a number of bucket indexes that had apparently
been resharded, and removed the legacy index using the radosgw-admin
bi purge --bucket ${bucket} ${marker}. I am still able to perform a
radosgw-admin metadata get bucket.instance:${bucket}:${marker}
successfully, however now when I run rados -p .rgw.buckets.index ls |
grep ${marker} nothing is returned. Even after this, we were still
seeing extremely high disk usage of our OSDs containing the bucket
indexes (we have a dedicated pool for this). I then modified the one
liner referenced in the previous link as follows:
grep -E '"bucket"|"id"|"marker"' bucket-stats.out | awk -F ":"
'{print $2}' | tr -d '",' | while read -r bucket; do read -r id; read
-r marker; [ "$id" == "$marker" ] && true || NEWID=`radosgw-admin --id
rgw.ceph-rgw-1 metadata get bucket.instance:${bucket}:${marker} |
python -c 'import sys, json; print
json.load(sys.stdin)["data"]["bucket_info"]["new_bucket_instance_id"]'`;
while [ ${NEWID} ]; do if [ "${NEWID}" != "${marker}" ] && [ ${NEWID}
!= ${bucket} ] ; then echo "$bucket $NEWID"; fi; NEWID=`radosgw-admin
--id rgw.ceph-rgw-1 metadata get bucket.instance:${bucket}:${NEWID} |
python -c 'import sys, json; print
json.load(sys.stdin)["data"]["bucket_info"]["new_bucket_instance_id"]'`;
done; done > buckets_with_multiple_reindexes2.txt
This loops through the buckets that have a different marker/bucket_id,
and looks to see if a new_bucket_instance_id is there, and if so will
loop through until there is no longer a "new_bucket_instance_id".
After letting this complete, this suggests that I have over 5000
indexes for 74 buckets, some of these buckets have > 100 indexes
apparently.
:~# awk '{print $1}' buckets_with_multiple_reindexes2.txt | uniq | wc -l
74
~# wc -l buckets_with_multiple_reindexes2.txt
5813 buckets_with_multiple_reindexes2.txt
This is running a single realm, multiple zone configuration, and no
multi site sync, but the closest I can find to this issue is this bug
https://tracker.ceph.com/issues/24603
Should I be OK to loop through these indexes and remove any with a
reshard_status of 2, a new_bucket_instance_id that does not match the
bucket_instance_id returned by the command:
radosgw-admin bucket stats --bucket ${bucket}
I'd ideally like to get to a point where I can turn dynamic sharding
back on safely for this cluster.
Thanks for any assistance, let me know if there's any more information
I should provide
Chris
On Thu, 4 Oct 2018 at 18:22 Chris Sarginson > wrote:
Hi,
Thanks for the response - I am still unsure as to what will
Re: [ceph-users] cephfs poor performance
On 08.10.2018 at 10:29, Yan, Zheng wrote: On Mon, Oct 8, 2018 at 3:38 PM Tomasz Płaza wrote: On 08.10.2018 at 09:21, Yan, Zheng wrote: On Mon, Oct 8, 2018 at 1:54 PM Tomasz Płaza wrote: Hi, Can someone please help me, how do I improve performance on our CephFS cluster? System in use is: Centos 7.5 with ceph 12.2.7. The hardware in use are as follows: 3xMON/MGR: 1xIntel(R) Xeon(R) Bronze 3106 16GB RAM 2xSSD for system 1Gbe NIC 2xMDS: 2xIntel(R) Xeon(R) Bronze 3106 64GB RAM 2xSSD for system 10Gbe NIC 6xOSD: 1xIntel(R) Xeon(R) Silver 4108 2xSSD for system 6xHGST HUS726060ALE610 SATA HDD's 1xINTEL SSDSC2BB150G7 for osd db`s (10G partitions) rest for OSD to place cephfs_metadata 10Gbe NIC pools (default crush rule aware of device class): rbd with 1024 pg crush rule replicated_hdd cephfs_data with 256 pg crush rule replicated_hdd cephfs_metadata with 32 pg crush rule replicated_ssd test done by fio: fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=1G --readwrite=randrw --rwmixread=75 kernel version? maybe cephfs driver in your kernel does not support AIO (--iodepth is 1 effectively) Yan, Zheng Kernel is 3.10.0-862.9.1.el7.x86_64 (I can update it to 3.10.0-862.14.4.el7) but do not know how to check aio support in kernel drive if it is relevant because I mounted it with ceph-fuse -n client.cephfs -k /etc/ceph/ceph.client.cephfs.keyring -m 192.168.10.1:6789 /mnt/cephfs Tom Płaza please try kernel mount. kernel mount did the trick, now performance is 3298/1101. Thank You. Tom Płaza shows iops write/read performance as folows: rbd 3663/1223 cephfs (fuse) 205/68 (wich is a little lower than raw performance of one hdd used in cluster) Everything is connected to one Cisco 10Gbe switch. Please help. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com Spółki Grupy Wirtualna Polska: Wirtualna Polska Holding Spółka Akcyjna z siedzibą w Warszawie, ul. Jutrzenki 137A, 02-231 Warszawa, wpisana do Krajowego Rejestru Sądowego - Rejestru Przedsiębiorców prowadzonego przez Sąd Rejonowy dla m.st. Warszawy w Warszawie pod nr KRS: 407130, kapitał zakładowy: 1 445 199,00 zł (w całości wpłacony), Numer Identyfikacji Podatkowej (NIP): 521-31-11-513 Wirtualna Polska Media Spółka Akcyjna z siedzibą w Warszawie, ul. Jutrzenki 137A, 02-231 Warszawa, wpisana do Krajowego Rejestru Sądowego - Rejestru Przedsiębiorców prowadzonego przez Sąd Rejonowy dla m.st. Warszawy w Warszawie pod nr KRS: 580004, kapitał zakładowy: 320 005 950,00 zł (w całości wpłacony), Numer Identyfikacji Podatkowej (NIP): 527-26-45-593 Administratorem udostępnionych danych osobowych jest Wirtualna Polska Media S.A. z siedzibą w Warszawie (dalej „WPM”). WPM przetwarza Twoje dane osobowe, które zostały podane przez Ciebie dobrowolnie w trakcie dotychczasowej współpracy, w związku z zawarciem umowy lub zostały zebrane ze źródeł powszechnie dostępnych, w szczególności: imię i nazwisko, adres email, numer telefonu. Przetwarzamy te dane w celach opisanych w polityce prywatności<https://onas.wp.pl/poufnosc.html>, między innymi w celu realizacji współpracy, realizacji obowiązków przewidzianych prawem, w celach marketingowych WP. Podstawą prawną przetwarzania Twoich danych osobowych w celach marketingowych jest prawnie uzasadniony interes jakim jest m.in. przesyłanie informacji marketingowych o usługach WP, w tym zaproszeń na konferencje branżowe, informacje o publikacjach. Twoje dane możemy przekazywać podmiotom przetwarzającym je na nasze zlecenie oraz podmiotom uprawnionym do uzyskania danych na podstawie obowiązującego prawa. Masz prawo m.in. do żądania dostępu do danych, sprostowania, usunięcia lub ograniczenia ich przetwarzania, jak również prawo do zgłoszenia sprzeciwu w przewidzianych w prawie sytuacjach. Prawa te oraz sposób ich realizacji opisaliśmy w polityce prywatności<https://onas.wp.pl/poufnosc.html>. Tam też znajdziesz informacje jak zakomunikować nam Twoją wolę skorzystania z tych praw. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] cephfs poor performance
On 08.10.2018 at 09:21, Yan, Zheng wrote: On Mon, Oct 8, 2018 at 1:54 PM Tomasz Płaza wrote: Hi, Can someone please help me, how do I improve performance on our CephFS cluster? System in use is: Centos 7.5 with ceph 12.2.7. The hardware in use are as follows: 3xMON/MGR: 1xIntel(R) Xeon(R) Bronze 3106 16GB RAM 2xSSD for system 1Gbe NIC 2xMDS: 2xIntel(R) Xeon(R) Bronze 3106 64GB RAM 2xSSD for system 10Gbe NIC 6xOSD: 1xIntel(R) Xeon(R) Silver 4108 2xSSD for system 6xHGST HUS726060ALE610 SATA HDD's 1xINTEL SSDSC2BB150G7 for osd db`s (10G partitions) rest for OSD to place cephfs_metadata 10Gbe NIC pools (default crush rule aware of device class): rbd with 1024 pg crush rule replicated_hdd cephfs_data with 256 pg crush rule replicated_hdd cephfs_metadata with 32 pg crush rule replicated_ssd test done by fio: fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=1G --readwrite=randrw --rwmixread=75 kernel version? maybe cephfs driver in your kernel does not support AIO (--iodepth is 1 effectively) Yan, Zheng Kernel is 3.10.0-862.9.1.el7.x86_64 (I can update it to 3.10.0-862.14.4.el7) but do not know how to check aio support in kernel drive if it is relevant because I mounted it with ceph-fuse -n client.cephfs -k /etc/ceph/ceph.client.cephfs.keyring -m 192.168.10.1:6789 /mnt/cephfs Tom Płaza shows iops write/read performance as folows: rbd 3663/1223 cephfs (fuse) 205/68 (wich is a little lower than raw performance of one hdd used in cluster) Everything is connected to one Cisco 10Gbe switch. Please help. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] cephfs poor performance
Hi, Can someone please help me, how do I improve performance on our CephFS cluster? System in use is: Centos 7.5 with ceph 12.2.7. The hardware in use are as follows: 3xMON/MGR: 1xIntel(R) Xeon(R) Bronze 3106 16GB RAM 2xSSD for system 1Gbe NIC 2xMDS: 2xIntel(R) Xeon(R) Bronze 3106 64GB RAM 2xSSD for system 10Gbe NIC 6xOSD: 1xIntel(R) Xeon(R) Silver 4108 2xSSD for system 6xHGST HUS726060ALE610 SATA HDD's 1xINTEL SSDSC2BB150G7 for osd db`s (10G partitions) rest for OSD to place cephfs_metadata 10Gbe NIC pools (default crush rule aware of device class): rbd with 1024 pg crush rule replicated_hdd cephfs_data with 256 pg crush rule replicated_hdd cephfs_metadata with 32 pg crush rule replicated_ssd test done by fio: fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=1G --readwrite=randrw --rwmixread=75 shows iops write/read performance as folows: rbd 3663/1223 cephfs (fuse) 205/68 (wich is a little lower than raw performance of one hdd used in cluster) Everything is connected to one Cisco 10Gbe switch. Please help. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] how dynamic bucket sharding works
Hi Cephers,
Could someone explain me how dynamic bucket index sharding works?
I have created a test bucket with 4 million objects on ceph 12.2.8 and
it showed 80 shards (ver, master_ver, max_marker fomr 0 to 79 in bucket
stats) and leave it for a night. Next day in the morning I found this in
reshard list:
"time": "2018-09-21 06:15:12.094792Z",
"tenant": "",
"bucket_name": "test",
"bucket_id": "_id_.7827818.1",
"new_instance_id": "test:_id_.25481437.10",
"old_num_shards": 8,
"new_num_shards": 16
During this reshard bucket stats showed 16 shards (counting ver,
master_ver, max_marker from bucket stats on marker _id_.7827818.1).
After deleting and re adding 2 objects reshard kicked in once more, this
time from 16 to 80 shards.
Actual bucket stats are:
{
"bucket": "test",
"zonegroup": "84d584b4-3e95-49f8-8285-4a704f8252e3",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "_id_.25481803.6",
"marker": "_id_.7827818.1",
"index_type": "Normal",
"owner": "test",
"ver":
"0#789,1#785,2#787,3#782,4#790,5#798,6#784,7#784,8#782,9#791,10#788,11#785,12#786,13#792,14#783,15#783,16#786,17#776,18#787,19#783,20#784,21#785,22#786,23#782,24#787,25#794,26#786,27#789,28#794,29#781,30#785,31#779,32#780,33#776,34#790,35#775,36#780,37#781,38#779,39#782,40#778,41#776,42#774,43#781,44#779,45#785,46#778,47#779,48#783,49#778,50#784,51#779,52#780,53#782,54#781,55#779,56#789,57#783,58#774,59#780,60#779,61#782,62#780,63#775,64#783,65#783,66#781,67#785,68#777,69#785,70#781,71#782,72#778,73#778,74#778,75#777,76#783,77#775,78#790,79#792",
"master_ver":
"0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0,64#0,65#0,66#0,67#0,68#0,69#0,70#0,71#0,72#0,73#0,74#0,75#0,76#0,77#0,78#0,79#0",
"mtime": "2018-09-21 08:40:33.652235",
"max_marker":
"0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#,11#,12#,13#,14#,15#,16#,17#,18#,19#,20#,21#,22#,23#,24#,25#,26#,27#,28#,29#,30#,31#,32#,33#,34#,35#,36#,37#,38#,39#,40#,41#,42#,43#,44#,45#,46#,47#,48#,49#,50#,51#,52#,53#,54#,55#,56#,57#,58#,59#,60#,61#,62#,63#,64#,65#,66#,67#,68#,69#,70#,71#,72#,73#,74#,75#,76#,77#,78#,79#",
"usage": {
"rgw.none": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 2
},
"rgw.main": {
"size": 419286170636,
"size_actual": 421335109632,
"size_utilized": 0,
"size_kb": 409459152,
"size_kb_actual": 411460068,
"size_kb_utilized": 0,
"num_objects": 401
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}
My question is: Why on earth did ceph reshard this bucket to 8 shards
and after than to 16 shards, and than to 80 after re adding 2 objects?
Additional question: Why do we need rgw_reshard_bucket_lock_duration if
https://ceph.com/community/new-luminous-rgw-dynamic-bucket-sharding/
states: "...Furthermore, there is no need to stop IO operations that go
to the bucket (although some concurrent operations may experience
additional latency when resharding is in progress)..." From My
experience it blocks write completely, only read works.
--
Thanks
Tom
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] What is rgw.none
Hi all,
I have a bucket with a vary big num_objects in rgw.none:
{
"bucket": "dyna",
"zonegroup": "84d584b4-3e95-49f8-8285-4a704f8252e3",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "default.rgw.buckets.data",
"data_extra_pool": "default.rgw.buckets.non-ec",
"index_pool": "default.rgw.buckets.index"
},
"id": "99a07ed8-2112-429b-9f94-81383220a95b.3676457.19",
"marker": "99a07ed8-2112-429b-9f94-81383220a95b.3676457.19",
"index_type": "Normal",
"owner": "dyna",
"ver":
"0#106001,1#106724,2#105646,3#105913,4#106485,5#105165,6#105712,7#107249",
"master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0",
"mtime": "2017-09-25 11:24:25.568108",
"max_marker": "0#,1#,2#,3#,4#,5#,6#,7#",
"usage": {
"rgw.none": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 18446744073709551615
},
"rgw.main": {
"size": 28380702832,
"size_actual": 28810874880,
"size_utilized": 28380702832,
"size_kb": 27715531,
"size_kb_actual": 28135620,
"size_kb_utilized": 27715531,
"num_objects": 210206
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 0
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}
What is rgw.none and is this big number OK?
--
Best regards
Tomasz Płaza
Spółki Grupy Wirtualna Polska:
Wirtualna Polska Holding Spółka Akcyjna z siedzibą w Warszawie, ul. Jutrzenki
137A, 02-231 Warszawa, wpisana do Krajowego Rejestru Sądowego - Rejestru
Przedsiębiorców prowadzonego przez Sąd Rejonowy dla m.st. Warszawy w Warszawie
pod nr KRS: 407130, kapitał zakładowy: 1 445 199,00 zł (w całości
wpłacony), Numer Identyfikacji Podatkowej (NIP): 521-31-11-513
Wirtualna Polska Media Spółka Akcyjna z siedzibą w Warszawie, ul. Jutrzenki
137A, 02-231 Warszawa, wpisana do Krajowego Rejestru Sądowego - Rejestru
Przedsiębiorców prowadzonego przez Sąd Rejonowy dla m.st. Warszawy w Warszawie
pod nr KRS: 580004, kapitał zakładowy: 317 957 900,00 zł (w całości
wpłacony), Numer Identyfikacji Podatkowej (NIP): 527-26-45-593
Administratorem udostępnionych danych osobowych jest Wirtualna Polska Media S.A. z siedzibą
w Warszawie (dalej „WPM”). WPM przetwarza Twoje dane osobowe, które zostały podane przez
Ciebie dobrowolnie w trakcie dotychczasowej współpracy, w związku z zawarciem umowy lub
zostały zebrane ze źródeł powszechnie dostępnych, w szczególności: imię i nazwisko, adres
email, numer telefonu. Przetwarzamy te dane w celach opisanych w polityce
prywatności<https://onas.wp.pl/poufnosc.html>, między innymi w celu realizacji
współpracy, realizacji obowiązków przewidzianych prawem, w celach marketingowych WP.
Podstawą prawną przetwarzania Twoich danych osobowych w celach marketingowych jest prawnie
uzasadniony interes jakim jest m.in. przesyłanie informacji marketingowych o usługach WP, w
tym zaproszeń na konferencje branżowe, informacje o publikacjach. Twoje dane możemy
przekazywać podmiotom przetwarzającym je na nasze zlecenie oraz podmiotom uprawnionym do
uzyskania danych na podstawie obowiązującego prawa. Masz prawo m.in. do żądania dostępu do
danych, sprostowania, usunięcia lub ograniczenia ich przetwarzania, jak również prawo do
zgłoszenia sprzeciwu w przewidzianych w prawie sytuacjach. Prawa te oraz sposób ich
realizacji opisaliśmy w polityce prywatności<https://onas.wp.pl/poufnosc.html>. Tam
też znajdziesz informacje jak zakomunikować nam Twoją wolę skorzystania z tych praw.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph as storage for docker registry
Hi All, We are running ceph 12.2.3 as storage for docker registry with swift API. This is the only workload with swift API on our ceph cluster. We need to run radosgw-admin bucket check --fix --check-objects --bucket docker-registry from time to time to fix issue described https://github.com/docker/distribution/issues/2430. Last time I had to push some tags manually with s3client because swift client showed err.detail="swift: Timeout expired while waiting for segments of /docker/" In ceph conf We have: [client] rgw resolve cname = true rgw thread pool size = 256 rgw num rados handles = 8 rgw override bucket index max shards = 8 log file = /dev/null rgw swift url = http://someurl.com Is there any recommended configuration for ceph and docker-registry on swift API? Thanks, Tom Spółki Grupy Wirtualna Polska: Wirtualna Polska Holding Spółka Akcyjna z siedzibą w Warszawie, ul. Jutrzenki 137A, 02-231 Warszawa, wpisana do Krajowego Rejestru Sądowego - Rejestru Przedsiębiorców prowadzonego przez Sąd Rejonowy dla m.st. Warszawy w Warszawie pod nr KRS: 407130, kapitał zakładowy: 1 440 487,60 zł (w całości wpłacony), Numer Identyfikacji Podatkowej (NIP): 521-31-11-513 Wirtualna Polska Media Spółka Akcyjna z siedzibą w Warszawie, ul. Jutrzenki 137A, 02-231 Warszawa, wpisana do Krajowego Rejestru Sądowego - Rejestru Przedsiębiorców prowadzonego przez Sąd Rejonowy dla m.st. Warszawy w Warszawie pod nr KRS: 580004, kapitał zakładowy: 317 957 900,00 zł (w całości wpłacony), Numer Identyfikacji Podatkowej (NIP): 527-26-45-593 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
