[ceph-users] Re: Dedicated radosgw gateways
Another not sure …. For syncing, the RGWs - afaik - use the URLs set as endpoints in the zone configurations. So, if RadosGW1-4 on the West cluster are configured as endpoints then RadosGW4 on the East cluster will pull data from all of them if there is anything to be pulled. And the same in the other direction. If you want to use only the two RadosGW4 for syncing then only they should be endpoints for their cluster. For customer S3 traffic the other RGWs just have to “be there” and you can talk to them, if the network allows. No need to set them as endpoints. (Disclaimer: I haven’t looked at the code. This is just my experience :) ) What I also usually do is - using your example - put a HAProxy/keepalived (Ingress) in front of RadosGW1-3. Then I only have one customer URL per cluster and HAProxy nicely load balances the requests over the 3 RGWs. Maybe this helps? Ciao, Uli PS: And only the "ceph config set client.rgw.clb.node3.nxumka rgw_run_sync_thread false” has an effect on which RGW tries to sync. The other two are unrelated to syncing, afaict, but might just be useful to remove load from the “customer” RGWs. > On 19. May 2023, at 15:24, Tarrago, Eli (RIS-BCT) > wrote: > > Thanks Ulrich, using `ceph config set` instead of specifying in the config > file did end up working for me. > > What I expected to happen with these series of changes would be: > RadosGW4 would only sync with RadosGW 4 of the opposing site. Looking at the > logs, I see that RadosGW4 talks to WestGW1,2,3,4 instead of just 4 and 4 > exclusively. I see this same pattern of activity from both sides of the > zonegroup. > > Is this expected? > > My expectation was RadosGW4 would exclusivly talk to RadosGW4 to remove sync > traffic from client facing radosgw. > > I have included a diagram below. I am unsure if pictures come thru the > mailing lists > > > > > From: Ulrich Klein <mailto:ulrich.kl...@ulrichklein.de>> > Date: Friday, May 19, 2023 at 5:57 AM > To: Tarrago, Eli (RIS-BCT) <mailto:eli.tarr...@lexisnexisrisk.com>> > Cc: ceph-users mailto:ceph-users@ceph.io>> > Subject: Re: [ceph-users] Dedicated radosgw gateways > > [You don't often get email from ulrich.kl...@ulrichklein.de > <mailto:ulrich.kl...@ulrichklein.de>. Learn why this is important at > https://aka.ms/LearnAboutSenderIdentification ] > > *** External email: use caution *** > > > > Not sure this will help: > > I did some experiments on small test clusters with separate RGWs for > "customers" and for syncing. Both are Cephadm/containerized clusters. > Cluster A primary with zone A with 4 nodes, one RGW on each, node 1 and 2 > behind one HAProxy/ingress URL for syncing via http://clusterA > <http://clustera/>, node3 and 4 behind one HAPRoxy/ingress for customers via > https://customersA <https://customersa/> > Cluster B secondary with zone B with 4 nodes, one RGW on each, node 1 and 2 > behind one HAProxy/ingress URL for syncing via http://clusterB > <http://clusterb/>, node 3 and 4 behind one HAPRoxy/ingress for customers via > https://customersB <https://customersb/> > > Endpoint for cluster A is http://clusterA <http://clustera/> > Endpoint for cluster B is http://clusterB <http://clusterb/> > > I set in cluster B: > ceph config set client.rgw.clb.node3.nxumka rgw_run_sync_thread false > ceph config set client.rgw.clb.node4.xeeqqq rgw_run_sync_thread false > > ceph config set client.rgw.clb.node3.nxumka rgw_enable_lc_threads false > ceph config set client.rgw.clb.node4.xeeqqq rgw_enable_lc_threads false > > ceph config set client.rgw.clb.node3.nxumka rgw_enable_gc_threads false > ceph config set client.rgw.clb.node4.xeeqqq rgw_enable_gc_threads false > > ceph orch restart rgw.customer # That's the "service" for customer RGW on > nodes 3 and 4. > > Now I uploaded a 32G file to cluster A (via https://customersA) > <https://customersa)/> > Once it's uploaded cluster B pulls it over. As the clusters are otherwise > completely idle I can check the network load (via SNMP) on the nodes to see > which one is doing the transfers. > Without the restart cluster B seems to use a random RGW for pulling the sync > data, for me actually always node 3 or 4. > But after the restart it uses only nodes 1 and 2 for pulling (although I only > ran the test 4 times). > > So, looks like one has to restart the RGWs to pick up the configuration. > > Ciao, Uli > > > On 18. May 2023, at 21:14, Tarrago, Eli (RIS-BCT) > > mailto:eli.tarr...@lexisnexisrisk.com>> > > wrote: > > > > Adding a bit more context to this thread. > > > > I a
[ceph-users] Re: Dedicated radosgw gateways
/admin/log/?type=data=58=1_.1=true=
> HTTP/1.1" 200 312 - - - latency=0.196007445s
> 2023-05-18T19:06:49.987+ 7fb27c750700 1 == starting new request
> req=0x7fb3e81b06f0 =
>
>
>
>
> radosgw-admin zonegroup get
> {
>"id": "x",
>"name": "eastWestceph",
>"api_name": "EastWestCeph",
>"is_master": "true",
>"endpoints": [
>"http://east01.noam.lnrm.net:8080;,
>"http://east02.noam.lnrm.net:8080;,
>"http://east03.noam.lnrm.net:8080;,
>"http://east04.noam.lnrm.net:8080;, << sync node
>"http://west01.noam.lnrm.net:8080;,
>"http://west02.noam.lnrm.net:8080;,
>"http://west03.noam.lnrm.net:8080;,
>"http://west04.noam.lnrm.net:8080; << sync node
>],
> ...
>],
>"hostnames_s3website": [],
>"master_zone": "x",
>"zones": [
>{
>"id": "x",
>"name": "rgw-west",
>"endpoints": [
>"http://west01.noam.lnrm.net:8080;,
>"http://west02.noam.lnrm.net:8080;,
>"http://west03.noam.lnrm.net:8080;,
>"http://west04.noam.lnrm.net:8080; << -- sync node
> ],
>"log_meta": "false",
>"log_data": "true",
>"bucket_index_max_shards": 0,
>"read_only": "false",
>"tier_type": "",
>"sync_from_all": "true",
>"sync_from": [],
>"redirect_zone": ""
>},
>{
>"id": "x",
>"name": "rgw-east",
>"endpoints": [
>"http://east01.noam.lnrm.net:8080;,
>"http://east02.noam.lnrm.net:8080;,
>"http://east03.noam.lnrm.net:8080;,
>"http://east04.noam.lnrm.net:8080; << -- sync node
>
> From: Ulrich Klein
> Date: Tuesday, May 16, 2023 at 10:22 AM
> To: Konstantin Shalygin
> Cc: Michal Strnad , ceph-users
> Subject: [ceph-users] Re: Dedicated radosgw gateways
> [You don't often get email from ulrich.kl...@ulrichklein.de. Learn why this
> is important at https://aka.ms/LearnAboutSenderIdentification ]
>
> *** External email: use caution ***
>
>
>
> Hi,
>
> Might be a dumb question …
> I'm wondering how I can set those config variables in some but not all RGW
> processes?
>
> I'm on a cephadm 17.2.6. On 3 nodes I have RGWs. The ones on 8080 are behind
> haproxy for users. the ones one 8081 I'd like for sync only.
>
> # ceph orch ps | grep rgw
> rgw.max.maxvm4.lmjaef maxvm4 *:8080 running (51m) 4s ago 2h
> 262M- 17.2.6 d007367d0f3c 315f47a4f164
> rgw.max.maxvm4.lwzxpf maxvm4 *:8081 running (51m) 4s ago 2h
> 199M- 17.2.6 d007367d0f3c 7ae82e5f6ef2
> rgw.max.maxvm5.syxpnb maxvm5 *:8081 running (51m) 4s ago 2h
> 137M- 17.2.6 d007367d0f3c c0635c09ba8f
> rgw.max.maxvm5.wtpyfk maxvm5 *:8080 running (51m) 4s ago 2h
> 267M- 17.2.6 d007367d0f3c b4ad91718094
> rgw.max.maxvm6.ostneb maxvm6 *:8081 running (51m) 4s ago 2h
> 150M- 17.2.6 d007367d0f3c 83b2af8f787a
> rgw.max.maxvm6.qfulra maxvm6 *:8080 running (51m) 4s ago 2h
> 262M- 17.2.6 d007367d0f3c 81d01bf9e21d
>
> # ceph config show rgw.max.maxvm4.lwzxpf
> Error ENOENT: no config state for daemon rgw.max.maxvm4.lwzxpf
>
> # ceph config set rgw.max.maxvm4.lwzxpf rgw_enable_lc_threads false
> Error EINVAL: unrecognized config target 'rgw.max.maxvm4.lwzxpf'
> (Not surprised)
>
> # ceph tell rgw.max.maxvm4.lmjaef get rgw_enable_lc_threads
> error handling command target: local variable 'poolid' referenced before
> assignment
>
> # ceph tell rgw.max.maxvm4.lmjaef set rgw_enable_lc_threads false
> error handling command target: local variable 'poolid' referenced before
> assignment
>
> Is there any way to set the config for specific RGWs in a containerized env?
>
> (ceph.conf doesn't work. Doesn't do anything and gets overwritten with a
> minimall version at "unpredictable" intervalls)
>
> Thanks for any ideas.
>
> Ciao, Uli
>
>> On 15
[ceph-users] Re: Dedicated radosgw gateways
Hi, Might be a dumb question … I'm wondering how I can set those config variables in some but not all RGW processes? I'm on a cephadm 17.2.6. On 3 nodes I have RGWs. The ones on 8080 are behind haproxy for users. the ones one 8081 I'd like for sync only. # ceph orch ps | grep rgw rgw.max.maxvm4.lmjaef maxvm4 *:8080 running (51m) 4s ago 2h 262M- 17.2.6 d007367d0f3c 315f47a4f164 rgw.max.maxvm4.lwzxpf maxvm4 *:8081 running (51m) 4s ago 2h 199M- 17.2.6 d007367d0f3c 7ae82e5f6ef2 rgw.max.maxvm5.syxpnb maxvm5 *:8081 running (51m) 4s ago 2h 137M- 17.2.6 d007367d0f3c c0635c09ba8f rgw.max.maxvm5.wtpyfk maxvm5 *:8080 running (51m) 4s ago 2h 267M- 17.2.6 d007367d0f3c b4ad91718094 rgw.max.maxvm6.ostneb maxvm6 *:8081 running (51m) 4s ago 2h 150M- 17.2.6 d007367d0f3c 83b2af8f787a rgw.max.maxvm6.qfulra maxvm6 *:8080 running (51m) 4s ago 2h 262M- 17.2.6 d007367d0f3c 81d01bf9e21d # ceph config show rgw.max.maxvm4.lwzxpf Error ENOENT: no config state for daemon rgw.max.maxvm4.lwzxpf # ceph config set rgw.max.maxvm4.lwzxpf rgw_enable_lc_threads false Error EINVAL: unrecognized config target 'rgw.max.maxvm4.lwzxpf' (Not surprised) # ceph tell rgw.max.maxvm4.lmjaef get rgw_enable_lc_threads error handling command target: local variable 'poolid' referenced before assignment # ceph tell rgw.max.maxvm4.lmjaef set rgw_enable_lc_threads false error handling command target: local variable 'poolid' referenced before assignment Is there any way to set the config for specific RGWs in a containerized env? (ceph.conf doesn't work. Doesn't do anything and gets overwritten with a minimall version at "unpredictable" intervalls) Thanks for any ideas. Ciao, Uli > On 15. May 2023, at 14:15, Konstantin Shalygin wrote: > > Hi, > >> On 15 May 2023, at 14:58, Michal Strnad wrote: >> >> at Cephalocon 2023, it was mentioned several times that for service tasks >> such as data deletion via garbage collection or data replication in S3 via >> zoning, it is good to do them on dedicated radosgw gateways and not mix them >> with gateways used by users. How can this be achieved? How can we isolate >> these tasks? Will using dedicated keyrings instead of admin keys be >> sufficient? How do you operate this in your environment? > > Just: > > # don't put client traffic to "dedicated radosgw gateways" > # disable lc/gc on "gateways used by users" via `rgw_enable_lc_threads = > false` & `rgw_enable_gc_threads = false` > > > k > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Veeam backups to radosgw seem to be very slow
Hi, I’ve tested that combination once last year. My experience was similar. It was dead-slow. But if I remember correctly my conclusion was that Veeam was sending very slowly lots of rather small objects without any parallelism. But apart from the cruel slowness I didn’t have problems of the “bucket does not exist”/“permission denied” type. I don’t have the setup anymore. But I think it might be worth checking what kind of objects Veeam puts into its buckets. Ciao, Uli > On 25. Apr 2023, at 15:01, Boris Behrens wrote: > > We have a customer that tries to use veeam with our rgw objectstorage and > it seems to be blazingly slow. > > What also seems to be strange, that veeam sometimes show "bucket does not > exist" or "permission denied". > > I've tested parallel and everything seems to work fine from the s3cmd/aws > cli standpoint. > > Does anyone here ever experienced veeam problems with rgw? > > Cheers > Boris > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Yet another question about OSD memory usage ...
Hi, Yet another question about OSD memory usage ... I have a test cluster running. When I do a ceph orch ps I see for my osd.11: ceph orch ps --refresh NAME HOSTPORTSSTATUSREFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID osd.11 ceph01 running (2h)97s ago 2h 23.0G13.1G 17.2.5 cc65afd6173a 5d1062e8d392 When I chek via top on the machine I see: PID USER PR NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND 39807 ceph 20 0 6254956 3.7g 9228 S 31.2 3.0 846:21.63 /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-lo+ Now, where does ceph orch ps get those 23.0G from, when top just shows 3.7G resident and 6.2G virtual for osd.11? (I do understand that the MEM LIM n the ceph orch ps list is not really the limit) Anyone know where that discrepancy comes from? Ciao, Uli ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: excluding from host_pattern
I use something like "^ceph(0[1-9])|(1[0-9])$", but in a script that checks a
parameter for a "correct" ceph node name like in:
wantNum=$1
if [[ $wantNum =~ ^ceph(0[2-9]|1[0-9])$ ]] ; then
wantNum=${BASH_REMATCH[1]}
Which gives me the number, if it is in the range 02-19
Dunno, if that helps :)
Ciao, Uli
> On 27. Jan 2023, at 18:17, E Taka <0eta...@gmail.com> wrote:
>
> Hi,
>
> I wonder if it is possible to define a host pattern, which includes the
> host names
> ceph01…ceph19, but no other hosts, especially not ceph00. That means, this
> pattern is wrong: ceph[01][0-9] , since it includes ceph00.
>
> Not really a problem, but it seems that the "“host-pattern” is a regex that
> matches against hostnames and returns only matching hosts"¹ is not defined
> more precisely in the docs.
>
> 1) https://docs.ceph.com/en/latest/cephadm/host-management/
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] No Authentication/Authorization for creating topics on RGW?
Hi, I'm experimenting with notifications for S3 buckets. I got it working with notifications to HTTP(S) endpoints. What I did: Create a topic: # cat create_topic.data Action=CreateTopic =topictest2 =verify-ssl=false =use-ssl=false =OpaqueData=Hallodrio =push-endpoint=http://helper.example.com/cgi-bin/topictest =persistent=false =cloudevents=false # curl --request POST 'https://rgw.example.com' --data @create_topic.data https://sns.amazonaws.com/doc/2010-03-31/;>arn:aws:sns:example::topictest2f0904533-f4ed-4d60-886c-4125fcbed97b.4944109.3169009808426767767 And then created a notification for some user, which I received ok via http. What I'm wondering: There was no authentication/authorization necessary at all to create the topic?? Is that normal? Any <...> could create a million topics that way. Is there a way to prevent that from happening? I haven't found one in the docs. I guess - being new to the topic of notifications - that I'm missing something obvious? Ciao, Uli ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] RGW/S3 after a cluster is/was full
Hi, I have a problem with a full cluster and getting it back to a healthy state. Fortunately it's a small test cluster with no valuable data in it. It is used exclusively for RGW/S3, running 17.2.3. I intentionaly filled it up via rclone/S3 until it got into HEALTH_ERR, so see what would happen in that situation. At first it sort-of looks ok, as the cluster apparently goes into a read-only state. I can still get the stored data via S3. But then there seems to be no way to get out of the full state. Via S3 one can't delete any objects or buckets. Or did I miss anything? The requests just hang until they time out. So, I used "rados rm -p --force-full" to delete a bunch of those multipart corpses and other "old" objects. That got the cluster back into HEALTH_OK. But now the RGW gc seems to be screwed up: # radosgw-admin gc list --include-all | grep oid | wc -l 158109 # radosgw-admin gc process --include-all # radosgw-admin gc list --include-all | grep oid | wc -l 158109 I.e.. it has 158109 objects to clean up, but doesn’t clean up anything. I guess that's because the objects it wants to collect don't exist anymore, but are in some index or other list. Is there any way to reset or clean up? I'd appreciate any hints. Ciao, Uli ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: why rgw generates large quantities orphan objects?
Hi, I’m wondering if this problem will ever get fixed? This multipart-orphan-problem has made it now multiple times to the list, the tickets are up to 6 years old … and nothing changes. It screws up per-user space accounting and uses up space for nothing. I’d open another ticket with easy steps to reproduce, but can’t get an account on tracker.ceph.com (“pending administrator approval”) Ciao, Uli > On 13. 10 2022, at 17:44, Haas, Josh wrote: > > Hi Liang, > > My guess would be this bug: > > https://tracker.ceph.com/issues/44660 > https://www.spinics.net/lists/ceph-users/msg30151.html > > > It's actually existed for at least 6 years: > https://tracker.ceph.com/issues/16767 > > > Which occurs any time you reupload the same *part* in a single Multipart > Upload multiple times. For example, if my Multipart upload consists of 3 > parts, if I upload part #2 twice, then the first upload of part #2 becomes > orphaned. > > > If this was indeed the cause, you should have multiple "_multipart_" rados > objects for the same part in "rados ls". For example, here's all the rados > objects associated with a bugged bucket before I deleted it: > > cc79b188-89d1-4f47-acb1-ab90513e9bc9.23325574.228__multipart_file.txt.4vkWzU4C5XLd2R6unFgbQ6aZM26vPuq8.1 > cc79b188-89d1-4f47-acb1-ab90513e9bc9.23325574.228__multipart_file.txt.2~4zogSe4Ep0xvSC8j6aX71x_96cOgvQN.1 > cc79b188-89d1-4f47-acb1-ab90513e9bc9.23325574.228__shadow_file.txt.4vkWzU4C5XLd2R6unFgbQ6aZM26vPuq8.1_1 > cc79b188-89d1-4f47-acb1-ab90513e9bc9.23325574.228__shadow_file.txt.2~4zogSe4Ep0xvSC8j6aX71x_96cOgvQN.1_1 > > > If we look at just these two: > > cc79b188-89d1-4f47-acb1-ab90513e9bc9.23325574.228__multipart_file.txt.4vkWzU4C5XLd2R6unFgbQ6aZM26vPuq8.1 > cc79b188-89d1-4f47-acb1-ab90513e9bc9.23325574.228__multipart_file.txt.2~4zogSe4Ep0xvSC8j6aX71x_96cOgvQN.1 > > > They are in the format: > > $BUCKETID__multipart_$S3KEY.$PARTUID.$PARTNUM > > Because everything matches ($BUCKETID, $S3KEY, $PARTNUM) except for $PARTUID, > this S3 object has been affected by the bug. If you find instances of rados > keys that match on everything except $PARTUID, then this bug is probably the > cause. > > > Josh > > > From: 郑亮 > Sent: Wednesday, October 12, 2022 1:34:31 AM > To: ceph-users@ceph.io > Subject: [ceph-users] why rgw generates large quantities orphan objects? > > Hi all, > Description of problem: [RGW] Buckets/objects deletion is causing large > quantities orphan raods objects > > The cluster was running a cosbench workload, then remove the partial data > by deleting objects from the cosbench client, then we have deleted all the > buckets with the help of `s3cmd rb --recursive --force` command that > removed all the buckets, but that did not help in the space reclaimation. > > ``` > [root@node01 /]# rgw-orphan-list > > Available pools: > >device_health_metrics > >.rgw.root > >os-test.rgw.buckets.non-ec > >os-test.rgw.log > >os-test.rgw.control > >os-test.rgw.buckets.index > >os-test.rgw.meta > >os-test.rgw.buckets.data > >deeproute-replica-hdd-pool > >deeproute-replica-ssd-pool > >cephfs-metadata > >cephfs-replicated-pool > >.nfs > > Which pool do you want to search for orphans (for multiple, use > space-separated list)? os-test.rgw.buckets.data > > Pool is "os-test.rgw.buckets.data". > > Note: output files produced will be tagged with the current timestamp -- > 20221008062356. > running 'rados ls' at Sat Oct 8 06:24:05 UTC 2022 > > running 'rados ls' on pool os-test.rgw.buckets.data. > > > > running 'radosgw-admin bucket radoslist' at Sat Oct 8 06:43:21 UTC 2022 > computing delta at Sat Oct 8 06:47:17 UTC 2022 > > 39662551 potential orphans found out of a possible 39844453 (99%). > The results can be found in './orphan-list-20221008062356.out'. > >Intermediate files are './rados-20221008062356.intermediate' and > './radosgw-admin-20221008062356.intermediate'. > *** > > *** WARNING: This is EXPERIMENTAL code and the results should be used > *** only with CAUTION! > *** > Done at Sat Oct 8 06:48:07 UTC 2022. > > [root@node01 /]# radosgw-admin gc list > [] > > [root@node01 /]# cat orphan-list-20221008062356.out | wc -l > 39662551 > > [root@node01 /]# rados df > POOL_NAME USED OBJECTS CLONES COPIES > MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS > WR USED COMPR UNDER COMPR > .nfs 4.3 MiB 4 0 12 >00 0 77398 76 MiB146 79 > KiB 0 B 0 B > .rgw.root180 KiB16 0 48 >00 0 28749 28 MiB 0 > 0 B 0 B 0 B > cephfs-metadata 932 MiB 14772 0 44316 >00 01569690 3.8 GiB1258651 3.4 > GiB 0 B 0 B >
[ceph-users] Re: Wrong size actual?
I’m not sure anymore, but I think I tried that on a test system. Afterwards I
had to recreate the RGW pools and start over, so didn’t try it on a “real”
system.
But I can try again in about 2 weeks. It’s dead simple to recreate the problem
(https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/XOQXZYOWYMMQBWFXMHYDQUJ7LZZPFLSU
<https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/XOQXZYOWYMMQBWFXMHYDQUJ7LZZPFLSU>)
in any Ceph version since at least Pacific
Ciao, Uli
> On 6. Sep 2022, at 15:48, J. Eric Ivancich wrote:
>
> You could use `rgw-orphan-list` to determine rados objects that aren’t
> referenced by any bucket indices. Those objects could be removed after
> verification since this is an experimental feature.
>
> Eric
> (he/him)
>
>> On Sep 5, 2022, at 10:44 AM, Ulrich Klein
>> wrote:
>>
>> Looks like the old problem of lost multipart upload fragments. Has been
>> haunting me in all versions for more than a year. Haven‘t found any way of
>> getting rid of them.
>> Even deleting the bucket seems to leave the objects in the rados pool
>> forever.
>>
>> Ciao, Uli
>>
>>> Am 05.09.2022 um 15:19 schrieb Rok Jaklič :
>>>
>>> Hi,
>>>
>>> when I do:
>>> radosgw-admin user stats --uid=X --tenant=Y --sync-stats
>>>
>>> I get:
>>> {
>>> "stats": {
>>> "size": 2620347285776,
>>> "size_actual": 2620348436480,
>>> "size_utilized": 0,
>>> "size_kb": 2558932897,
>>> "size_kb_actual": 2558934020,
>>> "size_kb_utilized": 0,
>>> "num_objects": 593
>>> },
>>> "last_stats_sync": "2022-09-05T13:11:05.009648Z",
>>> "last_stats_update": "2022-09-05T13:11:04.97Z"
>>> }
>>>
>>> even though I have only two empty buckets inside account. Any ideas why?
>>>
>>> Account was previously full of objects, but objects were delete few days
>>> ago.
>>>
>>> Kind regards,
>>> Rok
>>> ___
>>> ceph-users mailing list -- ceph-users@ceph.io
>>> To unsubscribe send an email to ceph-users-le...@ceph.io
>> ___
>> ceph-users mailing list -- ceph-users@ceph.io
>> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Wrong size actual?
Hm, there are a couple, like https://tracker.ceph.com/issues/44660, but none
with a resolution.
It’s a real problem for us because it accumulates “lost” space and screws up
space accounting.
But the ticket is classified as “3 - minor”, ie. apparently not seen as urgent
for the last couple of years.
Ciao, Uli
> On 6. Sep 2022, at 11:06, Rok Jaklič wrote:
>
> Thanks for the info.
>
> Is there any bug report open?
>
> On Mon, Sep 5, 2022 at 4:44 PM Ulrich Klein <mailto:ulrich.kl...@ulrichklein.de>> wrote:
> Looks like the old problem of lost multipart upload fragments. Has been
> haunting me in all versions for more than a year. Haven‘t found any way of
> getting rid of them.
> Even deleting the bucket seems to leave the objects in the rados pool forever.
>
> Ciao, Uli
>
> > Am 05.09.2022 um 15:19 schrieb Rok Jaklič > <mailto:rjak...@gmail.com>>:
> >
> > Hi,
> >
> > when I do:
> > radosgw-admin user stats --uid=X --tenant=Y --sync-stats
> >
> > I get:
> > {
> >"stats": {
> >"size": 2620347285776,
> >"size_actual": 2620348436480,
> >"size_utilized": 0,
> >"size_kb": 2558932897,
> >"size_kb_actual": 2558934020,
> >"size_kb_utilized": 0,
> >"num_objects": 593
> >},
> >"last_stats_sync": "2022-09-05T13:11:05.009648Z",
> >"last_stats_update": "2022-09-05T13:11:04.97Z"
> > }
> >
> > even though I have only two empty buckets inside account. Any ideas why?
> >
> > Account was previously full of objects, but objects were delete few days
> > ago.
> >
> > Kind regards,
> > Rok
> > ___
> > ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io>
> > To unsubscribe send an email to ceph-users-le...@ceph.io
> > <mailto:ceph-users-le...@ceph.io>
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Wrong size actual?
Looks like the old problem of lost multipart upload fragments. Has been
haunting me in all versions for more than a year. Haven‘t found any way of
getting rid of them.
Even deleting the bucket seems to leave the objects in the rados pool forever.
Ciao, Uli
> Am 05.09.2022 um 15:19 schrieb Rok Jaklič :
>
> Hi,
>
> when I do:
> radosgw-admin user stats --uid=X --tenant=Y --sync-stats
>
> I get:
> {
>"stats": {
>"size": 2620347285776,
>"size_actual": 2620348436480,
>"size_utilized": 0,
>"size_kb": 2558932897,
>"size_kb_actual": 2558934020,
>"size_kb_utilized": 0,
>"num_objects": 593
>},
>"last_stats_sync": "2022-09-05T13:11:05.009648Z",
>"last_stats_update": "2022-09-05T13:11:04.97Z"
> }
>
> even though I have only two empty buckets inside account. Any ideas why?
>
> Account was previously full of objects, but objects were delete few days
> ago.
>
> Kind regards,
> Rok
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: not so empty bucket
Yo, I’m having the same problem and can easily reproduce it. See https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/XOQXZYOWYMMQBWFXMHYDQUJ7LZZPFLSU/ And similar ones. The problem still exists in Quincy 17.2.0. But looks like it’s too low priority to be fixed. Ciao, Uli > On 09. 05 2022, at 22:20, Christopher Durham wrote: > > > I am using pacific 16.2.7 on rocket 8.5 Linux 8.5. > I have a once heavily used radosgw bucket that is now empty. Let's call it > 'oldbucket' awscli now shows that there are no objects in the bucket. > > However, radosgw-admin bucket stats --bucket oldbucket shows num_objects in > rgw.main as 39 objects, with about 200 gigs used in the size field. > > radosgw-admin bucket check --bucket oldbucket indeed shows 39 objects in a > list. > Each of these objects are of the form: > _multipart_originalobjectname.someoriginalid.NN > radosgw-admin bucket radoslist --bucket oldbucket shows only 9 objects, but > thoseobjects are alll included as part of the bucket check command above. > This particular bucket did have alot of multipart uploads. All multipart > uploads have beenaborted with awscli. And of course awscli shows no objects > in the bucket > > radosgw-admin bucket list --bucket oldbucket shows all 39 objects, which is > weird thatI see object names which once were multipart object parts. > > None of the 39 objects exist in the pool (rados -p $pool get $obj /tmp/$obj > all return 1) > If I list all the index objects for this bucket in the index pool and then do > a listomapkeys for each of the index objects,I see only the 39 omapkeys. > So my question is, what do I need to do to fix this bucket (without delting > and recreating it?) Would just doing a rmomapkeyon each of the omap keys in > the listomapkeys output solve my problem, and reclaim the 200 Gigs? > Do I have to rebuild the index somehow (--fix in bucket check above did > nothing). > > Thanks for any thoughts/insight. > -Chris > > > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW/S3 losing multipart upload objects
Hi, I just tried again on a Quincy 17.2.0. Same procedure, same problem. I just wonder if nobody else sees that problem? Ciao, Uli > On 18. 03 2022, at 12:18, Ulrich Klein wrote: > > I tried it on a mini-cluster (4 Raspberries) with 16.2.7. > Same procedure, same effect. I just can’t get rid of these objects. > > Is there any method that would allow me to delete these objects without > damaging RGW? > > Ciao, Uli > >> On 17. 03 2022, at 15:30, Soumya Koduri wrote: >> >> On 3/17/22 17:16, Ulrich Klein wrote: >>> Hi, >>> >>> My second attempt to get help with a problem I'm trying to solve for about >>> 6 month now. >>> >>> I have a Ceph 16.2.6 test cluster, used almost exclusively for providing >>> RGW/S3 service. similar to a production cluster. >>> >>> The problem I have is this: >>> A client uploads (via S3) a bunch of large files into a bucket via >>> multiparts >>> The upload(s) get interrupted and retried >>> In the end from a client's perspective all the files are visible and >>> everything looks fine. >>> But on the cluster there are many more objects in the buckets >>> Even after cleaning out the incomplete multipart uploads there are too many >>> objects >>> Even after deleting all the visible objects from the bucket there are still >>> objects in the bucket >>> I have so far found no way to get rid of those left-over objects. >>> It's screwing up space accounting and I'm afraid I'll eventually have a >>> cluster full of those lost objects. >>> The only way to clean up seems to be to copy te contents of a bucket to a >>> new bucket and delete the screwed-up bucket. But on a production system >>> that's not always a real option. >>> >>> I've found a variety of older threads that describe a similar problem. None >>> of them decribing a solution :( >>> >>> >>> >>> I can pretty easily reproduce the problem with this sequence: >>> >>> On a client system create a directory with ~30 200MB files. (On a faster >>> system I'd probably need bigger or more files) >>> tstfiles/tst01 - tst29 >>> >>> run >>> $ rclone mkdir tester:/test-bucket # creates a bucket on the test system >>> with user tester >>> Run >>> $ rclone sync -v tstfiles tester:/test-bucket/tstfiles >>> a couple of times (6-8), interrupting each one via CNTRL-C >>> Eventually let one finish. >>> >>> Now I can use s3cmd to see all the files: >>> $ s3cmd ls -lr s3://test-bucket/tstfiles >>> 2022-03-16 17:11 200M ecb28853bd18eeae185b0b12bd47333c-40 STANDARD >>> s3://test-bucket/tstfiles/tst01 >>> ... >>> 2022-03-16 17:13 200M ecb28853bd18eeae185b0b12bd47333c-40 STANDARD >>> s3://test-bucket/tstfiles/tst29 >>> >>> ... and to list incomplete uploads: >>> $ s3cmd multipart s3://test-bucket >>> s3://test-bucket/ >>> Initiated PathId >>> 2022-03-16T17:11:19.074Zs3://test-bucket/tstfiles/tst05 >>> 2~1nElF0c3uq5FnZ9cKlsnGlXKATvjr0g >>> ... >>> 2022-03-16T17:12:41.583Zs3://test-bucket/tstfiles/tst28 >>> 2~exVQUILhVSmFqWxCuAflRa4Tfq4nUQa >>> >>> I can abort the uploads with >>> $ s3cmd abortmp s3://test-bucket/tstfiles/tst05 >>> 2~1nElF0c3uq5FnZ9cKlsnGlXKATvjr0g >>> ... >> >> >> >> On the latest master, I see that these objects are deleted immediately post >> abortmp. I believe this issue may have beenn fixed as part of [1], >> backported to v16.2.7 [2]. Maybe you could try upgrading your cluster and >> recheck. >> >> >> Thanks, >> >> Soumya >> >> >> [1] https://tracker.ceph.com/issues/53222 >> >> [2] https://tracker.ceph.com/issues/53291 >> >> >> >> ___ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Permission problem upgrading Raspi-cluster from 16.2.7 to 17.2.0
Yup, looks exactly like the problem described in that thread. For the next cluster I’ll try without the _admin labels. I think I added them because without the label the files in /etc/ceph ever so often “magically” disappeared. So I’ll save copies of them before doing the upgrade. Thanks, Uli > On 27. 04 2022, at 12:17, Kuo Gene wrote: > > Hi, > > There’s previous discussion about this issue. > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ZZZGEVD4L4SEMEBMTZYDW2OOVWQFXGOA/ > > Can you check if there is any host with _admin label on it? Removing the > label works for me. > > Regards, > Gene Kuo > >> On Apr 27, 2022, at 18:21, Ulrich Klein wrote: >> >> Hi, >> >> Yesterday I upgraded my smallest test system, 4 Raspberries 4B, from Pacific >> 16.2.7 (cephadm/containerized) to 17.2.0 using >> ceph orch upgrade start --ceph-version 17.2.0 >> >> It mostly worked ok, but wouldn't have finished without manual intervention. >> Apparently each time a mgr is upgraded the process creates new >> /etc/ceph/ceph.conf and /etc/ceph/ceph.client.admin.keyring files on all >> nodes. >> To do that it looks like it first copies the files to >> /tmp/etc/ceph/ceph.conf on the node, then changes owwner and permission and >> then tries to move the file into place. Unfortunately it changes >> owner/permission in way so that it doesn't have permission to write to and >> move the file resulting in somethig like this in an infinite (?) loop: >> >> 2022-04-27T09:03:45.032808+ mgr.ceph00.lpaijp (mgr.2314108) 605 : >> cephadm [ERR] executing refresh((['ceph00', 'ceph01', 'ceph02', 'ceph03'],)) >> failed. >> Traceback (most recent call last): >> File "/usr/share/ceph/mgr/cephadm/ssh.py", line 221, in _write_remote_file >> await asyncssh.scp(f.name, (conn, tmp_path)) >> File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp >> await source.run(srcpath) >> File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run >> self.handle_error(exc) >> File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in >> handle_error >> raise exc from None >> File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run >> await self._send_files(path, b'') >> File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files >> self.handle_error(exc) >> File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in >> handle_error >> raise exc from None >> File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files >> await self._send_file(srcpath, dstpath, attrs) >> File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file >> await self._make_cd_request(b'C', attrs, size, srcpath) >> File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in >> _make_cd_request >> self._fs.basename(path)) >> File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in >> make_request >> raise exc >> asyncssh.sftp.SFTPFailure: scp: /tmp/etc/ceph/ceph.conf.new: Permission >> denied >> >> During handling of the above exception, another exception occurred: >> >> Traceback (most recent call last): >> File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work >> return f(*arg) >> File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh >> self._write_client_files(client_files, host) >> File "/usr/share/ceph/mgr/cephadm/serve.py", line 1052, in >> _write_client_files >> self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) >> File "/usr/share/ceph/mgr/cephadm/ssh.py", line 238, in write_remote_file >> host, path, content, mode, uid, gid, addr)) >> File "/usr/share/ceph/mgr/cephadm/module.py", line 569, in wait_async >> return self.event_loop.get_result(coro) >> File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result >> return asyncio.run_coroutine_threadsafe(coro, self._loop).result() >> File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result >> return self.__get_result() >> File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in >> __get_result >> raise self._exception >> File "/usr/share/ceph/mgr/cephadm/ssh.py", line 226, in _write_remote_file >> raise OrchestratorError(msg) >> orchestrator._interface.OrchestratorError: U
[ceph-users] Permission problem upgrading Raspi-cluster from 16.2.7 to 17.2.0
Hi, Yesterday I upgraded my smallest test system, 4 Raspberries 4B, from Pacific 16.2.7 (cephadm/containerized) to 17.2.0 using ceph orch upgrade start --ceph-version 17.2.0 It mostly worked ok, but wouldn't have finished without manual intervention. Apparently each time a mgr is upgraded the process creates new /etc/ceph/ceph.conf and /etc/ceph/ceph.client.admin.keyring files on all nodes. To do that it looks like it first copies the files to /tmp/etc/ceph/ceph.conf on the node, then changes owwner and permission and then tries to move the file into place. Unfortunately it changes owner/permission in way so that it doesn't have permission to write to and move the file resulting in somethig like this in an infinite (?) loop: 2022-04-27T09:03:45.032808+ mgr.ceph00.lpaijp (mgr.2314108) 605 : cephadm [ERR] executing refresh((['ceph00', 'ceph01', 'ceph02', 'ceph03'],)) failed. Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 221, in _write_remote_file await asyncssh.scp(f.name, (conn, tmp_path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp await source.run(srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run await self._send_files(path, b'') File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files await self._send_file(srcpath, dstpath, attrs) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file await self._make_cd_request(b'C', attrs, size, srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request self._fs.basename(path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request raise exc asyncssh.sftp.SFTPFailure: scp: /tmp/etc/ceph/ceph.conf.new: Permission denied During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/utils.py", line 76, in do_work return f(*arg) File "/usr/share/ceph/mgr/cephadm/serve.py", line 265, in refresh self._write_client_files(client_files, host) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1052, in _write_client_files self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 238, in write_remote_file host, path, content, mode, uid, gid, addr)) File "/usr/share/ceph/mgr/cephadm/module.py", line 569, in wait_async return self.event_loop.get_result(coro) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 48, in get_result return asyncio.run_coroutine_threadsafe(coro, self._loop).result() File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/ssh.py", line 226, in _write_remote_file raise OrchestratorError(msg) orchestrator._interface.OrchestratorError: Unable to write ceph02:/etc/ceph/ceph.conf: scp: /tmp/etc/ceph/ceph.conf.new: Permission denied On each node I had to do cd /usr/bin mv chmod chmod_real ; ln -s true chmod mv chown chown_real ; ln -s true chown And then whenever the file(s) appeared: chmod_real 666 /tmp/etc/ceph/ceph.conf.new to make it get over that hurdle. And once finished restore the chown/chmod binaries and permissions. I wonder if anyone else has seen that on Intel/AMD machines? Looks like a pretty obvious problem with the process shooting itself in the permission foot, and on bigger clusters that process would be a time consuming pain. Ciao, Uli ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph RGW Multisite Multi Zonegroup Build Problems
After a bunch of attempts to get multiple zonegroups with RGW multi-site to
work I’d have a question:
Has anyone successfully created a working setup with multiple zonegroups with
RGW multi-site using a cephadm/ceph orch installation of pacific?
Ciao, Uli
> On 19. 04 2022, at 14:33, Ulrich Klein wrote:
>
> Hi,
>
> I'm trying to do the same as Mark. Basically the same problem. Can’t get it
> to work.
> The —-master doesn’t make much of a difference for me.
>
>
> Any other idea, maybe?
>
> Ciao, Uli
>
> On Cluster #1 ("nceph"):
>
> radosgw-admin realm create --rgw-realm=acme --default
> radosgw-admin zonegroup create --rgw-zonegroup=us --rgw-realm=acme --master
> --default --endpoints=http://nceph00.uli.home:8080
> radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-west-1 --master
> --default --endpoints=http://nceph00.uli.home:8080
> radosgw-admin user create --uid="sysuser" --display-name="System User"
> --system --access-key=N7Y6CM8KIN45UY2J5NQA
> --secret=a8QvbAMGpDwPBk8E3t3jHTyTSNqMQi4PK04yN9GX
> radosgw-admin zone modify --rgw-zone=us-west-1
> --access-key=N7Y6CM8KIN45UY2J5NQA
> --secret=a8QvbAMGpDwPBk8E3t3jHTyTSNqMQi4PK04yN9GX
> radosgw-admin period update --commit
> ceph orch host label add nceph00 rgw
> ceph orch apply rgw acme --realm=acme --zone=us-west-1 '--placement=label:rgw
> count-per-host:1' --port=8080
> echo -n "N7Y6CM8KIN45UY2J5NQA" > ac
> echo -n "a8QvbAMGpDwPBk8E3t3jHTyTSNqMQi4PK04yN9GX" > sc
> ceph dashboard set-rgw-api-access-key -i ac
> ceph dashboard set-rgw-api-secret-key -i sc
>
> radosgw-admin period update --commit
> {
>"id": "5b304997-e3ba-4cc2-9f80-af88a31827c3",
>"epoch": 2,
>"predecessor_uuid": "118ecc9a-3824-4560-afdf-98f901836fb2",
>"sync_status": [],
>"period_map": {
>"id": "5b304997-e3ba-4cc2-9f80-af88a31827c3",
>"zonegroups": [
>{
>"id": "1df9e729-8fa0-47fa-942f-b5159fad8360",
>"name": "us",
>"api_name": "us",
>"is_master": "true",
>"endpoints": [
>"http://nceph00.uli.home:8080;
>],
>"hostnames": [],
>"hostnames_s3website": [],
>"master_zone": "7ac5da6e-ea41-43ce-b6fc-f5b6e794933f",
>"zones": [
>{
>"id": "7ac5da6e-ea41-43ce-b6fc-f5b6e794933f",
>"name": "us-west-1",
>"endpoints": [
>"http://nceph00.uli.home:8080;
>],
>"log_meta": "false",
>"log_data": "false",
>"bucket_index_max_shards": 11,
>"read_only": "false",
>"tier_type": "",
>"sync_from_all": "true",
>"sync_from": [],
>"redirect_zone": ""
>}
>],
>"placement_targets": [
>{
>"name": "default-placement",
>"tags": [],
>"storage_classes": [
>"STANDARD"
>]
>}
>],
>"default_placement": "default-placement",
>"realm_id": "657b514d-be49-45c8-a69e-7ee474276c9a",
>"sync_policy": {
>"groups": []
>}
>}
>],
>"short_zone_ids": [
>{
>"key": "7ac5da6e-ea41-43ce-b6fc-f5b6e794933f",
>"val": 1454718312
>}
>]
>},
>"master_zonegroup": "1df9e729-8fa0-47fa-942f-b5159fad8360",
>"master_zone": "7ac5da6e-ea41-43ce-b6fc-f5b6e794933f",
>"period_config": {
>"bucket_quota": {
>"enabled": false
[ceph-users] Re: Ceph RGW Multisite Multi Zonegroup Build Problems
Hi,
I'm trying to do the same as Mark. Basically the same problem. Can’t get it to
work.
The —-master doesn’t make much of a difference for me.
Any other idea, maybe?
Ciao, Uli
On Cluster #1 ("nceph"):
radosgw-admin realm create --rgw-realm=acme --default
radosgw-admin zonegroup create --rgw-zonegroup=us --rgw-realm=acme --master
--default --endpoints=http://nceph00.uli.home:8080
radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-west-1 --master
--default --endpoints=http://nceph00.uli.home:8080
radosgw-admin user create --uid="sysuser" --display-name="System User" --system
--access-key=N7Y6CM8KIN45UY2J5NQA
--secret=a8QvbAMGpDwPBk8E3t3jHTyTSNqMQi4PK04yN9GX
radosgw-admin zone modify --rgw-zone=us-west-1
--access-key=N7Y6CM8KIN45UY2J5NQA
--secret=a8QvbAMGpDwPBk8E3t3jHTyTSNqMQi4PK04yN9GX
radosgw-admin period update --commit
ceph orch host label add nceph00 rgw
ceph orch apply rgw acme --realm=acme --zone=us-west-1 '--placement=label:rgw
count-per-host:1' --port=8080
echo -n "N7Y6CM8KIN45UY2J5NQA" > ac
echo -n "a8QvbAMGpDwPBk8E3t3jHTyTSNqMQi4PK04yN9GX" > sc
ceph dashboard set-rgw-api-access-key -i ac
ceph dashboard set-rgw-api-secret-key -i sc
radosgw-admin period update --commit
{
"id": "5b304997-e3ba-4cc2-9f80-af88a31827c3",
"epoch": 2,
"predecessor_uuid": "118ecc9a-3824-4560-afdf-98f901836fb2",
"sync_status": [],
"period_map": {
"id": "5b304997-e3ba-4cc2-9f80-af88a31827c3",
"zonegroups": [
{
"id": "1df9e729-8fa0-47fa-942f-b5159fad8360",
"name": "us",
"api_name": "us",
"is_master": "true",
"endpoints": [
"http://nceph00.uli.home:8080;
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "7ac5da6e-ea41-43ce-b6fc-f5b6e794933f",
"zones": [
{
"id": "7ac5da6e-ea41-43ce-b6fc-f5b6e794933f",
"name": "us-west-1",
"endpoints": [
"http://nceph00.uli.home:8080;
],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": 11,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": [],
"redirect_zone": ""
}
],
"placement_targets": [
{
"name": "default-placement",
"tags": [],
"storage_classes": [
"STANDARD"
]
}
],
"default_placement": "default-placement",
"realm_id": "657b514d-be49-45c8-a69e-7ee474276c9a",
"sync_policy": {
"groups": []
}
}
],
"short_zone_ids": [
{
"key": "7ac5da6e-ea41-43ce-b6fc-f5b6e794933f",
"val": 1454718312
}
]
},
"master_zonegroup": "1df9e729-8fa0-47fa-942f-b5159fad8360",
"master_zone": "7ac5da6e-ea41-43ce-b6fc-f5b6e794933f",
"period_config": {
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
},
"realm_id": "657b514d-be49-45c8-a69e-7ee474276c9a",
"realm_name": "acme",
"realm_epoch": 2
}
Dashboard works, too
On cluster #2 ("ceph")
--
radosgw-admin realm pull --url=http://nceph00.uli.home:8080
--access-key=N7Y6CM8KIN45UY2J5NQA
--secret=a8QvbAMGpDwPBk8E3t3jHTyTSNqMQi4PK04yN9GX
radosgw-admin zonegroup create --rgw-realm=acme --rgw-zonegroup=eu
--endpoints=http://ceph00.uli.home:8080
radosgw-admin zone create --rgw-zone=eu-west-1 --rgw-zonegroup=eu
--endpoints=http://ceph00.uli.home:8080
(With or without —default makes no difference)
radosgw-admin zone modify --rgw-zone=eu-west-1 --rgw-zonegroup=eu
--access-key=N7Y6CM8KIN45UY2J5NQA
--secret=a8QvbAMGpDwPBk8E3t3jHTyTSNqMQi4PK04yN9GX
ceph orch host label add ceph00 rgw
ceph orch apply rgw acme --realm=acme --zone=eu-west-1 '--placement=label:rgw
count-per-host:1' --port=8080
echo -n "N7Y6CM8KIN45UY2J5NQA" > ac
echo -n "a8QvbAMGpDwPBk8E3t3jHTyTSNqMQi4PK04yN9GX" > sc
ceph dashboard set-rgw-api-access-key -i ac
ceph dashboard set-rgw-api-secret-key -i sc
radosgw-admin period update --commit
couldn't init storage provider
ceph orch ps
[ceph-users] Re: RadosGW S3 range on a 0 byte object gives 416 Range Not Satisfiable
RFC 7233 4.4 <https://datatracker.ietf.org/doc/html/rfc7233#section-4.4>. 416 Range Not Satisfiable The 416 (Range Not Satisfiable) status code indicates that none of the ranges in the request's Range header field (Section 3.1 <https://datatracker.ietf.org/doc/html/rfc7233#section-3.1>) overlap the current extent of the selected resource or that the set of ranges requested has been rejected due to invalid ranges or an excessive request of small or overlapping ranges. For byte ranges, failing to overlap the current extent means that the first-byte-pos of all of the byte-range-spec values were greater than the current length of the selected representation. When this status code is generated in response to a byte-range request, the sender SHOULD generate a Content-Range header field specifying the current length of the selected representation (Section 4.2 <https://datatracker.ietf.org/doc/html/rfc7233#section-4.2>). For example: HTTP/1.1 416 Range Not Satisfiable Date: Fri, 20 Jan 2012 15:41:54 GMT Content-Range: bytes */47022 Note: Because servers are free to ignore Range, many implementations will simply respond with the entire selected representation in a 200 (OK) response. That is partly because most clients are prepared to receive a 200 (OK) to complete the task (albeit less efficiently) and partly because clients might not stop making an invalid partial request until they have received a complete representation. Thus, clients cannot depend on receiving a 416 (Range Not Satisfiable) response even when it is most appropriate. > On 21. 03 2022, at 15:11, Ulrich Klein wrote: > > With a bit of HTTP background I’d say: > bytes=0-100 means: First byte to to 100nd byte. First byte is byte #0 > On an empty object there is no first byte, i.e. not satisfiable ==> 416 > > Should be the same as on a single byte object and > bytes=1-100 > > 200 OK should only be correct, if the server or a proxy in between doesn’t > support range requests. > > Ciao, Uli > >> On 21. 03 2022, at 14:47, Kai Stian Olstad wrote: >> >> Hi >> >> Ceph v16.2.6. >> >> Using GET with Range: bytes=0-100 it fails with 416 if the object is 0 >> byte. >> I tried reading the http specification[1] on the subject but did not get any >> wiser unfortunately. >> >> I did a test with curl and range against a 0 byte file on Nginx and it >> returned 200 OK. >> >> Does anyone know it's correct to return 416 on 0 byte object with range or >> should this be considered a bug in Ceph. >> >> >> [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.1 >> >> -- >> Kai Stian Olstad >> ___ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RadosGW S3 range on a 0 byte object gives 416 Range Not Satisfiable
With a bit of HTTP background I’d say: bytes=0-100 means: First byte to to 100nd byte. First byte is byte #0 On an empty object there is no first byte, i.e. not satisfiable ==> 416 Should be the same as on a single byte object and bytes=1-100 200 OK should only be correct, if the server or a proxy in between doesn’t support range requests. Ciao, Uli > On 21. 03 2022, at 14:47, Kai Stian Olstad wrote: > > Hi > > Ceph v16.2.6. > > Using GET with Range: bytes=0-100 it fails with 416 if the object is 0 > byte. > I tried reading the http specification[1] on the subject but did not get any > wiser unfortunately. > > I did a test with curl and range against a 0 byte file on Nginx and it > returned 200 OK. > > Does anyone know it's correct to return 416 on 0 byte object with range or > should this be considered a bug in Ceph. > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35.1 > > -- > Kai Stian Olstad > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Questions / doubts about rgw users and zones
Hi, I'm not the expert, either :) So if someone with more experience wants to correct me, that’s fine. But I think I have a similar setup with a similar goal. I have two clusters, purely for RGW/S3. I have a realm R in which I created a zonegroup ZG (not the low tax Kanton:) ) On the primary cluster I have a zone ZA as master and on the second cluster a zone ZB. With all set up including the access keys for the zones, metadata and data is synced between the two. Users access only the primary cluster, the secondary is basically a very safe backup. But I want - for some users - that their data is NOT replicated to that secondary cluster, cheaper plan or short lived data. I found two ways to achieve that. One is similar to what I understand is your setup: Create another zone ZC in zonegroup ZG on the primary cluster: - Create zone ZC with endpoint on host hostxy:8080 - period update commit - Create new pools and placement targets (if necessary) - period update commit - Add another RGW # ceph orch host label add hostxy rgwnosync # ceph orch apply rgw ZC --realm=myrealm --zone=ZC '-- placement=label:rgwnosync count-per-host:1' --port=8080 # radosgw-admin zone modify --rgw-zone=ZC --access- key=sysuserkey -- secret=sysusersecret - period update commit (My cook book also has this step, giving the credentials once more to the dashboard: On the primar cluster: # echo -n "sysuserkey" > ac # echo -n "sysusersecret" > sc # ceph dashboard set-rgw-api-access-key -i ac # ceph dashboard set-rgw-api-secret-key -i sc Not sure it's necessary, but doesn't hurt. ) Now there are 3 zones that sync everything back and forth in all directions. Then limit what to sync from where: On the primary cluster # radosgw-admin zone modify --rgw-zone=ZA --sync-from-all=false # radosgw-admin zone modify --rgw-zone=ZC --sync-from-all=false # radosgw-admin period update --commit On the secondary cluster: # radosgw-admin zone modify --rgw-zone=ZC --sync-from-all=false # radosgw-admin period update --commit Now nothing gets synced anymore But I want to sync from ZA to ZB and only that sync. On the secondary cluster: # radosgw-admin zone modify --rgw-zone=ZB --sync-from=ZA # radosgw-admin period update --commit Now, when users push their data to the URL for zone ZA it gets replicated to zone ZB. If they push to the ZC URL it stays in zone ZC on the primary cluster only. What's a bit "non-intuitive" are a few things: - In the dashboard on the primary cluster one has to look close to figure out which RGW it's currently talking to, and possibly select a different one in that selection at the top. - Users and their credentials are synced across all zones, which is good in my case, as users don't have to use different credentials for ZA and ZC. - Bucket names are also synced across all zones, but not their objects, which creates "interesting" effects. My alternative solution was to turn on/off synchronization on buckets: For any existing (!) bucket one can simply turn off/on synchronization via # radosgw-admin bucket sync [enable/disable] --bucket= Problem is that it only works on existing buckets. I've found no way to turn synchronization off by default, and even less what I actually need, which is turn synchronization/replication on/off per RGW user. I discarded sync policies as they left the sync status in a suspicious state, were complicated in a strange way and the documentation "wasn't too clear to me" Dunno, if this helps, and I'm pretty sure their may be better ways. But this worked for me. Ciao, Uli PS: I use s3cmd, rclone and cyberduck for my simple testing. aws cli I found more AWS-centric and it also doen't work well with Ceph/RGW tenants. And, I'm not sure why you have so many endpoints in the zonegroup, but no load balancer a la RGW ingress, i.e. keepalived+haproxy. But that may be my lack of expertise. > On 19. Mar 2022, at 11:47, Arno Lehmann wrote: > > Hi all, > > I am currently playing around with a Ceph lab environment, using it to > develop some particular solution, at the same time learning about the > many-armed monster. > > My understanding of Ceph is accordingly somewhat superficial. > > What I have are two clusters I set set up and manage using cephadm, thus > containerized. The version is 16.2.7. > > Each cluster consists of five VMs, a few osds, managers, five monitor > instances, and all the other stuff that cephadm, when deploying, decided I > need. Network per cluster is very simple, there's exactly one network for > services access, management, and in-cluster traffic. The two clusters are > connected by an IP-IP tunnel through a VPN. The whole thing is pretty slow, > but functional enough to allow me to run > > - one realm (I called it europe) > - inside is one zonegroup (called ch-de) > - in that zonegroup live some zones > In one of the clusters, the zone "yv" has been created as the master zone for > the zonegroup. Some user accounts for object storage
[ceph-users] Re: RGW/S3 losing multipart upload objects
I tried it on a mini-cluster (4 Raspberries) with 16.2.7. Same procedure, same effect. I just can’t get rid of these objects. Is there any method that would allow me to delete these objects without damaging RGW? Ciao, Uli > On 17. 03 2022, at 15:30, Soumya Koduri wrote: > > On 3/17/22 17:16, Ulrich Klein wrote: >> Hi, >> >> My second attempt to get help with a problem I'm trying to solve for about 6 >> month now. >> >> I have a Ceph 16.2.6 test cluster, used almost exclusively for providing >> RGW/S3 service. similar to a production cluster. >> >> The problem I have is this: >> A client uploads (via S3) a bunch of large files into a bucket via multiparts >> The upload(s) get interrupted and retried >> In the end from a client's perspective all the files are visible and >> everything looks fine. >> But on the cluster there are many more objects in the buckets >> Even after cleaning out the incomplete multipart uploads there are too many >> objects >> Even after deleting all the visible objects from the bucket there are still >> objects in the bucket >> I have so far found no way to get rid of those left-over objects. >> It's screwing up space accounting and I'm afraid I'll eventually have a >> cluster full of those lost objects. >> The only way to clean up seems to be to copy te contents of a bucket to a >> new bucket and delete the screwed-up bucket. But on a production system >> that's not always a real option. >> >> I've found a variety of older threads that describe a similar problem. None >> of them decribing a solution :( >> >> >> >> I can pretty easily reproduce the problem with this sequence: >> >> On a client system create a directory with ~30 200MB files. (On a faster >> system I'd probably need bigger or more files) >> tstfiles/tst01 - tst29 >> >> run >> $ rclone mkdir tester:/test-bucket # creates a bucket on the test system >> with user tester >> Run >> $ rclone sync -v tstfiles tester:/test-bucket/tstfiles >> a couple of times (6-8), interrupting each one via CNTRL-C >> Eventually let one finish. >> >> Now I can use s3cmd to see all the files: >> $ s3cmd ls -lr s3://test-bucket/tstfiles >> 2022-03-16 17:11 200M ecb28853bd18eeae185b0b12bd47333c-40 STANDARD >> s3://test-bucket/tstfiles/tst01 >> ... >> 2022-03-16 17:13 200M ecb28853bd18eeae185b0b12bd47333c-40 STANDARD >> s3://test-bucket/tstfiles/tst29 >> >> ... and to list incomplete uploads: >> $ s3cmd multipart s3://test-bucket >> s3://test-bucket/ >> InitiatedPathId >> 2022-03-16T17:11:19.074Z s3://test-bucket/tstfiles/tst05 >> 2~1nElF0c3uq5FnZ9cKlsnGlXKATvjr0g >> ... >> 2022-03-16T17:12:41.583Z s3://test-bucket/tstfiles/tst28 >> 2~exVQUILhVSmFqWxCuAflRa4Tfq4nUQa >> >> I can abort the uploads with >> $ s3cmd abortmp s3://test-bucket/tstfiles/tst05 >> 2~1nElF0c3uq5FnZ9cKlsnGlXKATvjr0g >> ... > > > > On the latest master, I see that these objects are deleted immediately post > abortmp. I believe this issue may have beenn fixed as part of [1], backported > to v16.2.7 [2]. Maybe you could try upgrading your cluster and recheck. > > > Thanks, > > Soumya > > > [1] https://tracker.ceph.com/issues/53222 > > [2] https://tracker.ceph.com/issues/53291 > > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW/S3 losing multipart upload objects
Yo, that one is one of the threads that looks very similar to my problem, just with no resolution for me. I have a multi-site setup, so no resharding. Tried it — setup RGW from scratch due to all the “funny” errors. So, hoping for 16.2.7 or the “in the works” fix :) Ciao, Uli > On 17. 03 2022, at 15:34, Matt Benjamin wrote: > > Thanks, Soumya. > > It's also possible that what's reproducing is the known (space) leak > during re-upload of multipart parts, described here: > https://tracker.ceph.com/issues/44660. > A fix for this is being worked on, it's taking a while. > > Matt > > On Thu, Mar 17, 2022 at 10:31 AM Soumya Koduri wrote: >> >> On 3/17/22 17:16, Ulrich Klein wrote: >>> Hi, >>> >>> My second attempt to get help with a problem I'm trying to solve for about >>> 6 month now. >>> >>> I have a Ceph 16.2.6 test cluster, used almost exclusively for providing >>> RGW/S3 service. similar to a production cluster. >>> >>> The problem I have is this: >>> A client uploads (via S3) a bunch of large files into a bucket via >>> multiparts >>> The upload(s) get interrupted and retried >>> In the end from a client's perspective all the files are visible and >>> everything looks fine. >>> But on the cluster there are many more objects in the buckets >>> Even after cleaning out the incomplete multipart uploads there are too many >>> objects >>> Even after deleting all the visible objects from the bucket there are still >>> objects in the bucket >>> I have so far found no way to get rid of those left-over objects. >>> It's screwing up space accounting and I'm afraid I'll eventually have a >>> cluster full of those lost objects. >>> The only way to clean up seems to be to copy te contents of a bucket to a >>> new bucket and delete the screwed-up bucket. But on a production system >>> that's not always a real option. >>> >>> I've found a variety of older threads that describe a similar problem. None >>> of them decribing a solution :( >>> >>> >>> >>> I can pretty easily reproduce the problem with this sequence: >>> >>> On a client system create a directory with ~30 200MB files. (On a faster >>> system I'd probably need bigger or more files) >>> tstfiles/tst01 - tst29 >>> >>> run >>> $ rclone mkdir tester:/test-bucket # creates a bucket on the test system >>> with user tester >>> Run >>> $ rclone sync -v tstfiles tester:/test-bucket/tstfiles >>> a couple of times (6-8), interrupting each one via CNTRL-C >>> Eventually let one finish. >>> >>> Now I can use s3cmd to see all the files: >>> $ s3cmd ls -lr s3://test-bucket/tstfiles >>> 2022-03-16 17:11 200M ecb28853bd18eeae185b0b12bd47333c-40 STANDARD >>> s3://test-bucket/tstfiles/tst01 >>> ... >>> 2022-03-16 17:13 200M ecb28853bd18eeae185b0b12bd47333c-40 STANDARD >>> s3://test-bucket/tstfiles/tst29 >>> >>> ... and to list incomplete uploads: >>> $ s3cmd multipart s3://test-bucket >>> s3://test-bucket/ >>> Initiated PathId >>> 2022-03-16T17:11:19.074Z s3://test-bucket/tstfiles/tst05 >>> 2~1nElF0c3uq5FnZ9cKlsnGlXKATvjr0g >>> ... >>> 2022-03-16T17:12:41.583Z s3://test-bucket/tstfiles/tst28 >>> 2~exVQUILhVSmFqWxCuAflRa4Tfq4nUQa >>> >>> I can abort the uploads with >>> $ s3cmd abortmp s3://test-bucket/tstfiles/tst05 >>> 2~1nElF0c3uq5FnZ9cKlsnGlXKATvjr0g >>> ... >> >> >> >> On the latest master, I see that these objects are deleted immediately >> post abortmp. I believe this issue may have beenn fixed as part of [1], >> backported to v16.2.7 [2]. Maybe you could try upgrading your cluster >> and recheck. >> >> >> Thanks, >> >> Soumya >> >> >> [1] https://tracker.ceph.com/issues/53222 >> >> [2] https://tracker.ceph.com/issues/53291 >> >> >> >> ___ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io >> > > > -- > > Matt Benjamin > Red Hat, Inc. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://www.redhat.com/en/technologies/storage > > tel. 734-821-5101 > fax. 734-769-8938 > cel. 734-216-5309 > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: RGW/S3 losing multipart upload objects
Ok, I’ll try again on 16.2.7. Only downside is that then I can’t use the dashboard on Safari i.e. iPads for monitoring anymore. And to make sure: From a user’s/client’s perspective the objects do disappear. Only on the Ceph/RGW-side - including accounting - they are still there and can’t be removed. Ciao, Uli > On 17. 03 2022, at 15:30, Soumya Koduri wrote: > > On 3/17/22 17:16, Ulrich Klein wrote: >> Hi, >> >> My second attempt to get help with a problem I'm trying to solve for about 6 >> month now. >> >> I have a Ceph 16.2.6 test cluster, used almost exclusively for providing >> RGW/S3 service. similar to a production cluster. >> >> The problem I have is this: >> A client uploads (via S3) a bunch of large files into a bucket via multiparts >> The upload(s) get interrupted and retried >> In the end from a client's perspective all the files are visible and >> everything looks fine. >> But on the cluster there are many more objects in the buckets >> Even after cleaning out the incomplete multipart uploads there are too many >> objects >> Even after deleting all the visible objects from the bucket there are still >> objects in the bucket >> I have so far found no way to get rid of those left-over objects. >> It's screwing up space accounting and I'm afraid I'll eventually have a >> cluster full of those lost objects. >> The only way to clean up seems to be to copy te contents of a bucket to a >> new bucket and delete the screwed-up bucket. But on a production system >> that's not always a real option. >> >> I've found a variety of older threads that describe a similar problem. None >> of them decribing a solution :( >> >> >> >> I can pretty easily reproduce the problem with this sequence: >> >> On a client system create a directory with ~30 200MB files. (On a faster >> system I'd probably need bigger or more files) >> tstfiles/tst01 - tst29 >> >> run >> $ rclone mkdir tester:/test-bucket # creates a bucket on the test system >> with user tester >> Run >> $ rclone sync -v tstfiles tester:/test-bucket/tstfiles >> a couple of times (6-8), interrupting each one via CNTRL-C >> Eventually let one finish. >> >> Now I can use s3cmd to see all the files: >> $ s3cmd ls -lr s3://test-bucket/tstfiles >> 2022-03-16 17:11 200M ecb28853bd18eeae185b0b12bd47333c-40 STANDARD >> s3://test-bucket/tstfiles/tst01 >> ... >> 2022-03-16 17:13 200M ecb28853bd18eeae185b0b12bd47333c-40 STANDARD >> s3://test-bucket/tstfiles/tst29 >> >> ... and to list incomplete uploads: >> $ s3cmd multipart s3://test-bucket >> s3://test-bucket/ >> InitiatedPathId >> 2022-03-16T17:11:19.074Z s3://test-bucket/tstfiles/tst05 >> 2~1nElF0c3uq5FnZ9cKlsnGlXKATvjr0g >> ... >> 2022-03-16T17:12:41.583Z s3://test-bucket/tstfiles/tst28 >> 2~exVQUILhVSmFqWxCuAflRa4Tfq4nUQa >> >> I can abort the uploads with >> $ s3cmd abortmp s3://test-bucket/tstfiles/tst05 >> 2~1nElF0c3uq5FnZ9cKlsnGlXKATvjr0g >> ... > > > > On the latest master, I see that these objects are deleted immediately post > abortmp. I believe this issue may have beenn fixed as part of [1], backported > to v16.2.7 [2]. Maybe you could try upgrading your cluster and recheck. > > > Thanks, > > Soumya > > > [1] https://tracker.ceph.com/issues/53222 > > [2] https://tracker.ceph.com/issues/53291 > > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] RGW/S3 losing multipart upload objects
Hi,
My second attempt to get help with a problem I'm trying to solve for about 6
month now.
I have a Ceph 16.2.6 test cluster, used almost exclusively for providing RGW/S3
service. similar to a production cluster.
The problem I have is this:
A client uploads (via S3) a bunch of large files into a bucket via multiparts
The upload(s) get interrupted and retried
In the end from a client's perspective all the files are visible and everything
looks fine.
But on the cluster there are many more objects in the buckets
Even after cleaning out the incomplete multipart uploads there are too many
objects
Even after deleting all the visible objects from the bucket there are still
objects in the bucket
I have so far found no way to get rid of those left-over objects.
It's screwing up space accounting and I'm afraid I'll eventually have a cluster
full of those lost objects.
The only way to clean up seems to be to copy te contents of a bucket to a new
bucket and delete the screwed-up bucket. But on a production system that's not
always a real option.
I've found a variety of older threads that describe a similar problem. None of
them decribing a solution :(
I can pretty easily reproduce the problem with this sequence:
On a client system create a directory with ~30 200MB files. (On a faster system
I'd probably need bigger or more files)
tstfiles/tst01 - tst29
run
$ rclone mkdir tester:/test-bucket # creates a bucket on the test system with
user tester
Run
$ rclone sync -v tstfiles tester:/test-bucket/tstfiles
a couple of times (6-8), interrupting each one via CNTRL-C
Eventually let one finish.
Now I can use s3cmd to see all the files:
$ s3cmd ls -lr s3://test-bucket/tstfiles
2022-03-16 17:11 200M ecb28853bd18eeae185b0b12bd47333c-40 STANDARD
s3://test-bucket/tstfiles/tst01
...
2022-03-16 17:13 200M ecb28853bd18eeae185b0b12bd47333c-40 STANDARD
s3://test-bucket/tstfiles/tst29
... and to list incomplete uploads:
$ s3cmd multipart s3://test-bucket
s3://test-bucket/
Initiated PathId
2022-03-16T17:11:19.074Zs3://test-bucket/tstfiles/tst05
2~1nElF0c3uq5FnZ9cKlsnGlXKATvjr0g
...
2022-03-16T17:12:41.583Zs3://test-bucket/tstfiles/tst28
2~exVQUILhVSmFqWxCuAflRa4Tfq4nUQa
I can abort the uploads with
$ s3cmd abortmp s3://test-bucket/tstfiles/tst05
2~1nElF0c3uq5FnZ9cKlsnGlXKATvjr0g
...
Afterwards s3cmd multipart s3://test-bucket doesn't list anything.
Now I delete all the objects in the bucket
$ s3 rm -r s3://test-bucket/tstfiles
.. and as client I think everything is cleaned up. No data stored to pay for.
Then I go to the Ceph cluster
In the dashboard I see 80MB in 16 objects in the bucket test-bucket
# radosgw-admin bi list --bucket test-bucket | grep idx
"idx": "_multipart_tstfiles/tst11.2~mki7cgVbg1dAh22EMZKJgmlUMveKBjx.4",
"idx": "_multipart_tstfiles/tst11.2~mki7cgVbg1dAh22EMZKJgmlUMveKBjx.5",
"idx": "_multipart_tstfiles/tst11.2~mki7cgVbg1dAh22EMZKJgmlUMveKBjx.6",
"idx": "_multipart_tstfiles/tst11.2~mki7cgVbg1dAh22EMZKJgmlUMveKBjx.7",
"idx": "_multipart_tstfiles/tst10.2~QlUi7pxm5KnQZEzPh-osjXGRe0jI-hS.3",
"idx": "_multipart_tstfiles/tst10.2~QlUi7pxm5KnQZEzPh-osjXGRe0jI-hS.5",
"idx": "_multipart_tstfiles/tst10.2~QlUi7pxm5KnQZEzPh-osjXGRe0jI-hS.6",
"idx": "_multipart_tstfiles/tst05.2~5l2zkUs__QvrD7pMevtFFhia47pdHLO.5",
"idx": "_multipart_tstfiles/tst05.2~5l2zkUs__QvrD7pMevtFFhia47pdHLO.6",
"idx": "_multipart_tstfiles/tst05.2~5l2zkUs__QvrD7pMevtFFhia47pdHLO.7",
"idx": "_multipart_tstfiles/tst05.2~5l2zkUs__QvrD7pMevtFFhia47pdHLO.8",
"idx": "_multipart_tstfiles/tst29.2~7ZrDo14zBvVBSASif1c8KmqXdwyzvzn.14",
"idx": "_multipart_tstfiles/tst29.2~7ZrDo14zBvVBSASif1c8KmqXdwyzvzn.15",
"idx": "_multipart_tstfiles/tst29.2~7ZrDo14zBvVBSASif1c8KmqXdwyzvzn.17",
"idx": "_multipart_tstfiles/tst29.2~7ZrDo14zBvVBSASif1c8KmqXdwyzvzn.19",
"idx": "_multipart_tstfiles/tst18.2~VaXq_AAM1JUb4Gtuw7zfYiKpBaYJSbs.1",
# radosgw-admin bucket stats --bucket test-bucket
{
"bucket": "test-bucket",
"num_shards": 30,
"tenant": "",
"zonegroup": "c44f5696-8f78-4c74-b657-15f611de1969",
"placement_rule": "ec21",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "3d925f45-51bd-4442-817d-acc4d77f6ba3.1925174.1",
"marker": "3d925f45-51bd-4442-817d-acc4d77f6ba3.1925174.1",
"index_type": "Normal",
"owner": "tester",
"ver":
"0#538,1#785,2#360,3#352,4#213,5#4,6#174,7#4,8#4,9#186,10#246,11#244,12#182,13#4,14#4,15#4,16#354,17#4,18#4,19#172,20#183,21#378,22#211,23#172,24#4,25#4,26#4,27#208,28#224,29#363",
"master_ver":
"0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0",
"mtime": "2022-03-16T17:02:38.754412Z",
"creation_time": "2022-03-16T17:02:38.715908Z",
[ceph-users] Re: Ceph Pacific 16.2.7 dashboard doesn't work with Safari
Replying to my self :)
It seems to be this function:
replaceBraces(e) {
==> return e.replace(/(?<=\d)\s*-\s*(?=\d)/g, "..").
replace(/\(/g, "{").
replace(/\)/g, "}").
replace(/\[/g, "{").
replace(/]/g, "}")
}
The first regular expression is what breaks it for Safari. I must admit that
I’m not sure what it does.
I removed it
replaceBraces(e) {
return e.replace(/\(/g, "{").
replace(/\)/g, "}").
replace(/\[/g, "{").
replace(/]/g, "}")
}
and now the dashboard shows again in Safari.
> On 08. 03 2022, at 10:37, Ulrich Klein wrote:
>
> Hi,
>
> I just upgraded a small test cluster on Raspberries from pacific 16.2.6 to
> 16.2.7.
> The upgrade went without major problems.
>
> But now the Ceph Dashboard doesn't work anymore in Safari.
> It complains about main..js "Line 3 invalid regular expression:
> invalid group specifier name".
> It works with Firefox and Edge on the Mac. But on an iPad I don't have that
> choice.
>
> Has anyone seen that problem, and maybe have a solution?
> Is that also a problem on “real” clusters” on x86_64?
>
> The javascript code is just one long line, i.e. unreadable.
>
> Ciao, Uli
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Ceph Pacific 16.2.7 dashboard doesn't work with Safari
Hi, I just upgraded a small test cluster on Raspberries from pacific 16.2.6 to 16.2.7. The upgrade went without major problems. But now the Ceph Dashboard doesn't work anymore in Safari. It complains about main..js "Line 3 invalid regular expression: invalid group specifier name". It works with Firefox and Edge on the Mac. But on an iPad I don't have that choice. Has anyone seen that problem, and maybe have a solution? Is that also a problem on “real” clusters” on x86_64? The javascript code is just one long line, i.e. unreadable. Ciao, Uli ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Understanding RGW multi zonegroup replication topology
Hi, Disclaimer: I'm in no way a Ceph expert. Have just been tinkering with Ceph/RGW for a larger installation for a while. My understanding is that the data between zones in a zonegroup is synced by default. And that works well, most of the time. If you, as I had to, want to restrict what data is synced, because e.g. some users' data should not be copied to a second site/zone, then there are a couple of ways to achieve that. Sync policies I discarded because I couldn't get them to work without "sync status" reporting all non-synced data as "not yet" synced, and there wasn't any per-user setting anyway. The sync settings on zones and buckets work fine. But the setting "sync_from..." on a zone is all or nothing and the setting on buckets can only be applied once the bucket exists. No setting on a per user basis. As an alternative I used two zonegroups A and B. On the primary cluster both had a zone A1 and B1, on the secondary cluster only one of them had zone A2. That way what's written to to A1 is replicated to A2. What's written to B1 is not replicated/synced. The good and bad part is that metadata (?) is synced across the zonegroups. That's good (for me) because the same user credentials work in both zonegroups, i.e. user selects the "replication" by using different S3 URLs. On the other hand it's annoying because also bucket names - if I remember correctly - are considered metadata and thus synced across all zones in the realm, which can create "interesting" effects. As I said, I'm not the expert. Just my experience so far. Ciao, Uli > On 01. 03 2022, at 01:54, Mark Selby wrote: > > I am designing a Ceph RGW multisite configuration. I have done a fair bit if > reading but still am having trouble groking the utility of having multiple > zonegroups within a realm. I know that all meta data is replicated between > zonegroups and that replication can be setup between zones across zonegroups. > I am having trouble understanding the benefits that a multi zonegroup > topology would provide. I really want to understand this as I want to make > sure that I design a topology that best meets the company’s needs. > > > > Any and all help is greatly appreciated. Thanks! > > > > -- > > Mark Selby > > mse...@voleon.com > > > > This email is subject to important conditions and disclosures that are listed > on this web page: https://voleon.com/disclaimer/. > > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: question about radosgw-admin bucket check
Hi Francois, > For the mpu's it is less important as I can fix them with some scripts. Would you mind sharing how you get rid of these left-over mpu objects? I’ve been trying to get rid of them without much success. The "radosgw-admin bucket check --bucket --fix --check-objects” I tried, but it didn’t seem to have any effect. But I’m on pacific Thanks in advance. Ciao, Uli > On 15. 02 2022, at 16:16, Scheurer François > wrote: > > Dear Ceph Experts, > > > The docu about this rgw command is a bit unclear: >radosgw-admin bucket check --bucket --fix --check-objects > > Is this command still maintained and safe to use? (we are still on nautilus) > Is it working with sharded buckets? and also in multi-site? > > I heard it will clear invalid mpu's from the index and correct wrong bucket > stats. > > The most important part for us would be to correct the bucket stats. > For the mpu's it is less important as I can fix them with some scripts. > > Thank you for your help! > > > Cheers > Francois > > > PS: > I can also reshard a bucket to correct its stats, but in multi-site this will > also require to delete this bucket in secondary zones and resync it from > scratch, which is sub-optimal ;-) > > > > > -- > > > EveryWare AG > François Scheurer > Senior Systems Engineer > Zurlindenstrasse 52a > CH-8003 Zürich > > tel: +41 44 466 60 00 > fax: +41 44 466 60 10 > mail: francois.scheu...@everyware.ch > web: http://www.everyware.ch > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Changing prometheus default alerts with cephadm
… and a „chattr +i“ on the file will preserve your changes from being over-/re-written at arbitrary points in time :) Been there… Ciao, Uli > Am 04.02.2022 um 11:05 schrieb Eugen Block : > > Hi, > > you should be able to change in the config file: > > /var/lib/ceph//prometheus.ses7-host1/etc/prometheus/alerting/ceph_alerts.yml > > and restart the containers. > > Regards, > Eugen > > > Zitat von Manuel Holtgrewe : > >> Dear all, >> >> I wonder how I can adjust the default alerts generated by prometheus >> when having deployed with cephadm? I need to adjust the network >> package drop thresholds a bit. >> >> Best wishes, >> Manuel >> ___ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io > > > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Pacific 16.2.6: Trying to get an RGW running for a scond zonegroup in an existing realm
Well, looks like not many people have tried this. And to me it looks like a bug/omission in "ceph orch apply rgw". After digging through the setup I figured out that the unit.run file for the new rgw.zone21 process/container doesn't get the --rgw-zonegroup (or --rgw-region) parameter for radosgw. Ceph orch apply rgw doesn’t seem to forward those parameters. Once I added "--rgw-zone=zone21 --rgw-zonegroup=zg2 --rgw-region=zg2" to the command line in the unit.run file and restarted the service on that node, the process came up and seems to work correctly(?) The behavior is somewhat unexpected to me: Buckets created in any of my 3 zones and their contents get synced to all other zones, regardless of zonegroups. Only if I create buckets/objects in a zone of one zonegroup I can't delete them in another zonegroup. I get an 301 without redirect URI. Not sure, if that's a bug or a feature. Ciao, Uli > On 01. 02 2022, at 18:09, Ulrich Klein wrote: > > Hi, > > Maybe someone who knows the commands can help me with my problem > > I have a small 6-node cluster running with 16.2.6 using cephadm and another > one with the same versions. > Both clusters are exlusively used for RGW/S3. > I have a realm myrealm, a zonegroup zg1 and zones on both cluster, one is > zone1, master, the other zone2 > I have RGWs running in both clusters behind HAproxy deployed via ceph orch > and labels and they work just fine. > > Now I want to add a second zonegroup zg2 on the first cluster, with a zone > zone21, master, and no secondary zone. > And, to use that zone I want to deploy another RGW for it. > > So, I did mostly the same as for the first zonegroup (nceph03 is a node in > the first cluster, the label rgwnosync is set and the keys are correct): > > radosgw-admin zonegroup create --rgw-zonegroup=zg2 > --endpoints=http://nceph03.example.com:8080 --rgw-realm=myrealm > radosgw-admin zone create --rgw-zonegroup=zg2 --rgw-zone=zone21 --master > --endpoints=http://nceph03.example.com:8080 > > radosgw-admin zone modify --rgw-zonegroup=zg2 --rgw-zone=zone21 > --access-key=ackey --secret=sekey > > radosgw-admin zonegroup add --rgw-zonegroup=zg2 --rgw-zone=zone21 > > radosgw-admin period update --commit > > Everything looks as expected (by me) up to this point. But then I try to add > an RGW process for that zone: > > ceph orch apply rgw zone21 --realm=myrealm --rgw-zonegroup=zg2 --zone=zone21 > '--placement=label:rgwnosync count-per-host:1' --port=8080 > > The process comes up and dies spitting out the messages in the log: > > ... 1 rgw main: Cannot find zone id=f9151746-09e9-4854-9159-9df35a3457bf > (name=zone21), switching to local zonegroup configuration > ... > ... > ... -1 rgw main: Cannot find zone id=f9151746-09e9-4854-9159-9df35a3457bf > (name=zone21) > ... > ... > ... 0 rgw main: ERROR: failed to start notify service ((22) Invalid argument > ... > ... > ... 0 rgw main: ERROR: failed to init services (ret=(22) Invalid argument) > ... > ... > ... -1 Couldn't init storage provider (RADOS) > > Somehow it looks like radosgw is looking for that zone in the default master > zonegroup zg1 and I can't figure out how to tell it to use zg2. > > I've tried variations of the "ceph orch apply rgw" command with and w/o > zonegroup, with and w/o realm=, and a whole lot more, but nothing seems to > make any difference. > Even tried putting stuff in various ceph.conf files (although I already know > it's ignored) > > Do I miss some step or command or setting or ... > > Any help would be appreciated. > > Ciao, Uli > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Pacific 16.2.6: Trying to get an RGW running for a scond zonegroup in an existing realm
Hi, Maybe someone who knows the commands can help me with my problem I have a small 6-node cluster running with 16.2.6 using cephadm and another one with the same versions. Both clusters are exlusively used for RGW/S3. I have a realm myrealm, a zonegroup zg1 and zones on both cluster, one is zone1, master, the other zone2 I have RGWs running in both clusters behind HAproxy deployed via ceph orch and labels and they work just fine. Now I want to add a second zonegroup zg2 on the first cluster, with a zone zone21, master, and no secondary zone. And, to use that zone I want to deploy another RGW for it. So, I did mostly the same as for the first zonegroup (nceph03 is a node in the first cluster, the label rgwnosync is set and the keys are correct): radosgw-admin zonegroup create --rgw-zonegroup=zg2 --endpoints=http://nceph03.example.com:8080 --rgw-realm=myrealm radosgw-admin zone create --rgw-zonegroup=zg2 --rgw-zone=zone21 --master --endpoints=http://nceph03.example.com:8080 radosgw-admin zone modify --rgw-zonegroup=zg2 --rgw-zone=zone21 --access-key=ackey --secret=sekey radosgw-admin zonegroup add --rgw-zonegroup=zg2 --rgw-zone=zone21 radosgw-admin period update --commit Everything looks as expected (by me) up to this point. But then I try to add an RGW process for that zone: ceph orch apply rgw zone21 --realm=myrealm --rgw-zonegroup=zg2 --zone=zone21 '--placement=label:rgwnosync count-per-host:1' --port=8080 The process comes up and dies spitting out the messages in the log: ... 1 rgw main: Cannot find zone id=f9151746-09e9-4854-9159-9df35a3457bf (name=zone21), switching to local zonegroup configuration ... ... ... -1 rgw main: Cannot find zone id=f9151746-09e9-4854-9159-9df35a3457bf (name=zone21) ... ... ... 0 rgw main: ERROR: failed to start notify service ((22) Invalid argument ... ... ... 0 rgw main: ERROR: failed to init services (ret=(22) Invalid argument) ... ... ... -1 Couldn't init storage provider (RADOS) Somehow it looks like radosgw is looking for that zone in the default master zonegroup zg1 and I can't figure out how to tell it to use zg2. I've tried variations of the "ceph orch apply rgw" command with and w/o zonegroup, with and w/o realm=, and a whole lot more, but nothing seems to make any difference. Even tried putting stuff in various ceph.conf files (although I already know it's ignored) Do I miss some step or command or setting or ... Any help would be appreciated. Ciao, Uli ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Question about multi-site sync policies
Hi, My first question on this list …. 2nd attempt because the first one didn’t make it (I hope)? I'm trying out RGW multi-site sync policies. I have a test/PoC setup with 16.2.6 using cephadm (which, by the way, I DO like) I only use RGW/S3 There is one realm (myrealm), one zonegroup (myzg) and 3 zones: Zone A, zone B and zone DR Zone A and B are on the same ceph instance with different rgw processes and pools, zone DR is on a second cluster, in case that matters. Without any sync policies all data is synced perfectly fine between all zones What I want to see is: Data from zone A is synced directionally to zone DR, but not to zone B Data from zone B is not synced anywhere Data from zone DR is not synced back/anywhere i.e. clients writing to zone A get their data "backed up" to zone DR, while clients writng to zone B don't get their data "backed" up to a second zone. Clients don't have access to zone DR. I used sync policies this way: radosgw-admin sync group create --group-id=drsync --status=allowed radosgw-admin sync group flow create --group-id=drsync --flow-id=a2dr --flow-type=directional --source-zone=a --dest-zone=dr radosgw-admin sync group pipe create --group-id=drsync --pipe-id=allbuck --source-zones='*' --source-bucket='*' --dest-zones='*' --dest-bucket='*' radosgw-admin sync group modify --group-id=drsync --status=enabled radosgw-admin period update --commit Now, from a data-in-buckets perspective all looks fine Add data to bucket buck on A -> appears on buck on DR, but not in B Add data to bucket buck on B -> doesn't appear in A or DR Same for all other combinations, just as I want it. BUT radosgw-admin sync status with --rgw-zone=A or DR or B after adding or removing data always shows some "data is behind on X shards", apparently for all the shards that are - intentionally - not synced. Those “behind” shards accumulate over time and never go away again. Is that just annoying but normal? Or is that a bug? Or is my configuration just "bad" and could be changed so I don't have those sort-of errors with the sync status. When I used bucket level sync policies (only sync certain buckets from zone A to DR, no zone B) I had iirc the same effect. What I'm really trying to achieve is something like "user sync policies", i.e. - User X data should be synced from A to DR - User Y data should only stay in A I’m trying to emulate that by using the existing/documented sync policies. User X gets URL for zone A, user Y gets URL for zone B (or with bucket level policies both get URL for zone A and for user X syncing is turned on “on demand” for certain existing buckets - inconvenient) Best would be if I could flip a flag and change the behaviour per user :) And even better if it was easy to understand, i.e. if user's sync-flag is turned on all his data is synced, not just new data. If user's sync-flag is turned off, all his data is removed from the DR zone. Thanks for any input :) Ciao, Uli ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
