[ceph-users] Re: S3website range requests - possible issue
Hello Casey, Thanks a lot for that. I’ve forgot to mention that in my previous message that I was able to trigger the prefetch by header bytes=1-10 You can see the the read 1~10 in the osd logs I’ve sent here - https://pastebin.com/nGQw4ugd Which is wierd as it seems that it is not the same you were able to replicate. Ondrej > On 22. 9. 2023, at 21:52, Casey Bodley wrote: > > hey Ondrej, > > thanks for creating the tracker issue > https://tracker.ceph.com/issues/62938. i added a comment there, and > opened a fix in https://github.com/ceph/ceph/pull/53602 for the only > issue i was able to identify > > On Wed, Sep 20, 2023 at 9:20 PM Ondřej Kukla wrote: >> >> I was checking the tracker again and I found already fixed issue that seems >> to be connected with this issue. >> >> https://tracker.ceph.com/issues/44508 >> >> Here is the PR that fixes it https://github.com/ceph/ceph/pull/33807 >> >> What I’m still not understanding is why this is only happening when using >> s3website api. >> >> Is there someone who could shed some light on this? >> >> Regards, >> >> Ondrej >> ___ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: S3website range requests - possible issue
that first "read 0~4194304" is probably what i fixed in https://github.com/ceph/ceph/pull/53602, but it's hard to tell from osd log where these osd ops are coming from. why are there several [read 1~10] requests after that? the rgw log would be more useful for debugging, with --debug-rgw=20 and --debug-ms=1 to show the osd ops/replies On Fri, Sep 22, 2023 at 4:00 PM Ondřej Kukla wrote: > > Hello Casey, > > Thanks a lot for that. > > I’ve forgot to mention that in my previous message that I was able to trigger > the prefetch by header bytes=1-10 > > You can see the the read 1~10 in the osd logs I’ve sent here - > https://pastebin.com/nGQw4ugd > > Which is wierd as it seems that it is not the same you were able to replicate. > > Ondrej > > On 22. 9. 2023, at 21:52, Casey Bodley wrote: > > hey Ondrej, > > thanks for creating the tracker issue > https://tracker.ceph.com/issues/62938. i added a comment there, and > opened a fix in https://github.com/ceph/ceph/pull/53602 for the only > issue i was able to identify > > On Wed, Sep 20, 2023 at 9:20 PM Ondřej Kukla wrote: > > > I was checking the tracker again and I found already fixed issue that seems > to be connected with this issue. > > https://tracker.ceph.com/issues/44508 > > Here is the PR that fixes it https://github.com/ceph/ceph/pull/33807 > > What I’m still not understanding is why this is only happening when using > s3website api. > > Is there someone who could shed some light on this? > > Regards, > > Ondrej > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: S3website range requests - possible issue
Hello Casey, Thanks a lot for that. I’ve forgot to mention that in my previous message that I was able to trigger the prefetch by header bytes=1-10 You can see the the read 1~10 in the osd logs I’ve sent here - https://pastebin.com/nGQw4ugd Which is wierd as it seems that it is not the same you were able to replicate. Ondrej > On 22. 9. 2023, at 21:52, Casey Bodley wrote: > > hey Ondrej, > > thanks for creating the tracker issue > https://tracker.ceph.com/issues/62938. i added a comment there, and > opened a fix in https://github.com/ceph/ceph/pull/53602 for the only > issue i was able to identify > > On Wed, Sep 20, 2023 at 9:20 PM Ondřej Kukla wrote: >> >> I was checking the tracker again and I found already fixed issue that seems >> to be connected with this issue. >> >> https://tracker.ceph.com/issues/44508 >> >> Here is the PR that fixes it https://github.com/ceph/ceph/pull/33807 >> >> What I’m still not understanding is why this is only happening when using >> s3website api. >> >> Is there someone who could shed some light on this? >> >> Regards, >> >> Ondrej >> ___ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: S3website range requests - possible issue
hey Ondrej, thanks for creating the tracker issue https://tracker.ceph.com/issues/62938. i added a comment there, and opened a fix in https://github.com/ceph/ceph/pull/53602 for the only issue i was able to identify On Wed, Sep 20, 2023 at 9:20 PM Ondřej Kukla wrote: > > I was checking the tracker again and I found already fixed issue that seems > to be connected with this issue. > > https://tracker.ceph.com/issues/44508 > > Here is the PR that fixes it https://github.com/ceph/ceph/pull/33807 > > What I’m still not understanding is why this is only happening when using > s3website api. > > Is there someone who could shed some light on this? > > Regards, > > Ondrej > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: S3website range requests - possible issue
I was checking the tracker again and I found already fixed issue that seems to be connected with this issue. https://tracker.ceph.com/issues/44508 Here is the PR that fixes it https://github.com/ceph/ceph/pull/33807 What I’m still not understanding is why this is only happening when using s3website api. Is there someone who could shed some light on this? Regards, Ondrej ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: S3website range requests - possible issue
When checking the RGW logs I can confirm that it is in fact the same issue as
the one in the issue.
2023-09-20T12:52:06.670+ 7f216d702700 1 -- xxx.xxx.58.15:0/758879303 -->
[v2:xxx.xxx.58.2:6816/8556,v1:xxx.xxx.58.2:6817/8556] -- osd_op(unknown.0.0:238
18.651
18:8a75a7b2:::39078a70-7768-48c8-96a5-1e13ced83b5b.58017020.1_videos%2f7.mp4:head
[getxattrs,stat,read 0~4194304] snapc 0=[]
ondisk+read+known_if_redirected+supports_pool_eio e60419) v8 -- 0x7f21dc00a420
con 0x7f21dc007820
You can find the OSD part of the log here - https://pastebin.com/nGQw4ugd
For the record the version of the cluster when I’m able to replicate this is
ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
Regards,
Ondrej
> On 20. 9. 2023, at 11:25, Ondřej Kukla wrote:
>
> I was checking the tracker again and I found already fixed issue that seems
> to be connected with this issue.
>
> https://tracker.ceph.com/issues/44508
>
> Here is the PR that fixes it https://github.com/ceph/ceph/pull/33807
>
> What I’m still not understanding is why this is only happening when using
> s3website api.
>
> Is there someone who could shed some light on this?
>
> Regards,
>
> Ondrej
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
> On 19. 9. 2023, at 10:49, Ondřej Kukla wrote:
>
> Hello,
>
> In our deployment we are using the mix of s3 and s3website RGW. I’ve noticed
> strange behaviour when sending range requests to the s3website RGWs that I’m
> not able to replicate on the s3 ones.
>
> I’ve created a simple wrk LUA script to test sending range requests on tiny
> ranges so the issue is easily seen.
>
> When sending these requests against s3 RGW I can see that the amount of data
> read from Ceph is ± equivalent to what the RGW sends to the client. This
> change very dramatically when I’m doing the same test against s3website RGW.
> The read from Ceph is huge (3Gb/s compared to ~22Mb/s on s3 RGW) I seems to
> me like the RGW is reading the whole files and then sending just the range
> which is different then what s3 does.
>
> I do not understand why would s3website need to read that much from Ceph and
> I believe this is a bug - I was looking through the tracker and wasn’t able
> to find anything related to s3website and range requests.
>
> Did anyone else noticed this issue?
>
> You can replicate it by running this wrk command wrk -t56 -c500 -d5m
> http://${rgwipaddress}:8080/${bucket}/videos/ -s wrk-range-small.lua
>
> wrk script
>
> -- Initialize the pseudo random number generator
> math.randomseed( os.time())
> math.random(); math.random(); math.random()
>
> i = 1
>
> function request()
>if i == 8
>then
>i = 1
>end
>
>local nrangefrom = math.random()
>local nrangeto = math.random(100)
>local path = wrk.path
>url = path..i..".mp4"
>wrk.headers["Range"] = nrangefrom.."-"..nrangeto
>i = i+1
>return wrk.format(nil, url)
> end
>
> Kind regards,
>
> Ondrej
> ___
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
___
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: S3website range requests - possible issue
I was checking the tracker again and I found already fixed issue that seems to be connected with this issue. https://tracker.ceph.com/issues/44508 Here is the PR that fixes it https://github.com/ceph/ceph/pull/33807 What I’m still not understanding is why this is only happening when using s3website api. Is there someone who could shed some light on this? Regards, Ondrej ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
