Hi,
Can you please explain what S3 operations you perform on the RGW?
I tried the 2nd script with "make bucket" and "put object", and got:
2024-04-09T13:54:00.608+ 7f93b3987640 20 Lua INFO: bucket operation
logs:
2024-04-09T13:54:00.608+ 7f93b3987640 20 Lua INFO: Name: fish
Hi GM,
sorry for the late reply. anmyway, you are right.
in "quincy" (v17) only the owner of the bucket was allowed to set a
notification on the bucket.
in "reef" (v18) we fixed that, so that we follow the permissions set on the
bucket.
you can use the "s3PutBucketNotification" policy on the
Hi Manuel,
I looked into the nautilus documentation [1]. could not find anything about
amqps there.
Yuval
[1] https://docs.ceph.com/en/nautilus/radosgw/notifications/#create-a-topic
On Mon, Mar 11, 2024 at 12:50 AM Manuel Negron wrote:
> Hello, ive been trying to setup bucket notifications
Lokendra and Kushagra,
We don't have such an enhancement on the roadmap. Would think of 2 options:
(1) implement the special logic using lua scripting. We have an example on
how to send notifications to a NATS broker from lua [1]. you can easily
adjust that to kafka. the 2 main drawbacks with this
Hi Rok,
Please have a look here:
https://github.com/ceph/ceph/tree/main/examples/rgw/boto3
It has both CLI and python examples. Let me know if you need more
information.
Yuval
On Tue, Nov 28, 2023 at 10:21 PM Rok Jaklič wrote:
> Hi,
>
> I would like to get info if the bucket or object got
Hi Vahideh,
Lua scripting was added in pacific. Did you try uploading that file to a
"pacific" RGW?
What is failing there?
Yuval
On Mon, Oct 30, 2023 at 5:04 PM Vahideh Alinouri
wrote:
> Dear Ceph Users,
>
> I am requesting the backporting changes related to the nats_adapter.lua.
> This
Hi Stephan,
Currently only some of the fields in the HTTP header are exposed, and,
sadly, "REMORE_ADDR" is not.
Created the following PR: https://github.com/ceph/ceph/pull/54211 to expose
all HTTP header fields to lua (note that you can also change them if you
want).
The fix is simple, so I would
Hi Artem,
Thanks for noting that. Submitted a doc fix here:
https://github.com/ceph/ceph/pull/54136
Yuval
On Sat, Oct 21, 2023 at 3:04 AM Zac Dover wrote:
> Artem,
>
> Thanks for this, but I don't quite understand what should be changed. Your
> suggested change is present in the documentation
Hi Ondřej,
As you said, you can't add a new header in the response, but maybe you can
override one of the existing response fields?
e.g. Request.Response.Message
let me know if that works for you?
Yuval
On Mon, Sep 4, 2023 at 1:33 PM Ondřej Kukla wrote:
> Hello,
>
> We have a RGW setup that
Hi Daniel,
I assume you are using persistent topics?
We had a bug that we recently fixed, where the queue of a persistent
notification was not deleted when the deletion was done from radosgw-admin.
see: https://tracker.ceph.com/issues/61311
However, there are no plans to backport that to pacific.
On Thu, Jun 15, 2023 at 4:48 PM Casey Bodley wrote:
> On Thu, Jun 15, 2023 at 7:23 AM Christian Rohmann
> wrote:
> >
> > Hello Ceph-Users,
> >
> > context or motivation of my question is S3 bucket policies and other
> > cases using the source IP address as condition.
> >
> > I was wondering if
, there are several race conditions that happened with kafka and
persistent notifications and also exists for amqp. Will be fixing that as
part of: https://tracker.ceph.com/issues/61639
Yuval
On Sun, Jun 11, 2023 at 11:48 AM Yuval Lifshitz wrote:
> Hi Stefan,
> Thanks for the inputs. Replied
Hi Stefan,
Thanks for the inputs. Replied inline
On Fri, Jun 9, 2023 at 6:53 PM Stefan Reuter
wrote:
> Hi Yuval,
>
> Thanks for having a look at bucket notifications and collecting
> feedback. I also see potential for improvement in the area of bucket
> notifications.
>
> We have observed
Dear Community,
I would like to collect your feedback on this issue. This is a followup
from a discussion that started in the RGW refactoring meeting on 31-May-23
(thanks @Krunal Chheda for bringing up this
topic!).
Currently persistent notifications are retried indefinitely.
The only limiting
you are right, the object's time has an issue. opened a tracker:
https://tracker.ceph.com/issues/61310
to get the bucket owner, use: Request.Bucket.User.Id
you can also get that from the user: Request.ObjectOwner.User.Id
the "tenant" will be set only if you defined a tenant in your system.
and
I think that information on the object exists only in the postRequest
context (because only then we accessed the object).
to get the size of the object in the preRequest context you need to take it
from "Request.ContentLength".
see:
Hi,
The RGWDebugLog() function is writing debug logs at level 20, so you have
to make sure that debug_rgw = 20
Also, if Request.HTTP.StorageClass does not exist in the message (is "nil")
the string concat operator would fail, and the script would exit. So, it is
probably better to check that in
pull/51422
On Wed, May 10, 2023 at 12:20 PM Yuval Lifshitz wrote:
> thanks Thomas!
> opened this tracker: https://tracker.ceph.com/issues/59697 should cover
> the missing dependencies for luarocks on the centos8 container (feel free
> to add anything missing there...).
> still tryin
he pre request context.
>
> Cheers,
> Tom
>
> On Fri, 28 Apr 2023 at 15:19, Thomas Bennett wrote:
>
>> Hey Yuval,
>>
>> No problem. It was interesting to me to figure out how it all fits
>> together and works. Thanks for opening an issue on the tracker.
&g
t if you can backport this fix to Quincy.
>
> Thank you!
> Vladimir.
>
> -Original Message-----
> *From*: Yuval Lifshitz >
> *To*: Ondřej Kukla <%3d%3futf-8%3fq%3fond%3dc5%3d99ej%3f%3d%20kukla%20%3cond...@kuuk.la%3e>>
> *Cc*: ceph-users@ceph.io
> *Subject*
Hi Ondřej,
Greater to hear that you use lua. You are right, this field has become
writable only in reef.
I can backport the fix to quincy, so that you can use it in the next quincy
release (not sure when it is).
A better option would be to allow setting the failure from lua - but this
would be a
Hi Istvan,
Looks like you are using user/password and SSL on the communication
channels between RGW and the Kafka broker.
Maybe the issue is around the certificate? could you please increase RGW
debug logs to 20 and see if there are any kafka related errors there?
Yuval
On Tue, Apr 25, 2023 at
Hi Thomas,
Thanks for the detailed info!
RGW lua scripting was never tested in a cephadm deployment :-(
Opened a tracker: https://tracker.ceph.com/issues/59574 to make sure this
would work out of the box.
Yuval
On Tue, Apr 25, 2023 at 10:25 PM Thomas Bennett wrote:
> Hi ceph users,
>
> I've
- it's the CopyFrom is causing an issue :)
>
> I've got some other example Lua scripts working now.
>
> Kind regards,
> Thomas
>
>
>
> On Sun, 9 Apr 2023 at 11:41, Yuval Lifshitz wrote:
>
>> Hi Thomas,
>> I think you found a crash when using the
Hi Thomas,
I think you found a crash when using the lua "CopyFrom" field.
Opened a tracker: https://tracker.ceph.com/issues/59381
Will fix SASP and keep you updated.
Yuval
On Wed, Apr 5, 2023 at 6:58 PM Thomas Bennett wrote:
> Hi,
>
> We're currently testing out lua scripting in the Ceph
Hi Ulrich,
You are correct, there is no specific authorization needed for creating
topics. User authentication is done as with any other REST call, but there
are no restrictions and any user can create a topic.
Would probably make sense to limit that ability. Would appreciate if you
could open a
Hi Steven,
When using synchronous (=non-persistent) notifications, the overall rate is
dependent on the latency between the RGW and the endpoint to which you are
sending the notifications. The protocols for sending the
notifications (kafka/amqp) are using batches and are usually very
efficient.
Hi Yee Ting,
The RGW replied with 400 "Invalid Input". can you please send the RGW logs?
They should hold more information about the nature of the problem.
Regarding the certificate file, it is not loaded when configured, but only
when the connection to the broker is created and the first
Thilo and Stefan,
Thanks for your help!
Both fixes are now under review;
https://github.com/ceph/ceph/pull/48875
https://github.com/ceph/ceph/pull/48877
it might take a little longer, but the fixes should also be backported to
quincy.
Yuval
On Mon, Nov 14, 2022 at 11:50 AM Yuval Lifshitz wrote
22 at 11:15 AM Thilo-Alexander Ginkel
wrote:
> Hello Yuval,
>
> On Sun, Nov 13, 2022 at 10:24 AM Yuval Lifshitz
> wrote:
>
>> The issue was indeed fixed (both with regard to filtering and sending the
>> metadata) in the tracker that you mentioned but was broken by ano
Hi Thilo,
The issue was indeed fixed (both with regard to filtering and sending the
metadata) in the tracker that you mentioned but was broken by another fix (
https://github.com/ceph/ceph/pull/39192) :-(
I will re-apply the fix (and make sure we regression tests, so it does not
break again).
Hi David,
Using bucket notifications sounds like the right approach here. One option
with the policy would be to move the suspicious objects to a quarantine
bucket, so they are not lost in case of false positives, but not harmful
either.
I did something similar, but with inline processing using
> Sr Linux Administrator, The Voleon Group
>
> mse...@voleon.com
>
>
>
> This email is subject to important conditions and disclosures that are
> listed on this web page: https://voleon.com/disclaimer/.
>
>
>
>
>
> *From: *Yuval Lifshitz
> *Date: *Thursday
raub wrote:
> > Can maybe leverage one of the other calls to check for upload
> completion:
> > list multipart uploads and/or list parts. The latter should work if
> you
> > have the upload id at hand.
> >
> > Yehuda
> >
> > O
> This email is subject to important conditions and disclosures that are
> listed on this web page: https://voleon.com/disclaimer/.
>
>
>
>
>
> *From: *Yuval Lifshitz
> *Date: *Monday, July 18, 2022 at 9:33 PM
> *To: *Mark Selby
> *Cc: *"ceph-users@ceph.io&q
Hi Mark,
It is in quincy but wasn't backported to pacific yet.
I can do this backport, but I'm not sure when is the next pacific release.
Yuval
On Tue, Jul 19, 2022 at 5:04 AM Mark Selby wrote:
> I am trying to use RGW Bucket Notifications to trigger events on object
> creation and have into a
Hi Koldo,
this might be related to the containerized deployment.
the error code (-10) is returned when we cannot find the "luarocks" binary.
assuming it is installed on the host (just check: "luarocks --version"), it
might not be accessible from inside the RGW container.
if this is the case, can
Hi Koldo,
Great to hear that you are using Lua scripting!
Some of the fields that you use: Request.Object and Request.Bucket are
marked as "optional" in the table. Meaning, that for some operations they
may have nil value. Ideally, the script would check that, either by
checking Request.RGWOp or
Hi Alex,
I also seemed to miss your email :-)
On Mon, Oct 18, 2021 at 11:32 AM Alex Kershaw
wrote:
> Hi Yuval,
>
> Apologies - I'm having some trouble with my microsoft spam filter and I'm
> not sure this email reached you. If it did please excuse the duplicate.
> This is in response to:
>
r "acks" that serves the same purpose
>
> If pubsub is likely to be deprecated, we'll need to start moving fast.
> What's the latest thinking on this?
>
>
we are not going to deprecate that until we have an alternative solution in
ceph.
> Cheers,
>
> Dave
>
&
Hi Alex,
How many overall zones do you have configured in the system?
We have an issue with pubsub based notifications, where we may get as many
as (#zone-1) duplicates per object.
This, however, won't explain 13 events per object.
Did you verify that these are indeed the same events? For the
ansibleuser@ceprgw01 ~]$ ceph --version
> ceph version 14.2.11-181.el7cp (68fea1005601531fe60d2979c56ea63bc073c84f)
> nautilus (stable)
> [ansibleuser@ceprgw01 ~]$ aws --version
> aws-cli/2.0.30 Python/3.7.3 Linux/3.10.0-1160.25.1.el7.x86_64
> botocore/2.0.0dev34
> [ansibleuser@ceprgw01 ~]$
>
> Best
change":
> "topic_logs"}, set-topic-attributes
>
> I am using Ceph 4.2.
>
> [root@ceprgw01 ~]# aws --version
> aws-cli/1.14.28 Python/2.7.5 Linux/3.10.0-1160.25.1.el7.x86_64 botocore/1.8.35
>
>
>
> Best regards,
> Sanjeev
>
> -
Hi Istvan,
Hope this is still relevant... but you may want to have a look at this
example:
https://github.com/ceph/ceph/blob/master/examples/lua/prometheus_adapter.lua
https://github.com/ceph/ceph/blob/master/examples/lua/prometheus_adapter.md
where we log RGW object sizes to Prometheus.
would
ou can use the example from the AWS
docs).
if the topics and notifications are more static, you can create them up
front using the AWS cli, and then you don't need to do anything in your
Java code.
> Best regards,
> Sanjeev
>
>
> --
> *From:* Yuval
Hi Sanjeev,
Welcome to the Ceph community!
Which protocol do you intend to use in ActiveMQ?
If you want to use AMQP1.0, you will have to wait, as this is still not
officially supported [1].
Currently, we support AMQP0.9.1, Kafka, and HTTP.
As for the more general question.
To make bucket
b.com/ceph/ceph/tree/master/examples/boto3#aws-cli
>
> Regards
> Daniel
>
>
> On Tue, 22 Jun 2021 at 18:34, Yuval Lifshitz wrote:
>
>> Hi Daniel,
>> You are correct, currently, only v2 auth is supported for topic
>> management.
>> (tracked here: https
On Wed, Jun 23, 2021 at 6:39 PM Daniel Iwan wrote:
> this looks like a bug, the topic should be created in the right tenant.
>> please submit a tracker for that.
>>
>
> Thank you for confirming.
> Created here https://tracker.ceph.com/issues/51331
>
>
thanks
> yes. topics are owned by the
On Wed, Jun 23, 2021 at 2:21 PM Daniel Iwan wrote:
> Hi
>
> I'm using Ceph Pacific 16.2.1
>
> I'm creating a topic as a user which belongs to a non-default tenant.
> I'm using AWS CLI 2 with v3 authentication enabled
>
> aws --profile=ceph-myprofile --endpoint=$HOST_S3_API --region="" sns
>
Hi Daniel,
You are correct, currently, only v2 auth is supported for topic management.
(tracked here: https://tracker.ceph.com/issues/50039)
It should be fixed soon but may take some time before it is backported to
Pacific (will keep the list posted).
Best Regards,
Yuval
On Tue, Jun 22, 2021
> Agoda Services Co., Ltd.
> e: istvan.sz...@agoda.com
> -------
>
> -Original Message-
> From: Yuval Lifshitz
> Sent: Tuesday, April 27, 2021 11:49 PM
> To: Szabo, Istvan (A
> Istvan Szabo
> Senior Infrastructure Engineer
> ---
> Agoda Services Co., Ltd.
> e: istvan.sz...@agoda.com
> ---
>
>
>
> *From:* Yuval Lifshitz
> *Sent:* Wednesday, Ap
Hi Istvan,
Can you please share the relevant part for the radosgw log, indicating
which input was invalid?
The only way I managed to reproduce that error is by sending the request to
a non-HTTPS radosgw (which does not seem to be your case). In such a case
it replies with "InvalidInput" because we
the "Filter" tag is optional in the XML, so I don't think this is the issue.
Note that the bucket and topic have to exist when you create the
notification.
Can you try creating the notification using the AWS CLI tool instead of
CURL?
You can see examples here:
Hi David,
Don't have any good idea for "octopus" (other than ops log), but you can do
that (and more) in "pacific" using lua scripting on the RGW:
https://docs.ceph.com/en/pacific/radosgw/lua-scripting/
Yuval
On Thu, Apr 1, 2021 at 7:11 PM David Orman wrote:
> Hi,
>
> Is there any way to log
and Wales, No.11266143. Funded by UK Research and Innovation through
> the Engineering and Physical Sciences Research Council.
>
> On 29 Jan 2021, at 09:14, Yuval Lifshitz wrote:
>
>
>
> On Fri, Jan 29, 2021 at 9:18 AM Schoonjans, Tom (RFI,RAL,-) <
> tom.schoonj...@rfi.a
179810 Company Limited by Guarantee Registered in England
> and Wales, No.11266143. Funded by UK Research and Innovation through
> the Engineering and Physical Sciences Research Council.
>
> On 28 Jan 2021, at 18:16, Yuval Lifshitz wrote:
>
>
>
> On Thu, Jan 28, 2021 at 7
Institute
> Harwell Science & Innovation Campus
> Didcot
> Oxfordshire
> OX11 0FA
> United Kingdom
>
> https://www.rfi.ac.uk
>
> The Rosalind Franklin Institute is a registered charity in England and
> Wales, No. 1179810 Company Limited by Guarantee Registered in Engl
United Kingdom
>
> https://www.rfi.ac.uk
>
> The Rosalind Franklin Institute is a registered charity in England and
> Wales, No. 1179810 Company Limited by Guarantee Registered in England
> and Wales, No.11266143. Funded by UK Research and Innovation through
> the Engineerin
No.11266143. Funded by UK Research and Innovation through
> the Engineering and Physical Sciences Research Council.
>
> On 26 Jan 2021, at 19:37, Yuval Lifshitz wrote:
>
> Hi Tom,
> Did you create the exchange in rabbitmq? The RGW does not create it and
> assume it is
Hi Tom,
Did you create the exchange in rabbitmq? The RGW does not create it and
assume it is already created?
Could you increase the log level in RGW and see if there are more log
messages that have "AMQP" in them?
Thanks,
Yuval
On Tue, Jan 26, 2021 at 7:33 PM Byrne, Thomas (STFC,RAL,SC) <
Thanks Anantha!
On Sat, Nov 28, 2020 at 2:37 AM Adiga, Anantha
wrote:
> Hi Yuval,
>
>
>
> Your questions have been added.
>
>
>
> Thank you,
>
> Anantha
>
>
>
> *From:* Yuval Lifshitz
> *Sent:* Wednesday, November 25, 2020 6:30 AM
> *T
Hi Mike,
Could we add more questions on RGW usecases and functionality adoption?
For instance:
bucket notifications:
* do you use "bucket notifications"?
* if so, which endpoint do you use: kafka, amqp, http?
* which other endpoints would you like to see there?
sync modules:
* do you use the
Hi Krasaev,
Thanks for pointing out this issue! This is currently under review here:
[1], and tracked here: [2].
Once merged, the fix would be available on the master development branch,
and the plan is to backport the fix to Octopus in the future.
Yuval
[1]
Dear Community,
Since Nautilus, we have 2 mechanisms for notifying 3rd parties on changes
in buckets and objects: "bucket notifications" [1] and "pubsub" [2].
In "bucket notifications" (="push mode") the events are sent from the RGW
to an external entity (kafka, rabbitmq etc.), while in "pubsub"
Hi Dustin,
This is an issue that will happen regardless of pubsub configuration.
Tracked here: https://tracker.ceph.com/issues/45816
Yuval
On Sun, May 31, 2020 at 11:00 AM Yuval Lifshitz wrote:
> Hi Dustin,
> Did you create a pubsub zone [1] in your cluster?
> (note that this is
Hi Dustin,
Did you create a pubsub zone [1] in your cluster?
(note that this is currently not supported in rook, so it had to be done
manually).
Yuval
[1]
https://docs.ceph.com/docs/master/radosgw/pubsub-module/#pubsub-zone-configuration
On Fri, May 29, 2020 at 7:16 PM Dustin Guerrero
Hi Alexis,
Which version are you using? There was a bug in 14.2.8 with topic creation.
See: https://tracker.ceph.com/issues/44614
Also note that for topic operations we are using a different signature
version (ver3) than the one used by default by the aws CLI tools.
Please see here:
On Thu, Apr 23, 2020 at 8:28 AM Andreas Unterkircher
wrote:
> Dear Yuval!
>
> > The message format you tried to use is the standard one (the one being
> > emitted from boto3, or any other AWS SDK [1]).
> > It passes the arguments using 'x-www-form-urlencoded'. For example:
>
> Thank you for your
Hi Andreas,
The message format you tried to use is the standard one (the one being
emitted from boto3, or any other AWS SDK [1]).
It passes the arguments using 'x-www-form-urlencoded'. For example:
POST / HTTP/1.1
Host: localhost:8000
Accept-Encoding: identity
Date: Tue, 21 Apr 2020 08:52:35 GMT
which ceph version are you using?
note that topic creation methods got broken in v14.2.8 (working in v14.2.7).
Yuval
On Tue, Mar 17, 2020 at 9:25 AM wrote:
> Hello, I try to set up s3 http notifications but cant seem to figure it
> out.
>
> I tried with python boto3 like this:
>
>
yes, this is a regression issue with the new version:
https://tracker.ceph.com/issues/44614
On Thu, Mar 12, 2020 at 8:44 PM 曹 海旺 wrote:
> I think it is a bug . I reinstall the cluster . The response of create
> topic still 405 .methodnotallowed, anynoe konw why? Thank you very much !
>
>
Hi,
Which version of ceph are you using?
Yuval
On Wed, Mar 11, 2020 at 6:59 AM 曹 海旺 wrote:
> HI,
> I'm sorry to bother you again
>
> I want to use kafka to queue the notifications , I add a topic named
> kafka,and put the notification config xml
>
> The topic info:
>
>
the endpoint is not the RGW endpoint, it is the server to which you
want to send the bucket notifications to.
E.g. if you have a rabbitmq server running at address: 1.2.3.4, you should use:
push-endpoint=amqp://1.2.3.4
note that in such a case the: amqp-exchange parameter must be set as well.
74 matches
Mail list logo
