oscerd opened a new pull request, #3026: URL: https://github.com/apache/camel-kamelets/pull/3026
Fixes #2303. Builds what @yuvalif asked for and confirmed on the issue. Ceph RGW **pushes** bucket notifications rather than being polled, and his deployment uses Kafka for that — so this consumes those notifications and optionally fetches the object each one refers to, mirroring what `aws-s3-event-based-source` does for SQS. ## The parsing could not be shared with the AWS Kamelet This is the part that justifies a separate Kamelet rather than a parameter on the existing one. Ceph emits the **classic S3 envelope** — a `Records` array — while `aws-s3-event-based-source` parses the **EventBridge** shape: | | event name | bucket | key | |---|---|---|---| | **Ceph** | `Records[0].eventName` | `Records[0].s3.bucket.name` | `Records[0].s3.object.key` | | **AWS** | `$.detail.reason` | `$.detail.bucket.name` | `$.detail.object.key` | Completely different paths. There is no shared parsing to factor out. ## What it reuses Object fetching goes through the **existing Ceph path** — `aws2-s3` with `uriEndpointOverride` against the RGW endpoint, exactly as `ceph-source` and `ceph-sink` already do. No new component is involved. The Kafka properties mirror `kafka-source` (`saslAuthType` / `saslUsername` / `saslPassword`, `autoOffsetReset`, `consumerGroup`) so the two read alike. ## What stays out of scope, and why Configuring the notification is **not** something this Kamelet does, and the description says so. It needs two API calls Camel cannot currently make: - `PutBucketNotificationConfiguration` — absent from `camel-aws2-s3`'s 27 operations - an SNS-style `CreateTopic` carrying `push-endpoint: kafka://...` — `camel-aws2-sns` can point at RGW via `uriEndpointOverride` but exposes no operation enum and no topic attributes, so the push endpoint cannot be expressed That keeps this consistent with the position in the thread that the component will not grow Ceph-specific configuration. The Kamelet consumes a notification someone else configured. ## Verification Ran against a live broker with the event payload taken verbatim from the [Ceph documentation](https://docs.ceph.com/en/latest/radosgw/notifications/#events) that @yuvalif linked. **Consuming** — the notification arrives and is emitted unchanged: ``` CEPHEVENT >>> {"Records":[{"eventVersion":"2.1","eventSource":"ceph:s3", ... "eventName":"ObjectCreated:Put" ...}]} ``` **Parsing**, with `getObject` enabled and the fetch swapped for a log so the extracted values are visible: ``` PARSED event=[ObjectCreated:Put] bucket=[mybucket1] key=[myimage1.jpg] ``` All three paths resolve against the real envelope. `script/validator` reports no errors, `script/generator` adds the `nav.adoc` entry, `mvn clean install` passes with tests from the repository root. ## One change outside the Kamelet `script/validator/validator.go` gains one line — this Kamelet joins the `verifyUsedParams` exemption list, for exactly the reason `aws-s3-event-based-source` is already on it: `getObject` is consumed by a build-time `precondition`, which the declared-parameter check cannot see. That list growing by hand is the smell I noted on #1173. I have followed the existing mechanism rather than changing the validator's contract in a Kamelet PR, but it is one more argument for replacing the hardcoded paths with an annotation the Kamelet itself carries. ## Open question for @yuvalif The Ceph extension fields — `s3.bucket.id`, `s3.object.metadata`, `s3.object.tags`, and the record-level `eventId` and `opaqueData` — are passed through in the body but not surfaced as headers. `eventId` in particular is documented as usable for acking, so it would make a natural idempotency key. Happy to add them as headers in a follow-up if that is useful; I did not want to guess at which ones matter. --- _Claude Code on behalf of Andrea Cosentino_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
