davsclaus opened a new pull request, #3193:
URL: https://github.com/apache/camel-kamelets/pull/3193
Part of #3170: `azure-eventhubs-source` can read with its own consumer
group, so two integrations can each receive every event of the same event hub.
Until now it always read as `$Default`.
### The option
`consumerGroupName` is a string: the consumer group to read as. Each
consumer group receives every event and has its own saved position (checkpoint)
in `blobContainerName`, so integrations with different consumer groups can
share the checkpoint container.
It is mapped as `consumerGroupName={{?consumerGroupName}}` in the existing
`&` query string of the template URI, and deliberately has no `default`. When
it is not set, the optional placeholder drops the parameter, so the endpoint
URI is exactly today's and the component reads as `$Default`.
The option description, the "Good to know" paragraph and the partial page (a
new section with an example) say that the consumer group must exist, how to
create it in the Azure portal, and what happens when it does not exist.
`eventPosition` is not offered. The component option is a `Map` of partition
id to `EventPosition`, which cannot be set from a URI string:
`eventPosition=earliest` fails with `No type converter available to convert
from type: java.lang.String to the required type: java.util.Map`. It would need
a bean, and the component has no simpler option to start from the earliest or
latest position. Without a saved position the Kamelet keeps starting with the
events that arrive after it starts, as the description already says.
### Verification
Tested with Camel JBang 4.24.0-SNAPSHOT against the Azure Event Hubs
emulator with Azurite, with an event hub `eh1` of 2 partitions and the consumer
groups `cg-a` and `cg-b` declared in the emulator config (`$Default` is always
there).
The Kamelet cannot reach the emulator on its own: the component builds the
Event Hubs address as `<namespaceName>.servicebus.windows.net` and the blob
address as `https://<blobAccountName>.blob.core.windows.net`, and the Kamelet
has no option to change either. The test therefore ran the Kamelet unchanged
and set two things on the `azure-eventhubs` component: `connectionString` with
`UseDevelopmentEmulator=true` (application property), and a
`BlobCheckpointStore` on Azurite (a small test-only Java class). The Kamelet
still passes its own parameters; `camel get endpoint --wide-uri` showed the
endpoint URI it produces.
| Case | Result |
|---|---|
| `consumerGroupName: cg-a` and `cg-b`, two integrations running, 5 events
sent (twice) | Each integration received all 5 events, both partitions, each
time |
| Option unset, running next to them | Received all 5 events too; endpoint
URI has no `consumerGroupName`; the checkpoint blobs are under `$default` |
| Option unset, Kamelet from `main` and from this PR | Same endpoint URI
(`...&blobContainerName=...&credentialType=CONNECTION_STRING`); the `main`
Kamelet continued from the `$default` position saved by the PR Kamelet and
received only the new events |
| Checkpoint blobs in Azurite | `eh1/$default/...`, `eh1/cg-a/...`,
`eh1/cg-b/...`: one position per consumer group |
| `cg-b` stopped, 5 events sent, `cg-b` restarted | `cg-b` received the 5
events sent while it was stopped (its own saved position) |
| `cg-a` started with no saved position (empty checkpoint container) after
20 events were in the hub | Received none of the older events, then all 5 sent
afterwards |
| `consumerGroupName: no-such-group` | The integration starts and the route
is `Started`, but receives nothing. For each partition it logs a WARN `Error
processing exchange ... AmqpException - The messaging entity
'emulatorns1:eventHub:eh1~16383\|no-such-group' could not be found`
(`amqp:not-found`), and tries again about every 2 minutes |
| `eventPosition=earliest` on a plain `azure-eventhubs` route | Fails to
start: `No type converter available to convert from type: java.lang.String to
the required type: java.util.Map` |
`./mvnw clean verify -DskipTests` and then `./mvnw clean verify` passed. The
copy of the Kamelet in `library/camel-kamelets` was refreshed with the build's
own copy step (`resources:copy-resources@copy-resource-one`, which normally
runs at `install`), and `kamelets-index.json` was regenerated by the build.
_Claude Code on behalf of Claus Ibsen_
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]