davsclaus opened a new pull request, #30296: URL: https://github.com/apache/camel/pull/30296
Follow-up to #30295 ([CAMEL-27130](https://issues.apache.org/jira/browse/CAMEL-27130)): Camel JBang downloads the serde jars of the chosen `schemaRegistryVendor`, so no `--dep` is needed. For Confluent, there is now a `--repos=confluent` shorthand. > **Stacked on #30295.** This branch is built on the CAMEL-27130 branch, so until #30295 is merged its diff includes that PR. Review only the last two commits. ### 1. Camel JBang downloads the serdes **Mappings.** All 16 serializer and deserializer classes of the four vendors are mapped by class name in `dsl/camel-kamelet-main/src/main/known-third-party-libraries.properties`. That is the existing source that `prepare-kamelet-main` turns into `camel-thirdparty-known-dependencies.properties` with the versions from the parent pom. **How the class reaches Kafka.** No change was needed for this part: - camel-kafka already resolves `value.serializer` and `value.deserializer` through the Camel `ClassResolver` (`KafkaEndpoint.updateClassProperties`), and passes the resolved `Class` to Kafka. - Under JBang, that resolver is `DependencyDownloaderClassResolver`, which downloads the mapped artifact. **Several artifacts per mapping.** A mapping can now list several artifacts, separated by commas, which are downloaded together as one dependency graph. - Azure needs this. `azure-schemaregistry-kafka-avro` pulls in `azure-core` 1.35.0, while its `azure-identity` needs 1.54.1 or later. The first send then failed with `NoSuchMethodError: TokenRequestContext.isCaeEnabled()`. - The Azure classes now map to the serde plus `com.azure:azure-core` from the Azure SDK BOM. - `KnownDependenciesResolver` gets `mavenGavsForClass`, `mavenGavsForImport` and `download(...)`. The class resolver, the Java imports, the properties component and the property-binding listener now download every artifact of a mapping. **A failed download is remembered.** The class resolver now remembers a known dependency whose download failed, so that it is not attempted again for each class loader. Without the Confluent repository, this gives one error instead of five stack traces. **camel-kafka checks the serde.** - When `schemaRegistryVendor` is set and its serde class cannot be loaded, the producer, consumer and share consumer fail at start, naming the Maven artifact to add. - For Confluent, the message also gives the repository URL and `camel run --repos=confluent`. - With a custom `KafkaClientFactory`, which may supply the class another way, it only logs a warning. - An explicit `valueSerializer` or `valueDeserializer` is never checked. ### 2. Versions in `parent/pom.xml` | Property | Version | Why | |---|---|---| | `confluent-serializer-version` | 8.3.3 | Confluent Platform 8.3.x goes with Kafka 4.3.x, which Camel uses (`kafka-version` 4.3.2). 8.4.1 is the newest | | `aws-glue-schema-registry-version` | 2.0.2 | Newest on Maven Central | | `azure-schemaregistry-kafka-avro-version` | 1.1.4 | Newest on Maven Central | | `apicurio-registry-sdk-version` | 3.3.3 (reused) | The serdes come from the same Apicurio Registry release as the SDK | `azure-core` comes from `azure-sdk-bom-version`. `PrepareKameletMainMojo` now: - accepts several artifacts per line; - verifies the class keys, not only packages; - adds extra repositories while verifying, with `camel.known-dependencies.verify.repos`, which defaults to the Confluent repository. `-Dcamel.known-dependencies.verify=true` checked 167 entries against 154 jars. ### 3. The `confluent` repository alias - `known-maven-repos.properties` gets `confluent = https://packages.confluent.io/maven/`, next to `redhat`. - The aliases were not documented, so `camel-jbang-running.adoc` now lists all six. - The Confluent repository is not added automatically: the user asks for it with `--repos=confluent`. - The camel-kafka "Schema registry" docs now say that Camel JBang downloads the jars. ### End to end Run with the Camel JBang launcher built from this branch, against Kafka 4.4 in Docker, without `--dep`: | Vendor | Result | |---|---| | Confluent, no `--repos` | Fails at start, naming `io.confluent:kafka-avro-serializer`, the repository and `--repos=confluent` | | Confluent, `--repos=confluent` | Serde 8.3.3 downloaded. An Avro `GenericRecord` was sent and received, and the subject registered | | Apicurio Registry 3.3.4 | Serde 3.3.3 downloaded. Sent and received, and the artifact registered | | AWS Glue | Serde 2.0.2 downloaded and configured. Fails only on the fake credentials | | Azure | Serde 1.1.4 and `azure-core` 1.57.0 downloaded. Fails only on the fake tenant, with no `NoSuchMethodError`. With `--dep` on the serde only, the `NoSuchMethodError` comes back, which confirms that the new path is what fixes it | ### Tests New tests: - `DefaultRepositoryResolverTest` - `KnownDependenciesResolverTest`: the 16 classes against the parent pom versions, plus several artifacts per mapping - `MavenDependencyDownloaderBatchTest` - `KafkaShareSchemaRegistrySerdeTest` - more cases in `KafkaSchemaRegistryVendorTest` `mvn test` passed for camel-package-maven-plugin, camel-tooling-maven, camel-kafka-common, camel-kafka (1405 tests), camel-kafka-share and camel-kamelet-main (1033 tests). ### Notes - The new version properties are only read by the generator, so Dependabot will not bump them. - If the Azure serde is added with `--dep` instead, JBang does not bring `azure-core`. The docs keep the note about it. - A dependency whose download failed is not retried during the same run. - JSON Schema and Protobuf were not run end to end; their class names are verified in the jars. _Claude Code on behalf of Claus Ibsen_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
