davsclaus opened a new pull request, #30295: URL: https://github.com/apache/camel/pull/30295
[CAMEL-27130](https://issues.apache.org/jira/browse/CAMEL-27130): camel-kafka can set up a schema registry from two options. The user then does not need to know the serde class names, or each vendor's names for the registry properties. ### New options - **`schemaRegistryVendor`:** `confluent`, `apicurio`, `aws-glue` or `azure`. No default. - **`schemaFormat`:** `avro` (default), `json` or `protobuf`. Only used when a vendor is set. Values are case-insensitive. An unknown value, or a vendor/format combination that the vendor does not support, fails when the producer or consumer starts. For example, Azure has a released Avro serde only. From the two options, camel-kafka sets: - the value serializer, on the producer, and the value deserializer, on the consumer; - the registry properties of the vendor, such as the URL from `schemaRegistryURL`, auto-registration of schemas on the producer, and the specific Avro reader when `specificAvroReader` is set. Every class name and property key was taken from the vendor jars (with `javap`). The classes are referred to by name only, so camel-kafka has no dependency on any vendor. **Precedence:** - A `valueSerializer` or `valueDeserializer` other than the camel-kafka default is kept. - `additionalProperties` are applied last, so they always win. - Key serdes are never changed. Without a vendor, nothing changes: `schemaRegistryURL` and `specificAvroReader` behave as before. | Vendor | Formats | Registry properties set | |---|---|---| | `confluent` | avro, json, protobuf | `schema.registry.url`, `specific.avro.reader` | | `apicurio` | avro, json, protobuf | `apicurio.registry.url`, `apicurio.registry.auto-register`, `apicurio.registry.use-specific-avro-reader` | | `aws-glue` | avro, json, protobuf | `dataFormat`, `schemaAutoRegistrationEnabled`, `avroRecordType`, `endpoint` when a URL is given. The region and the registry name come from the AWS defaults or `additionalProperties` | | `azure` | avro | `schema.registry.url` (the Event Hubs namespace), `auto.register.schemas`, `use.azure.credential`. The schema group comes from `additionalProperties`. Flags are set as `Boolean`, which the Azure serde requires | The share consumer (`kafka-share`) gets the deserializer side too, as the options live in the shared `KafkaClientConfiguration`. ### Producer fix The producer converted the message body using the configured value serializer, which is still the default `StringSerializer` when the vendor's serializer is set as a property. An Avro `GenericRecord` was therefore turned into a String, and Confluent registered the schema `"string"`. The new `KafkaConfiguration.resolveValueSerializer()` returns the vendor's serializer in that case. `KafkaProducer` and `KeyValueHolderIterator` use it. ### Docs `kafka-component.adoc` has a new section, "Schema registry". It covers: - the mapping and the serde artifacts of each vendor; - the precedence rules; - Java and YAML examples for each vendor; - that the user adds the serde jars. The Confluent jars are not on Maven Central, only in the public `https://packages.confluent.io/maven/` repository; - a note on the `azure-core` version conflict among the Azure serde's own dependencies. ### Tests - **New `KafkaSchemaRegistryVendorTest`:** 23 tests. All 10 vendor and format combinations, with the exact producer and consumer properties; defaults; precedence; Azure credential and schema group through `#bean`; endpoint and component options; failures for unsupported or unknown values; `resolveValueSerializer`. - **`KafkaShareConfigurationTest`:** 2 new tests. - **Unit test suites:** camel-kafka-common 11, camel-kafka-share 1031, camel-kafka 1386. All pass. **End to end** (Camel JBang, Kafka 4.4 in Docker, an Avro `GenericRecord` sent and received): | Vendor | Result | |---|---| | Confluent | Against `cp-schema-registry:8.0.0`, 3 records sent and received, and the record schema registered. Before the producer fix, the schema `"string"` was registered | | Apicurio | Against Apicurio Registry 3.3.4, with serde 3.3.4 and 3.0.8, records sent and received, and the artifact auto-registered | | AWS Glue | No emulator. With the real serde and fake credentials, it reaches the Glue API and fails only on the token | | Azure | No emulator. With the real serde, it reaches authentication with `DefaultAzureCredential` and fails only on the fake tenant, once `azure-core` is aligned | JSON Schema and protobuf were not run end to end; their class names are verified. ### Follow-up A separate PR will let Camel JBang download the serde jars of the chosen vendor, and add the Confluent Maven repository. This is needed by the Kafka Kamelets of camel-kamelets, which will offer a schema registry with a few options instead of one Kamelet per vendor: apache/camel-kamelets#3183. _Claude Code on behalf of Claus Ibsen_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
