davsclaus opened a new pull request, #30295:
URL: https://github.com/apache/camel/pull/30295

   [CAMEL-27130](https://issues.apache.org/jira/browse/CAMEL-27130): 
camel-kafka can set up a schema registry from two options. The user then does 
not need to know the serde class names, or each vendor's names for the registry 
properties.
   
   ### New options
   
   - **`schemaRegistryVendor`:** `confluent`, `apicurio`, `aws-glue` or 
`azure`. No default.
   - **`schemaFormat`:** `avro` (default), `json` or `protobuf`. Only used when 
a vendor is set.
   
   Values are case-insensitive. An unknown value, or a vendor/format 
combination that the vendor does not support, fails when the producer or 
consumer starts. For example, Azure has a released Avro serde only.
   
   From the two options, camel-kafka sets:
   
   - the value serializer, on the producer, and the value deserializer, on the 
consumer;
   - the registry properties of the vendor, such as the URL from 
`schemaRegistryURL`, auto-registration of schemas on the producer, and the 
specific Avro reader when `specificAvroReader` is set.
   
   Every class name and property key was taken from the vendor jars (with 
`javap`). The classes are referred to by name only, so camel-kafka has no 
dependency on any vendor.
   
   **Precedence:**
   
   - A `valueSerializer` or `valueDeserializer` other than the camel-kafka 
default is kept.
   - `additionalProperties` are applied last, so they always win.
   - Key serdes are never changed.
   
   Without a vendor, nothing changes: `schemaRegistryURL` and 
`specificAvroReader` behave as before.
   
   | Vendor | Formats | Registry properties set |
   |---|---|---|
   | `confluent` | avro, json, protobuf | `schema.registry.url`, 
`specific.avro.reader` |
   | `apicurio` | avro, json, protobuf | `apicurio.registry.url`, 
`apicurio.registry.auto-register`, `apicurio.registry.use-specific-avro-reader` 
|
   | `aws-glue` | avro, json, protobuf | `dataFormat`, 
`schemaAutoRegistrationEnabled`, `avroRecordType`, `endpoint` when a URL is 
given. The region and the registry name come from the AWS defaults or 
`additionalProperties` |
   | `azure` | avro | `schema.registry.url` (the Event Hubs namespace), 
`auto.register.schemas`, `use.azure.credential`. The schema group comes from 
`additionalProperties`. Flags are set as `Boolean`, which the Azure serde 
requires |
   
   The share consumer (`kafka-share`) gets the deserializer side too, as the 
options live in the shared `KafkaClientConfiguration`.
   
   ### Producer fix
   
   The producer converted the message body using the configured value 
serializer, which is still the default `StringSerializer` when the vendor's 
serializer is set as a property. An Avro `GenericRecord` was therefore turned 
into a String, and Confluent registered the schema `"string"`. The new 
`KafkaConfiguration.resolveValueSerializer()` returns the vendor's serializer 
in that case. `KafkaProducer` and `KeyValueHolderIterator` use it.
   
   ### Docs
   
   `kafka-component.adoc` has a new section, "Schema registry". It covers:
   
   - the mapping and the serde artifacts of each vendor;
   - the precedence rules;
   - Java and YAML examples for each vendor;
   - that the user adds the serde jars. The Confluent jars are not on Maven 
Central, only in the public `https://packages.confluent.io/maven/` repository;
   - a note on the `azure-core` version conflict among the Azure serde's own 
dependencies.
   
   ### Tests
   
   - **New `KafkaSchemaRegistryVendorTest`:** 23 tests. All 10 vendor and 
format combinations, with the exact producer and consumer properties; defaults; 
precedence; Azure credential and schema group through `#bean`; endpoint and 
component options; failures for unsupported or unknown values; 
`resolveValueSerializer`.
   - **`KafkaShareConfigurationTest`:** 2 new tests.
   - **Unit test suites:** camel-kafka-common 11, camel-kafka-share 1031, 
camel-kafka 1386. All pass.
   
   **End to end** (Camel JBang, Kafka 4.4 in Docker, an Avro `GenericRecord` 
sent and received):
   
   | Vendor | Result |
   |---|---|
   | Confluent | Against `cp-schema-registry:8.0.0`, 3 records sent and 
received, and the record schema registered. Before the producer fix, the schema 
`"string"` was registered |
   | Apicurio | Against Apicurio Registry 3.3.4, with serde 3.3.4 and 3.0.8, 
records sent and received, and the artifact auto-registered |
   | AWS Glue | No emulator. With the real serde and fake credentials, it 
reaches the Glue API and fails only on the token |
   | Azure | No emulator. With the real serde, it reaches authentication with 
`DefaultAzureCredential` and fails only on the fake tenant, once `azure-core` 
is aligned |
   
   JSON Schema and protobuf were not run end to end; their class names are 
verified.
   
   ### Follow-up
   
   A separate PR will let Camel JBang download the serde jars of the chosen 
vendor, and add the Confluent Maven repository.
   
   This is needed by the Kafka Kamelets of camel-kamelets, which will offer a 
schema registry with a few options instead of one Kamelet per vendor: 
apache/camel-kamelets#3183.
   
   _Claude Code on behalf of Claus Ibsen_
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to