Croway commented on code in PR #30295:
URL: https://github.com/apache/camel/pull/30295#discussion_r4602984373
##########
components/camel-kafka/src/main/java/org/apache/camel/component/kafka/KafkaProducer.java:
##########
@@ -296,7 +296,7 @@ protected ProducerRecord<Object, Object>
createRecord(Exchange exchange, Message
// must convert each entry of the iterator into the value according to
// the serializer
- Object value = tryConvertToSerializedType(exchange, message.getBody(),
configuration.getValueSerializer());
+ Object value = tryConvertToSerializedType(exchange, message.getBody(),
configuration.resolveValueSerializer());
Review Comment:
This fixes the conversion when the vendor sets the serializer, but the
conversion still follows the configured `valueSerializer`, not the serializer
the Kafka producer gets. Setting the registry serializer with
`additionalProperties.value.serializer`, the way it was done before this PR,
still turns an Avro record into a String: the "schema `"string"` registered"
case of the description.
Reproduced with a value serializer that records the type it gets, and a body
that is not a String but has a `toString()`, like a `GenericRecord`:
-
`kafka:t?schemaRegistryVendor=confluent&schemaRegistryURL=http://registry:8081`:
the serializer gets the record
-
`kafka:t?schemaRegistryURL=http://registry:8081&additionalProperties.value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer`:
the serializer gets a `java.lang.String`
`additionalProperties.value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer`
with a String body fails the same way, with or without a vendor:
`ClassCastException: class java.lang.String cannot be cast to class [B`.
None of this is new with this PR, but the fix could cover it. In `doStart`,
read `value.serializer` from the final properties (after
`applyAdditionalProperties` and `updateClassProperties`, so it is a `Class` or
a name), keep its class name in a field, and use it here and in
`KeyValueHolderIterator`. The vendor, `additionalProperties` and
`valueSerializer` cases then work the same way, and the vendor and format are
no longer checked again for every record.
_Claude Code on behalf of Croway_
##########
components/camel-kafka-common/src/main/java/org/apache/camel/component/kafka/KafkaClientConfiguration.java:
##########
@@ -413,7 +434,60 @@ protected void applyCommonClientProperties(Properties
props) {
addPropertyIfNotEmpty(props,
CommonClientConfigs.RECONNECT_BACKOFF_MAX_MS_CONFIG,
getReconnectBackoffMaxMs());
addPropertyIfNotEmpty(props,
CommonClientConfigs.RETRY_BACKOFF_MS_CONFIG, getRetryBackoffMs());
addPropertyIfNotEmpty(props,
CommonClientConfigs.RETRY_BACKOFF_MAX_MS_CONFIG, getRetryBackoffMaxMs());
- addPropertyIfNotEmpty(props, "schema.registry.url",
getSchemaRegistryURL());
+ KafkaSchemaRegistry.applyUrl(props, resolveSchemaRegistryVendor(),
getSchemaRegistryURL());
+ }
+
+ /**
+ * Adds the value serializer and the registry properties of the
schemaRegistryVendor to the producer properties. The
+ * value serializer is only set when the configured one is the default, so
that an explicit valueSerializer wins.
+ * Does nothing if no schemaRegistryVendor is set.
+ *
+ * @param props the producer properties
+ * @param valueSerializer the configured value serializer
+ * @throws IllegalArgumentException if the vendor or format is unknown, or
the vendor does not support the format
+ */
+ protected void applySchemaRegistryProducerProperties(Properties props,
String valueSerializer) {
+ String vendor = resolveSchemaRegistryVendor();
+ if (vendor != null) {
+ boolean explicit =
!KafkaConstants.KAFKA_DEFAULT_SERIALIZER.equals(valueSerializer);
Review Comment:
A `valueSerializer=org.apache.kafka.common.serialization.StringSerializer`
set on purpose cannot be told apart from the default, so the vendor serializer
replaces it (line 468 does the same for
`valueDeserializer=StringDeserializer`). This shows when the vendor is set on
the component and one endpoint sends plain text, such as a dead letter topic.
Reproduced with a `KafkaComponent` with
`setSchemaRegistryVendor("confluent")`, and the producer properties of:
-
`kafka:dlq?valueSerializer=org.apache.kafka.common.serialization.StringSerializer`:
`value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer`
-
`kafka:dlq2?valueSerializer=org.apache.kafka.common.serialization.ByteArraySerializer`:
`ByteArraySerializer`, kept
- `kafka:dlq3?schemaRegistryVendor=`: `StringSerializer`
So an empty `schemaRegistryVendor` on the endpoint opts out, but it is not
documented. Either the docs give it as the way to opt out for one endpoint, or
Camel tells a serde the user set from the default (for example, the field stays
null until it is set, and the getter returns the default).
_Claude Code on behalf of Croway_
##########
components/camel-kafka-common/src/main/java/org/apache/camel/component/kafka/KafkaSchemaRegistry.java:
##########
@@ -0,0 +1,233 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.kafka;
+
+import java.util.Locale;
+import java.util.Properties;
+
+import org.apache.camel.util.ObjectHelper;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.producer.ProducerConfig;
+
+/**
+ * The value serializer, value deserializer and registry properties that the
schemaRegistryVendor and schemaFormat
+ * options set up for a schema registry vendor.
+ * <p/>
+ * The serializer and deserializer are class names and the properties are keys
of the vendor serde libraries, so that
+ * Camel does not depend on any vendor library: the user adds the serde jar of
the vendor. The keys and class names are
+ * the ones of the vendor libraries (Confluent kafka-avro-serializer 8.0,
Apicurio Registry 3.x serdes, AWS Glue
+ * schema-registry-serde 1.1 and Azure azure-schemaregistry-kafka-avro 1.1).
+ */
+final class KafkaSchemaRegistry {
+
+ static final String CONFLUENT = "confluent";
+ static final String APICURIO = "apicurio";
+ static final String AWS_GLUE = "aws-glue";
+ static final String AZURE = "azure";
+
+ static final String AVRO = "avro";
+ static final String JSON = "json";
+ static final String PROTOBUF = "protobuf";
+
+ static final String VENDORS = CONFLUENT + "," + APICURIO + "," + AWS_GLUE
+ "," + AZURE;
+ static final String FORMATS = AVRO + "," + JSON + "," + PROTOBUF;
+
+ // Confluent (and Azure, which uses the same key names)
+ static final String CONFLUENT_URL = "schema.registry.url";
+ static final String CONFLUENT_SPECIFIC_AVRO_READER =
"specific.avro.reader";
+ // Apicurio Registry 3.x (SchemaResolverConfig and AvroSerdeConfig)
+ static final String APICURIO_URL = "apicurio.registry.url";
+ static final String APICURIO_AUTO_REGISTER =
"apicurio.registry.auto-register";
+ static final String APICURIO_SPECIFIC_AVRO_READER =
"apicurio.registry.use-specific-avro-reader";
+ // AWS Glue (AWSSchemaRegistryConstants)
+ static final String GLUE_ENDPOINT = "endpoint";
+ static final String GLUE_DATA_FORMAT = "dataFormat";
+ static final String GLUE_AUTO_REGISTRATION =
"schemaAutoRegistrationEnabled";
+ static final String GLUE_AVRO_RECORD_TYPE = "avroRecordType";
+ // Azure (AbstractKafkaSerdeConfig and KafkaAvroSerializerConfig)
+ static final String AZURE_USE_AZURE_CREDENTIAL = "use.azure.credential";
+ static final String AZURE_CREDENTIAL = "schema.registry.credential";
+ static final String AZURE_AUTO_REGISTER = "auto.register.schemas";
+
+ private static final String GLUE_SERIALIZER
+ =
"com.amazonaws.services.schemaregistry.serializers.GlueSchemaRegistryKafkaSerializer";
+ private static final String GLUE_DESERIALIZER
+ =
"com.amazonaws.services.schemaregistry.deserializers.GlueSchemaRegistryKafkaDeserializer";
+
+ private KafkaSchemaRegistry() {
+ }
+
+ /**
+ * The vendor in lower case, or null when no vendor is set. Fails for an
unknown vendor, an unknown format, or a
+ * format the vendor does not support.
+ */
+ static String vendor(String vendor, String format) {
+ if (ObjectHelper.isEmpty(vendor)) {
+ return null;
+ }
+ String v = vendor.trim().toLowerCase(Locale.ROOT);
+ if (!CONFLUENT.equals(v) && !APICURIO.equals(v) && !AWS_GLUE.equals(v)
&& !AZURE.equals(v)) {
+ throw new IllegalArgumentException(
+ "Unknown schemaRegistryVendor: " + vendor + ". The
supported values are " + VENDORS
+ + ". Leave schemaRegistryVendor
unset to configure the serializer, "
+ + "deserializer and registry
properties yourself.");
+ }
+ String f = format(format);
+ if (AZURE.equals(v) && !AVRO.equals(f)) {
+ throw new IllegalArgumentException(
+ "schemaRegistryVendor azure supports only schemaFormat
avro, not " + f
+ + ": the Azure Schema Registry
serde for Kafka "
+ +
"(com.microsoft.azure:azure-schemaregistry-kafka-avro) is for Avro only.");
+ }
+ return v;
+ }
+
+ private static String format(String format) {
+ if (ObjectHelper.isEmpty(format)) {
+ return AVRO;
+ }
+ String f = format.trim().toLowerCase(Locale.ROOT);
+ if (!AVRO.equals(f) && !JSON.equals(f) && !PROTOBUF.equals(f)) {
+ throw new IllegalArgumentException(
+ "Unknown schemaFormat: " + format + ". The supported
values are " + FORMATS + ".");
+ }
+ return f;
+ }
+
+ /**
+ * The value serializer class name of the vendor and format.
+ */
+ static String serializer(String vendor, String format) {
+ String f = format(format);
+ return switch (vendor) {
+ case CONFLUENT -> switch (f) {
+ case JSON ->
"io.confluent.kafka.serializers.json.KafkaJsonSchemaSerializer";
+ case PROTOBUF ->
"io.confluent.kafka.serializers.protobuf.KafkaProtobufSerializer";
+ default ->
"io.confluent.kafka.serializers.KafkaAvroSerializer";
+ };
+ case APICURIO -> switch (f) {
+ case JSON ->
"io.apicurio.registry.serde.jsonschema.JsonSchemaKafkaSerializer";
+ case PROTOBUF ->
"io.apicurio.registry.serde.protobuf.ProtobufKafkaSerializer";
+ default ->
"io.apicurio.registry.serde.avro.AvroKafkaSerializer";
+ };
+ case AWS_GLUE -> GLUE_SERIALIZER;
+ default ->
"com.microsoft.azure.schemaregistry.kafka.avro.KafkaAvroSerializer";
+ };
+ }
+
+ /**
+ * The value deserializer class name of the vendor and format.
+ */
+ static String deserializer(String vendor, String format) {
+ String f = format(format);
+ return switch (vendor) {
+ case CONFLUENT -> switch (f) {
+ case JSON ->
"io.confluent.kafka.serializers.json.KafkaJsonSchemaDeserializer";
+ case PROTOBUF ->
"io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializer";
+ default ->
"io.confluent.kafka.serializers.KafkaAvroDeserializer";
+ };
+ case APICURIO -> switch (f) {
+ case JSON ->
"io.apicurio.registry.serde.jsonschema.JsonSchemaKafkaDeserializer";
+ case PROTOBUF ->
"io.apicurio.registry.serde.protobuf.ProtobufKafkaDeserializer";
+ default ->
"io.apicurio.registry.serde.avro.AvroKafkaDeserializer";
+ };
+ case AWS_GLUE -> GLUE_DESERIALIZER;
+ default ->
"com.microsoft.azure.schemaregistry.kafka.avro.KafkaAvroDeserializer";
+ };
+ }
+
+ /**
+ * Adds the registry URL under the key of the vendor. Without a vendor,
the URL is set as schema.registry.url, the
+ * key of Confluent and of the registries that follow its serde
configuration.
+ */
+ static void applyUrl(Properties props, String vendor, String url) {
+ if (ObjectHelper.isEmpty(url)) {
+ return;
+ }
+ if (vendor == null) {
+ props.put(CONFLUENT_URL, url);
+ return;
+ }
+ switch (vendor) {
+ case APICURIO -> props.put(APICURIO_URL, url);
+ // Glue is addressed by region and registry name, the URL is an
optional endpoint override
+ case AWS_GLUE -> props.put(GLUE_ENDPOINT, url);
+ default -> props.put(CONFLUENT_URL, url);
+ }
+ }
+
+ /**
+ * Sets the value serializer (unless the user set one) and the producer
properties of the vendor.
+ */
+ static void applyProducer(Properties props, String vendor, String format,
boolean explicitSerializer) {
+ if (!explicitSerializer) {
+ props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
serializer(vendor, format));
+ }
+ switch (vendor) {
+ // Confluent registers the schemas by default
(auto.register.schemas=true)
+ case APICURIO -> props.put(APICURIO_AUTO_REGISTER, Boolean.TRUE);
+ case AWS_GLUE -> {
+ props.put(GLUE_DATA_FORMAT,
format(format).toUpperCase(Locale.ROOT));
+ props.put(GLUE_AUTO_REGISTRATION, Boolean.TRUE);
+ }
+ case AZURE -> {
+ // the Azure serde reads its flags as java.lang.Boolean (a
String fails)
+ props.put(AZURE_AUTO_REGISTER, Boolean.TRUE);
Review Comment:
The Azure serde reads its flags with `(Boolean) props.getOrDefault(...)`
(`KafkaAvroSerializerConfig.getAutoRegisterSchemas()`,
`AbstractKafkaSerdeConfig.createDefaultAzureCredential()`,
`KafkaAvroDeserializerConfig.getAvroSpecificReader()`), and
`additionalProperties` from a URI or YAML are Strings. So for these flags, the
docs' "any `additionalProperties`, such as a property of the vendor that Camel
also sets" fails when the producer is created.
Reproduced with
`kafka:t?schemaRegistryVendor=azure&schemaRegistryURL=ns.servicebus.windows.net&additionalProperties.schema.group=g`:
- with `&additionalProperties.auto.register.schemas=false`:
```
FailedToCreateProducerException: Failed to create Producer for endpoint:
kafka://t?additionalProperties.auto.register.schemas=false&... Reason:
java.lang.ClassCastException: class java.lang.String cannot be cast to class
java.lang.Boolean
```
- without it, `configure` passes, and the first send reaches the
`azure-core` `NoSuchMethodError` (`TokenRequestContext.isCaeEnabled()`) that
the docs describe
- with
`&additionalProperties.auto.register.schemas=#valueAs(boolean):false`, the
property is the `Boolean` `false`, which works
`auto.register.schemas` is the one users will want to turn off: Camel turns
registration on for apicurio, aws-glue and azure, where the serdes have it off
by default (`AUTO_REGISTER_ARTIFACT_DEFAULT`, `schemaAutoRegistrationEnabled`,
`AUTO_REGISTER_SCHEMAS_CONFIG_DEFAULT` are all `false`). Either the Azure part
of the docs shows `#valueAs(boolean):false`, or, when the vendor is azure,
Camel converts `"true"`/`"false"` to `Boolean` for `auto.register.schemas`,
`use.azure.credential` and `specific.avro.reader` after
`applyAdditionalProperties`.
_Claude Code on behalf of Croway_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]