n3nash commented on a change in pull request #2619:
URL: https://github.com/apache/hudi/pull/2619#discussion_r597938553
##########
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/AvroKafkaSource.java
##########
@@ -43,17 +46,34 @@
public class AvroKafkaSource extends AvroSource {
private static final Logger LOG =
LogManager.getLogger(AvroKafkaSource.class);
-
private final KafkaOffsetGen offsetGen;
-
private final HoodieDeltaStreamerMetrics metrics;
public AvroKafkaSource(TypedProperties props, JavaSparkContext sparkContext,
SparkSession sparkSession,
SchemaProvider schemaProvider, HoodieDeltaStreamerMetrics metrics) {
super(props, sparkContext, sparkSession, schemaProvider);
- this.metrics = metrics;
+
props.put("key.deserializer", StringDeserializer.class);
- props.put("value.deserializer", KafkaAvroDeserializer.class);
+ String deserializerClassName =
props.getString(DataSourceWriteOptions.KAFKA_AVRO_VALUE_DESERIALIZER(), "");
+
+ if (deserializerClassName.isEmpty()) {
+ props.put("value.deserializer", KafkaAvroDeserializer.class);
+ } else {
+ try {
+ if (schemaProvider == null) {
+ throw new HoodieIOException("SchemaProvider has to be set to use
custom Deserializer");
+ }
+ props.put(DataSourceWriteOptions.SCHEMA_PROVIDER_CLASS_PROP(),
schemaProvider.getClass().getName());
+ props.put(DataSourceWriteOptions.JAVA_SPARK_CONTEXT_PROP(),
sparkContext);
+ props.put("value.deserializer", Class.forName(deserializerClassName));
Review comment:
What is `value.deserializer` ? Is there a CONSTANT for this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
