gauravg1977 commented on issue #11598: URL: https://github.com/apache/hudi/issues/11598#issuecomment-2225908980
Lets say I can do without using schema registry and am happy to use a hardcoded schema. Is there a way to consume proto from kafka using that? I noticed a property called **hoodie.streamer.schemaprovider.proto.class.name** as described here: https://hudi.apache.org/docs/configurations/#Kafka-Source-Configs and used it as following in the properties file **hoodie.streamer.schemaprovider.proto.class.name=com.gaurav.data.vol.Volsurf.VolSurface** (this is my protoc compiled class) I tried using it and also omitted specifying any schema registry properties (with the intention of skipping the usage of schema registry), but looks like properties like schema.registry.url and hoodie.streamer.schemaprovider.registry.url are mandatory. Without these I get exceptions like Caused by: io.confluent.common.config.ConfigException: Missing required configuration "schema.registry.url" which has no default value. at io.confluent.common.config.ConfigDef.parse(ConfigDef.java:251) at io.confluent.common.config.AbstractConfig.<init>(AbstractConfig.java:78) at io.confluent.kafka.serializers.AbstractKafkaSchemaSerDeConfig.<init>(AbstractKafkaSchemaSerDeConfig.java:159) at io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializerConfig.<init>(KafkaProtobufDeserializerConfig.java:63) at io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializer.configure(KafkaProtobufDeserializer.java:55) at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:717) ... 17 more and Caused by: org.apache.hudi.exception.HoodieNotSupportedException: Required property hoodie.streamer.schemaprovider.registry.url is missing at org.apache.hudi.common.util.ConfigUtils.lambda$checkRequiredConfigProperties$2(ConfigUtils.java:275) at java.base/java.util.Collections$SingletonList.forEach(Collections.java:4856) at org.apache.hudi.common.util.ConfigUtils.checkRequiredConfigProperties(ConfigUtils.java:273) at org.apache.hudi.utilities.schema.SchemaRegistryProvider.<init>(SchemaRegistryProvider.java:168) If I put those properties back and also hoodie.streamer.schemaprovider.registry.schemaconverter=org.apache.hudi.utilities.schema.converter.ProtoSchemaToAvroSchemaConverter (without that, it tries to use the schema as an avro schema), I get the original error reported in this issue. I will appreciate if you can provide guidance on whether it is possible to consume protos from kafka without using schema registry as that will also work for me for now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
