[ 
https://issues.apache.org/jira/browse/CAMEL-27329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Claus Ibsen resolved CAMEL-27329.
---------------------------------
    Resolution: Fixed

Fixed by https://github.com/apache/camel/pull/30550 (merged as 34ddddcb3eeb): 
the schema and messageName options of the protobuf data format, with the new 
camel-protobuf-schema module.

_Claude Code on behalf of Claus Ibsen_

> camel-protobuf - a schema option with the .proto text, to read and write a 
> DynamicMessage without generated classes (new camel-protobuf-schema)
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-27329
>                 URL: https://issues.apache.org/jira/browse/CAMEL-27329
>             Project: Camel
>          Issue Type: New Feature
>            Reporter: Claus Ibsen
>            Assignee: Claus Ibsen
>            Priority: Major
>             Fix For: 4.24.0
>
>
> The protobuf data format (library GoogleProtobuf) needs {{instanceClass}}, a 
> class generated by protoc. A low-code route, such as a YAML route or a 
> Kamelet, has no generated classes, so it cannot turn JSON into a Protobuf 
> message or read Protobuf binary data.
> This matters most for Kafka with a schema registry: with 
> {{schemaRegistryVendor}} and {{schemaFormat=protobuf}}, the Confluent 
> {{KafkaProtobufSerializer}} and the Apicurio {{ProtobufKafkaSerializer}} need 
> a {{com.google.protobuf.Message}} as the record value. Today a YAML route can 
> only build one with a script (the Kamelet tests build a {{DynamicMessage}} in 
> Groovy from a hand-made {{FileDescriptorProto}}).
> h3. Proposal
> Two new options of the protobuf data format, used instead of 
> {{instanceClass}}:
> * {{schema}}: the text of a .proto file, or a resource such as 
> {{classpath:order.proto}}, {{file:order.proto}} or an http URL (a Confluent 
> registry serves the text of a schema at 
> {{/subjects/<subject>/versions/latest/schema}}).
> * {{messageName}}: the message type to use, such as {{Order}}, {{shop.Order}} 
> or the nested {{Order.Line}}. By default the first message type of the schema.
> With {{schema}}, the default instance is 
> {{DynamicMessage.getDefaultInstance(descriptor)}}, so everything the data 
> format does today works on it:
> * unmarshal JSON ({{contentTypeFormat=json}}) or Protobuf binary data into a 
> {{DynamicMessage}};
> * marshal a {{DynamicMessage}}, or a {{Map}} (through the existing 
> {{ProtobufConverter.toProto}}), to binary data or JSON.
> {code:yaml}
> - unmarshal:
>     protobuf:
>       contentTypeFormat: json
>       schema: "resource:classpath:order.proto"
>       messageName: Order
> - to:
>     uri: kafka:orders
>     parameters:
>       schemaRegistryVendor: confluent
>       schemaRegistryURL: http://localhost:8081
>       schemaFormat: protobuf
> {code}
> h3. Two modules, so camel-protobuf stays light
> protobuf-java cannot parse .proto text, and the only maintained Java parser 
> is the one of Square Wire, which brings Kotlin. So:
> * *camel-protobuf* gets the two options and a small SPI, 
> {{org.apache.camel.dataformat.protobuf.ProtobufSchemaParser}} ({{Descriptor 
> parse(String schema, String messageName)}}). The data format takes a single 
> bean of that type from the registry, or else the implementation named in 
> {{META-INF/services/org/apache/camel/protobuf-schema-parser}} (the usual 
> FactoryFinder lookup, through {{ResolverHelper.resolveService}}). With 
> {{schema}} set and no implementation, it fails at startup with: "The schema 
> option of the protobuf data format needs a parser of .proto files: add 
> camel-protobuf-schema to the classpath (camel:protobuf-schema as a Kamelet 
> dependency)". camel-protobuf gets no new dependency.
> * *camel-protobuf-schema* (new, an "other" module like camel-jsoup) depends 
> on camel-protobuf and holds {{WireProtobufSchemaParser}}, registered as the 
> {{protobuf-schema-parser}} service. It parses and links the .proto text with 
> {{com.squareup.wire:wire-schema-jvm}} ({{squareup-wire-version}} 6.4.7, 
> already managed in camel-parent, and the same version that the Confluent 
> 8.4.1 and Apicurio 3.3.4 serdes use). wire's {{SchemaEncoder}} writes the 
> {{FileDescriptorProto}}, and {{FileDescriptor.buildFrom}} makes the 
> descriptor. Imports of the well-known types of {{google/protobuf}} 
> (timestamp, duration, struct, wrappers, any, empty, field_mask) use the 
> descriptors compiled into protobuf-java, so that JsonFormat handles them in 
> their JSON form. Other imports fail with a clear message. The code generators 
> of wire (javapoet, kotlinpoet) are excluded; the dependencies it adds are 
> wire-schema-jvm (490 KB), wire-runtime-jvm (257 KB), okio-jvm (373 KB), 
> kotlin-stdlib (1.7 MB) and guava (3 MB).
> Other Kotlin-free parsers were checked and are not usable: 
> com.squareup:protoparser (archived in 2018, last release 2015), 
> io.protostuff:protostuff-parser (no release since 2022, needs the ANTLR tool, 
> Guava and Guice, and gives its own model, not descriptors), and 
> com.github.os72:protoc-jar (runs a bundled native protoc, last release 2020).
> A prototype passes unit tests for proto2 and proto3: nested messages, enums, 
> maps, oneof, repeated fields, an imported {{google.protobuf.Timestamp}}, JSON 
> -> DynamicMessage, Map -> binary -> DynamicMessage, the errors for an unknown 
> message name, a syntax error and an unsupported import, the startup error 
> without camel-protobuf-schema, and a parser bean from the registry. Built 
> this way, a DynamicMessage is written by {{kafka}} with 
> {{schemaFormat=protobuf}} through both a Confluent API registry (Redpanda) 
> and Apicurio Registry 3.3.4, and read back by the matching deserializers.
> Also needed: the two options in the model ({{ProtobufDataFormat}} in 
> camel-core-model, its reifier, the XML and YAML DSL), the docs of the data 
> format, the {{@BodyType}} descriptions, and the new module in the BOM, 
> camel-allcomponents, the catalog and the "others" docs. 
> {{camel:protobuf-schema}} resolves to 
> {{org.apache.camel:camel-protobuf-schema}} in Camel JBang and as a Kamelet 
> dependency.
> Motivation: the Kamelet catalog (apache/camel-kamelets) can then offer a 
> Kamelet that turns JSON into a Protobuf message for {{kafka-sink}} with 
> {{schemaFormat}} {{protobuf}}, for YAML users.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to