Claus Ibsen created CAMEL-27329:
-----------------------------------

             Summary: camel-protobuf - a schema option with the .proto text, to 
read and write a DynamicMessage without generated classes (new 
camel-protobuf-schema)
                 Key: CAMEL-27329
                 URL: https://issues.apache.org/jira/browse/CAMEL-27329
             Project: Camel
          Issue Type: New Feature
            Reporter: Claus Ibsen
             Fix For: 4.24.0


The protobuf data format (library GoogleProtobuf) needs {{instanceClass}}, a 
class generated by protoc. A low-code route, such as a YAML route or a Kamelet, 
has no generated classes, so it cannot turn JSON into a Protobuf message or 
read Protobuf binary data.

This matters most for Kafka with a schema registry: with 
{{schemaRegistryVendor}} and {{schemaFormat=protobuf}}, the Confluent 
{{KafkaProtobufSerializer}} and the Apicurio {{ProtobufKafkaSerializer}} need a 
{{com.google.protobuf.Message}} as the record value. Today a YAML route can 
only build one with a script (the Kamelet tests build a {{DynamicMessage}} in 
Groovy from a hand-made {{FileDescriptorProto}}).

h3. Proposal

Two new options of the protobuf data format, used instead of {{instanceClass}}:
* {{schema}}: the text of a .proto file, or a resource such as 
{{classpath:order.proto}}, {{file:order.proto}} or an http URL (a Confluent 
registry serves the text of a schema at 
{{/subjects/<subject>/versions/latest/schema}}).
* {{messageName}}: the message type to use, such as {{Order}}, {{shop.Order}} 
or the nested {{Order.Line}}. By default the first message type of the schema.

With {{schema}}, the default instance is 
{{DynamicMessage.getDefaultInstance(descriptor)}}, so everything the data 
format does today works on it:
* unmarshal JSON ({{contentTypeFormat=json}}) or Protobuf binary data into a 
{{DynamicMessage}};
* marshal a {{DynamicMessage}}, or a {{Map}} (through the existing 
{{ProtobufConverter.toProto}}), to binary data or JSON.

{code:yaml}
- unmarshal:
    protobuf:
      contentTypeFormat: json
      schema: "resource:classpath:order.proto"
      messageName: Order
- to:
    uri: kafka:orders
    parameters:
      schemaRegistryVendor: confluent
      schemaRegistryURL: http://localhost:8081
      schemaFormat: protobuf
{code}

h3. Two modules, so camel-protobuf stays light

protobuf-java cannot parse .proto text, and the only maintained Java parser is 
the one of Square Wire, which brings Kotlin. So:
* *camel-protobuf* gets the two options and a small SPI, 
{{org.apache.camel.dataformat.protobuf.ProtobufSchemaParser}} ({{Descriptor 
parse(String schema, String messageName)}}). The data format takes a single 
bean of that type from the registry, or else the implementation named in 
{{META-INF/services/org/apache/camel/protobuf-schema-parser}} (the usual 
FactoryFinder lookup, through {{ResolverHelper.resolveService}}). With 
{{schema}} set and no implementation, it fails at startup with: "The schema 
option of the protobuf data format needs a parser of .proto files: add 
camel-protobuf-schema to the classpath (camel:protobuf-schema as a Kamelet 
dependency)". camel-protobuf gets no new dependency.
* *camel-protobuf-schema* (new, an "other" module like camel-jsoup) depends on 
camel-protobuf and holds {{WireProtobufSchemaParser}}, registered as the 
{{protobuf-schema-parser}} service. It parses and links the .proto text with 
{{com.squareup.wire:wire-schema-jvm}} ({{squareup-wire-version}} 6.4.7, already 
managed in camel-parent, and the same version that the Confluent 8.4.1 and 
Apicurio 3.3.4 serdes use). wire's {{SchemaEncoder}} writes the 
{{FileDescriptorProto}}, and {{FileDescriptor.buildFrom}} makes the descriptor. 
Imports of the well-known types of {{google/protobuf}} (timestamp, duration, 
struct, wrappers, any, empty, field_mask) use the descriptors compiled into 
protobuf-java, so that JsonFormat handles them in their JSON form. Other 
imports fail with a clear message. The code generators of wire (javapoet, 
kotlinpoet) are excluded; the dependencies it adds are wire-schema-jvm (490 
KB), wire-runtime-jvm (257 KB), okio-jvm (373 KB), kotlin-stdlib (1.7 MB) and 
guava (3 MB).

Other Kotlin-free parsers were checked and are not usable: 
com.squareup:protoparser (archived in 2018, last release 2015), 
io.protostuff:protostuff-parser (no release since 2022, needs the ANTLR tool, 
Guava and Guice, and gives its own model, not descriptors), and 
com.github.os72:protoc-jar (runs a bundled native protoc, last release 2020).

A prototype passes unit tests for proto2 and proto3: nested messages, enums, 
maps, oneof, repeated fields, an imported {{google.protobuf.Timestamp}}, JSON 
-> DynamicMessage, Map -> binary -> DynamicMessage, the errors for an unknown 
message name, a syntax error and an unsupported import, the startup error 
without camel-protobuf-schema, and a parser bean from the registry. Built this 
way, a DynamicMessage is written by {{kafka}} with {{schemaFormat=protobuf}} 
through both a Confluent API registry (Redpanda) and Apicurio Registry 3.3.4, 
and read back by the matching deserializers.

Also needed: the two options in the model ({{ProtobufDataFormat}} in 
camel-core-model, its reifier, the XML and YAML DSL), the docs of the data 
format, the {{@BodyType}} descriptions, and the new module in the BOM, 
camel-allcomponents, the catalog and the "others" docs. 
{{camel:protobuf-schema}} resolves to 
{{org.apache.camel:camel-protobuf-schema}} in Camel JBang and as a Kamelet 
dependency.

Motivation: the Kamelet catalog (apache/camel-kamelets) can then offer a 
Kamelet that turns JSON into a Protobuf message for {{kafka-sink}} with 
{{schemaFormat}} {{protobuf}}, for YAML users.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to