Claus Ibsen created CAMEL-27329:
-----------------------------------
Summary: camel-protobuf - a schema option with the .proto text, to
read and write a DynamicMessage without generated classes (new
camel-protobuf-schema)
Key: CAMEL-27329
URL: https://issues.apache.org/jira/browse/CAMEL-27329
Project: Camel
Issue Type: New Feature
Reporter: Claus Ibsen
Fix For: 4.24.0
The protobuf data format (library GoogleProtobuf) needs {{instanceClass}}, a
class generated by protoc. A low-code route, such as a YAML route or a Kamelet,
has no generated classes, so it cannot turn JSON into a Protobuf message or
read Protobuf binary data.
This matters most for Kafka with a schema registry: with
{{schemaRegistryVendor}} and {{schemaFormat=protobuf}}, the Confluent
{{KafkaProtobufSerializer}} and the Apicurio {{ProtobufKafkaSerializer}} need a
{{com.google.protobuf.Message}} as the record value. Today a YAML route can
only build one with a script (the Kamelet tests build a {{DynamicMessage}} in
Groovy from a hand-made {{FileDescriptorProto}}).
h3. Proposal
Two new options of the protobuf data format, used instead of {{instanceClass}}:
* {{schema}}: the text of a .proto file, or a resource such as
{{classpath:order.proto}}, {{file:order.proto}} or an http URL (a Confluent
registry serves the text of a schema at
{{/subjects/<subject>/versions/latest/schema}}).
* {{messageName}}: the message type to use, such as {{Order}}, {{shop.Order}}
or the nested {{Order.Line}}. By default the first message type of the schema.
With {{schema}}, the default instance is
{{DynamicMessage.getDefaultInstance(descriptor)}}, so everything the data
format does today works on it:
* unmarshal JSON ({{contentTypeFormat=json}}) or Protobuf binary data into a
{{DynamicMessage}};
* marshal a {{DynamicMessage}}, or a {{Map}} (through the existing
{{ProtobufConverter.toProto}}), to binary data or JSON.
{code:yaml}
- unmarshal:
protobuf:
contentTypeFormat: json
schema: "resource:classpath:order.proto"
messageName: Order
- to:
uri: kafka:orders
parameters:
schemaRegistryVendor: confluent
schemaRegistryURL: http://localhost:8081
schemaFormat: protobuf
{code}
h3. Two modules, so camel-protobuf stays light
protobuf-java cannot parse .proto text, and the only maintained Java parser is
the one of Square Wire, which brings Kotlin. So:
* *camel-protobuf* gets the two options and a small SPI,
{{org.apache.camel.dataformat.protobuf.ProtobufSchemaParser}} ({{Descriptor
parse(String schema, String messageName)}}). The data format takes a single
bean of that type from the registry, or else the implementation named in
{{META-INF/services/org/apache/camel/protobuf-schema-parser}} (the usual
FactoryFinder lookup, through {{ResolverHelper.resolveService}}). With
{{schema}} set and no implementation, it fails at startup with: "The schema
option of the protobuf data format needs a parser of .proto files: add
camel-protobuf-schema to the classpath (camel:protobuf-schema as a Kamelet
dependency)". camel-protobuf gets no new dependency.
* *camel-protobuf-schema* (new, an "other" module like camel-jsoup) depends on
camel-protobuf and holds {{WireProtobufSchemaParser}}, registered as the
{{protobuf-schema-parser}} service. It parses and links the .proto text with
{{com.squareup.wire:wire-schema-jvm}} ({{squareup-wire-version}} 6.4.7, already
managed in camel-parent, and the same version that the Confluent 8.4.1 and
Apicurio 3.3.4 serdes use). wire's {{SchemaEncoder}} writes the
{{FileDescriptorProto}}, and {{FileDescriptor.buildFrom}} makes the descriptor.
Imports of the well-known types of {{google/protobuf}} (timestamp, duration,
struct, wrappers, any, empty, field_mask) use the descriptors compiled into
protobuf-java, so that JsonFormat handles them in their JSON form. Other
imports fail with a clear message. The code generators of wire (javapoet,
kotlinpoet) are excluded; the dependencies it adds are wire-schema-jvm (490
KB), wire-runtime-jvm (257 KB), okio-jvm (373 KB), kotlin-stdlib (1.7 MB) and
guava (3 MB).
Other Kotlin-free parsers were checked and are not usable:
com.squareup:protoparser (archived in 2018, last release 2015),
io.protostuff:protostuff-parser (no release since 2022, needs the ANTLR tool,
Guava and Guice, and gives its own model, not descriptors), and
com.github.os72:protoc-jar (runs a bundled native protoc, last release 2020).
A prototype passes unit tests for proto2 and proto3: nested messages, enums,
maps, oneof, repeated fields, an imported {{google.protobuf.Timestamp}}, JSON
-> DynamicMessage, Map -> binary -> DynamicMessage, the errors for an unknown
message name, a syntax error and an unsupported import, the startup error
without camel-protobuf-schema, and a parser bean from the registry. Built this
way, a DynamicMessage is written by {{kafka}} with {{schemaFormat=protobuf}}
through both a Confluent API registry (Redpanda) and Apicurio Registry 3.3.4,
and read back by the matching deserializers.
Also needed: the two options in the model ({{ProtobufDataFormat}} in
camel-core-model, its reifier, the XML and YAML DSL), the docs of the data
format, the {{@BodyType}} descriptions, and the new module in the BOM,
camel-allcomponents, the catalog and the "others" docs.
{{camel:protobuf-schema}} resolves to
{{org.apache.camel:camel-protobuf-schema}} in Camel JBang and as a Kamelet
dependency.
Motivation: the Kamelet catalog (apache/camel-kamelets) can then offer a
Kamelet that turns JSON into a Protobuf message for {{kafka-sink}} with
{{schemaFormat}} {{protobuf}}, for YAML users.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)