[
https://issues.apache.org/jira/browse/NIFI-14878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lukas Kucharski updated NIFI-14878:
-----------------------------------
Description:
h2. Summary
Implement StandardProtobufReader Controller Service for Protobuf Record
Processing
h2. Description
This enhancement completes the Protobuf support implementation by introducing
the *StandardProtobufReader* controller service. This represents Phase 2 of the
Confluent Protobuf integration, building upon the foundational services
delivered in Phase 1 (linked).
The new reader addresses limitations of the existing *ProtobufReader*
component, which requires manual proto file management and cannot effectively
operate in clustered environments without manual schema distribution across
nodes. Also, the existing reader is designed to work with filesystem source
only, as the 'Proto Directory' property is always required, even when the
schema access strategy is set to 'Text Property'
h2. Key Components Implemented
*StandardProtobufReader* Controller Service
- New Record Reader Implementation: Controller service extending
*SchemaRegistryService* for Protobuf record processing
- Multiple Schema Access Strategies: Supports flexible schema resolution
methods without requiring local proto files
h3. Schema Access Strategy Support
The service supports the following schema access strategies:
1. Schema Text: Direct schema definition input
2. Schema Name from Registry: Reference schemas by name from the configured
registry
3. Schema Reference Reader: Use the schema id embedded in the protobuf payload
header. Then use the schema registry to fetch the schema by the the id.
h2. Key points
- {*}ProtobufSchemaCompiler{*}: component for compiling and managing Protobuf
schemas
- Schema Caching: (200 schemas, 1-hour expiration)
- Reference Resolution: Automatic resolution of imported/referenced schemas
from registry
- Compiles raw Protobuf schema definitions using wire library
- Handles complex schema dependencies and imports
- Schema Resolution: Determines schema source based on configured strategy
- Message Name Resolution: Identifies target message type from payload or
configuration
- Record Deserialization: Converts Protobuf binary data to NiFi records
h3. Design Decisions
- New Implementation: Created as separate component rather than extending
existing ProtobufReader to avoid breaking changes
- Focused Scope: Does not support "Generate From Proto File" strategy,
focusing on registry-based approaches
- Clean API: Streamlined property configuration without legacy constraints
was:
h2. Summary
Implement StandardProtobufReader Controller Service for Protobuf Record
Processing
h2. Description
This enhancement completes the Protobuf support implementation by introducing
the *StandardProtobufReader* controller service. This represents Phase 2 of the
Confluent Protobuf integration, building upon the foundational services
delivered in Phase 1 (linked).
The new reader addresses limitations of the existing *ProtobufReader*
component, which requires manual proto file management and cannot effectively
operate in clustered environments without manual schema distribution across
nodes. Also, the existing reader is designed to work with filesystem source
only, as the 'Proto Directory' property is always required, even when the
schema access strategy is set to 'Text Property'
h2. Key Components Implemented
*StandardProtobufReader* Controller Service
- New Record Reader Implementation: Controller service extending
*SchemaRegistryService* for Protobuf record processing
- Multiple Schema Access Strategies: Supports flexible schema resolution
methods without requiring local proto files
h3. Schema Access Strategy Support
The service supports the following schema access strategies:
1. Schema Text: Direct schema definition input
2. Schema Name from Registry: Reference schemas by name from the configured
registry
3. Schema Reference Reader: Use the schema id embedded in the protobuf payload
header. Then use the schema registry to fetch the schema by the the id.
h2. Key points
- {*}ProtobufSchemaCompiler{*}: component for compiling and managing Protobuf
schemas
- Schema Caching: (200 schemas, 1-hour expiration)
- Reference Resolution: Automatic resolution of imported/referenced schemas
from registry
- Compiles raw Protobuf schema definitions using wire library
- Handles complex schema dependencies and imports
- Schema Resolution: Determines schema source based on configured strategy
- Message Name Resolution: Identifies target message type from payload or
configuration
- Record Deserialization: Converts Protobuf binary data to NiFi records
h3. Design Decisions
- New Implementation: Created as separate component rather than extending
existing ProtobufReader to avoid breaking changes
- Focused Scope: Does not support "Generate From Proto File" strategy,
focusing on registry-based approaches
- Clean API: Streamlined property configuration without legacy constraints
Use Case Scenarios
Enterprise Kafka Integration
- Process Protobuf messages from Kafka topics using Confluent Schema Registry
- Support for complex message hierarchies and nested schemas
- Dynamic message type resolution from wire format headers
Multi-Schema Environments
- Handle topics containing multiple Protobuf message types
- Automatic schema evolution support through registry integration
- Efficient processing of high-volume Protobuf data streams
Configuration Properties
- Schema Access Strategy: Selection of schema resolution method
- Schema Registry: Reference to configured schema registry service
- Schema Reference Reader: Service for reading embedded schema references
- Message Name Resolver: Service for dynamic message name resolution
- Cache Configuration: Tunable caching parameters for performance optimization
Business Value
- Simplified Protobuf Processing: Eliminates manual proto file management in
production environments
- Enterprise-Ready: Full support for clustered deployments and schema registry
integration
- Performance Optimized: Intelligent caching reduces schema compilation
overhead
- Future-Proof: Extensible design supporting evolving Protobuf use cases
Verification
- Successfully built and tested with Maven and JDK 21
- Integration tested with Confluent Schema Registry environments
- Performance validated with schema caching mechanisms
- Cluster deployment scenarios verified
- Requires foundational services from NIFI-14424 Phase 1
(ConfluentProtobufMessageNameResolver, SchemaDefinition, MessageNameResolver
API)
> Implement StandardProtobufReader Controller Service for Protobuf Record
> Processing
> ----------------------------------------------------------------------------------
>
> Key: NIFI-14878
> URL: https://issues.apache.org/jira/browse/NIFI-14878
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 2.6.0
> Reporter: Lukas Kucharski
> Assignee: Lukas Kucharski
> Priority: Major
>
> h2. Summary
> Implement StandardProtobufReader Controller Service for Protobuf Record
> Processing
> h2. Description
> This enhancement completes the Protobuf support implementation by introducing
> the *StandardProtobufReader* controller service. This represents Phase 2 of
> the Confluent Protobuf integration, building upon the foundational services
> delivered in Phase 1 (linked).
> The new reader addresses limitations of the existing *ProtobufReader*
> component, which requires manual proto file management and cannot effectively
> operate in clustered environments without manual schema distribution across
> nodes. Also, the existing reader is designed to work with filesystem source
> only, as the 'Proto Directory' property is always required, even when the
> schema access strategy is set to 'Text Property'
> h2. Key Components Implemented
> *StandardProtobufReader* Controller Service
> - New Record Reader Implementation: Controller service extending
> *SchemaRegistryService* for Protobuf record processing
> - Multiple Schema Access Strategies: Supports flexible schema resolution
> methods without requiring local proto files
> h3. Schema Access Strategy Support
> The service supports the following schema access strategies:
> 1. Schema Text: Direct schema definition input
> 2. Schema Name from Registry: Reference schemas by name from the configured
> registry
> 3. Schema Reference Reader: Use the schema id embedded in the protobuf
> payload header. Then use the schema registry to fetch the schema by the the
> id.
> h2. Key points
> - {*}ProtobufSchemaCompiler{*}: component for compiling and managing
> Protobuf schemas
> - Schema Caching: (200 schemas, 1-hour expiration)
> - Reference Resolution: Automatic resolution of imported/referenced schemas
> from registry
> - Compiles raw Protobuf schema definitions using wire library
> - Handles complex schema dependencies and imports
> - Schema Resolution: Determines schema source based on configured strategy
> - Message Name Resolution: Identifies target message type from payload or
> configuration
> - Record Deserialization: Converts Protobuf binary data to NiFi records
>
> h3. Design Decisions
> - New Implementation: Created as separate component rather than extending
> existing ProtobufReader to avoid breaking changes
> - Focused Scope: Does not support "Generate From Proto File" strategy,
> focusing on registry-based approaches
> - Clean API: Streamlined property configuration without legacy constraints
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
