[ https://issues.apache.org/jira/browse/NIFI-11259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Handermann resolved NIFI-11259. ------------------------------------- Fix Version/s: 2.0.0-M4 Resolution: Fixed > Redesign Kafka Component Integration > ------------------------------------ > > Key: NIFI-11259 > URL: https://issues.apache.org/jira/browse/NIFI-11259 > Project: Apache NiFi > Issue Type: Epic > Components: Extensions > Reporter: David Handermann > Assignee: Paul Grey > Priority: Major > Fix For: 2.0.0-M4 > > Time Spent: 4.5h > Remaining Estimate: 0h > > NiFi has supported integration with Kafka across a number of major release > versions. As Kafka has continued to release new major versions, NiFi > integration has followed a general pattern of duplicating existing code, > updating Kafka libraries, and making small modifications. Although this > approach has allowed NiFi to support a range of Kafka versions, it has made > Kafka components difficult to maintain. > A new approach should be designed and implemented that abstracts access to > Kafka library components. This approach should provide more maintainable > Kafka Processors, and also define a better path to support new major Kafka > releases. > The general approach should consist of a NiFi Controller Service and > supporting classes that abstract Kafka operations for publishing and > consuming messages. The Controller Service interface should not have any > dependencies on Kafka libraries, making it independent of Kafka versions. The > interface should define a contract for accessing Kafka Brokers so that > Controller Service implementations will be responsible for connection and > authentication settings. Controller Service implementations should be aligned > to major Kafka versions, allowing minor dependency version updates without > changing Controller Service implementations. > New Kafka Processors should be designed and implemented to use the Controller > Service interface, without requiring knowledge of Kafka connection details. > New Processors should support the same basic production and consumption > capabilities, with configurable strategies for Kafka Record handling. These > Processors should support the same capabilities as current Processors, > including treating entire FlowFiles as Kafka Records, as well as processing > FlowFiles using a configurable delimiter or using NiFi Record services. > New Kafka components should build on the pattern of the nifi-kafka-shared > module to promote code reuse across major Kafka versions. > The new decoupled Kafka Processors and Controller Services should enable > better unit testing. The Testcontainers library should considered for > integration testing with different Kafka runtime versions. > Documentation should be written to describe common migration scenarios from > existing Kafka Processors to new components. -- This message was sent by Atlassian Jira (v8.20.10#820010)