exceptionfactory commented on PR #10538: URL: https://github.com/apache/nifi/pull/10538#issuecomment-3729449487
Thanks for the reply @tpalfy. I agree UX and DevX are the two major areas of concern, but the response does not address the fundamental issues I have raised with each. This is admittedly a grey area, which is the reason for this extended conversation. The question is not about SDLC, but about cohesion of functionality. Coming at this from one additional angle, this feature was introduced in NiFi 1.13.0, which was prior to the implementation of Stateless execution. Now that the framework supports configuring any Process Group with Stateless execution, it is easier to design a coherent pipeline where the NiFi consumer does not commit offsets until after successful processing. It still requires careful pipeline design, but it means that records for a specific partition can be processed to completion, and if rebalancing occurs, the new consumer receives a coherent set of records. In a different direction, Kafka 4.0 introduced [Queues](cwiki.apache.org/confluence/display/KAFKA/Queues+for+Kafka+(KIP-932)+-+Early+Access+Release+Notes) which provide another more coherent alternative to attempted static partition assignments on the consumer side. Although this static partition assignment may have been a useful feature at the time, now over five years ago, these are some of the reasons I do not support bringing it back in the same way. I remain open to considering an approach with a separate Processor, and I am willing to work through a review to build that in an optimal way, although I now question whether it is even needed. With different capabilities available, and a different module design, I do not support bringing back this feature as it was previously implemented. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
