Re: Processor Extensibility

2020-07-07 Thread Matt Burgess
This is probably better suited for the dev list (not sure if you're subscribed but please do, BCC'ing users and moving to dev), but the implementations (components and their NARs) are not designed to be subclassed for custom extensions outside the codebase, can you describe your use case (and

Processor Extensibility

2020-07-07 Thread Eric Secules
Hello, I was wondering if there was a recommendation on how to extend the functionality of nifi processors without forking the nifi repository. I'm looking for a way to include a processor's nar in my project and extend from it. I'd also like to be able to extend that processor's test suite so I

Re: Hive_1_1 Processors and Controllers Missing in NiFi 1.11.4

2020-07-07 Thread Matt Burgess
Harsha, The nifi-hive-services-api-nar is included with the NiFi assembly/release and includes the definition of what the Hive1_1ConnectionPool should be, but the actual component is in the nifi-hive1_1-nar, which is not included with the NiFi release (due to space concerns) and must be manually

Re: Hive_1_1 Processors and Controllers Missing in NiFi 1.11.4

2020-07-07 Thread Sri Harsha Chavali
Hi Matt, Restarting NiFi did the trick. I removed the old nifi-hive_1_1-nar-1.9.2.nar from the extensions directory and "restarted" NiFi and all the processors and controllers automatically picked the newer versions. Earlier I removed the old version and placed the new version of the nar in

Re: Hive_1_1 Processors and Controllers Missing in NiFi 1.11.4

2020-07-07 Thread Matt Burgess
Harsha, There are two NARs associated with Hive components, nifi-hive-services-api-nar which has the Hive1_1ConnectionPool service (actually an interface, but that's under the hood), and the nifi-hive1_1-nar which has the processors that declare themselves as users of that interface (and the

Re: NiFi 1.11.4 -- "Unable to access lib/bootstrap to create bootstrap classloader"

2020-07-07 Thread Pierre Villard
Hi Joseph, It looks like there is an issue with the location of the directories/files. What did you change? Did you solve your issue? Thanks, Pierre Le mar. 30 juin 2020 à 16:44, Joseph Wheeler a écrit : > Hello! > > > > I’m trying to deploy NiFi 1.11.4 to a new environment. After configuring

Re: ConsumeKafka_2_0 Max Poll Records

2020-07-07 Thread Jason Iannone
Mark, agreed. I like to refactor by making sure I have tests, that way I know what I broke. I'm hoping this is an area I can help with once I get through the red tape. On Tue, Jul 7, 2020 at 10:14 AM Mark Payne wrote: > Adding unit tests is not a bad idea. But really I think we should just >

Re: ConsumeKafka_2_0 Max Poll Records

2020-07-07 Thread Mark Payne
Adding unit tests is not a bad idea. But really I think we should just refactor this and properly use the properties explicitly. The current implementation is a bit “magical” and not at all straight forward. On Jul 7, 2020, at 9:58 AM, Jason Iannone mailto:bread...@gmail.com>> wrote: Hi

Re: ConsumeKafka_2_0 Max Poll Records

2020-07-07 Thread Jason Iannone
Hi Bryan, Thanks, I completely missed that! It also makes more sense why some of the properties use dot notation over dash. Definitely something that there should be tests around to make sure it isn't accidentally removed. Thanks, Jason On Tue, Jul 7, 2020 at 9:48 AM Bryan Bende wrote: > Hi

Re: ConsumeKafka_2_0 Max Poll Records

2020-07-07 Thread Bryan Bende
Hi Jason, It is a bit confusing, but in the createConsumerPool there is a line: KafkaProcessorUtils.buildCommonKafkaProperties(context, ConsumerConfig.class, props); That ends up looping through all the properties of the processor and checking if their name matches any of the names in

ConsumeKafka_2_0 Max Poll Records

2020-07-07 Thread Jason Iannone
Hi all, I have been digging through the ConsumeKafka_2_0 (and record) code and noticed that the PropertyDescriptor for MAX_POLL_RECORDS isn't connected to anything. Is this intentional, and a "deprecated field" or am I missing something? Thanks, Jason