Thanks Chris, after looking more closely at the ConsumeKafka processors, I realized that they don't accept incoming flow files.
It makes the flow more complex, and subject to potential queuing volume concerns, but another option might be to use the Wait and Notify processors. At a basic level, ConsumeKafka would retrieve files and queue them up with the Wait processor. The HandleHttpRequest processor would trigger the Notify processor, which would allow the Wait processor to release the files for HandleHttpResponse. Depending on the volume of messages from Kafka, and the interval at which the REST client connects, this could be problematic, but it is one potential option. Regards, David Handermann On Thu, Feb 4, 2021 at 8:50 AM Ralph Vercauteren <r...@qad.com> wrote: > Thanks, Chris, > > This is the same problem I figured, > > Is it possible to create an executeScript reading Kafka? > > With regards, > Mit freundlichem Gruß, > Met vriendelijke groet, > > *Ralph Vercauteren* > Technical Architect QAD Automation Solutions > Mobile NL: +31 6 5397 7230 > r...@qad.com > > This e-mail may contain QAD proprietary information and should be treated > as confidential. > > > On Thu, Feb 4, 2021 at 3:45 PM Chris Sampson <chris.samp...@naimuri.com> > wrote: > >> Suspect the problem there would be that the ConsumeKafka processors >> (according to the documentation) don't accept incoming connections, so >> you've no way of sending the FlowFile from InvokeHttp/HandleHttpRequest to >> ConsumeKafka to fetch the messages (which you'd possibly then want to >> MergeContent/MergeRecord to send everything back as a single response). >> >> I'm not very familiar with Kafka processors though, so hopefully someone >> else can suggest an alternative (or correct me and the NiFi docs if >> ConsumeKafka isn't just a source processor). >> >> >> >> --- >> *Chris Sampson* >> IT Consultant >> chris.samp...@naimuri.com >> <https://www.naimuri.com/> >> >> >> On Thu, 4 Feb 2021 at 14:26, David Handermann <exceptionfact...@gmail.com> >> wrote: >> >>> Ralph, >>> >>> Thanks for the reply and additional details. I may be misunderstanding, >>> but just for clarification, in your diagram, are you expecting NiFi to act >>> as the REST server? If that is the case, then you should be able to use >>> one of the ConsumeKafka processors to retrieve messages and return them to >>> the REST client. At a very basic level, the flow would go from >>> HandleHttpRequest to ConsumeKafka to HandleHttpResponse. Depending on the >>> format of the messages retrieved from Kafka, you may need to introduce >>> additional processors prior to HandleHttpResponse in order to format the >>> output for delivery to the REST client. >>> >>> Regards, >>> David Handermann >>> >>> On Thu, Feb 4, 2021 at 7:47 AM Ralph Vercauteren <r...@qad.com> wrote: >>> >>>> Hi David, >>>> >>>> Thanks for responding. >>>> >>>> I understand that part, still have the issue to see how we get the >>>> kafka messages from a topic. >>>> >>>> [image: image.png] >>>> >>>> With regards, >>>> Mit freundlichem Gruß, >>>> Met vriendelijke groet, >>>> >>>> *Ralph Vercauteren* >>>> Technical Architect QAD Automation Solutions >>>> Mobile NL: +31 6 5397 7230 >>>> r...@qad.com >>>> >>>> This e-mail may contain QAD proprietary information and should be >>>> treated as confidential. >>>> >>>> >>>> On Thu, Feb 4, 2021 at 2:21 PM David Handermann < >>>> exceptionfact...@gmail.com> wrote: >>>> >>>>> Hi Ralph, >>>>> >>>>> If you are looking to create a REST service for receiving and >>>>> processing messages in NiFi, take a look at the ListenHTTP processor, as >>>>> well as the HandleHttpRequest and HandleHttpResponse processors. The >>>>> ListenHTTP processor supports receiving HTTP requests and sending the >>>>> contents to other processors. The ListenHTTP processor provides the >>>>> easiest approach for getting started. If you need to perform validation >>>>> or >>>>> additional processing before sending back a response to REST clients, then >>>>> using HandleHttpRequest and HandleHttpResponse with other processors in >>>>> between can support that approach. When using HandleHttpRequest and >>>>> HandleHttpResponse, it is important to optimize intermediate as much as >>>>> possible to avoid HTTP clients timing out while waiting for a response. >>>>> Both processors support a number of configuration properties, so be sure >>>>> to >>>>> read through the documentation for each one in order to determine the >>>>> optimal settings for your configuration. >>>>> >>>>> Regards, >>>>> David Handermann >>>>> >>>>> On Thu, Feb 4, 2021 at 3:32 AM Ralph Vercauteren <r...@qad.com> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> Not sure if I need to ask this in this mailing list or the list of >>>>>> the developers. >>>>>> >>>>>> I want to create a REST server with NIFI. >>>>>> >>>>>> I have a Kafka queue storing messages. When something is connected to >>>>>> the REST server, the Kafka queue has to be read and all waiting messages >>>>>> should be collected and send back as a response to the REST client, >>>>>> >>>>>> I tried to google for a hint or solution, but I couldn't find it. >>>>>> >>>>>> With regards, >>>>>> Mit freundlichem Gruß, >>>>>> Met vriendelijke groet, >>>>>> >>>>>> *Ralph Vercauteren* >>>>>> Technical Architect QAD Automation Solutions >>>>>> Mobile NL: +31 6 5397 7230 >>>>>> r...@qad.com >>>>>> >>>>>> This e-mail may contain QAD proprietary information and should be >>>>>> treated as confidential. >>>>>> >>>>>
