Hello,
Connect the success relationship of the query part to update attribute and
use the expression language to keep counter so that you can control how
long to poll for. The expression can be something like
"${retry.counter:replaceNull('0'):plus(1)}" (on retry.counter attribute).
Then use RouteOnAttribute to discard flowfiles with retry_counter greater
than some value, and to poll the endpoint otherwise. Parse the output of
the poll for the completion flag (e.g. ScanContent/ExtractText). The
success relationship will give you the completed ones, so you can route
that to retrieve the actual results. Route the failed ones back to the
start of the loop (UpdateAttribute).
If you want to poll forever, then you do not need the
UpdateAttribute+RouteOnAttribute, just route the failed ExtractText back to
your poll, but limiting the retries looks best practice for me.
Hope this helps and let me know if you find an easier way to achieve this
J
On Sun, Nov 8, 2015 at 7:09 PM, M Singh <[email protected]> wrote:
> Hi:
>
> I am working on a data flow that involves repeated polling a http endpoint
> and based on a status flag in the response, fetch the results or poll the
> endpoint again.
>
> So, the flow is
>
> - Submit a query
> - Check the results of the query at a status endpoint returned from
> the query submission. If
> - job status is complete, then get the results from results endpoint
> and save it in a file
> - job status is still in progress, then make another request to the
> status endpoint in a loop until the status is complete
>
>
> I am stuck on how to implement the second part of step 2 above (repeatedly
> poll a http endpoint for status complete using a status flag).
>
> I could not figure out if there is any Nifi processor that can create a
> loop among the processors (in my case GetHttp). If there any pattern that
> combines the processors for creating the loop, please let me know.
>
> Thanks
>
>
>
