Re: Dynamic URLs using InvokeHttp from an array

[Adam Taft]

You are probably missing the necessary change to the following file:

META-INF/services/org.apache.nifi.processor.Processor

​If you haven't modified this file to include your processor, this would be
the problem.

Adam
​


On Fri, Apr 1, 2016 at 2:53 PM, kkang  wrote:

> I looked into the links, but they didn't quite give me what I was looking
> for.  Instead, I thought I would try a different route.  I "borrowed" the
> code from InvokeHttp and created another processor called InvokeHttpLooped
> (I know...not very original).
>
> I modified it so that I could put place holders in the URL (example:
> "http://somedns.com/${0}?query=${1}"...I tried following the same type of
> pattern already established).  I then added another property
> (ContextDataList) that should contain 1 or more rows with comma delimited
> values.  In a loop for the number of line in ContextDataList, I create a
> separate FlowFile, fill in the placeholders matching the sequence in the
> comma separated values, and finally do what InvokeHttp did for each
> iteration of the loop for the number of rows.
>
> Since this was the first time using Maven, I finally succeeded in getting a
> build done; however, when I copied in the NAR file, the InvokeHttpLooped
> did
> not show up in the UI.
>
> I probably missed some steps...any suggestions?
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Dynamic-URLs-using-InvokeHttp-from-an-array-tp8638p8722.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Dynamic URLs using InvokeHttp from an array

[kkang]

I looked into the links, but they didn't quite give me what I was looking
for.  Instead, I thought I would try a different route.  I "borrowed" the
code from InvokeHttp and created another processor called InvokeHttpLooped
(I know...not very original).  

I modified it so that I could put place holders in the URL (example:
"http://somedns.com/${0}?query=${1}"...I tried following the same type of
pattern already established).  I then added another property
(ContextDataList) that should contain 1 or more rows with comma delimited
values.  In a loop for the number of line in ContextDataList, I create a
separate FlowFile, fill in the placeholders matching the sequence in the
comma separated values, and finally do what InvokeHttp did for each
iteration of the loop for the number of rows.

Since this was the first time using Maven, I finally succeeded in getting a
build done; however, when I copied in the NAR file, the InvokeHttpLooped did
not show up in the UI.

I probably missed some steps...any suggestions?



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Dynamic-URLs-using-InvokeHttp-from-an-array-tp8638p8722.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Dynamic URLs using InvokeHttp from an array

[Adam Taft]

Yeah, these solutions won't work for thousands of iterations.  Andy's
suggestion for using ExecuteScript starts to sound very compelling,
especially if you are algorithmically generating your term values.

Another thought for you.  Uwe Geercken was experimenting with a processor
which could read in a CSV file and output a flowfile attribute for every
cell in the CSV data.  Something like this might work for you.

Basically you'd have a single column CSV file with all your terms.  For
every line in the file, a new flowfile would be produced.  Each "column"
from each line would be stored as a flowfile attribute.  You'd end up with
a new flowfile for every term, with a flowfile attribute containing that
term.

Here's a link to his work:

https://github.com/uwegeercken/nifi_processors

Here's an archive from the mailing list discussion:

http://mail-archives.apache.org/mod_mbox/nifi-dev/201603.mbox/%3Ctrinity-4e63574c-9f19-459f-b048-ca40667e964c-1458542998682@3capp-webde-bs02%3E

Something like this might be worth considering as well.

On Thu, Mar 31, 2016 at 9:10 AM, kkang  wrote:

> Thanks, but unfortunately I have thousands of iterations that must occur so
> this would probably be too tedious; however, it is a technique that may
> come
> in handy with smaller looped scenarios.  I am still looking at the
> solutions
> that Andy sent earlier.
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Dynamic-URLs-using-InvokeHttp-from-an-array-tp8638p8658.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Dynamic URLs using InvokeHttp from an array

[Adam Taft]

OK, one more "out the box" idea to consider.

UpdateAttribute also has a mode which "clones" the flowfile if multiple
rules are matched.  Here's the specific quote from the UpdateAttribute
documentation:

"If the FlowFile policy is set to "use clone", and multiple rules match,
then a copy of the incoming FlowFile is created, such that the number of
outgoing FlowFiles is equal to the number of rules that match. In other
words, if two rules (A and B) both match, then there will be two outgoing
FlowFiles, one for Rule A and one for Rule B. This can be useful in
situations where you want to add an attribute to use as a flag for routing
later. In this example, there will be two copies of the file available, one
to route for the A path, and one to route for the B path"

If you used the Advanced UI, you might be able to create rules which always
match, but alter the value of the $foo parameter to your liking.  If the
"use clone" option was set, it would create a new flowfile for every rule
matched.  Thus if your array had 10 values, you'd have 10 rules, each one
would set $foo to a different value.  Out from UpdateAttribute, you'd end
up with 10 flowfiles that could be sent to InvokeHTTP.

That might be a fun way to solve this.  :)


On Thu, Mar 31, 2016 at 9:55 AM, Adam Taft  wrote:

> One (possibly bad) idea would be to try and loop your flow around the
> UpdateAttribute processor using RouteOnAttribute.  UpdateAttribute has an
> "advanced" mode which would let you do logic something like:
>
> if $foo == "" then set $foo = "step 1";
> if $foo == "step 1" then set $foo = "step 2";
> if $foo == "step 2" then set $foo = "step 3";
> ...
> if $foo == "step n" then set $foo = "finished";
>
> The next part would be RouteOnAttribute, which would read the value of
> $foo and if set to "finished" break the loop.  Otherwise it would pass to
> InvokeHTTP and then back to UpdateAttribute.  The setup for this would be
> tedious, but I think it would technically work.
>
> Just putting this out there for brainstorming purposes.
>
>
>
>
> On Wed, Mar 30, 2016 at 6:25 PM, kkang  wrote:
>
>> I have been able to figure out how to GenerateFlowFile -> UpdateAttribute
>> ->
>> InvokeHttp to dynamically send a URL (example:
>> https://somedomain.com?parameterx=${foo}); however, I need to do this N
>> number of times and replace ${foo} with a known set of values.  Is there a
>> way to call InvokeHttp multiple times and use the next value for ${foo}
>> automatically?
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-nifi-developer-list.39713.n7.nabble.com/Dynamic-URLs-using-InvokeHttp-from-an-array-tp8638.html
>> Sent from the Apache NiFi Developer List mailing list archive at
>> Nabble.com.
>>
>
>

Re: Dynamic URLs using InvokeHttp from an array

[kkang]

Thanks, but unfortunately I have thousands of iterations that must occur so
this would probably be too tedious; however, it is a technique that may come
in handy with smaller looped scenarios.  I am still looking at the solutions
that Andy sent earlier.



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Dynamic-URLs-using-InvokeHttp-from-an-array-tp8638p8658.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Dynamic URLs using InvokeHttp from an array

[Adam Taft]

One (possibly bad) idea would be to try and loop your flow around the
UpdateAttribute processor using RouteOnAttribute.  UpdateAttribute has an
"advanced" mode which would let you do logic something like:

if $foo == "" then set $foo = "step 1";
if $foo == "step 1" then set $foo = "step 2";
if $foo == "step 2" then set $foo = "step 3";
...
if $foo == "step n" then set $foo = "finished";

The next part would be RouteOnAttribute, which would read the value of $foo
and if set to "finished" break the loop.  Otherwise it would pass to
InvokeHTTP and then back to UpdateAttribute.  The setup for this would be
tedious, but I think it would technically work.

Just putting this out there for brainstorming purposes.




On Wed, Mar 30, 2016 at 6:25 PM, kkang  wrote:

> I have been able to figure out how to GenerateFlowFile -> UpdateAttribute
> ->
> InvokeHttp to dynamically send a URL (example:
> https://somedomain.com?parameterx=${foo}); however, I need to do this N
> number of times and replace ${foo} with a known set of values.  Is there a
> way to call InvokeHttp multiple times and use the next value for ${foo}
> automatically?
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Dynamic-URLs-using-InvokeHttp-from-an-array-tp8638.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Dynamic URLs using InvokeHttp from an array

[Andy LoPresto]

This sounds like a good candidate for the `ExecuteScript` processor. Matt 
Burgess has written some good tutorials on using that here [1] [2]. You could 
also write a custom processor that extends `InvokeHTTP` and uses the new state 
management features [3] to keep a counter value, an iteration limit, and the 
known values.

[1] 
http://funnifi.blogspot.com/2016/02/writing-reusable-scripted-processors-in.html
 

[2] 
http://funnifi.blogspot.com/2016/03/executescript-json-to-json-revisited_14.html
 

[3] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#state_management
 


Andy LoPresto
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Mar 30, 2016, at 3:25 PM, kkang  wrote:
> 
> I have been able to figure out how to GenerateFlowFile -> UpdateAttribute ->
> InvokeHttp to dynamically send a URL (example:
> https://somedomain.com?parameterx=${foo}); however, I need to do this N
> number of times and replace ${foo} with a known set of values.  Is there a
> way to call InvokeHttp multiple times and use the next value for ${foo}
> automatically?
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Dynamic-URLs-using-InvokeHttp-from-an-array-tp8638.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.



signature.asc
Description: Message signed with OpenPGP using GPGMail