Re: How to send back a reply from HandleHTTPRequest

[Pierre Villard]

James,

If you always want to return 200 as HTTP response whatever the request is,
then you could have:
HandleHttpRequest directly linked to HandleHttpResponse with property
status code to 200, and HandleHttpRequest linked to your flow to perform
the expected tasks.

Now let's say that you want to return a response based on the request, then
you can set processors between the HandleHttpRequest and the
HandleHttpResponse and then use the expression language to set the response
code to the value contained by an attribute of the flow file based on the
previous processors. Note that the content of the flow file will be used as
content of the HTTP response.

One example could be to expose a web service to retrieve files from HDFS (a
bit like WebHDFS): the user sends a request with the path of the file to
retrieve, then you link HandleHttpRequest to a FetchHDFS processor, then if
the file exists (success relationship) you can link to a HandleHttpResponse
with code 200 and the user will be able to get the file from HDFS, and if
the file does not exist (or if there is any kind of issue, failure
relationship), you can link the FetchHDFS to a HandleHttpResponse with an
error code 4xx. Obviously you could add some complexity to your workflow:
you could also allow users to send data to HDFS, etc, etc.

Hope this clarify a bit.

-Pierre


2017-02-07 23:51 GMT+01:00 James McMahon :

> Can the response be a standard http code that is automatically returned by
> the HandleHTTPResponse processor? Also, is it the Handle processor that is
> determining the correct response and sending it back, or a different
> processor? I guess I am still confused because I did not see any provision
> in the configuration for HandleHTTPResponse that sent back an http status.
> -Jim
>
> On Tue, Feb 7, 2017 at 4:08 PM, Aldrin Piri  wrote:
>
>> HI James,
>>
>> This would occur via the HandleHTTPResponse.  At a high level, the
>> request flowfile is routed to the HandleHTTPResponse processor.  Of course,
>> any kind of processing could occur between the two points.
>>
>> A simple example can be located on the sample templates page with the
>> Hello_NiFi_Web_Service.xml [1].  In this case, it is performing a very
>> simple replacement of text that is returned to the caller.
>>
>> Please let us know if you have any additional questions.
>>
>> --aldrin
>>
>> [1] https://cwiki.apache.org/confluence/display/NIFI/Example
>> +Dataflow+Templates
>>
>> On Tue, Feb 7, 2017 at 4:04 PM, James McMahon 
>> wrote:
>>
>>> Good evening. I have a number of customer applications that will be
>>> posting content to a NiFi HandleHTTPRequest processor. These apps need an
>>> http reply so that they know status. I can't find in the configuration any
>>> way to do that. I know that I must be overlooking what must be a
>>> commonplace requirement. Can anyone tell me how they configured their
>>> workflow so that it provides a response and status to the request? Thanks
>>> very much. -Jim
>>>
>>
>>
>

Flowfile handling in C# is possible or not?

[prabhu Mahendran]

Since processors code written as java  which handles flow file read and
write with relationship transfer.

Is it possible for flow file handling in .net application?

Many thanks,
prabhu

Re: How to send back a reply from HandleHTTPRequest

[James McMahon]

Can the response be a standard http code that is automatically returned by
the HandleHTTPResponse processor? Also, is it the Handle processor that is
determining the correct response and sending it back, or a different
processor? I guess I am still confused because I did not see any provision
in the configuration for HandleHTTPResponse that sent back an http status.
-Jim

On Tue, Feb 7, 2017 at 4:08 PM, Aldrin Piri  wrote:

> HI James,
>
> This would occur via the HandleHTTPResponse.  At a high level, the request
> flowfile is routed to the HandleHTTPResponse processor.  Of course, any
> kind of processing could occur between the two points.
>
> A simple example can be located on the sample templates page with the
> Hello_NiFi_Web_Service.xml [1].  In this case, it is performing a very
> simple replacement of text that is returned to the caller.
>
> Please let us know if you have any additional questions.
>
> --aldrin
>
> [1] https://cwiki.apache.org/confluence/display/NIFI/
> Example+Dataflow+Templates
>
> On Tue, Feb 7, 2017 at 4:04 PM, James McMahon 
> wrote:
>
>> Good evening. I have a number of customer applications that will be
>> posting content to a NiFi HandleHTTPRequest processor. These apps need an
>> http reply so that they know status. I can't find in the configuration any
>> way to do that. I know that I must be overlooking what must be a
>> commonplace requirement. Can anyone tell me how they configured their
>> workflow so that it provides a response and status to the request? Thanks
>> very much. -Jim
>>
>
>

Re: GetTwitter - Security/Certificate Issue

[Aldrin Piri]

Hi Dan,

Was this with an updated ca-certificates package or the original one listed
when this conversation started?

Should have asked from this initially, but could you also please provide
the output with verbose logging for the curl command?

curl -v  https://stream.twitter.com/

--Aldrin

On Tue, Feb 7, 2017 at 4:39 PM, Dan Giannone  wrote:

> Hi Aldrin,
>
>
>
> Here is a screenshot of the result. Looks like there is definitely an
> issue. Please let me know if this sheds any light on the issue.
>
>
>
> Thanks,
>
>
>
> Dan
>
>
>
> *From:* Aldrin Piri [mailto:aldrinp...@gmail.com]
> *Sent:* Monday, February 06, 2017 9:07 PM
>
> *To:* users@nifi.apache.org
> *Subject:* Re: GetTwitter - Security/Certificate Issue
>
>
>
> Hi Dan,
>
>
>
> Just as a quick diagnostic, are you able to curl
> https://stream.twitter.com/?  This will report in being unauthorized, but
> will at least confirm that the network connectivity with the associated
> endpoint used by the processor has appropriate access.  I have seen in
> certain environments that network proxies/filters can attempt to intervene
> in such requests causing similar errors to manifest.
>
>
>
> Please let us know your results.
>
>
>
> --aldrin
>
>
>
> On Fri, Feb 3, 2017 at 8:32 AM, Dan Giannone  wrote:
>
> Hi Aldrin,
>
>
>
> The machine in question is a linux server that we use as our ‘sandbox’ to
> try new things (nifi in this case), so I can definitely upgrade the yum
> package. As for your second question, the server runs on my company’s
> network, but other than that I don’t see any other considerations. Any
> thoughts?
>
>
>
> -Dan
>
>
>
> *From:* Aldrin Piri [mailto:aldrinp...@gmail.com]
> *Sent:* Wednesday, February 01, 2017 5:05 PM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: GetTwitter - Security/Certificate Issue
>
>
>
> Hi Dan,
>
>
>
> I did a bit of poking around and was not able to find that exact RPM
> version, but was not able to recreate with the CA certs from similar RPMs.
> As a quick check, is upgrading the mentioned yum package a possibility on
> the system?
>
>
>
> Are there any intervening network considerations or is the machine in
> question directly accessing the internet?
>
>
>
> On Wed, Feb 1, 2017 at 12:35 PM, Dan Giannone 
> wrote:
>
> Hi Aldrin,
>
>
>
> The version of jdk being used is 1.8. The details of the packages are
> attached in the PNG files. Please let me know if you need any additional
> info to help diagnose the issue!
>
>
>
> Thanks,
>
>
>
> Dan
>
>
>
>
>
> *From:* Aldrin Piri [mailto:aldrinp...@gmail.com]
> *Sent:* Tuesday, January 31, 2017 2:20 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: GetTwitter - Security/Certificate Issue
>
>
>
> Hi Dan,
>
>
>
> The GetTwitter processor does not make use of an Apache NiFi
> SSLContextService so the certificate chain issues are likely more tied to
> the JVM/OS specifically.  Did a quick check on some of the instances I am
> running and Twitter seems to be operating normally.
>
>
>
> Could you share some more details about your environment, specifically JRE
> being used?  If you are running a Linux variant, is your ca-certificates
> package (Yum based: ca-certificates, Aptitude based: 
> ca-cerificates/ca-certificates-java)
> up to date?  If so, what version is the package (Yum based: yum info
> ca-certificates, Aptitude based: apt-cache showpkg ca-certificates)?
>
>
>
> Thanks,
>
> Aldrin
>
>
>
>
>
> On Tue, Jan 31, 2017 at 1:28 PM, Andy LoPresto 
> wrote:
>
> Hi Dan,
>
>
>
> Yes, currently your processor is saying that it receives a certificate
> identifying https://www.twitter.com (or whatever the actual URL is) but
> it cannot build a complete chain between the presented certificate and a
> known CA/trusted certificate. This is because by default, NiFi doesn’t know
> any trusted certificates.
>
>
>
> You can configure a StandardSSLContextService in Controller Services which
> points the *truststore file* to $JRE_HOME/lib/security/cacerts (for
> example, on my Mac, it is /Library/Java/JavaVirtualMachines/jdk1.8.0_
> 101.jdk/Contents/Home/jre/lib/security/cacerts), and set the *truststore
> type* to JKS and the *truststore password* to “changeit”.
>
>
>
> There is an existing Jira discussing adding this by default [1], but there
> are pros and cons to that decision.
>
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-1477?jql=
> text%20~%20%22truststore%22%20AND%20project%20%3D%20%22Apache%20NiFi%22
> 
>
>
>
> Andy LoPresto
>
> alopre...@apache.org
>
> *alopresto.apa...@gmail.com *
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
>
>
> On Jan 31, 2017, at 6:40 AM, Dan Giannone  wrote:
>
>
>
> Hello,
>
>
>
> I am attempting to configure the GetTwitter processor. I’ve set the
> required properties such as consumer key and access token. However, when I
> turn it on I get the following error:
>
>
>
> Received error CONNE

RE: GetTwitter - Security/Certificate Issue

[Dan Giannone]

Hi Aldrin,

Here is a screenshot of the result. Looks like there is definitely an issue. 
Please let me know if this sheds any light on the issue.

Thanks,

Dan

From: Aldrin Piri [mailto:aldrinp...@gmail.com]
Sent: Monday, February 06, 2017 9:07 PM
To: users@nifi.apache.org
Subject: Re: GetTwitter - Security/Certificate Issue

Hi Dan,

Just as a quick diagnostic, are you able to curl https://stream.twitter.com/?  
This will report in being unauthorized, but will at least confirm that the 
network connectivity with the associated endpoint used by the processor has 
appropriate access.  I have seen in certain environments that network 
proxies/filters can attempt to intervene in such requests causing similar 
errors to manifest.

Please let us know your results.

--aldrin

On Fri, Feb 3, 2017 at 8:32 AM, Dan Giannone 
mailto:dgiann...@humana.com>> wrote:
Hi Aldrin,

The machine in question is a linux server that we use as our ‘sandbox’ to try 
new things (nifi in this case), so I can definitely upgrade the yum package. As 
for your second question, the server runs on my company’s network, but other 
than that I don’t see any other considerations. Any thoughts?

-Dan

From: Aldrin Piri [mailto:aldrinp...@gmail.com]
Sent: Wednesday, February 01, 2017 5:05 PM

To: users@nifi.apache.org
Subject: Re: GetTwitter - Security/Certificate Issue

Hi Dan,

I did a bit of poking around and was not able to find that exact RPM version, 
but was not able to recreate with the CA certs from similar RPMs.  As a quick 
check, is upgrading the mentioned yum package a possibility on the system?

Are there any intervening network considerations or is the machine in question 
directly accessing the internet?

On Wed, Feb 1, 2017 at 12:35 PM, Dan Giannone 
mailto:dgiann...@humana.com>> wrote:
Hi Aldrin,

The version of jdk being used is 1.8. The details of the packages are attached 
in the PNG files. Please let me know if you need any additional info to help 
diagnose the issue!

Thanks,

Dan


From: Aldrin Piri [mailto:aldrinp...@gmail.com]
Sent: Tuesday, January 31, 2017 2:20 PM
To: users@nifi.apache.org
Subject: Re: GetTwitter - Security/Certificate Issue

Hi Dan,

The GetTwitter processor does not make use of an Apache NiFi SSLContextService 
so the certificate chain issues are likely more tied to the JVM/OS 
specifically.  Did a quick check on some of the instances I am running and 
Twitter seems to be operating normally.

Could you share some more details about your environment, specifically JRE 
being used?  If you are running a Linux variant, is your ca-certificates 
package (Yum based: ca-certificates, Aptitude based: 
ca-cerificates/ca-certificates-java) up to date?  If so, what version is the 
package (Yum based: yum info ca-certificates, Aptitude based: apt-cache showpkg 
ca-certificates)?

Thanks,
Aldrin


On Tue, Jan 31, 2017 at 1:28 PM, Andy LoPresto 
mailto:alopre...@apache.org>> wrote:
Hi Dan,

Yes, currently your processor is saying that it receives a certificate 
identifying https://www.twitter.com (or whatever the actual URL is) but it 
cannot build a complete chain between the presented certificate and a known 
CA/trusted certificate. This is because by default, NiFi doesn’t know any 
trusted certificates.

You can configure a StandardSSLContextService in Controller Services which 
points the *truststore file* to $JRE_HOME/lib/security/cacerts (for example, on 
my Mac, it is 
/Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/security/cacerts),
 and set the *truststore type* to JKS and the *truststore password* to 
“changeit”.

There is an existing Jira discussing adding this by default [1], but there are 
pros and cons to that decision.

[1] 
https://issues.apache.org/jira/browse/NIFI-1477?jql=text%20~%20%22truststore%22%20AND%20project%20%3D%20%22Apache%20NiFi%22

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jan 31, 2017, at 6:40 AM, Dan Giannone 
mailto:dgiann...@humana.com>> wrote:

Hello,

I am attempting to configure the GetTwitter processor. I’ve set the required 
properties such as consumer key and access token. However, when I turn it on I 
get the following error:

Received error CONNECTION_ERROR: sun.security.validator.validatorexception pkix 
path building failed 
sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
valid certification path to requested target. Will attempt to reconnect

It’s pretty clear there is some sort of certificate/security issue. How would I 
go about correcting this?

Thanks,

Dan Giannone


The information transmitted is intended only for the

Re: How to send back a reply from HandleHTTPRequest

[James McMahon]

Outstanding. I will check this out this evening and try it first thing in
the morning when I am back at site Aldrin. Thank you very much, sir. -Jim
Mc.

On Tue, Feb 7, 2017 at 4:08 PM, Aldrin Piri  wrote:

> HI James,
>
> This would occur via the HandleHTTPResponse.  At a high level, the request
> flowfile is routed to the HandleHTTPResponse processor.  Of course, any
> kind of processing could occur between the two points.
>
> A simple example can be located on the sample templates page with the
> Hello_NiFi_Web_Service.xml [1].  In this case, it is performing a very
> simple replacement of text that is returned to the caller.
>
> Please let us know if you have any additional questions.
>
> --aldrin
>
> [1] https://cwiki.apache.org/confluence/display/NIFI/
> Example+Dataflow+Templates
>
> On Tue, Feb 7, 2017 at 4:04 PM, James McMahon 
> wrote:
>
>> Good evening. I have a number of customer applications that will be
>> posting content to a NiFi HandleHTTPRequest processor. These apps need an
>> http reply so that they know status. I can't find in the configuration any
>> way to do that. I know that I must be overlooking what must be a
>> commonplace requirement. Can anyone tell me how they configured their
>> workflow so that it provides a response and status to the request? Thanks
>> very much. -Jim
>>
>
>

Re: Writing back through a python stream callback when the flowfile content is a mix of character and binary

[James McMahon]

There are still issues without using bytearray[], Matt. I tried using a
json function in its place to format my data as json, and it still occurs.

I still have this problem but have implemented a temp workaround. I don't
think it is a very good one, as it turns out. As one of our previous
collaborators suggested, I SplitContent and then later after operating on
just the text data in the header I MergeContent to bring the pieces back
together. Problem with this is that there are limits to the number of
"split" flowfiles you can try to bring back at any one time. And to make it
15-20K, you need to up a parameter in nifi.properties. If I bump it up to
20,000 , let's say, then as soon as the 20001 fragment appears it rolls the
oldest one off to Failure. I can't have this happen. My flow volume is much
too high to throttle it down to this level. While I got it to work for the
time being by restricting my ListFile prior to my FetchFile, my approach
will nto scale to my customer's needs.

I hope this makes some modest sense. I am typing all this in her from home
without my NiFi flow and etails in front of me. Cheers and thanks again fro
any future insights you may have. -Jim Mc.

On Fri, Feb 3, 2017 at 10:39 PM, Matt Burgess  wrote:

> James,
>
> I haven't had a chance to dig into this yet, but one thing I noticed
> about your script was an issue identified by Bryan Rosander (NiFi
> committer and all-around good guy :) as the probable cause of the
> TypeError, namely the calling of bytearray() after encode() (the
> latter of which already returns a byte array) [1]. Does removing the
> call to bytearray() fix your script, or are there still issues with
> decoding the input stream?
>
> Regards,
> Matt
>
> [1] https://community.hortonworks.com/questions/81291/nifi-
> executescript-processor-error-using-string-in.html
>
>
> On Thu, Feb 2, 2017 at 5:38 PM, James McMahon 
> wrote:
> > I have a flowfile that has tagged character information I need to get at
> > throughout the first few sections of the file. I need to use regex in
> python
> > to select some of those values and to transform others. I am using an
> > ExecuteScript processor to execute my python code. Here is my approach:
> >
> >
> >
> > = = = = =
> >
> > class PyStreamCallback(StreamCallback) :
> >
> >def __init__ (self) :
> >
> >def process(self, inputSteam, outputStream) :
> >
> >   stuff = IOUtils.toString(inputStream, StandardCharsets.UTF_8)  #
> what
> > happens to my binary and extreme chars when they get passed through this
> > step?
> >
> >  .
> >
> >  . (transform and pick out select content)
> >
> >  .
> >
> >  outputStream.write(bytearray(stuff.encode(‘utf-8’ # am I
> using
> > the wrong functions to put my text chars and my binary and my extreme
> chars
> > back on the stream as a byte stream? What should I be doing to handle the
> > variety of data?
> >
> >
> >
> > flowFile = session.get()
> >
> > if (flowFile!= None)
> >
> >incoming = flowFile.getAttribute(‘filename’)
> >
> >logging.info(‘about to process file: %s’, incoming)
> >
> >flowFile = session.write(flowFile, PyStreamCallback())   # line 155
> in my
> > code
> >
> >session.transfer(flowFile, REL_SUCCESS)
> >
> >session.commit()
> >
> >
> >
> > = = = = =
> >
> >
> >
> > When my incoming flowfile is all character content - such as tagged xml
> - my
> > code works fine. All the flowfiles that also contain some binary data
> and/or
> > characters at the extremes such as foreign language characters don’t
> work.
> > They error out. I suspect it has to do with the way I am writing back to
> the
> > flowfile stream.
> >
> >
> >
> > Here is the error I am getting:
> >
> > Org.apache.nifi.processor.exception.ProcessException:
> > javax.script.ScriptException: TypeError: write(): 1st arg can’t be
> coerced
> > to int, byte[] in 

Re: How to send back a reply from HandleHTTPRequest

[Aldrin Piri]

HI James,

This would occur via the HandleHTTPResponse.  At a high level, the request
flowfile is routed to the HandleHTTPResponse processor.  Of course, any
kind of processing could occur between the two points.

A simple example can be located on the sample templates page with the
Hello_NiFi_Web_Service.xml [1].  In this case, it is performing a very
simple replacement of text that is returned to the caller.

Please let us know if you have any additional questions.

--aldrin

[1]
https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates

On Tue, Feb 7, 2017 at 4:04 PM, James McMahon  wrote:

> Good evening. I have a number of customer applications that will be
> posting content to a NiFi HandleHTTPRequest processor. These apps need an
> http reply so that they know status. I can't find in the configuration any
> way to do that. I know that I must be overlooking what must be a
> commonplace requirement. Can anyone tell me how they configured their
> workflow so that it provides a response and status to the request? Thanks
> very much. -Jim
>

How to send back a reply from HandleHTTPRequest

[James McMahon]

Good evening. I have a number of customer applications that will be posting
content to a NiFi HandleHTTPRequest processor. These apps need an http
reply so that they know status. I can't find in the configuration any way
to do that. I know that I must be overlooking what must be a commonplace
requirement. Can anyone tell me how they configured their workflow so that
it provides a response and status to the request? Thanks very much. -Jim

Re: Apache NIFI Connector to MS Dynamics CRM (cloud) ?

[James Wing]

Mike,

I am not familiar with a specialized connector for MS Dynamics, but NiFi
has good general-purpose support for databases via JDBC and HTTP(S) web
services (see InvokeHTTP).  It is likely that NiFi can do at least some
interaction with Dynamics using only the built-in processors.

But NiFi is also a great framework for building custom components with both
scripting languages and Java, if you go that route.

Thanks,

James

On Tue, Feb 7, 2017 at 11:58 AM, Michael Porter  wrote:

> Is there an already developed connector to communicate from apache nifi
> (for example an MS SQLserver db) TO a MS Dynamics CRM cloud instance?  My
> search seems to not turn up anything related.  If there is not, is there
> another nifi round-about method?
>
>
>
>
>
> Thx in advance,
>
>
>
> Mike
>
>
>

Apache NIFI Connector to MS Dynamics CRM (cloud) ?

[Michael Porter]

Is there an already developed connector to communicate from apache nifi (for 
example an MS SQLserver db) TO a MS Dynamics CRM cloud instance?  My search 
seems to not turn up anything related.  If there is not, is there another nifi 
round-about method?


Thx in advance,

Mike

Re: Problem when using backpressure to distribute load over nodes in a cluster

[Bas van Kortenhof]

Hi Koji,

Thanks for the quick response. I have set the batch size to 1 indeed, and
the flow you describe works, but my problem is a bit more complex. I'll try
to show it with an example:



In this case Node 1 is parsing a flow file (indicated by the X in the
connection between FetchS3Object and Parse). Both connections have a
backpressure threshold of 1, but because the object is already fetched, the
first connection is empty and can thus be filled. This means that, if a new
item becomes available in the queue, both of the following cases can happen
with equal probability:



I'd like to force the second case to happen, because node 2 has more
resources available.

I hope this explains the situation a bit better. So basically I want the
backpressure to occur based on a threshold on the whole flow, not an
individual connection. I haven't found a way to do this up to this point.

Hopefully you have an idea how to achieve this.

Regards,
Bas



--
View this message in context: 
http://apache-nifi-users-list.2361937.n4.nabble.com/Problem-when-using-backpressure-to-distribute-load-over-nodes-in-a-cluster-tp863p877.html
Sent from the Apache NiFi Users List mailing list archive at Nabble.com.