Re: POST multipart/form-data with Invokehttp

[Otto Fowler]

 Well it is in review right now, and there will be changes up coming.

On April 27, 2020 at 23:12:21, Luis Carmona (lcarm...@openpartner.cl) wrote:

Hi Otto,

Compiled your version, and It DID WORK !!.

How risky is to use this "version" in a production environment ?

Thanks a lot.

LC




On Mon, 2020-04-27 at 11:05 -0700, Otto Fowler wrote:
> No, Luis, if the PR is accepted and lands, it will be in the next
> released version of nifi after that.
> br/>> If you build nifi yourself, it will be availablle when you build
> master after it lands. br/>> br/>> On April 27, 2020 at 13:56:36, Luis
Carmona (lcarmona@@openpartner.cl)
> wrote:
> > Thank you all, br/>> > br/>> > Wesley and Etienne, is there any
documentation source aabout how to br/>> > connect a script in javascript
to nifi ressources, InputStream, br/>> > OutputStream, Erros, and so on ??
br/>> > br/>> > br/>> > Otto, sure I can give it a try, I am desperate for
thiss solution.
> > What br/>> > you mention means I have to lookk for a tutorial about
adding a
> > custom br/>> > processor right ?? br/>&> > br/>> > br/>> > Thanks
again, br/>> > br/>> > LC br/>> > br/>> > br/>> > br/>> > br/>> > On Mon,
2020-04-27 at 14:52 -0300, Wesley C. Dias de Olliveira
> > wrote: br/>> > > Owh!! br/>> >; > br/>> Great, Otto!! br/>> > > br/>>
GGood news!! br/>> > > br/>> Em seg., 27 de abr. dee 2020 às 14:50, Ottto
Fowler < br/>> > > ottobaackwa...@gmail.com> escreveu: br/>> > > > What
good timing, I just did : br/>> >
> > https:://github.com/apache/nifi/pull/4234 br/>> > > > If you can build
and try that would be sweet! or maybe a
> > review! br/>> > br/>> > On April 27, 2020 at 13:45:42, Etienne
> > Jouvin ( < br/>> > > > lapinoujouj...@gmail.com) wrote: br/>> > > > >
Hello. br/>> > > > > br/>> > > I did it with a prrocessor
EExecuteGroovyScript. br/>> > > > > br/> > > The script body is
somethinng like : br/>> > >; > > br/>> > > import
> > org.apache.http.entiity.mime.MultipartEntityBuilder
> > > > > import org.apache.http.entity.ContentType
> > > > > br/>> > > flowFFileList = session.get(100) br/>> > > > >
if(!!flowFileList.isEmpty()) { br/>> > > > > flowFFileList.each { flowFile
-> br//>> > > def multipart br/>> > &> > > String text =
flowFile.read().getText("UTF-8")
> > > > > br/>> > > flowFFile.write{streamIn, streamOut-> br/>> > > > >
multipart = MultipartEntityBuilder.create() br/>> > > > > //speciffy
multipart entries here br/>> > > > > .addTextBody(""object", text, br/>> >
> > > ContentType.APPLICATTION_JSON) br/>> > > > >
.addBinaryBody(""content", new br/>> > > > >
FFile(flowFile.'document.content.path'), br/>> > > > >
ContentType.create(flowFFile.'document.mime.type'), br/>> > > > >
flowFFile.'document.name') br/>> > > > > .build() br/>> > > > >
multipart.writeTo(streamOut) br/>> > > > > }} br/>> > > > > ///set the
`documentum.action.rest.content.type` attribute to br/>&> > > > > be used
as `Content-Type` in InvokeHTTP
> > > > > flowFile.'document.content.type' =
> > > > > multipart.getContentType().getValue() bbr/>> > > > >
session.transfer(flowFile, REL_SUCCESS) br/>> > > > > }} br/>> > > > > }}
br/>> > > > > br/>> > > br/>> > > Attributes are : < br/>> > > > >
document.conntent.path : content path br/>> > > > > document.mime..type :
content mime type br/>> > > > > document.name : binaire content name br/>>
> > > > br/>> >; > Output update attribute < br/>> > > > >
docuument.content.type : multipart content type. br/>> > > > > br/>> > >
You need some extra librairries : br/> > > > > httpcore-4.4.12.jar
br/>> > > >  httpmime-4.5.10.jar br/>> > > > > br/>> >; > This will
build a multipartt as the flowfile
> > content and you can br/>> > > > > usse it for the call after. br/>> > >
> > br/>>  > br/>> > > Etienne < br/>> > > > &> br/>> > > br/>> > > Le
lun. 27 avr. 2020 à 19:21, Luis
> > Carmona < br/>> > > > > lcarmona@@openpartner.cl> a écrit : br/>> > > >
> &> Hi everyone, br/>> > > > > > br/>> > > > Hoping everybody is doing ok,
wherever you are,
> > need some help br/>> > > > > > pllease. br/>> > > > > > br/>> > > >
DDoes anyone has ssent a file and parameters to
> > a REST point br/>> > > > > > usinng br/>> > > > > > Invokehhtp with
multipart/form-ddata as mime-type ? br/>> > > > > > br/>> &> > > I can't
figure ouut how to include the -F
> > , speaking br/>> > > >; > > in terms br/>> > > > > > of curl
syntaxx. br/>> > > > > > br/>> > > > I reaally need thiis done throught
NIFIso any
> > help will be highly br/>> > > > >  apreciated. br/>> > > > > >
br/>> >  > Thanks in advancee. br/>> > > > > > br/> > > > LC <
br/>> > > > > > br/> br/>> br/> < br/> <

Re: POST multipart/form-data with Invokehttp

[Luis Carmona]

Hi Otto,

Compiled your version, and It DID WORK !!.

How risky is to use this "version" in a production environment ?

Thanks a lot.

LC




On Mon, 2020-04-27 at 11:05 -0700, Otto Fowler wrote:
> No, Luis,  if the PR is accepted and lands, it will be in the next
> released version of nifi after that.
> 
> If you build nifi yourself, it will be available when you build
> master after it lands. 
> 
> On April 27, 2020 at 13:56:36, Luis Carmona (lcarm...@openpartner.cl)
> wrote:
> > Thank you all, 
> > 
> > Wesley and Etienne, is there any documentation source about how to 
> > connect a script in javascript to nifi resources, InputStream, 
> > OutputStream, Erros, and so on ? 
> > 
> > 
> > Otto, sure I can give it a try, I am desperate for this solution.
> > What 
> > you mention means I have to look for a tutorial about adding a
> > custom 
> > processor right ? 
> > 
> > 
> > Thanks again, 
> > 
> > LC 
> > 
> > 
> > 
> > 
> > On Mon, 2020-04-27 at 14:52 -0300, Wesley C. Dias de Oliveira
> > wrote: 
> > > Owh! 
> > > br/>> Great, Otto!! 
> > > br/>> Good news!! 
> > > br/>> Em seg., 27 de abr. de 2020 às 14:50, Ottto Fowler < 
> > > ottobackwa...@gmail.com> escreveu: 
> > > > What good timing, I just did : br/>> >
> > https:://github.com/apache/nifi/pull/4234 
> > > > If you can build and try that would be sweet! or maybe a
> > review! br/>> > br/>> > On April 27, 2020 at 13:45:42, Etienne
> > Jouvin ( < 
> > > > lapinoujou...@gmail.com) wrote: 
> > > > > Hello. 
> > > > > br/>> > > I did it with a processor EExecuteGroovyScript. 
> > > > > br/>> > > The script body is somethinng like : 
> > > > > br/>> > > import
> > org.apache.http.entiity.mime.MultipartEntityBuilder 
> > > > > import org.apache.http.entity.ContentType 
> > > > > br/>> > > flowFFileList = session.get(100) 
> > > > > if(!flowFileList.isEmpty()) { 
> > > > > flowFileList.each { flowFile -> br//>> > > def multipart 
> > > > > String text = flowFile.read().getText("UTF-8") 
> > > > > br/>> > > flowFFile.write{streamIn, streamOut-> 
> > > > > multipart = MultipartEntityBuilder.create() 
> > > > > //specify multipart entries here 
> > > > > .addTextBody("object", text, 
> > > > > ContentType.APPLICATION_JSON) 
> > > > > .addBinaryBody("content", new 
> > > > > File(flowFile.'document.content.path'), 
> > > > > ContentType.create(flowFile.'document.mime.type'), 
> > > > > flowFile.'document.name') 
> > > > > .build() 
> > > > > multipart.writeTo(streamOut) 
> > > > > } 
> > > > > //set the `documentum.action.rest.content.type` attribute to 
> > > > > be used as `Content-Type` in InvokeHTTP 
> > > > > flowFile.'document.content.type' = 
> > > > > multipart.getContentType().getValue() 
> > > > > session.transfer(flowFile, REL_SUCCESS) 
> > > > > } 
> > > > > } 
> > > > > br/>> > > br/>> > > Attributes are : < 
> > > > > document.content.path : content path 
> > > > > document.mime.type : content mime type 
> > > > > document.name : binaire content name 
> > > > > br/>> > > Output update attribute < 
> > > > > document.content.type : multipart content type. 
> > > > > br/>> > > You need some extra librairries : 
> > > > > httpcore-4.4.12.jar 
> > > > > httpmime-4.5.10.jar 
> > > > > br/>> > > This will build a multipartt as the flowfile
> > content and you can 
> > > > > use it for the call after. 
> > > > > br/>> > > br/>> > > Etienne < 
> > > > > br/>> > > br/>> > > Le lun. 27 avr. 2020 à 19:21, Luis
> > Carmona < 
> > > > > lcarm...@openpartner.cl> a écrit : 
> > > > > > Hi everyone, 
> > > > > > br/>> > > > Hoping everybody is doing ok, wherever you are,
> > need some help 
> > > > > > please. 
> > > > > > br/>> > > > Does anyone has ssent a file and parameters to
> > a REST point 
> > > > > > using 
> > > > > > Invokehhtp with multipart/form-data as mime-type ? 
> > > > > > br/>> > > > I can't figure ouut how to include the -F
> > , speaking 
> > > > > > in terms 
> > > > > > of curl syntax. 
> > > > > > br/>> > > > I really need thiis done throught NIFIso any
> > help will be highly 
> > > > > > apreciated. 
> > > > > > br/>> > > > Thanks in advancee. 
> > > > > > br/>> > > > LC < 
> > > > > > br/>> br/>> br/> < 

Re: POST multipart/form-data with Invokehttp

[Andy LoPresto]

ExecuteProcess and ExecuteStreamCommand both allow shell commands to be run; 
ExecuteProcess does not allow incoming flowfiles but ExecuteStreamCommand does. 

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Apr 27, 2020, at 2:31 PM, Luis Carmona  wrote:
> 
> Hi Wesley,
> 
> couldn't use Execute Processor as it doesn't receive an input FlowFile
> ( or I didn't find out how to connect it ) and I need to give to the
> procesor the file that should be sent.
> 
> Thanks anyway.
> 
> Will try with a script now.
> 
> LC
> 
> 
> 
> 
> On Mon, 2020-04-27 at 14:26 -0300, Wesley C. Dias de Oliveira wrote:
>> Hello, Luis.
>> 
>> Have you tried to send with ExecuteProcessor?
>> 
>> 
>> 
>> Using that way you can invoke curl explicit to run your command.
>> 
>> Em seg., 27 de abr. de 2020 às 14:21, Luis Carmona <
>> lcarm...@openpartner.cl> escreveu:
>>> Hi everyone,
>>> 
>>> Hoping everybody is doing ok, wherever you are, need some help
>>> please.
>>> 
>>> Does anyone has sent a file and parameters to a REST point using
>>> Invokehhtp with multipart/form-data as mime-type ?
>>> 
>>> I can't figure out how to include the -F , speaking in
>>> terms
>>> of curl syntax.
>>> 
>>> I really need this done throught NIFIso any help will be highly
>>> apreciated.
>>> 
>>> Thanks in advance.
>>> 
>>> LC
>>> 
>> 
>> 
> 

Re: POST multipart/form-data with Invokehttp

[Luis Carmona]

Hi Wesley,

couldn't use Execute Processor as it doesn't receive an input FlowFile
( or I didn't find out how to connect it ) and I need to give to the
procesor the file that should be sent.

Thanks anyway.

Will try with a script now.

LC




On Mon, 2020-04-27 at 14:26 -0300, Wesley C. Dias de Oliveira wrote:
> Hello, Luis.
> 
> Have you tried to send with ExecuteProcessor?
> 
> 
> 
> Using that way you can invoke curl explicit to run your command.
> 
> Em seg., 27 de abr. de 2020 às 14:21, Luis Carmona <
> lcarm...@openpartner.cl> escreveu:
> > Hi everyone,
> > 
> > Hoping everybody is doing ok, wherever you are, need some help
> > please.
> > 
> > Does anyone has sent a file and parameters to a REST point using
> > Invokehhtp with multipart/form-data as mime-type ?
> > 
> > I can't figure out how to include the -F , speaking in
> > terms
> > of curl syntax.
> > 
> > I really need this done throught NIFIso any help will be highly
> > apreciated.
> > 
> > Thanks in advance.
> > 
> > LC
> > 
> 
> 

Re: OIDC Redirect loop

[Ami Goldenberg]

Actually with a set of 1 this would not have mattered. I think the problem
is not with the sticky sessions...

On Mon, Apr 27, 2020 at 9:43 PM Ami Goldenberg  wrote:

> Great idea Andy
>
> I reduced the scale to 1 and it is still doing the same redirect loop.
> I guess the load balancer is hitting a different node even if sticky is
> set up? Even if eventually the service does clientAffinity maybe the client
> IP is not taken correctly?
>
> What are your thoughts?
>
> On Mon, Apr 27, 2020 at 9:25 PM Andy LoPresto 
> wrote:
>
>> Can you verify the initial redirect to OIDC and the callback are going to
>> the same node in NiFi? I see your LB configs are set to sticky sessions,
>> but it may be that if the callback is originating from the OIDC IDP server
>> rather than the actual client IP, the session affinity is not being
>> applied. Regardless, the error appears to indicate that the JWT provided in
>> the request to NiFi isn’t able to be validated, which indicates that the
>> key used to sign it isn’t present on that node, which is likely due to the
>> request being sent to a node other than the one that signed it.
>>
>> Quick and easy way to validate this would be to change the stateful set #
>> to 1 node and attempt the same sequence of operations.
>>
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> He/Him
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Apr 27, 2020, at 8:12 AM, Ami Goldenberg  wrote:
>>
>> Hi Nathan,
>> Indeed, that's the case
>>
>> On Mon, Apr 27, 2020 at 5:57 PM Nathan Gough  wrote:
>>
>>> Hi Ami,
>>>
>>> Just to confirm, the OAuth Client ID redirect URL in OIDC is set to "
>>> https://${nifi.hostname}:${nifi.port}/nifi-api/access/oidc/callback;
>>> and the NiFi property is set "nifi.security.user.oidc.discovery.url=
>>> https://accounts.google.com/.well-known/openid-configuration;.
>>>
>>> Nathan
>>>
>>> On Mon, Apr 27, 2020 at 12:37 AM Ami Goldenberg 
>>> wrote:
>>>
 Hi,

 We are trying to deploy NiFi on kubernetes after successfully using it
 for a while.
 The issue we are having is that every time we enter our nifi URL it
 will redirect us to Google and once we sign in we just get redirected 
 again.

 *The error I see on users.log is:*
 o.a.n.w.s.NiFiAuthenticationFilter Attempting request for ()
 GET https://XXX.XXX./nifi-api/flow/current-user
  (source ip:
 172.32.34.99)
 2020-04-25T19:48:06.256605759Z 2020-04-25 19:48:05,983 ERROR [NiFi
 Web Server-16] o.a.nifi.web.security.jwt.JwtService There was an error
 validating the JWT
 2020-04-25T19:48:06.256610178Z 2020-04-25 19:48:05,983 ERROR [NiFi Web
 Server-16] o.a.nifi.web.security.jwt.JwtService Unable to validate
 the access token.
 2020-04-25T19:48:06.256613727Z Caused by: JWT signature does not
 match locally computed signature. JWT validity cannot be asserted and
 should not be trusted.
 2020-04-25T19:48:06.256617124Z 2020-04-25 19:48:05,984 WARN [NiFi Web
 Server-16] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web
 api:Unable to validate the access token.

 *We're trying to follow practices from blogs and pvillard's repo:*

-

 https://github.com/pvillard31/nifi-gcp-terraform/tree/master/gcp-cluster-secured-nifi-oidc
-
https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect
-
https://medium.com/swlh/operationalising-nifi-on-kubernetes-1a8e0ae16a6c

 *Our set up is as such:*

- OIDC provider is Google
- TLS-toolkit running in server mode inside k8s
- StatefulSet of 3 replicas
- Zookeeper in K8s
- Ingress that is set up to create a load balancer in AWS - with
sticky sessions (based on cookie)
- Service that is set up with sessionAffinity: ClientIP


 Any idea which direction I should be checking next?anks!

>>>
>>

Re: OIDC Redirect loop

[Ami Goldenberg]

Great idea Andy

I reduced the scale to 1 and it is still doing the same redirect loop.
I guess the load balancer is hitting a different node even if sticky is set
up? Even if eventually the service does clientAffinity maybe the client IP
is not taken correctly?

What are your thoughts?

On Mon, Apr 27, 2020 at 9:25 PM Andy LoPresto  wrote:

> Can you verify the initial redirect to OIDC and the callback are going to
> the same node in NiFi? I see your LB configs are set to sticky sessions,
> but it may be that if the callback is originating from the OIDC IDP server
> rather than the actual client IP, the session affinity is not being
> applied. Regardless, the error appears to indicate that the JWT provided in
> the request to NiFi isn’t able to be validated, which indicates that the
> key used to sign it isn’t present on that node, which is likely due to the
> request being sent to a node other than the one that signed it.
>
> Quick and easy way to validate this would be to change the stateful set #
> to 1 node and attempt the same sequence of operations.
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Apr 27, 2020, at 8:12 AM, Ami Goldenberg  wrote:
>
> Hi Nathan,
> Indeed, that's the case
>
> On Mon, Apr 27, 2020 at 5:57 PM Nathan Gough  wrote:
>
>> Hi Ami,
>>
>> Just to confirm, the OAuth Client ID redirect URL in OIDC is set to "
>> https://${nifi.hostname}:${nifi.port}/nifi-api/access/oidc/callback; and
>> the NiFi property is set "nifi.security.user.oidc.discovery.url=
>> https://accounts.google.com/.well-known/openid-configuration;.
>>
>> Nathan
>>
>> On Mon, Apr 27, 2020 at 12:37 AM Ami Goldenberg 
>> wrote:
>>
>>> Hi,
>>>
>>> We are trying to deploy NiFi on kubernetes after successfully using it
>>> for a while.
>>> The issue we are having is that every time we enter our nifi URL it will
>>> redirect us to Google and once we sign in we just get redirected again.
>>>
>>> *The error I see on users.log is:*
>>> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for ()
>>> GET https://XXX.XXX./nifi-api/flow/current-user
>>>  (source ip:
>>> 172.32.34.99)
>>> 2020-04-25T19:48:06.256605759Z 2020-04-25 19:48:05,983 ERROR [NiFi
>>> Web Server-16] o.a.nifi.web.security.jwt.JwtService There was an error
>>> validating the JWT
>>> 2020-04-25T19:48:06.256610178Z 2020-04-25 19:48:05,983 ERROR [NiFi Web
>>> Server-16] o.a.nifi.web.security.jwt.JwtService Unable to validate
>>> the access token.
>>> 2020-04-25T19:48:06.256613727Z Caused by: JWT signature does not
>>> match locally computed signature. JWT validity cannot be asserted and
>>> should not be trusted.
>>> 2020-04-25T19:48:06.256617124Z 2020-04-25 19:48:05,984 WARN [NiFi Web
>>> Server-16] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web
>>> api:Unable to validate the access token.
>>>
>>> *We're trying to follow practices from blogs and pvillard's repo:*
>>>
>>>-
>>>
>>> https://github.com/pvillard31/nifi-gcp-terraform/tree/master/gcp-cluster-secured-nifi-oidc
>>>-
>>>https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect
>>>-
>>>https://medium.com/swlh/operationalising-nifi-on-kubernetes-1a8e0ae16a6c
>>>
>>> *Our set up is as such:*
>>>
>>>- OIDC provider is Google
>>>- TLS-toolkit running in server mode inside k8s
>>>- StatefulSet of 3 replicas
>>>- Zookeeper in K8s
>>>- Ingress that is set up to create a load balancer in AWS - with
>>>sticky sessions (based on cookie)
>>>- Service that is set up with sessionAffinity: ClientIP
>>>
>>>
>>> Any idea which direction I should be checking next?anks!
>>>
>>
>

Re: POST multipart/form-data with Invokehttp

[Luis Carmona]

OK Otto,

got it.

LC



On Mon, 2020-04-27 at 11:05 -0700, Otto Fowler wrote:
> No, Luis,  if the PR is accepted and lands, it will be in the next
> released version of nifi after that.
> 
> If you build nifi yourself, it will be available when you build
> master after it lands. 
> 
> On April 27, 2020 at 13:56:36, Luis Carmona (lcarm...@openpartner.cl)
> wrote:
> > Thank you all, 
> > 
> > Wesley and Etienne, is there any documentation source about how to 
> > connect a script in javascript to nifi resources, InputStream, 
> > OutputStream, Erros, and so on ? 
> > 
> > 
> > Otto, sure I can give it a try, I am desperate for this solution.
> > What 
> > you mention means I have to look for a tutorial about adding a
> > custom 
> > processor right ? 
> > 
> > 
> > Thanks again, 
> > 
> > LC 
> > 
> > 
> > 
> > 
> > On Mon, 2020-04-27 at 14:52 -0300, Wesley C. Dias de Oliveira
> > wrote: 
> > > Owh! 
> > > br/>> Great, Otto!! 
> > > br/>> Good news!! 
> > > br/>> Em seg., 27 de abr. de 2020 às 14:50, Ottto Fowler < 
> > > ottobackwa...@gmail.com> escreveu: 
> > > > What good timing, I just did : br/>> >
> > https:://github.com/apache/nifi/pull/4234 
> > > > If you can build and try that would be sweet! or maybe a
> > review! br/>> > br/>> > On April 27, 2020 at 13:45:42, Etienne
> > Jouvin ( < 
> > > > lapinoujou...@gmail.com) wrote: 
> > > > > Hello. 
> > > > > br/>> > > I did it with a processor EExecuteGroovyScript. 
> > > > > br/>> > > The script body is somethinng like : 
> > > > > br/>> > > import
> > org.apache.http.entiity.mime.MultipartEntityBuilder 
> > > > > import org.apache.http.entity.ContentType 
> > > > > br/>> > > flowFFileList = session.get(100) 
> > > > > if(!flowFileList.isEmpty()) { 
> > > > > flowFileList.each { flowFile -> br//>> > > def multipart 
> > > > > String text = flowFile.read().getText("UTF-8") 
> > > > > br/>> > > flowFFile.write{streamIn, streamOut-> 
> > > > > multipart = MultipartEntityBuilder.create() 
> > > > > //specify multipart entries here 
> > > > > .addTextBody("object", text, 
> > > > > ContentType.APPLICATION_JSON) 
> > > > > .addBinaryBody("content", new 
> > > > > File(flowFile.'document.content.path'), 
> > > > > ContentType.create(flowFile.'document.mime.type'), 
> > > > > flowFile.'document.name') 
> > > > > .build() 
> > > > > multipart.writeTo(streamOut) 
> > > > > } 
> > > > > //set the `documentum.action.rest.content.type` attribute to 
> > > > > be used as `Content-Type` in InvokeHTTP 
> > > > > flowFile.'document.content.type' = 
> > > > > multipart.getContentType().getValue() 
> > > > > session.transfer(flowFile, REL_SUCCESS) 
> > > > > } 
> > > > > } 
> > > > > br/>> > > br/>> > > Attributes are : < 
> > > > > document.content.path : content path 
> > > > > document.mime.type : content mime type 
> > > > > document.name : binaire content name 
> > > > > br/>> > > Output update attribute < 
> > > > > document.content.type : multipart content type. 
> > > > > br/>> > > You need some extra librairries : 
> > > > > httpcore-4.4.12.jar 
> > > > > httpmime-4.5.10.jar 
> > > > > br/>> > > This will build a multipartt as the flowfile
> > content and you can 
> > > > > use it for the call after. 
> > > > > br/>> > > br/>> > > Etienne < 
> > > > > br/>> > > br/>> > > Le lun. 27 avr. 2020 à 19:21, Luis
> > Carmona < 
> > > > > lcarm...@openpartner.cl> a écrit : 
> > > > > > Hi everyone, 
> > > > > > br/>> > > > Hoping everybody is doing ok, wherever you are,
> > need some help 
> > > > > > please. 
> > > > > > br/>> > > > Does anyone has ssent a file and parameters to
> > a REST point 
> > > > > > using 
> > > > > > Invokehhtp with multipart/form-data as mime-type ? 
> > > > > > br/>> > > > I can't figure ouut how to include the -F
> > , speaking 
> > > > > > in terms 
> > > > > > of curl syntax. 
> > > > > > br/>> > > > I really need thiis done throught NIFIso any
> > help will be highly 
> > > > > > apreciated. 
> > > > > > br/>> > > > Thanks in advancee. 
> > > > > > br/>> > > > LC < 
> > > > > > br/>> br/>> br/> < 

Re: OIDC Redirect loop

[Wyllys Ingersoll]

I have a very similar configuration and similar problem.  After
authenticating with the OIDC server (Keycloak), I often get multiple
failures in verifying the JWT from the nifi servers and have to reload the
browser multiple times until it eventually hits the right one.

On Mon, Apr 27, 2020 at 2:25 PM Andy LoPresto  wrote:

> Can you verify the initial redirect to OIDC and the callback are going to
> the same node in NiFi? I see your LB configs are set to sticky sessions,
> but it may be that if the callback is originating from the OIDC IDP server
> rather than the actual client IP, the session affinity is not being
> applied. Regardless, the error appears to indicate that the JWT provided in
> the request to NiFi isn’t able to be validated, which indicates that the
> key used to sign it isn’t present on that node, which is likely due to the
> request being sent to a node other than the one that signed it.
>
> Quick and easy way to validate this would be to change the stateful set #
> to 1 node and attempt the same sequence of operations.
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> He/Him
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Apr 27, 2020, at 8:12 AM, Ami Goldenberg  wrote:
>
> Hi Nathan,
> Indeed, that's the case
>
> On Mon, Apr 27, 2020 at 5:57 PM Nathan Gough  wrote:
>
>> Hi Ami,
>>
>> Just to confirm, the OAuth Client ID redirect URL in OIDC is set to "
>> https://${nifi.hostname}:${nifi.port}/nifi-api/access/oidc/callback; and
>> the NiFi property is set "nifi.security.user.oidc.discovery.url=
>> https://accounts.google.com/.well-known/openid-configuration;.
>>
>> Nathan
>>
>> On Mon, Apr 27, 2020 at 12:37 AM Ami Goldenberg 
>> wrote:
>>
>>> Hi,
>>>
>>> We are trying to deploy NiFi on kubernetes after successfully using it
>>> for a while.
>>> The issue we are having is that every time we enter our nifi URL it will
>>> redirect us to Google and once we sign in we just get redirected again.
>>>
>>> *The error I see on users.log is:*
>>> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for ()
>>> GET https://XXX.XXX./nifi-api/flow/current-user
>>>  (source ip:
>>> 172.32.34.99)
>>> 2020-04-25T19:48:06.256605759Z 2020-04-25 19:48:05,983 ERROR [NiFi
>>> Web Server-16] o.a.nifi.web.security.jwt.JwtService There was an error
>>> validating the JWT
>>> 2020-04-25T19:48:06.256610178Z 2020-04-25 19:48:05,983 ERROR [NiFi Web
>>> Server-16] o.a.nifi.web.security.jwt.JwtService Unable to validate
>>> the access token.
>>> 2020-04-25T19:48:06.256613727Z Caused by: JWT signature does not
>>> match locally computed signature. JWT validity cannot be asserted and
>>> should not be trusted.
>>> 2020-04-25T19:48:06.256617124Z 2020-04-25 19:48:05,984 WARN [NiFi Web
>>> Server-16] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web
>>> api:Unable to validate the access token.
>>>
>>> *We're trying to follow practices from blogs and pvillard's repo:*
>>>
>>>-
>>>
>>> https://github.com/pvillard31/nifi-gcp-terraform/tree/master/gcp-cluster-secured-nifi-oidc
>>>-
>>>https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect
>>>-
>>>https://medium.com/swlh/operationalising-nifi-on-kubernetes-1a8e0ae16a6c
>>>
>>> *Our set up is as such:*
>>>
>>>- OIDC provider is Google
>>>- TLS-toolkit running in server mode inside k8s
>>>- StatefulSet of 3 replicas
>>>- Zookeeper in K8s
>>>- Ingress that is set up to create a load balancer in AWS - with
>>>sticky sessions (based on cookie)
>>>- Service that is set up with sessionAffinity: ClientIP
>>>
>>>
>>> Any idea which direction I should be checking next?anks!
>>>
>>
>

Re: OIDC Redirect loop

[Andy LoPresto]

Can you verify the initial redirect to OIDC and the callback are going to the 
same node in NiFi? I see your LB configs are set to sticky sessions, but it may 
be that if the callback is originating from the OIDC IDP server rather than the 
actual client IP, the session affinity is not being applied. Regardless, the 
error appears to indicate that the JWT provided in the request to NiFi isn’t 
able to be validated, which indicates that the key used to sign it isn’t 
present on that node, which is likely due to the request being sent to a node 
other than the one that signed it. 

Quick and easy way to validate this would be to change the stateful set # to 1 
node and attempt the same sequence of operations. 


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Apr 27, 2020, at 8:12 AM, Ami Goldenberg  wrote:
> 
> Hi Nathan,
> Indeed, that's the case
> 
> On Mon, Apr 27, 2020 at 5:57 PM Nathan Gough  > wrote:
> Hi Ami,
> 
> Just to confirm, the OAuth Client ID redirect URL in OIDC is set to 
> "https://${nifi.hostname}:${nifi.port}/nifi-api/access/oidc/callback; and the 
> NiFi property is set 
> "nifi.security.user.oidc.discovery.url=https://accounts.google.com/.well-known/openid-configu
>  ration".
> 
> Nathan
> 
> On Mon, Apr 27, 2020 at 12:37 AM Ami Goldenberg  > wrote:
> Hi,
> 
> We are trying to deploy NiFi on kubernetes after successfully using it for a 
> while.
> The issue we are having is that every time we enter our nifi URL it will 
> redirect us to Google and once we sign in we just get redirected again.
> 
> The error I see on users.log is:
> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for () GET 
> https://XXX.XXX./nifi-api/flow/current-user 
>  (source ip: 172.32.34.99) 
> 2020-04-25T19:48:06.256605759Z 2020-04-25 19:48:05,983 ERROR [NiFi Web 
> Server-16] o.a.nifi.web.security.jwt.JwtService There was an error validating 
> the JWT
> 2020-04-25T19:48:06.256610178Z 2020-04-25 19:48:05,983 ERROR [NiFi Web 
> Server-16] o.a.nifi.web.security.jwt.JwtService Unable to validate the access 
> token.
> 2020-04-25T19:48:06.256613727Z Caused by: JWT signature does not match 
> locally computed signature. JWT validity cannot be asserted and should not be 
> trusted.
> 2020-04-25T19:48:06.256617124Z 2020-04-25 19:48:05,984 WARN [NiFi Web 
> Server-16] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web 
> api:Unable to validate the access token.
> 
> We're trying to follow practices from blogs and pvillard's repo:
> https://github.com/pvillard31/nifi-gcp-terraform/tree/master/gcp-cluster-secured-nifi-oidc
>  
> 
> https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect 
> 
> https://medium.com/swlh/operationalising-nifi-on-kubernetes-1a8e0ae16a6c 
> Our 
> set up is as such:
> OIDC provider is Google
> TLS-toolkit running in server mode inside k8s
> StatefulSet of 3 replicas
> Zookeeper in K8s
> Ingress that is set up to create a load balancer in AWS - with sticky 
> sessions (based on cookie)
> Service that is set up with sessionAffinity: ClientIP
> 
> Any idea which direction I should be checking next?anks!

Re: POST multipart/form-data with Invokehttp

[Luis Carmona]

Thank you all,

Wesley and Etienne, is there any documentation source about how to
connect a script in javascript to nifi resources, InputStream,
OutputStream, Erros, and so on ?


Otto, sure I can give it a try, I am desperate for this solution. What
you mention means I have to look for a tutorial about adding a custom
processor right ?


Thanks again,

LC




On Mon, 2020-04-27 at 14:52 -0300, Wesley C. Dias de Oliveira wrote:
> Owh!
> 
> Great, Otto!
> 
> Good news!
> 
> Em seg., 27 de abr. de 2020 às 14:50, Otto Fowler <
> ottobackwa...@gmail.com> escreveu:
> > What good timing, I just did : 
> > https://github.com/apache/nifi/pull/4234
> > If you can build and try that would be sweet!  or maybe a review! 
> > 
> > On April 27, 2020 at 13:45:42, Etienne Jouvin (
> > lapinoujou...@gmail.com) wrote:
> > > Hello.
> > > 
> > > I did it with a processor ExecuteGroovyScript.
> > > 
> > > The script body is something like :
> > > 
> > > import org.apache.http.entity.mime.MultipartEntityBuilder
> > > import org.apache.http.entity.ContentType
> > > 
> > > flowFileList = session.get(100)
> > > if(!flowFileList.isEmpty()) {
> > >   flowFileList.each { flowFile -> 
> > > def multipart
> > > String text = flowFile.read().getText("UTF-8")
> > > 
> > > flowFile.write{streamIn, streamOut->
> > >   multipart = MultipartEntityBuilder.create()
> > > //specify multipart entries here
> > > .addTextBody("object", text,
> > > ContentType.APPLICATION_JSON)
> > > .addBinaryBody("content", new
> > > File(flowFile.'document.content.path'),
> > > ContentType.create(flowFile.'document.mime.type'),
> > > flowFile.'document.name')
> > > .build()
> > >   multipart.writeTo(streamOut)
> > > }
> > > //set the `documentum.action.rest.content.type` attribute to
> > > be used as `Content-Type` in InvokeHTTP
> > > flowFile.'document.content.type' =
> > > multipart.getContentType().getValue()
> > > session.transfer(flowFile, REL_SUCCESS)
> > >   }
> > > }
> > > 
> > > 
> > > Attributes are :
> > > document.content.path : content path
> > > document.mime.type : content mime type
> > > document.name : binaire content name
> > > 
> > > Output update attribute
> > > document.content.type : multipart content type.
> > > 
> > > You need some extra librairies :
> > > httpcore-4.4.12.jar
> > > httpmime-4.5.10.jar
> > > 
> > > This will build a multipart as the flowfile content and you can
> > > use it for the call after.
> > > 
> > > 
> > > Etienne
> > > 
> > > 
> > > Le lun. 27 avr. 2020 à 19:21, Luis Carmona <
> > > lcarm...@openpartner.cl> a écrit :
> > > > Hi everyone,
> > > > 
> > > > Hoping everybody is doing ok, wherever you are, need some help
> > > > please.
> > > > 
> > > > Does anyone has sent a file and parameters to a REST point
> > > > using
> > > > Invokehhtp with multipart/form-data as mime-type ?
> > > > 
> > > > I can't figure out how to include the -F , speaking
> > > > in terms
> > > > of curl syntax.
> > > > 
> > > > I really need this done throught NIFIso any help will be highly
> > > > apreciated.
> > > > 
> > > > Thanks in advance.
> > > > 
> > > > LC
> > > > 
> 
> 

Re: POST multipart/form-data with Invokehttp

[Otto Fowler]

 No, Luis,  if the PR is accepted and lands, it will be in the next
released version of nifi after that.

If you build nifi yourself, it will be available when you build master
after it lands.

On April 27, 2020 at 13:56:36, Luis Carmona (lcarm...@openpartner.cl) wrote:

Thank you all,

Wesley and Etienne, is there any documentation source about how to
connect a script in javascript to nifi resources, InputStream,
OutputStream, Erros, and so on ?


Otto, sure I can give it a try, I am desperate for this solution. What
you mention means I have to look for a tutorial about adding a custom
processor right ?


Thanks again,

LC




On Mon, 2020-04-27 at 14:52 -0300, Wesley C. Dias de Oliveira wrote:
> Owh!
> br/>> Great, Otto!!
> br/>> Good news!!
> br/>> Em seg., 27 de abr. de 2020 às 14:50, Ottto Fowler <
> ottobackwa...@gmail.com> escreveu:
> > What good timing, I just did : br/>> > https:://
github.com/apache/nifi/pull/4234
> > If you can build and try that would be sweet! or maybe a review! br/>>
> br/>> > On April 27, 2020 at 13:45:42, Etienne Jouvin ( <
> > lapinoujou...@gmail.com) wrote:
> > > Hello.
> > > br/>> > > I did it with a processor EExecuteGroovyScript.
> > > br/>> > > The script body is somethinng like :
> > > br/>> > > import org.apache.http.entiity.mime.MultipartEntityBuilder
> > > import org.apache.http.entity.ContentType
> > > br/>> > > flowFFileList = session.get(100)
> > > if(!flowFileList.isEmpty()) {
> > > flowFileList.each { flowFile -> br//>> > > def multipart
> > > String text = flowFile.read().getText("UTF-8")
> > > br/>> > > flowFFile.write{streamIn, streamOut->
> > > multipart = MultipartEntityBuilder.create()
> > > //specify multipart entries here
> > > .addTextBody("object", text,
> > > ContentType.APPLICATION_JSON)
> > > .addBinaryBody("content", new
> > > File(flowFile.'document.content.path'),
> > > ContentType.create(flowFile.'document.mime.type'),
> > > flowFile.'document.name')
> > > .build()
> > > multipart.writeTo(streamOut)
> > > }
> > > //set the `documentum.action.rest.content.type` attribute to
> > > be used as `Content-Type` in InvokeHTTP
> > > flowFile.'document.content.type' =
> > > multipart.getContentType().getValue()
> > > session.transfer(flowFile, REL_SUCCESS)
> > > }
> > > }
> > > br/>> > > br/>> > > Attributes are : <
> > > document.content.path : content path
> > > document.mime.type : content mime type
> > > document.name : binaire content name
> > > br/>> > > Output update attribute <
> > > document.content.type : multipart content type.
> > > br/>> > > You need some extra librairries :
> > > httpcore-4.4.12.jar
> > > httpmime-4.5.10.jar
> > > br/>> > > This will build a multipartt as the flowfile content and
you can
> > > use it for the call after.
> > > br/>> > > br/>> > > Etienne <
> > > br/>> > > br/>> > > Le lun. 27 avr. 2020 à 19:21, Luis Carmona <
> > > lcarm...@openpartner.cl> a écrit :
> > > > Hi everyone,
> > > > br/>> > > > Hoping everybody is doing ok, wherever you are, need
some help
> > > > please.
> > > > br/>> > > > Does anyone has ssent a file and parameters to a REST
point
> > > > using
> > > > Invokehhtp with multipart/form-data as mime-type ?
> > > > br/>> > > > I can't figure ouut how to include the -F ,
speaking
> > > > in terms
> > > > of curl syntax.
> > > > br/>> > > > I really need thiis done throught NIFIso any help will
be highly
> > > > apreciated.
> > > > br/>> > > > Thanks in advancee.
> > > > br/>> > > > LC <
> > > > br/>> br/>> br/> <

Re: POST multipart/form-data with Invokehttp

[Wesley C. Dias de Oliveira]

Owh!

Great, Otto!

Good news!

Em seg., 27 de abr. de 2020 às 14:50, Otto Fowler 
escreveu:

> What good timing, I just did : https://github.com/apache/nifi/pull/4234
> If you can build and try that would be sweet!  or maybe a review!
>
> On April 27, 2020 at 13:45:42, Etienne Jouvin (lapinoujou...@gmail.com)
> wrote:
>
> Hello.
>
> I did it with a processor ExecuteGroovyScript.
>
> The script body is something like :
>
> import org.apache.http.entity.mime.MultipartEntityBuilder
> import org.apache.http.entity.ContentType
>
> flowFileList = session.get(100)
> if(!flowFileList.isEmpty()) {
>   flowFileList.each { flowFile ->
> def multipart
> String text = flowFile.read().getText("UTF-8")
>
> flowFile.write{streamIn, streamOut->
>   multipart = MultipartEntityBuilder.create()
> //specify multipart entries here
> .addTextBody("object", text, ContentType.APPLICATION_JSON)
> .addBinaryBody("content", new
> File(flowFile.'document.content.path'),
> ContentType.create(flowFile.'document.mime.type'), flowFile.'document.name
> ')
> .build()
>   multipart.writeTo(streamOut)
> }
> //set the `documentum.action.rest.content.type` attribute to be used
> as `Content-Type` in InvokeHTTP
> flowFile.'document.content.type' =
> multipart.getContentType().getValue()
> session.transfer(flowFile, REL_SUCCESS)
>   }
> }
>
>
> Attributes are :
>
>- document.content.path : content path
>- document.mime.type : content mime type
>- document.name : binaire content name
>
>
> Output update attribute
> document.content.type : multipart content type.
>
> You need some extra librairies :
>
>- httpcore-4.4.12.jar
>- httpmime-4.5.10.jar
>
>
> This will build a multipart as the flowfile content and you can use it for
> the call after.
>
>
> Etienne
>
>
> Le lun. 27 avr. 2020 à 19:21, Luis Carmona  a
> écrit :
>
>> Hi everyone,
>>
>> Hoping everybody is doing ok, wherever you are, need some help please.
>>
>> Does anyone has sent a file and parameters to a REST point using
>> Invokehhtp with multipart/form-data as mime-type ?
>>
>> I can't figure out how to include the -F , speaking in terms
>> of curl syntax.
>>
>> I really need this done throught NIFIso any help will be highly
>> apreciated.
>>
>> Thanks in advance.
>>
>> LC
>>
>>

-- 
Grato,
Wesley C. Dias de Oliveira.

Linux User nº 576838.

Re: POST multipart/form-data with Invokehttp

[Otto Fowler]

 What good timing, I just did : https://github.com/apache/nifi/pull/4234
If you can build and try that would be sweet!  or maybe a review!

On April 27, 2020 at 13:45:42, Etienne Jouvin (lapinoujou...@gmail.com)
wrote:

Hello.

I did it with a processor ExecuteGroovyScript.

The script body is something like :

import org.apache.http.entity.mime.MultipartEntityBuilder
import org.apache.http.entity.ContentType

flowFileList = session.get(100)
if(!flowFileList.isEmpty()) {
  flowFileList.each { flowFile ->
def multipart
String text = flowFile.read().getText("UTF-8")

flowFile.write{streamIn, streamOut->
  multipart = MultipartEntityBuilder.create()
//specify multipart entries here
.addTextBody("object", text, ContentType.APPLICATION_JSON)
.addBinaryBody("content", new
File(flowFile.'document.content.path'),
ContentType.create(flowFile.'document.mime.type'), flowFile.'document.name')
.build()
  multipart.writeTo(streamOut)
}
//set the `documentum.action.rest.content.type` attribute to be used as
`Content-Type` in InvokeHTTP
flowFile.'document.content.type' = multipart.getContentType().getValue()
session.transfer(flowFile, REL_SUCCESS)
  }
}


Attributes are :

   - document.content.path : content path
   - document.mime.type : content mime type
   - document.name : binaire content name


Output update attribute
document.content.type : multipart content type.

You need some extra librairies :

   - httpcore-4.4.12.jar
   - httpmime-4.5.10.jar


This will build a multipart as the flowfile content and you can use it for
the call after.


Etienne


Le lun. 27 avr. 2020 à 19:21, Luis Carmona  a
écrit :

> Hi everyone,
>
> Hoping everybody is doing ok, wherever you are, need some help please.
>
> Does anyone has sent a file and parameters to a REST point using
> Invokehhtp with multipart/form-data as mime-type ?
>
> I can't figure out how to include the -F , speaking in terms
> of curl syntax.
>
> I really need this done throught NIFIso any help will be highly
> apreciated.
>
> Thanks in advance.
>
> LC
>
>

Re: POST multipart/form-data with Invokehttp

[Luis Carmona]

Hi Wesley,

no, haven't used any processor related to do things out of NIFI itself.

Will give it a try thanks.

LC



On Mon, 2020-04-27 at 14:26 -0300, Wesley C. Dias de Oliveira wrote:
> Hello, Luis.
> 
> Have you tried to send with ExecuteProcessor?
> 
> 
> 
> Using that way you can invoke curl explicit to run your command.
> 
> Em seg., 27 de abr. de 2020 às 14:21, Luis Carmona <
> lcarm...@openpartner.cl> escreveu:
> > Hi everyone,
> > 
> > Hoping everybody is doing ok, wherever you are, need some help
> > please.
> > 
> > Does anyone has sent a file and parameters to a REST point using
> > Invokehhtp with multipart/form-data as mime-type ?
> > 
> > I can't figure out how to include the -F , speaking in
> > terms
> > of curl syntax.
> > 
> > I really need this done throught NIFIso any help will be highly
> > apreciated.
> > 
> > Thanks in advance.
> > 
> > LC
> > 
> 
> 

Re: POST multipart/form-data with Invokehttp

[Etienne Jouvin]

Hello.

I did it with a processor ExecuteGroovyScript.

The script body is something like :

import org.apache.http.entity.mime.MultipartEntityBuilder
import org.apache.http.entity.ContentType

flowFileList = session.get(100)
if(!flowFileList.isEmpty()) {
  flowFileList.each { flowFile ->
def multipart
String text = flowFile.read().getText("UTF-8")

flowFile.write{streamIn, streamOut->
  multipart = MultipartEntityBuilder.create()
//specify multipart entries here
.addTextBody("object", text, ContentType.APPLICATION_JSON)
.addBinaryBody("content", new
File(flowFile.'document.content.path'),
ContentType.create(flowFile.'document.mime.type'), flowFile.'document.name')
.build()
  multipart.writeTo(streamOut)
}
//set the `documentum.action.rest.content.type` attribute to be used as
`Content-Type` in InvokeHTTP
flowFile.'document.content.type' = multipart.getContentType().getValue()
session.transfer(flowFile, REL_SUCCESS)
  }
}


Attributes are :

   - document.content.path : content path
   - document.mime.type : content mime type
   - document.name : binaire content name


Output update attribute
document.content.type : multipart content type.

You need some extra librairies :

   - httpcore-4.4.12.jar
   - httpmime-4.5.10.jar


This will build a multipart as the flowfile content and you can use it for
the call after.


Etienne


Le lun. 27 avr. 2020 à 19:21, Luis Carmona  a
écrit :

> Hi everyone,
>
> Hoping everybody is doing ok, wherever you are, need some help please.
>
> Does anyone has sent a file and parameters to a REST point using
> Invokehhtp with multipart/form-data as mime-type ?
>
> I can't figure out how to include the -F , speaking in terms
> of curl syntax.
>
> I really need this done throught NIFIso any help will be highly
> apreciated.
>
> Thanks in advance.
>
> LC
>
>

POST multipart/form-data with Invokehttp

[Luis Carmona]

Hi everyone,

Hoping everybody is doing ok, wherever you are, need some help please.

Does anyone has sent a file and parameters to a REST point using
Invokehhtp with multipart/form-data as mime-type ?

I can't figure out how to include the -F , speaking in terms
of curl syntax.

I really need this done throught NIFIso any help will be highly
apreciated.

Thanks in advance.

LC

Re: POST multipart/form-data with Invokehttp

[Wesley C. Dias de Oliveira]

Hello, Luis.

Have you tried to send with ExecuteProcessor?

[image: image.png]

Using that way you can invoke curl explicit to run your command.

Em seg., 27 de abr. de 2020 às 14:21, Luis Carmona 
escreveu:

> Hi everyone,
>
> Hoping everybody is doing ok, wherever you are, need some help please.
>
> Does anyone has sent a file and parameters to a REST point using
> Invokehhtp with multipart/form-data as mime-type ?
>
> I can't figure out how to include the -F , speaking in terms
> of curl syntax.
>
> I really need this done throught NIFIso any help will be highly
> apreciated.
>
> Thanks in advance.
>
> LC
>
>

-- 
Grato,
Wesley C. Dias de Oliveira.

Linux User nº 576838.

Re: OIDC Redirect loop

[Ami Goldenberg]

Hi Nathan,
Indeed, that's the case

On Mon, Apr 27, 2020 at 5:57 PM Nathan Gough  wrote:

> Hi Ami,
>
> Just to confirm, the OAuth Client ID redirect URL in OIDC is set to "
> https://${nifi.hostname}:${nifi.port}/nifi-api/access/oidc/callback; and
> the NiFi property is set "nifi.security.user.oidc.discovery.url=
> https://accounts.google.com/.well-known/openid-configuration;.
>
> Nathan
>
> On Mon, Apr 27, 2020 at 12:37 AM Ami Goldenberg 
> wrote:
>
>> Hi,
>>
>> We are trying to deploy NiFi on kubernetes after successfully using it
>> for a while.
>> The issue we are having is that every time we enter our nifi URL it will
>> redirect us to Google and once we sign in we just get redirected again.
>>
>> *The error I see on users.log is:*
>> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for ()
>> GET https://XXX.XXX./nifi-api/flow/current-user (source ip:
>> 172.32.34.99)
>> 2020-04-25T19:48:06.256605759Z 2020-04-25 19:48:05,983 ERROR [NiFi
>> Web Server-16] o.a.nifi.web.security.jwt.JwtService There was an error
>> validating the JWT
>> 2020-04-25T19:48:06.256610178Z 2020-04-25 19:48:05,983 ERROR [NiFi Web
>> Server-16] o.a.nifi.web.security.jwt.JwtService Unable to validate
>> the access token.
>> 2020-04-25T19:48:06.256613727Z Caused by: JWT signature does not
>> match locally computed signature. JWT validity cannot be asserted and
>> should not be trusted.
>> 2020-04-25T19:48:06.256617124Z 2020-04-25 19:48:05,984 WARN [NiFi Web
>> Server-16] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web
>> api:Unable to validate the access token.
>>
>> *We're trying to follow practices from blogs and pvillard's repo:*
>>
>>-
>>
>> https://github.com/pvillard31/nifi-gcp-terraform/tree/master/gcp-cluster-secured-nifi-oidc
>>-
>>https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect
>>-
>>https://medium.com/swlh/operationalising-nifi-on-kubernetes-1a8e0ae16a6c
>>
>> *Our set up is as such:*
>>
>>- OIDC provider is Google
>>- TLS-toolkit running in server mode inside k8s
>>- StatefulSet of 3 replicas
>>- Zookeeper in K8s
>>- Ingress that is set up to create a load balancer in AWS - with
>>sticky sessions (based on cookie)
>>- Service that is set up with sessionAffinity: ClientIP
>>
>>
>> Any idea which direction I should be checking next?anks!
>>
>

Re: OIDC Redirect loop

[Nathan Gough]

Hi Ami,

Just to confirm, the OAuth Client ID redirect URL in OIDC is set to "
https://${nifi.hostname}:${nifi.port}/nifi-api/access/oidc/callback; and
the NiFi property is set "nifi.security.user.oidc.discovery.url=
https://accounts.google.com/.well-known/openid-configuration;.

Nathan

On Mon, Apr 27, 2020 at 12:37 AM Ami Goldenberg  wrote:

> Hi,
>
> We are trying to deploy NiFi on kubernetes after successfully using it
> for a while.
> The issue we are having is that every time we enter our nifi URL it will
> redirect us to Google and once we sign in we just get redirected again.
>
> *The error I see on users.log is:*
> o.a.n.w.s.NiFiAuthenticationFilter Attempting request for ()
> GET https://XXX.XXX./nifi-api/flow/current-user (source ip:
> 172.32.34.99)
> 2020-04-25T19:48:06.256605759Z 2020-04-25 19:48:05,983 ERROR [NiFi
> Web Server-16] o.a.nifi.web.security.jwt.JwtService There was an error
> validating the JWT
> 2020-04-25T19:48:06.256610178Z 2020-04-25 19:48:05,983 ERROR [NiFi Web
> Server-16] o.a.nifi.web.security.jwt.JwtService Unable to validate
> the access token.
> 2020-04-25T19:48:06.256613727Z Caused by: JWT signature does not
> match locally computed signature. JWT validity cannot be asserted and
> should not be trusted.
> 2020-04-25T19:48:06.256617124Z 2020-04-25 19:48:05,984 WARN [NiFi Web
> Server-16] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web
> api:Unable to validate the access token.
>
> *We're trying to follow practices from blogs and pvillard's repo:*
>
>-
>
> https://github.com/pvillard31/nifi-gcp-terraform/tree/master/gcp-cluster-secured-nifi-oidc
>-
>https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect
>-
>https://medium.com/swlh/operationalising-nifi-on-kubernetes-1a8e0ae16a6c
>
> *Our set up is as such:*
>
>- OIDC provider is Google
>- TLS-toolkit running in server mode inside k8s
>- StatefulSet of 3 replicas
>- Zookeeper in K8s
>- Ingress that is set up to create a load balancer in AWS - with
>sticky sessions (based on cookie)
>- Service that is set up with sessionAffinity: ClientIP
>
>
> Any idea which direction I should be checking next?anks!
>

Re: MergeRecord performance

[Mark Payne]

Robert,

What kind of performance degradation were you seeing here? I put together some 
simple flows to see if I could reproduce using 1.9.2 and current master.
My flow consisted of GenerateFlowFile (generating 2 CSV rows per FlowFile) -> 
ConvertRecord (to Avro) -> MergeRecord (read Avro, write Avro) -> 
UpdateAttribute to try to mimic what you’ve got, given the details that I have.

I did see a performance degradation on the order of about 10%. So on my laptop 
I went from processing 2.49 MM FlowFiles in 1.9.2 in 5 mins to 2.25 MM on the 
master branch. Interestingly, I saw no real change when I enabled Snappy 
compression.

For a point of reference, I also tried removing MergeRecord and just Generate 
-> Convert -> UpdateAttribute. I saw the same roughly 10% performance 
degradation.

I’m curious if you’re seeing more than that. If so, I think a template would be 
helpful to understand what’s different.

Thanks
-Mark


On Apr 24, 2020, at 4:50 PM, Robert R. Bruno 
mailto:rbru...@gmail.com>> wrote:

Joe,

In that part of the flow, we are using avro readers and writers.  We are using 
snappy compression (which could be part of the problem).  Since we are using 
avro at that point the embedded schema is being used by the reader and the 
writer is using the schema name property along with an internal schema registry 
in nifi.

I can see what could potentially be shared.

Thanks

On Fri, Apr 24, 2020 at 4:41 PM Joe Witt 
mailto:joe.w...@gmail.com>> wrote:
Robert,

Can you please detail the record readers and writers involved and how schemas 
are accessed?  There can be very important performance related changes in the 
parsers/serializers of the given formats.  And we've added a lot to make schema 
caching really capable but you have to opt into it.  It is of course possible 
MergeRecord itself is the culprit for performance reduction but lets get a more 
full picture here.

Are you able to share a template and sample data which we can use to replicate?

Thanks

On Fri, Apr 24, 2020 at 4:38 PM Robert R. Bruno 
mailto:rbru...@gmail.com>> wrote:
I wanted to see if anyone else has experienced performance issues with the 
newest version of nifi and MergeRecord?  We have been running on nifi 1.9.2 for 
awhile now, and recently upgraded to nifi 1.11.4.  Once upgraded, our identical 
flows were no longer able to keep up with our data mainly at MergeRecord 
processors.

We ended up downgrading back to nifi 1.9.2.  Once we downgraded, all was 
keeping up again.  There were no errors to speak of when we were running the 
flow with 1.11.4.  We did see higher load on the OS, but this may have been 
caused by the fact there was such a tremendous backlog built up in the flow.

Another side note, we saw one UpdateRecord processor producing errors when I 
tested the flow with nifi 1.11.4 with a small test flow.  I was able to fix 
this issue by changing some parameters in my RecordWriter.  So perhaps some 
underlying ways records are being handled since 1.9.2 caused the performance 
issue we saw?

Any insight anyone has would be greatly appreciated, as we very much would like 
to upgrade to nifi 1.11.4.  One thought was switching the MergeRecord 
processors to MergeContent since I've been told MergeContent seems to perform 
better, but not sure if this is actually true.  We are using the pattern of 
chaining a few MergeRecord processors together to help with performance.

Thanks in advance!

Re: OIDC Redirect loop

[Edward Armes]

Hi Ami,

Biased on the error you've got in the user log it looks like you've got 
a local trust issue. If you could tell us what you've already tried, 
someone might be able to help you a bit more.


Edward

On 27/04/2020 05:36, Ami Goldenberg wrote:

Hi,

We are trying to deploy NiFi on kubernetes after successfully using it 
for a while.
The issue we are having is that every time we enter our nifi URL it 
will redirect us to Google and once we sign in we just get 
redirected again.


_The error I see on users.log is:_
o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (token>) GET https://XXX.XXX./nifi-api/flow/current-user (source 
ip: 172.32.34.99)
2020-04-25T19:48:06.256605759Z 2020-04-25 19:48:05,983 ERROR [NiFi 
Web Server-16] o.a.nifi.web.security.jwt.JwtService There was an error 
validating the JWT
2020-04-25T19:48:06.256610178Z 2020-04-25 19:48:05,983 ERROR [NiFi Web 
Server-16] o.a.nifi.web.security.jwt.JwtService Unable to validate 
the access token.
2020-04-25T19:48:06.256613727Z Caused by: JWT signature does not 
match locally computed signature. JWT validity cannot be asserted and 
should not be trusted.
2020-04-25T19:48:06.256617124Z 2020-04-25 19:48:05,984 WARN [NiFi Web 
Server-16] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web 
api:Unable to validate the access token.


_We're trying to follow practices from blogs and pvillard's repo:_

  * 
https://github.com/pvillard31/nifi-gcp-terraform/tree/master/gcp-cluster-secured-nifi-oidc
  * https://bryanbende.com/development/2017/10/03/apache-nifi-openid-connect
  * https://medium.com/swlh/operationalising-nifi-on-kubernetes-1a8e0ae16a6c

_Our set up is as such:_

  * OIDC provider is Google
  * TLS-toolkit running in server mode inside k8s
  * StatefulSet of 3 replicas
  * Zookeeper in K8s
  * Ingress that is set up to create a load balancer in AWS - with
sticky sessions (based on cookie)
  * Service that is set up with sessionAffinity: ClientIP


Any idea which direction I should be checking next?anks!