RE: How to ingest files into HDFS via Apache NiFi from non-hadoop environment

[Takanobu Asanuma]

Hello Mothi86,

I think you can achieve it by using HttpFS on HDFS side. It is a part of hadoop 
library and a proxy server for HDFS.
https://hadoop.apache.org/docs/stable/hadoop-hdfs-httpfs/index.html

In your case, running HttpFS server on the management nodes or the edge node 
would be good. And set 'webhdfs://{HttpFS hostname}:{port}' to 'fs.defaultFS' 
in core-site.xml for NiFi's HDFS processors. Then, your NiFi cluster only need 
to access the HttpFS server and can access HDFS from non-hadoop environment.

Regards,
Takanobu

RE: Authorization problems of NiFi secured cluster

[Takanobu Asanuma]

Hi Matt and Koji,

Thanks for the information. So if there is not any flow.xml.gz in conf 
directory when a secured nifi cluster is starting, we need to add "Node 
Identity" (and "Initial Admin Identity") to the policies (each component or PG) 
explicitly, right? That's my case. After adding flow.xml.gz and then starting 
the secured cluster, I confirmed that the policies are set automatically.

-Original Message-
From: Koji Kawamura [mailto:ijokaruma...@gmail.com] 
Sent: Tuesday, June 27, 2017 10:06 PM
To: dev 
Subject: Re: Authorization problems of NiFi secured cluster

Thanks Matt for clarification. My cluster had an existing flow.xml I happened 
copied from another NiFi instance.

On Jun 27, 2017 9:14 PM, "Matt Gilman"  wrote:

Takanobu,

The dataflow-specific policies (any policies on the root Process Group) are 
only granted for new instances when there is an existing flow.xml.gz in your 
/conf directory. When there is no flow and the NiFi instance is 
joining a cluster the policies cannot be granted at start up because the 
components technically do not exist yet. However, your Initial Admin is given 
the required permissions to grant those dataflow-specific policies once the 
nodes have all joined the cluster. There is a short snippet in the Admin guide 
describing this behavior [1] (if you scroll down a little bit looking for the 
little info (i) icon on the left).

Hope that clears it up.

Matt

[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-
guide.html#authorizer-configuration

On Tue, Jun 27, 2017 at 6:03 AM, Takanobu Asanuma 
wrote:

> Hi Koji,
>
> Thank you very much for the confirmation. Hmm... I will continue to 
> investigate why my cluster does not work correctly.
>
> Thanks again,
> Takanobu
>
> -Original Message-
> From: Koji Kawamura [mailto:ijokaruma...@gmail.com]
> Sent: Tuesday, June 27, 2017 5:59 PM
> To: dev 
> Subject: Re: Authorization problems of NiFi secured cluster
>
> I just created a brand-new secured cluster now. NiFi automatically 
> created a policy "view the data" (and others) with the user defined as 
> "Initial Admin Identity" and "Node Identity" in conf/authorizers.xml.
> It seems working as expected.
>
> Koji
>
> On Tue, Jun 27, 2017 at 5:26 PM, Koji Kawamura 
> 
> wrote:
> > Hi Takanobu,
> >
> > Glad to hear that you have it fixed.
> >
> >> Although I defined the Node Identity before stating the cluster at 
> >> the
> first time, it seemed NiFi did not automatically create the policies 
> and I needed to add the Node Identity to the policy explicitly.
> >
> > Thanks for sharing, ideally NiFi cluster should work without adding 
> > the policy manually.
> > I will try to setup a brand-new secured NiFi cluster to see what 
> > initial policy setting will look like.
> > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.htm
> > l#
> > cluster-node-identities
> >
> > Thanks,
> > Koji
> >
> > On Tue, Jun 27, 2017 at 5:08 PM, Takanobu Asanuma 
> >  wrote:
> >> Hi Koji,
> >>
> >> Thank you for your quick and valuable answer! That's exactly what I
> need. After adding "Node Identity" of authorizers.xml to the "view the 
> data" policy, the authorized user can list the queue.
> >>
>  IIRC, if you define the Node Identity before starting the secured
> cluster at the first time, NiFi automatically creates necessary 
> policies for each node to proxy user request (I maybe wrong on this..).
> >>
> >> Although I defined the Node Identity before stating the cluster at 
> >> the
> first time, it seemed NiFi did not automatically create the policies 
> and I needed to add the Node Identity to the policy explicitly.
> >>
> >> Thanks again!
> >> Takanobu
> >>
> >> -Original Message-
> >> From: Koji Kawamura [mailto:ijokaruma...@gmail.com]
> >> Sent: Tuesday, June 27, 2017 2:32 PM
> >> To: dev 
> >> Subject: Re: Authorization problems of NiFi secured cluster
> >>
> >> Hello Takanobu,
> >>
> >> If the issue doesn't happen with standalone mode, I assume it 
> >> happens
> because the security policy does not allow NiFi node to "view the data".
> >>
> >> When a user sends a request to a node within a cluster, the node
> proxies the request to other nodes within the same cluster.
> >> I'd recommend to check if conf/authorizers.xml has Node Identity
> properties, looks like this:
> >>
> >> 
> >>   ...
> >>   CN=localhost, OU=NIFI 
> >> 
> >>
> >> IIRC, if you define the Node Identity before starting the secured
> cluster at the first time, NiFi automatically creates necessary 
> policies for each node to proxy user request (I maybe wrong on 
> this..). If you already have the cluster started, then you can add 
> NiFi node as a user
then
> add it to the "view the data" policy manually (probably at the root 
> PG's policy would be the most appropriate place).
> >>
> >> I 

Re: NiFi Assembly NOTICE/LICENSE Maintenance

[Tony Kurc]

For reference:
Netty 4.1 NOTICE
https://github.com/netty/netty/blob/4.1/NOTICE.txt
Netty 3.7  NOTICE
https://github.com/netty/netty/blob/3.7/NOTICE.txt




On Tue, Jun 27, 2017 at 10:21 PM, Tony Kurc  wrote:

> Background, I asked Mike how he put together the LICENSE, and why he added
> a separate section in the LICENSE for Google Protocol Buffers 3.3.1, and
> his answer that made sense was "well, what existed there was there had a
> version (2.5.0)".
>
> Interesting note, the Google Protocol Buffers LICENSE looks to be the
> same.
>
> Sort of the opposite issue with Netty. NOTICE didn't have a version of
> Netty, and the NOTICES between versions were fairly different.
>
>
>
> On Tue, Jun 27, 2017 at 10:16 PM, Michael Hogue <
> michael.p.hogu...@gmail.com> wrote:
>
>> Hello all,
>>
>>I'm attempting to merge a LICENSE and NOTICE i've created for a new
>> grpc
>> processor bundle [1,2] into the NiFi assembly. I've run into a couple of
>> things i don't know how to resolve:
>>
>> 1. If I add a new (transitive) dependency with a newer version than exists
>> elsewhere in the code _and_ the licenses are the same except for the
>> version, do the license for each of the versions need spelled out in the
>> nifi assembly LICENSE file?
>>
>> 2. One of the grpc dependencies i've added pulls in a version of netty
>> fairly different than what exists in the code already. Should there be a
>> separate block in the assembly NOTICE if they differ? Is it sufficient to
>> add to the existing netty block?
>>
>> PR reference:
>> https://github.com/apache/nifi/pull/1947/files#diff-c3a3e6d0
>> 27b17e530efdb23269e95968R1132
>>
>> Thanks,
>> Mike
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-4037
>> [2] https://issues.apache.org/jira/browse/NIFI-4038
>>
>
>

Re: NiFi Assembly NOTICE/LICENSE Maintenance

[Tony Kurc]

Background, I asked Mike how he put together the LICENSE, and why he added
a separate section in the LICENSE for Google Protocol Buffers 3.3.1, and
his answer that made sense was "well, what existed there was there had a
version (2.5.0)".

Interesting note, the Google Protocol Buffers LICENSE looks to be the same.

Sort of the opposite issue with Netty. NOTICE didn't have a version of
Netty, and the NOTICES between versions were fairly different.



On Tue, Jun 27, 2017 at 10:16 PM, Michael Hogue  wrote:

> Hello all,
>
>I'm attempting to merge a LICENSE and NOTICE i've created for a new grpc
> processor bundle [1,2] into the NiFi assembly. I've run into a couple of
> things i don't know how to resolve:
>
> 1. If I add a new (transitive) dependency with a newer version than exists
> elsewhere in the code _and_ the licenses are the same except for the
> version, do the license for each of the versions need spelled out in the
> nifi assembly LICENSE file?
>
> 2. One of the grpc dependencies i've added pulls in a version of netty
> fairly different than what exists in the code already. Should there be a
> separate block in the assembly NOTICE if they differ? Is it sufficient to
> add to the existing netty block?
>
> PR reference:
> https://github.com/apache/nifi/pull/1947/files#diff-
> c3a3e6d027b17e530efdb23269e95968R1132
>
> Thanks,
> Mike
>
> [1] https://issues.apache.org/jira/browse/NIFI-4037
> [2] https://issues.apache.org/jira/browse/NIFI-4038
>

NiFi Assembly NOTICE/LICENSE Maintenance

[Michael Hogue]

Hello all,

   I'm attempting to merge a LICENSE and NOTICE i've created for a new grpc
processor bundle [1,2] into the NiFi assembly. I've run into a couple of
things i don't know how to resolve:

1. If I add a new (transitive) dependency with a newer version than exists
elsewhere in the code _and_ the licenses are the same except for the
version, do the license for each of the versions need spelled out in the
nifi assembly LICENSE file?

2. One of the grpc dependencies i've added pulls in a version of netty
fairly different than what exists in the code already. Should there be a
separate block in the assembly NOTICE if they differ? Is it sufficient to
add to the existing netty block?

PR reference:
https://github.com/apache/nifi/pull/1947/files#diff-c3a3e6d027b17e530efdb23269e95968R1132

Thanks,
Mike

[1] https://issues.apache.org/jira/browse/NIFI-4037
[2] https://issues.apache.org/jira/browse/NIFI-4038

Re: Does PostHTTP support Multipart/form-data ?

[Adam Taft]

The multipart/form-data body would have to be preemptively created and
stored in your flowfile payload.  InvokeHTTP could then be used to POST the
message body to the remote server (after having set the appropriate
content-type).

i.e. you have to manually construct the multipart form in the flowfile
payload itself, the NIFI processors aren't going to do that for you like
curl does.

Does that make sense?

On Mon, Jun 26, 2017 at 1:44 PM, icreatedanaccount <
pierluc.boudr...@gmail.com> wrote:

> I know they work great for sending parameters in the request headers, but
> does the PostHTTP and InvokeHTTP processors support multipart/form-data as
> a
> content Content-Type ?
>
> Imagine something like this in CURL :
> curl -v -X POST -F "file=@/19010230.bin"
>
> Best,
> Luc
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/Does-PostHTTP-support-Multipart-
> form-data-tp16259.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: How to ingest files into HDFS via Apache NiFi from non-hadoop environment

[Adam Taft]

This is a bit outside of the box, but I have actually implemented this
solution previously.

My scenario was very similar.  NIFI was installed outside of the firewalled
HDFS cluster.  The only external access to the HDFS cluster was through SSH.

Therefore, my solution was to use SSH to call a remote command on the HDFS
node.  This was enabled using the ExecuteStreamCommand processor.  I used
the hadoop fs command line tools, piping in the contents of the flowfile.

The basic command (assuming put) would look something like this:

$>  cat file.ext | hadoop fs -put - /hdfs/path/file.ext

This would read from standard input and store the stream into file.ext.
Next you add the SSH execution to call the above.

$>  cat file.ext | ssh user@remote 'hadoop fs -put - /hdfs/path/file.ext'

Now we can try to put the above into the ExecuteStreamCommand processor.
We will extract the filename from the flowfile attribute.  I like using
bash to execute my script:

ExecuteStreamCommand
Command Path:  /bin/bash
Command Arguments: -c; "ssh user@remote 'hadoop fs -put -
/hdfs/path/${filename}'"* unsure of the quotes here

Not sure if the above helps, since it sounds like you're going for
something more than 'get' and 'put'.  But the above is an easy mechanism to
interact with an HDFS cluster if the NIFI node is not running on the
cluster.



On Fri, Jun 23, 2017 at 2:53 PM, Mothi86  wrote:

> Okay thanks so that clarifies that NiFi will not work in terms of
> integrating
> from local machine / non-hadoop environment to hadoop environment. It
> either
> has to be in edge node or built up a node similar restriction of edge or
> management node.
>
> Is this HDF recommended solution ?
>
> Will spinning a VM work ? Can you suggest me VM requirements for Apache
> NiFi
> ?
>
>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/How-to-ingest-files-into-HDFS-via-
> Apache-NiFi-from-non-hadoop-environment-tp16247p16252.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Expression Language For AMQP Processors

[Joey Frazee]

Sometimes it’s just an historical oversight that EL support isn’t available for 
certain properties, but in this case the connection is initialized only once, 
so the connection specific properties like hostname, port, username and 
password cannot be evaluated against FlowFile attributes in any meaningful way.

Now, since this processor was first contributed, the variable registry was 
added so there is now a scenario where you could use EL and properties from the 
variable registry. This would be a reasonable enhancement, though there’s a 
risk of confusion since it won’t behave correctly if the expressions are in 
terms of FlowFile attributes.

-joey

> On Jun 27, 2017, at 8:33 AM, hemantvsn  wrote:
> 
> Hi,
> 
> For publishing/consuming messages on Rabbit queue, NIFI provides support via
> PublishAMQP and ConsumeAMQP processors.
> 
> However, only limited properties support EL while others don't.
> 
> Properties supporting EL
> Exchange Name
> Routing Key
> 
> Thus we can dynamically control these.
> 
> However remaining properties like
> hostname
> port
> username
> password
> 
> aren't controlled by EL
> 
> Any idea why this was done and things to do to make them configurable
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Expression-Language-For-AMQP-Processors-tp16269.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Expression Language For AMQP Processors

[hemantvsn]

Hi,

For publishing/consuming messages on Rabbit queue, NIFI provides support via
PublishAMQP and ConsumeAMQP processors.

However, only limited properties support EL while others don't.

Properties supporting EL
Exchange Name
Routing Key

Thus we can dynamically control these.

However remaining properties like
hostname
port
username
password

aren't controlled by EL

Any idea why this was done and things to do to make them configurable



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Expression-Language-For-AMQP-Processors-tp16269.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Authorization problems of NiFi secured cluster

[Koji Kawamura]

Thanks Matt for clarification. My cluster had an existing flow.xml I
happened copied from another NiFi instance.

On Jun 27, 2017 9:14 PM, "Matt Gilman"  wrote:

Takanobu,

The dataflow-specific policies (any policies on the root Process Group) are
only granted for new instances when there is an existing flow.xml.gz in
your /conf directory. When there is no flow and the NiFi
instance is joining a cluster the policies cannot be granted at start up
because the components technically do not exist yet. However, your Initial
Admin is given the required permissions to grant those dataflow-specific
policies once the nodes have all joined the cluster. There is a short
snippet in the Admin guide describing this behavior [1] (if you scroll down
a little bit looking for the little info (i) icon on the left).

Hope that clears it up.

Matt

[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-
guide.html#authorizer-configuration

On Tue, Jun 27, 2017 at 6:03 AM, Takanobu Asanuma 
wrote:

> Hi Koji,
>
> Thank you very much for the confirmation. Hmm... I will continue to
> investigate why my cluster does not work correctly.
>
> Thanks again,
> Takanobu
>
> -Original Message-
> From: Koji Kawamura [mailto:ijokaruma...@gmail.com]
> Sent: Tuesday, June 27, 2017 5:59 PM
> To: dev 
> Subject: Re: Authorization problems of NiFi secured cluster
>
> I just created a brand-new secured cluster now. NiFi automatically created
> a policy "view the data" (and others) with the user defined as "Initial
> Admin Identity" and "Node Identity" in conf/authorizers.xml.
> It seems working as expected.
>
> Koji
>
> On Tue, Jun 27, 2017 at 5:26 PM, Koji Kawamura 
> wrote:
> > Hi Takanobu,
> >
> > Glad to hear that you have it fixed.
> >
> >> Although I defined the Node Identity before stating the cluster at the
> first time, it seemed NiFi did not automatically create the policies and I
> needed to add the Node Identity to the policy explicitly.
> >
> > Thanks for sharing, ideally NiFi cluster should work without adding
> > the policy manually.
> > I will try to setup a brand-new secured NiFi cluster to see what
> > initial policy setting will look like.
> > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#
> > cluster-node-identities
> >
> > Thanks,
> > Koji
> >
> > On Tue, Jun 27, 2017 at 5:08 PM, Takanobu Asanuma
> >  wrote:
> >> Hi Koji,
> >>
> >> Thank you for your quick and valuable answer! That's exactly what I
> need. After adding "Node Identity" of authorizers.xml to the "view the
> data" policy, the authorized user can list the queue.
> >>
>  IIRC, if you define the Node Identity before starting the secured
> cluster at the first time, NiFi automatically creates necessary policies
> for each node to proxy user request (I maybe wrong on this..).
> >>
> >> Although I defined the Node Identity before stating the cluster at the
> first time, it seemed NiFi did not automatically create the policies and I
> needed to add the Node Identity to the policy explicitly.
> >>
> >> Thanks again!
> >> Takanobu
> >>
> >> -Original Message-
> >> From: Koji Kawamura [mailto:ijokaruma...@gmail.com]
> >> Sent: Tuesday, June 27, 2017 2:32 PM
> >> To: dev 
> >> Subject: Re: Authorization problems of NiFi secured cluster
> >>
> >> Hello Takanobu,
> >>
> >> If the issue doesn't happen with standalone mode, I assume it happens
> because the security policy does not allow NiFi node to "view the data".
> >>
> >> When a user sends a request to a node within a cluster, the node
> proxies the request to other nodes within the same cluster.
> >> I'd recommend to check if conf/authorizers.xml has Node Identity
> properties, looks like this:
> >>
> >> 
> >>   ...
> >>   CN=localhost, OU=NIFI
> >> 
> >>
> >> IIRC, if you define the Node Identity before starting the secured
> cluster at the first time, NiFi automatically creates necessary policies
> for each node to proxy user request (I maybe wrong on this..). If you
> already have the cluster started, then you can add NiFi node as a user
then
> add it to the "view the data" policy manually (probably at the root PG's
> policy would be the most appropriate place).
> >>
> >> I confirmed that the issue can be reproduced by removing NiFi node user
> from "view the data" policy.
> >>
> >> Please try above and let us know if it addresses your issue.
> >>
> >> Thanks,
> >> Koji
> >>
> >> On Tue, Jun 27, 2017 at 1:12 PM, Takanobu Asanuma <
> tasan...@yahoo-corp.jp> wrote:
> >>> Hello experts,
> >>>
> >>> When I created a NiFi cluster with security, any users can't list any
> queues due to "insufficient permissions" though the users have the
> permissions.
> >>>
> >>> For example, there is a dataflow which contains processor-A and
> processor-B, and processor-A is connecting to processor-B. In this case,
> even if user1 has the policies 

Re: Authorization problems of NiFi secured cluster

[Matt Gilman]

Takanobu,

The dataflow-specific policies (any policies on the root Process Group) are
only granted for new instances when there is an existing flow.xml.gz in
your /conf directory. When there is no flow and the NiFi
instance is joining a cluster the policies cannot be granted at start up
because the components technically do not exist yet. However, your Initial
Admin is given the required permissions to grant those dataflow-specific
policies once the nodes have all joined the cluster. There is a short
snippet in the Admin guide describing this behavior [1] (if you scroll down
a little bit looking for the little info (i) icon on the left).

Hope that clears it up.

Matt

[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#authorizer-configuration

On Tue, Jun 27, 2017 at 6:03 AM, Takanobu Asanuma 
wrote:

> Hi Koji,
>
> Thank you very much for the confirmation. Hmm... I will continue to
> investigate why my cluster does not work correctly.
>
> Thanks again,
> Takanobu
>
> -Original Message-
> From: Koji Kawamura [mailto:ijokaruma...@gmail.com]
> Sent: Tuesday, June 27, 2017 5:59 PM
> To: dev 
> Subject: Re: Authorization problems of NiFi secured cluster
>
> I just created a brand-new secured cluster now. NiFi automatically created
> a policy "view the data" (and others) with the user defined as "Initial
> Admin Identity" and "Node Identity" in conf/authorizers.xml.
> It seems working as expected.
>
> Koji
>
> On Tue, Jun 27, 2017 at 5:26 PM, Koji Kawamura 
> wrote:
> > Hi Takanobu,
> >
> > Glad to hear that you have it fixed.
> >
> >> Although I defined the Node Identity before stating the cluster at the
> first time, it seemed NiFi did not automatically create the policies and I
> needed to add the Node Identity to the policy explicitly.
> >
> > Thanks for sharing, ideally NiFi cluster should work without adding
> > the policy manually.
> > I will try to setup a brand-new secured NiFi cluster to see what
> > initial policy setting will look like.
> > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#
> > cluster-node-identities
> >
> > Thanks,
> > Koji
> >
> > On Tue, Jun 27, 2017 at 5:08 PM, Takanobu Asanuma
> >  wrote:
> >> Hi Koji,
> >>
> >> Thank you for your quick and valuable answer! That's exactly what I
> need. After adding "Node Identity" of authorizers.xml to the "view the
> data" policy, the authorized user can list the queue.
> >>
>  IIRC, if you define the Node Identity before starting the secured
> cluster at the first time, NiFi automatically creates necessary policies
> for each node to proxy user request (I maybe wrong on this..).
> >>
> >> Although I defined the Node Identity before stating the cluster at the
> first time, it seemed NiFi did not automatically create the policies and I
> needed to add the Node Identity to the policy explicitly.
> >>
> >> Thanks again!
> >> Takanobu
> >>
> >> -Original Message-
> >> From: Koji Kawamura [mailto:ijokaruma...@gmail.com]
> >> Sent: Tuesday, June 27, 2017 2:32 PM
> >> To: dev 
> >> Subject: Re: Authorization problems of NiFi secured cluster
> >>
> >> Hello Takanobu,
> >>
> >> If the issue doesn't happen with standalone mode, I assume it happens
> because the security policy does not allow NiFi node to "view the data".
> >>
> >> When a user sends a request to a node within a cluster, the node
> proxies the request to other nodes within the same cluster.
> >> I'd recommend to check if conf/authorizers.xml has Node Identity
> properties, looks like this:
> >>
> >> 
> >>   ...
> >>   CN=localhost, OU=NIFI
> >> 
> >>
> >> IIRC, if you define the Node Identity before starting the secured
> cluster at the first time, NiFi automatically creates necessary policies
> for each node to proxy user request (I maybe wrong on this..). If you
> already have the cluster started, then you can add NiFi node as a user then
> add it to the "view the data" policy manually (probably at the root PG's
> policy would be the most appropriate place).
> >>
> >> I confirmed that the issue can be reproduced by removing NiFi node user
> from "view the data" policy.
> >>
> >> Please try above and let us know if it addresses your issue.
> >>
> >> Thanks,
> >> Koji
> >>
> >> On Tue, Jun 27, 2017 at 1:12 PM, Takanobu Asanuma <
> tasan...@yahoo-corp.jp> wrote:
> >>> Hello experts,
> >>>
> >>> When I created a NiFi cluster with security, any users can't list any
> queues due to "insufficient permissions" though the users have the
> permissions.
> >>>
> >>> For example, there is a dataflow which contains processor-A and
> processor-B, and processor-A is connecting to processor-B. In this case,
> even if user1 has the policies which are view/modify the component/data of
> processor-A and processor-B, he can't list the queue of the processors.
> >>>
> >>> This problem only occurs when the secured NiFi instance is 

RE: Authorization problems of NiFi secured cluster

[Takanobu Asanuma]

Hi Koji,

Thank you very much for the confirmation. Hmm... I will continue to investigate 
why my cluster does not work correctly.

Thanks again,
Takanobu

-Original Message-
From: Koji Kawamura [mailto:ijokaruma...@gmail.com] 
Sent: Tuesday, June 27, 2017 5:59 PM
To: dev 
Subject: Re: Authorization problems of NiFi secured cluster

I just created a brand-new secured cluster now. NiFi automatically created a 
policy "view the data" (and others) with the user defined as "Initial Admin 
Identity" and "Node Identity" in conf/authorizers.xml.
It seems working as expected.

Koji

On Tue, Jun 27, 2017 at 5:26 PM, Koji Kawamura  wrote:
> Hi Takanobu,
>
> Glad to hear that you have it fixed.
>
>> Although I defined the Node Identity before stating the cluster at the first 
>> time, it seemed NiFi did not automatically create the policies and I needed 
>> to add the Node Identity to the policy explicitly.
>
> Thanks for sharing, ideally NiFi cluster should work without adding 
> the policy manually.
> I will try to setup a brand-new secured NiFi cluster to see what 
> initial policy setting will look like.
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#
> cluster-node-identities
>
> Thanks,
> Koji
>
> On Tue, Jun 27, 2017 at 5:08 PM, Takanobu Asanuma 
>  wrote:
>> Hi Koji,
>>
>> Thank you for your quick and valuable answer! That's exactly what I need. 
>> After adding "Node Identity" of authorizers.xml to the "view the data" 
>> policy, the authorized user can list the queue.
>>
 IIRC, if you define the Node Identity before starting the secured cluster 
 at the first time, NiFi automatically creates necessary policies for each 
 node to proxy user request (I maybe wrong on this..).
>>
>> Although I defined the Node Identity before stating the cluster at the first 
>> time, it seemed NiFi did not automatically create the policies and I needed 
>> to add the Node Identity to the policy explicitly.
>>
>> Thanks again!
>> Takanobu
>>
>> -Original Message-
>> From: Koji Kawamura [mailto:ijokaruma...@gmail.com]
>> Sent: Tuesday, June 27, 2017 2:32 PM
>> To: dev 
>> Subject: Re: Authorization problems of NiFi secured cluster
>>
>> Hello Takanobu,
>>
>> If the issue doesn't happen with standalone mode, I assume it happens 
>> because the security policy does not allow NiFi node to "view the data".
>>
>> When a user sends a request to a node within a cluster, the node proxies the 
>> request to other nodes within the same cluster.
>> I'd recommend to check if conf/authorizers.xml has Node Identity properties, 
>> looks like this:
>>
>> 
>>   ...
>>   CN=localhost, OU=NIFI 
>> 
>>
>> IIRC, if you define the Node Identity before starting the secured cluster at 
>> the first time, NiFi automatically creates necessary policies for each node 
>> to proxy user request (I maybe wrong on this..). If you already have the 
>> cluster started, then you can add NiFi node as a user then add it to the 
>> "view the data" policy manually (probably at the root PG's policy would be 
>> the most appropriate place).
>>
>> I confirmed that the issue can be reproduced by removing NiFi node user from 
>> "view the data" policy.
>>
>> Please try above and let us know if it addresses your issue.
>>
>> Thanks,
>> Koji
>>
>> On Tue, Jun 27, 2017 at 1:12 PM, Takanobu Asanuma  
>> wrote:
>>> Hello experts,
>>>
>>> When I created a NiFi cluster with security, any users can't list any 
>>> queues due to "insufficient permissions" though the users have the 
>>> permissions.
>>>
>>> For example, there is a dataflow which contains processor-A and 
>>> processor-B, and processor-A is connecting to processor-B. In this case, 
>>> even if user1 has the policies which are view/modify the component/data of 
>>> processor-A and processor-B, he can't list the queue of the processors.
>>>
>>> This problem only occurs when the secured NiFi instance is clustering mode 
>>> (nifi.cluster.is.node=true). If secured NiFi instance is standalone mode, 
>>> the problem doesn't happen. I have faced this problem with the latest 
>>> release version, 1.3.0.
>>>
>>> Do you have any thoughts?
>>>
>>> Thanks,
>>> Takanobu Asanuma

Re: Authorization problems of NiFi secured cluster

[Koji Kawamura]

I just created a brand-new secured cluster now. NiFi automatically
created a policy "view the data" (and others) with the user defined as
"Initial Admin Identity" and "Node Identity" in conf/authorizers.xml.
It seems working as expected.

Koji

On Tue, Jun 27, 2017 at 5:26 PM, Koji Kawamura  wrote:
> Hi Takanobu,
>
> Glad to hear that you have it fixed.
>
>> Although I defined the Node Identity before stating the cluster at the first 
>> time, it seemed NiFi did not automatically create the policies and I needed 
>> to add the Node Identity to the policy explicitly.
>
> Thanks for sharing, ideally NiFi cluster should work without adding
> the policy manually.
> I will try to setup a brand-new secured NiFi cluster to see what
> initial policy setting will look like.
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#cluster-node-identities
>
> Thanks,
> Koji
>
> On Tue, Jun 27, 2017 at 5:08 PM, Takanobu Asanuma
>  wrote:
>> Hi Koji,
>>
>> Thank you for your quick and valuable answer! That's exactly what I need. 
>> After adding "Node Identity" of authorizers.xml to the "view the data" 
>> policy, the authorized user can list the queue.
>>
 IIRC, if you define the Node Identity before starting the secured cluster 
 at the first time, NiFi automatically creates necessary policies for each 
 node to proxy user request (I maybe wrong on this..).
>>
>> Although I defined the Node Identity before stating the cluster at the first 
>> time, it seemed NiFi did not automatically create the policies and I needed 
>> to add the Node Identity to the policy explicitly.
>>
>> Thanks again!
>> Takanobu
>>
>> -Original Message-
>> From: Koji Kawamura [mailto:ijokaruma...@gmail.com]
>> Sent: Tuesday, June 27, 2017 2:32 PM
>> To: dev 
>> Subject: Re: Authorization problems of NiFi secured cluster
>>
>> Hello Takanobu,
>>
>> If the issue doesn't happen with standalone mode, I assume it happens 
>> because the security policy does not allow NiFi node to "view the data".
>>
>> When a user sends a request to a node within a cluster, the node proxies the 
>> request to other nodes within the same cluster.
>> I'd recommend to check if conf/authorizers.xml has Node Identity properties, 
>> looks like this:
>>
>> 
>>   ...
>>   CN=localhost, OU=NIFI 
>> 
>>
>> IIRC, if you define the Node Identity before starting the secured cluster at 
>> the first time, NiFi automatically creates necessary policies for each node 
>> to proxy user request (I maybe wrong on this..). If you already have the 
>> cluster started, then you can add NiFi node as a user then add it to the 
>> "view the data" policy manually (probably at the root PG's policy would be 
>> the most appropriate place).
>>
>> I confirmed that the issue can be reproduced by removing NiFi node user from 
>> "view the data" policy.
>>
>> Please try above and let us know if it addresses your issue.
>>
>> Thanks,
>> Koji
>>
>> On Tue, Jun 27, 2017 at 1:12 PM, Takanobu Asanuma  
>> wrote:
>>> Hello experts,
>>>
>>> When I created a NiFi cluster with security, any users can't list any 
>>> queues due to "insufficient permissions" though the users have the 
>>> permissions.
>>>
>>> For example, there is a dataflow which contains processor-A and 
>>> processor-B, and processor-A is connecting to processor-B. In this case, 
>>> even if user1 has the policies which are view/modify the component/data of 
>>> processor-A and processor-B, he can't list the queue of the processors.
>>>
>>> This problem only occurs when the secured NiFi instance is clustering mode 
>>> (nifi.cluster.is.node=true). If secured NiFi instance is standalone mode, 
>>> the problem doesn't happen. I have faced this problem with the latest 
>>> release version, 1.3.0.
>>>
>>> Do you have any thoughts?
>>>
>>> Thanks,
>>> Takanobu Asanuma

Re: Authorization problems of NiFi secured cluster

[Koji Kawamura]

Hi Takanobu,

Glad to hear that you have it fixed.

> Although I defined the Node Identity before stating the cluster at the first 
> time, it seemed NiFi did not automatically create the policies and I needed 
> to add the Node Identity to the policy explicitly.

Thanks for sharing, ideally NiFi cluster should work without adding
the policy manually.
I will try to setup a brand-new secured NiFi cluster to see what
initial policy setting will look like.
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#cluster-node-identities

Thanks,
Koji

On Tue, Jun 27, 2017 at 5:08 PM, Takanobu Asanuma
 wrote:
> Hi Koji,
>
> Thank you for your quick and valuable answer! That's exactly what I need. 
> After adding "Node Identity" of authorizers.xml to the "view the data" 
> policy, the authorized user can list the queue.
>
>>> IIRC, if you define the Node Identity before starting the secured cluster 
>>> at the first time, NiFi automatically creates necessary policies for each 
>>> node to proxy user request (I maybe wrong on this..).
>
> Although I defined the Node Identity before stating the cluster at the first 
> time, it seemed NiFi did not automatically create the policies and I needed 
> to add the Node Identity to the policy explicitly.
>
> Thanks again!
> Takanobu
>
> -Original Message-
> From: Koji Kawamura [mailto:ijokaruma...@gmail.com]
> Sent: Tuesday, June 27, 2017 2:32 PM
> To: dev 
> Subject: Re: Authorization problems of NiFi secured cluster
>
> Hello Takanobu,
>
> If the issue doesn't happen with standalone mode, I assume it happens because 
> the security policy does not allow NiFi node to "view the data".
>
> When a user sends a request to a node within a cluster, the node proxies the 
> request to other nodes within the same cluster.
> I'd recommend to check if conf/authorizers.xml has Node Identity properties, 
> looks like this:
>
> 
>   ...
>   CN=localhost, OU=NIFI 
> 
>
> IIRC, if you define the Node Identity before starting the secured cluster at 
> the first time, NiFi automatically creates necessary policies for each node 
> to proxy user request (I maybe wrong on this..). If you already have the 
> cluster started, then you can add NiFi node as a user then add it to the 
> "view the data" policy manually (probably at the root PG's policy would be 
> the most appropriate place).
>
> I confirmed that the issue can be reproduced by removing NiFi node user from 
> "view the data" policy.
>
> Please try above and let us know if it addresses your issue.
>
> Thanks,
> Koji
>
> On Tue, Jun 27, 2017 at 1:12 PM, Takanobu Asanuma  
> wrote:
>> Hello experts,
>>
>> When I created a NiFi cluster with security, any users can't list any queues 
>> due to "insufficient permissions" though the users have the permissions.
>>
>> For example, there is a dataflow which contains processor-A and processor-B, 
>> and processor-A is connecting to processor-B. In this case, even if user1 
>> has the policies which are view/modify the component/data of processor-A and 
>> processor-B, he can't list the queue of the processors.
>>
>> This problem only occurs when the secured NiFi instance is clustering mode 
>> (nifi.cluster.is.node=true). If secured NiFi instance is standalone mode, 
>> the problem doesn't happen. I have faced this problem with the latest 
>> release version, 1.3.0.
>>
>> Do you have any thoughts?
>>
>> Thanks,
>> Takanobu Asanuma

RE: Authorization problems of NiFi secured cluster

[Takanobu Asanuma]

Hi Koji,

Thank you for your quick and valuable answer! That's exactly what I need. After 
adding "Node Identity" of authorizers.xml to the "view the data" policy, the 
authorized user can list the queue.

>> IIRC, if you define the Node Identity before starting the secured cluster at 
>> the first time, NiFi automatically creates necessary policies for each node 
>> to proxy user request (I maybe wrong on this..). 

Although I defined the Node Identity before stating the cluster at the first 
time, it seemed NiFi did not automatically create the policies and I needed to 
add the Node Identity to the policy explicitly.

Thanks again!
Takanobu

-Original Message-
From: Koji Kawamura [mailto:ijokaruma...@gmail.com] 
Sent: Tuesday, June 27, 2017 2:32 PM
To: dev 
Subject: Re: Authorization problems of NiFi secured cluster

Hello Takanobu,

If the issue doesn't happen with standalone mode, I assume it happens because 
the security policy does not allow NiFi node to "view the data".

When a user sends a request to a node within a cluster, the node proxies the 
request to other nodes within the same cluster.
I'd recommend to check if conf/authorizers.xml has Node Identity properties, 
looks like this:


  ...
  CN=localhost, OU=NIFI 


IIRC, if you define the Node Identity before starting the secured cluster at 
the first time, NiFi automatically creates necessary policies for each node to 
proxy user request (I maybe wrong on this..). If you already have the cluster 
started, then you can add NiFi node as a user then add it to the "view the 
data" policy manually (probably at the root PG's policy would be the most 
appropriate place).

I confirmed that the issue can be reproduced by removing NiFi node user from 
"view the data" policy.

Please try above and let us know if it addresses your issue.

Thanks,
Koji

On Tue, Jun 27, 2017 at 1:12 PM, Takanobu Asanuma  
wrote:
> Hello experts,
>
> When I created a NiFi cluster with security, any users can't list any queues 
> due to "insufficient permissions" though the users have the permissions.
>
> For example, there is a dataflow which contains processor-A and processor-B, 
> and processor-A is connecting to processor-B. In this case, even if user1 has 
> the policies which are view/modify the component/data of processor-A and 
> processor-B, he can't list the queue of the processors.
>
> This problem only occurs when the secured NiFi instance is clustering mode 
> (nifi.cluster.is.node=true). If secured NiFi instance is standalone mode, the 
> problem doesn't happen. I have faced this problem with the latest release 
> version, 1.3.0.
>
> Do you have any thoughts?
>
> Thanks,
> Takanobu Asanuma