Re: Insufficient Permissions - Unable to view the user interface - at WebUI

[Matthew Clarke]

1) "sAMAccountName"

First of I'm using "sAMAccountName"  only because: 1.1) it is mentioned in
official documentation
https://nifi.apache.org/documentation/nifi-2.0.0-M2/html/administration-guide.html#ldap_login_identity_provider
1.2)
I don't know any better 1.3) What would be your suggestion?

-- The documentation serves only an example. You'll need to work with your
LDAP/Active Directory admins to understand the structure of your LDAP users
and groups and what attributes are available to use in your specific
configuration. I am assuming since you stated that you saw "unable to view
user interface" in the NiFi UI, that your NiFi did present you with a login
window and this response can after entering your LDAP/AD username and
password? Seeing "unable to view the user interface" means that user
authentication was successful, but the authenticated user was not
authorized to view the user interface. Check the nifi-user.log to see the
exact case sensitive user identity being passed for authorization.  The
user identity string is what you want to be using as your initial admin
value. Also make sure in your nifi.properties file that
NiFi.security.user.login.identity.provider=ldap-provider is set.



2) Do you suggest that if one uses sAMAccountName as "User Search Filter"
in login-identity-providers.xml, one can't use DN as "Initial User Identity
1" or/and "Initial Admin Identity" in corresponding authorizers.xml?

-- I was NOT implying you must reconfigure your ldap-provider "identity
strategy" to USE_USERNAME from default USE_DN.  It was only a suggestion to
simplify user management later.

If using USE_DN -- upon successful user authentication via ldap-provider,
the user's full DN will be evaluated against any identity.mapping.patterns
configured in the NiFi.properties file. after which the useri identity
string is passed to NiFi authorization handled via the authorizers xml.

If using USE_USERNAME -- upon successful user authentication via
ldap-provider, the user's username entered in login window will be
evaluated against any identity.mapping.patterns configured in the
NiFi.properties file. after which the useri identity string is passed to
NiFi authorization handled via the authorizers.xml.

Whichever you decide to use, the resulting user identity post any
identity.mapping.value (if pattern regex matches) is the user string (case
sensitive) you would add as the initial admin identity in the
file-access-policy provider. Note: modifying the authorizers.xml will NOT
result in any modification to previously generated uses.xl or
authorizations.xml files. You'll need to remove it rename that files so new
are created from your changes.

Reference documentation links:

https://nifi.apache.org/documentation/nifi-2.0.0-M2/html/administration-guide.html#ldap_login_identity_provider

https://nifi.apache.org/documentation/nifi-2.0.0-M2/html/administration-guide.html#authorizers-setup

https://nifi.apache.org/documentation/nifi-2.0.0-M2/html/administration-guide.html#identity-mapping-properties

Hope this helps,

Matt

On Mon, Feb 26, 2024, 2:07 AM Alexei Rozenvaser 
wrote:

> Hello Mike,
>
>
>
> Can you elaborate a little bit more on your last post?
>
>
>
> 1) "sAMAccountName"
>
> First of I'm using "sAMAccountName"  only because: 1.1) it is mentioned in
> official documentation
> https://nifi.apache.org/documentation/nifi-2.0.0-M2/html/administration-guide.html#ldap_login_identity_provider
> 1.2) I don't know any better 1.3) What would be your suggestion?
>
>
>
> 2) Do you suggest that if one uses sAMAccountName as "User Search Filter"
> in login-identity-providers.xml, one can't use DN as "Initial User Identity
> 1" or/and "Initial Admin Identity" in corresponding authorizers.xml?
>
> What is the proper and coherent way to do it?
>
>
>
> 2.1) If I'm using "sAMAccountName" then I must use "USE_USERNAME" for
> "Identity Strategy" in login-identity-providers.xml and username like
> usern...@domain.com as "Initial User Identity 1" and "Initial Admin
> Identity" in authorizers.xml?
>
>
>
> 2.2) If one is using DN "Initial User Identity 1" and "Initial Admin
> Identity" in authorizers.xml and the "USE_DN" for "Identity Strategy" in
> login-identity-providers.xml then what should one use as  "User Search
> Filter" in login-identity-providers.xml?
>
>
>
> Thanks
>
>
>
> *From:* Matthew Clarke 
> *Sent:* Sunday, February 25, 2024 11:19 PM
> *To:* users@nifi.apache.org
> *Cc:* Alexei Rozenvaser 
> *Subject:* Re: Insufficient Permissions - Unable to view the user
> interface - at WebUI
>
>
>
> You don't often get email from matt.clarke..

Re: Insufficient Permissions - Unable to view the user interface - at WebUI

[Matthew Clarke]

Hello Alexei,

Since you are loving in using your  sAMAccountName, I recommend using them
identity string instead of the full DN of your AD users.  In your
ldap-provider, change USE_DN to USE_USERNAME. Upon successfully
authentication the username you provided in the login window will be posted
to configured authorized in your NiFi.  Using sAMAccountName instead of the
full DN will also make it easier setting up additional AD user
authorizations later via NiFi UI.

Then configure your initial user identity (file-user-group-provider) and
admin identity (file-access-policy-provider) to match your username.  Keep
in mind that NiFi is case sensitive.  You then need to delete or rename
your current users.xml and authorizations.xml files. These files are only
created on NiFi startup if they do NOT already exist. The content is never
modified in existing files when you make changes to the authorizers.xml.

Thanks,
Matt

On Thu, Feb 22, 2024, 9:24 AM Michael Moser  wrote:

> Hello Alexei,
>
> If you have configured an org.apache.nifi.ldap.LdapProvider in your NiFi
> login-identity-providers.xml for *authentication* then you will also have
> to configure an org.apache.nifi.ldap.tenants.LdapUserGroupProvider in your
> NiFi authorizers.xml for *authorization*.  Some instructions are in the
> NiFi Admin Guide.
>
> Also, if you use an LdapUserGroupProvider that will contain your identity
> in AD, then you will not need a "Initial User Identity 1" in your
> definition of FileUserGroupProvider.
>
> Kind regards,
> -- Mike
>
>
> On Thu, Feb 22, 2024 at 9:09 AM Alexei Rozenvaser <
> alexei@tomer-rs.co.il> wrote:
>
>> My ./config/authorizers.xml
>>
>>
>>
>> 
>>
>> 
>>
>> 
>>
>> file-user-group-provider
>>
>> org.apache.nifi.authorization.FileUserGroupProvider
>>
>> ./conf/users.xml
>>
>> CN=My
>> Name,OU=MyOU,DC=MyDomain
>>
>> 
>>
>> 
>>
>> file-access-policy-provider
>>
>>
>> org.apache.nifi.authorization.FileAccessPolicyProvider
>>
>> file-user-group-provider
>>
>> ./conf/authorizations.xml
>>
>> CN=My
>> Name,OU=MyOU,DC=MyDomain
>>
>> 
>>
>> 
>>
>> 
>>
>> 
>>
>> managed-authorizer
>>
>>
>> org.apache.nifi.authorization.StandardManagedAuthorizer
>>
>> file-access-policy-provider
>>
>> 
>>
>> 
>>
>>
>>
>>
>>
>> *From:* Alexei Rozenvaser 
>> *Sent:* Thursday, February 22, 2024 2:59 PM
>> *To:* users@nifi.apache.org
>> *Subject:* Insufficient Permissions - Unable to view the user interface
>> - at WebUI
>>
>>
>>
>> Hi Everyone
>>
>>
>>
>> I have a brand new installation of NiFi 2.0.
>>
>> I configured LDAP user authentication.
>> I have my AD DN defined as both "Initial User Identity 1" and "Initial
>> Admin Identity" in *authorizers.xml*
>>
>> 1.Now I can launch the NiFi server successfully.
>>
>> 2.I can successfully log in to NiFi's webUI with my AD user
>>
>> 3.But I get: "Insufficient Permissions" - "Unable to view the user
>> interface."  at WebUI
>>
>> 4.There is an AccessDeniedExeptionMapper identity [cn=My
>> Name,ou=MyOU,DC=MyDC], group[] does not have permission to access the
>> requested resource. Unable to view the user interface. Returning Forbidden
>> response. entry at nifi-user.log
>>
>> If I understand the situation correctly I can pass the authentication
>> phase but my user wasn't authorized for UI access?
>> What should I check first?
>>
>>
>>
>

Re: Inspecting content claims

[Matthew Clarke]

A content claim that has not been moved to an archive subdirectory within
the content_repository directory is considered still active.   Since a
content claim can contain the content for 1 too many FlowFiles and a
content claim can't be moved to archive until ALL FlowFiles referencing
that claim have reach there point of termination in your dataflows,  even a
small FlowFile left queen can hold up a large claim.  Not sure what version
of NiFi you are running, but if suggesting being in the most current and
making sure the spendable claim size set in your nifi.properties file is no
larger then 50kb.

On Tue, Apr 26, 2022, 8:20 AM Isha Lamboo 
wrote:

> Hi all,
>
>
>
> I’m investigating an issue where the content disk of one of our nifi 1.12
> clusters is filling up while the active usage is very low (10-12 GB active
> data per node while the cluster view shows 98% full of 600 GB per node).
>
> My first idea was that archiving wasn’t being run often enough or cleaning
> up enough data, but from what I read, I should see global backpressure
> being applied then.
>
>
>
> For reference, these are the settings:
>
> # Content Repository
>
> nifi.content.claim.max.appendable.size=1 MB
>
> nifi.content.repository.archive.max.retention.period=12 hours
>
> nifi.content.repository.archive.max.usage.percentage=80%
>
> nifi.content.repository.archive.enabled=true
>
> nifi.content.repository.always.sync=false
>
>
>
> Now my next suspect is a queue full of old flowfiles (50+ days in queue)
> in a disabled flow that keep content claims active amounting to much more
> diskspace than needed.
>
> I know I can look up the Content Claim identifier in the content
> repository directory, but while I can see the size of the total file, I
> have no idea whether those are all active flowfiles or mostly archived.
>
>
>
> Is there a more systematic way of inspecting active/inactive content in
> content claims?
>
>
>
> Regards,
>
>
>
> Isha
>

Re: Is it possible to completely disable RPGs, not just transmission?

[Matthew Clarke]

You are correct, the moment a RPG is added to the canvas a background
thread will attempt to fetch the Site-to-Site details from the configure
remote NiFi. There is no way to disable this.  RPGs only give you the
ability to enable or disable transmission of FlowFiles to/from remote ports.

On Wed, May 4, 2022, 6:11 AM Isha Lamboo 
wrote:

> Hi Juan,
>
>
>
> Thank you for the reply. I assume you mean the
> nifi.flowcontroller.autoResumeState=false setting. I’m already using that
> to ensure no flows started before we want them to. My problem is that the
> RPGs keep trying to retrieve the site-to-site details regardless.
>
>
>
> Isha
>
>
>
> *Van:* Juan Pablo Gardella 
> *Verzonden:* woensdag 4 mei 2022 11:36
> *Aan:* users@nifi.apache.org
> *Onderwerp:* Re: Is it possible to completely disable RPGs, not just
> transmission?
>
>
>
> There is a property on nifi.properties that you can start nifi and all
> processors are in stop state. I could not find it, but maybe it is useful
> for that scenario.
>
>
>
> On Wed, May 4, 2022 at 5:14 AM Isha Lamboo 
> wrote:
>
> Hi all,
>
>
>
> Is there a way to stop disabled Remote Process Groups from continually
> contacting the remote to update the contents?
>
>
>
> I’m migrating a cluster with hundreds of Remote Process Groups and the
> moment I start up the new cluster with all flows stopped/disabled, the RPGs
> all start contacting the remote, regardless of the RPG’s status.
>
> This results in various errors since firewall ports and remote nifi
> policies are not yet in place. I’m worried about the http threads on the
> remote NiFi cluster being overloaded and locally, all other errors are
> drowned out by the RPG errors.
>
>
>
> This seems like a limitation in the Remote Process Group. I can’t even see
> the enabled/disabled status while the RPG is failing to update from the
> remote instance.
>
>
>
> Regards,
>
>
>
> Isha
>
>
>
>

Re: Sort Processor list by Concurrent Tasks

[Matthew Clarke]

You can sort by "Run Status" column and the look at all your "Running"
processors. To the right of running you will see a number that will
indicate how many active threads existed at the time the summary was
conducted. You can click on the refresh icon in the lower right corner of
the summary window to update this info. You can use this to determine which
officials are actively using a high number of threads.

Matt

On Jan 6, 2017 7:25 PM, "Kevin Verhoeven"  wrote:

> I just noticed that my NiFi 1.1.1 cluster has started to use a much higher
> number of threads and I want to identify what Processors are using these
> threads by looking at the Concurrent Tasks assigned to the Processors. I
> have a few hundred Processors scattered across various Processor Groups so
> I can't right-click on each Processor. Is there a way to see Concurrent
> Tasks in the NiFi Summary Report in the UI? I could then sort by the column
> and locate which Processors are set too high.
>
> Any advice would be appreciated, thanks!
>
> Kevin
>

Re: problem creating simple cluster

[Matthew Clarke]

In the NiFi.properties file is a property just above your http.port=8080
for the http.host= . If that field is left blank. UI requested that are
replicated between node's may end up using localhost. Make sure this
property is set to the Nodes hostname or IP addresses on every node. A
restart will be needed before any changes to the NiFi.properties file to
take affect.

Thanks,
Matt

On Dec 7, 2016 8:21 PM, "Koji Kawamura"  wrote:

> Sorry, I overlooked the nifi.properties settings you shared.
> Would you share what you can see on the NiFi "Cluster window", from
> right top Hamburger menu, especially the 'Node Address' column?
>
> Thanks,
> Koji
>
> On Thu, Dec 8, 2016 at 10:10 AM, Koji Kawamura 
> wrote:
> > Hi Brian,
> >
> > Are those three node running on a same host using different port? Or
> > running on different hosts?
> > nifi.properties has nifi.cluster.node.address configuration, which is
> > used by a NiFi node to tell how other NiFi nodes should access the
> > node.
> >
> > If the property is blank, NiFi uses 'localhost' as node hostname.
> > I think that's why the node tried to replicate the request to
> 'localhost:8080'.
> >
> > If so, the property should be set with a hostname that is accessible
> > from other nodes.
> >
> > Or, if there's any firewall among nodes,
> > nifi.web.http.port
> > nifi.cluster.node.protocol.port
> > should be opened.
> >
> > I sometimes forget this on AWS with security group setting then get
> > timeout error.
> >
> > Thanks,
> > Koji
> >
> > On Thu, Dec 8, 2016 at 2:59 AM, Brian Jeltema 
> wrote:
> >> The cluster is not running securely, so I don’t believe that file is
> >> relavent. In the stack trace,
> >> the reference to /nifi-api/flow/current-user is misleading - I think any
> >> nifi-api call has problems.
> >>
> >> On Dec 7, 2016, at 12:37 PM, James Wing  wrote:
> >>
> >> Brian,
> >>
> >> Did you add entries for the node DNs in the conf/authorizers.xml file?
> >> Something like:
> >>
> >> 
> >> CN=node1.nifi, ...
> >> CN=node2.nifi, ...
> >> ...
> >>
> >> Thanks,
> >>
> >> James
> >>
> >> On Wed, Dec 7, 2016 at 8:28 AM, Brian Jeltema 
> wrote:
> >>>
> >>> I’m trying to create my first cluster using NiFi 1.1.0. It’s a simple
> >>> 3-node unsecure configuration with each node running embedded
> >>> zookeeper. The instances all come up and the zookeeper quarum is
> >>> reached.
> >>>
> >>> If I bring up the UI for the node that is elected as the
> >>> cluster coordinator, it works as expected, and shows that 3 nodes
> >>> are participating in the cluster.
> >>>
> >>> However, if I attempt to display the UI on the non-coordinator nodes,
> >>> after a delay of about 10 seconds an error page is returned. The
> >>> logs contains a stream of exceptions similar to the following:
> >>>
> >>> 2016-12-07 11:08:17,914 WARN [Replicate Request Thread-2]
> >>> o.a.n.c.c.h.r.ThreadPoolRequestReplicator Failed to replicate request
> GET
> >>> /nifi-api/flow/current-user to localhost:8080 due to {}
> >>> com.sun.jersey.api.client.ClientHandlerException:
> >>> java.net.SocketTimeoutException: Read timed out
> >>> at
> >>> com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(
> URLConnectionClientHandler.java:155)
> >>> ~[jersey-client-1.19.jar:1.19]
> >>> at com.sun.jersey.api.client.Client.handle(Client.java:652)
> >>> ~[jersey-client-1.19.jar:1.19]
> >>> at
> >>> com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(
> GZIPContentEncodingFilter.java:123)
> >>> ~[jersey-client-1.19.jar:1.19]
> >>> at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
> >>> ~[jersey-client-1.19.jar:1.19]
> >>> at com.sun.jersey.api.client.WebResource.access$200(
> WebResource.java:74)
> >>> ~[jersey-client-1.19.jar:1.19]
> >>> at com.sun.jersey.api.client.WebResource$Builder.get(
> WebResource.java:509)
> >>> ~[jersey-client-1.19.jar:1.19]
> >>> at
> >>> org.apache.nifi.cluster.coordination.http.replication.
> ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.
> java:578)
> >>> ~[nifi-framework-cluster-1.1.0.jar:1.1.0]
> >>> at
> >>> org.apache.nifi.cluster.coordination.http.replication.
> ThreadPoolRequestReplicator$NodeHttpRequest.run(
> ThreadPoolRequestReplicator.java:770)
> >>> ~[nifi-framework-cluster-1.1.0.jar:1.1.0]
> >>> at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
> >>> [na:1.8.0_101]
> >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_101]
> >>> at
> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> >>> [na:1.8.0_101]
> >>> at
> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> >>> [na:1.8.0_101]
> >>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> >>> Caused by: java.net.SocketTimeoutException: Read timed out
> >>> at java.net.SocketInputStream.socketRead0(Native 

Re: NiFi | Contoller Service is not getting updated

[Matthew Clarke]

When nifi starts it unpacks the nars in to a work directory. Try deleting
that NiFi work directory before restarting to see if your changes are seen.

On Sep 4, 2016 9:06 AM, "Shashi Vishwakarma" 
wrote:

> Hi
>
> I created a sample controller service - 'MyControllerService' and packaged
> it into nar and pasted into nifi lib directory. I restarted nifi service to
> see changes. I was able to see MyControllerService in Controller setting.
> After that i made small label change into controller service and followed
> same process but changes are not getting affected.
>
> Even i removed nar files from nifi/lib just to check if it getting removed
> from list. That too not happening.
>
> Even I dont see any exception in nifi/log.
>
> Any pointer for this issue?
>
> Thanks
> Shashi
>

Re: I need help configuring Site-to-Site in Secure Mode.

[Matthew Clarke]

Do the certs you created/obtained support being used for both client and
server auth.  If they were created for server auth only, this could explain
your issue.  NiFi instances need to act as a client and a server at times.

Thanks,
Matt

On Fri, Sep 2, 2016 at 10:59 AM, Peter Wicks (pwicks) 
wrote:

> Bryan,
>
>
>
> We’ve fixed our certs, with no change to the outcome.
>
>
>
> We have username/password authentication enabled, via Kerberos, are there
> issues having Kerberos enabled (username/password) and trying to do
> site-to-site? When I try to initiate site-to-site with an instance of NiFI
> configured for Kerberos all requests come through to the server as
> anonymous because no challenge appears to be sent.  We’ve debugged the code
> and even deep down in NiFiUserUtils.getNiFiUser the request is already
> marked as anonymous by the Spring framework. It appears to me that the
> client has a cert, and is waiting for a challenge(?) from the server, and
> the server is configured for Kerberos and it’s waiting for a ‘bearer’ token…
>
>
>
> We’ve debugged both client and server, the client sends the request and
> gets back a 401 (Unauthorized). SSL verifies good.
>
> Server doesn’t appear to get any authorization information of any kind.
>
>
>
> Looking for further guidance/next steps.
>
>
>
> Thanks,
>
>   Peter
>
>
>
> *From:* Bryan Bende [mailto:bbe...@gmail.com]
> *Sent:* Thursday, September 01, 2016 9:44 AM
>
> *To:* users@nifi.apache.org
> *Subject:* Re: I need help configuring Site-to-Site in Secure Mode.
>
>
>
> Peter,
>
>
>
> Yes, by no means am I saying everyone should use the TLS toolkit. I was
> just using that because many people are not familiar with how to create
> certificates, and for people trying to follow a tutorial it is the easiest
> option.
>
>
>
> In your case you definitely want to be using your CA. What you described
> about not having a cert for client authentication definitely sounds like it
> would be a problem. Let us know if everything works out after getting the
> new certs.
>
>
>
> -Bryan
>
>
>
>
>
> On Thu, Sep 1, 2016 at 11:34 AM, Peter Wicks (pwicks) 
> wrote:
>
> Bryan,
>
>
>
> Paul and I have been working on this, and I think our issue is related to
> certificates.
>
>
>
> In your blog posting you used TLS-Toolkit in your example, but I think
> that is unrealistic for many environments.  For example, this also creates
> the certificates for SSL right? But these will be self-signed and thus
> untrusted by default in web browsers.  In our environment we generated SSL
> certificates from our CA and loaded them into the KeyStore.  We then
> extracted public keys for the SSL certs and put them in each of the Trust
> Stores.  This I think is where our main problem is…
>
>
>
> I’m making a few assumptions here, so feel free to correct me, but my
> understanding is that when you use TLS-Toolkit it either creates multiple
> certs (SSL & Client Auth), or it creates a cert that you are allowed to use
> for both activities.  In our case we ONLY have SSL certs, and the certs are
> marked such that they aren’t allowed to be used for Client Authentication.
> I believe this is the reason why our requests are showing up as
> ‘Anonymous’, because there are no Client Authentication certificates in the
> KeyStore, just SSL certs.
>
>
>
> I’ve asked our security team for Client Authentication certs for each
> server, since it would be our preference to use our CA rather than having
> TLS-Toolkit be its own CA.
>
>
>
> Thoughts?
>
>
>
> Thanks,
>
>   Peter
>
>
>
> *From:* Bryan Bende [mailto:bbe...@gmail.com]
> *Sent:* Thursday, September 01, 2016 9:26 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: I need help configuring Site-to-Site in Secure Mode.
>
>
>
> Paul,
>
>
>
> Clustering is not a requirement for site-to-site... This sounds strange
> since "anonymous" is used to represent a user when NiFi is not secured.
>
>
>
> Can you double-check all your configs and make sure you have the following
> properties set...
>
>
>
> nifi.remote.input.secure=true
>
> nifi.web.https.host=
> nifi.web.https.port=
>
> nifi.security.keystore=
> nifi.security.keystoreType=
> nifi.security.keystorePasswd=
> nifi.security.keyPasswd=
> nifi.security.truststore=
> nifi.security.truststoreType=
> nifi.security.truststorePasswd=
>
>
>
> After your question the other day I went through the steps of setting
> secure site-to-site to make sure I knew what I was talking about :)
>
>
>
> I wrote up the steps here:  http://bryanbende.com/
> development/2016/08/30/apache-nifi-1.0.0-secure-site-to-site
>
>
>
> Thanks,
>
>
>
> Bryan
>
>
>
> On Thu, Sep 1, 2016 at 10:44 AM, Paul Gibeault (pagibeault) <
> pagibea...@micron.com> wrote:
>
> Bryan,
>
>
>
> Thanks for the reply.  After increasing the log level for Authentication I
> saw the target NiFi instance used the account “anonymous” for the
> Site-to-Site connection.  After creating a policy for “anonymous”, I was
> able to 

Re: Failure Handling

[Matthew Clarke]

Hello James,

Welcome to the NiFi community.

Whether it is a good idea or bad idea to loop a failure relationship back
on a processor for retry is completely dependent on the type or processor.
For example, looping the failure relationship on a PutFile is a very good
idea; however, looping failure on a processor like CompressContent
(configured to decompress) may not be the best idea.  A file that fails to
decompress will likely continue to fail to decompress creating a never
ending loop.  It may also be desirable to build a dataflow loop [1].  This
will allow you to retry only so many times before take some new action.
That action may be send out an email notification about the dataflow
problem.

The NiFi processors have a configurable "penalty duration".  The default
for this is 30 seconds.  Anytime a processor routes FlowFiles to a failure
relationship, they are penalized for this duration of time.  During the
Penalty period NiFi will ignore these penalized FlowFiles and work on other
FLowFiles in queue coming back to these when the penalty expires.  This
help with the race condition you mentioned.

In your example of disk failure, you need to take other things into
consideration. How much data can your NiFi afford to ingest before your
disk fills?  Have I followed best practices for deploying my NiFi instance?
[2]
Connections within NiFi provide a means for setting object or size back
pressure thresholds. [3]  This allow you to control per connection many
FlowFile can queue before the source processor for that connection is no
longer triggered to run.  You can set back pressure on every connection all
the way back to your dataflow ingest point(s) to essential halt your
dataflow before disks fill in the case of a major failure like you
described.  This also prevents one bad behaving dataflow on a canvas of
many dataflows from taking over all resources.

Hope this helps,
Matt

​[1]
https://cwiki.apache.org/confluence/download/attachments/57904847/Retry_Count_Loop.xml?version=1=1433271239000=v2
[2]
https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html
[3]
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Connecting_Components

Re: authentication problem

[Matthew Clarke]

Verify the user running your nifi has the correct permissions to read and
edit all the database files in the NiFi database repository directory. Also
make sure the user running NiFi had permissions to edit the
authorized-users.xml file. This file is only read on start-up. After nifi
is running it operates solely from the memory and DBs. My guess is here
that nifi can not update the authorized-users.xml file with the changes you
are making in the User management UI.

Thanks,
Matt

On Jul 26, 2016 12:47 PM, "Perko, Ralph J"  wrote:

Hi – whenever we restart Nifi half the accounts are disabled with this
message:

INFO [main] o.a.n.a.s.action.SeedUserAccountsAction User not authorized
with configured provider: . Disabling account...

The users are in the authorized-users.xml file.  Is this a configuration
issue on our part or a bug?  It is curious that not all the accounts get
disabled only some and it is always the same accounts.

To re-enable the accounts I go into the users page, select the disabled
user (click the little pencil) and click ‘apply’ with no changes - the
account is re-enabled.

Details:
Nifi 0.6.1
authority-providers.xml: default file provider
login-identy-management.xml: kerberos-provider (corporate system – everyone
is in it)
authorized-users.xml:setup for each user

Thanks,
Ralph

Re: Java Heap Size increasing without anything happening

[Matthew Clarke]

Old gen would be considered full garbage collection. You are good.
On Jun 30, 2016 8:22 PM, "Stéphane Maarek" 
wrote:

> Got it thanks,
> So I assume none of this below if full garbage collection? This has
> happened in less than a day
>
> *G1 Young Generation* 23568 times 00:07:49.718
> *G1 Old Generation* 20 times 00:00:13.226
>
> On Thu, Jun 30, 2016 at 10:22 PM Mark Payne  wrote:
>
>> Stephane,
>>
>> This is normal and expected. Even though there is no data flowing through
>> your system, the framework
>> does have some maintenance that it has to do, such as pruning old
>> archived data from the Content Repository
>> and Provenance Repository, checkpointing the FlowFile Repository, keeping
>> stats for all of the processors, etc., etc.
>>
>> If you were to see "Full Garbage Collections" taking place then there may
>> be some concern but minor garbage collections
>> are pretty frequent and shouldn't cause concern.
>>
>> Thanks
>> -Mark
>>
>>
>> On Jun 30, 2016, at 12:37 AM, Stéphane Maarek 
>> wrote:
>>
>> Heap (32.0%)
>> *Max* 5 GB
>> *Total* 4 GB
>> *Used* 1.62 GB
>> *Free* 2.38 GB
>> dd
>>
>> I have an issue, I followed this link to "optimize" HDF
>> https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html
>>
>>
>> Basically, I have two UDP receivers but they don't receive any data right
>> now. They produce 14,000 tasks each every 5 minutes (which is fine). The
>> issue is that my Heap increases to the top 4 GB every minute I would say,
>> triggering a garbage collection event (I use G1). Is that normal? I would
>> expect the memory to stay constant if no data goes through my system at all
>>
>> Thanks,
>> Stephane
>>
>>
>>

Re: Processor Question

[Matthew Clarke]

You may also want to look at using the modifyBytes processor for number 1.
On Jun 4, 2016 1:49 PM, "Thad Guidry"  wrote:

> For your 1st case, you can use either SegmentContent by your 256 bytes (or
> perhaps you can even use SplitText)
>
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.SegmentContent/index.html
>
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.SplitText/index.html
>
> For your 2nd case, you can use ReplaceText
>
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.ReplaceText/index.html
>
> Thad
> +ThadGuidry 
>
>

Re: Cluster Node Protocol Threads - will this configuration help reduce a high queue?

[Matthew Clarke]

Hey Kevin,
The DistributeLoad processor in itself does not distribute data across
to the other nodes your cluster.  Can you explain your flow in a little
more detail? I am afraid i an missing something here. If you want to spread
data across your nodes, I encourage you to use a remote process group. This
will auto scale and auto load balance data to all your nodes. The setting
you mentioned is for node communication but not data transmission between
nodes. Increasing this values can help when you have a large number of
nodes, but in your case with only four nodes it should not make much
impact. Again these nodes comms are not for data. What processors are
queues building behind?

Matt
On May 27, 2016 11:41 AM, "Kevin Verhoeven" 
wrote:

> I’m new to NiFi and I have a small problem. I’m running NiFi
> 0.7.0-SNAPSHOT on Windows 2012 R2 VMs. I’ve noticed that the queues on some
> of my Processors are very large, sometimes as high as 10,000 flowfiles. The
> flowfiles are eventually processed but at a slow pace.
>
>
>
> I run a cluster with 4 nodes. The initial Get Processor runs on the
> Primary Node to only request a single file and I then use the
> DistributeLoad Processor with the Next Available setting to spread the load
> across the cluster nodes. However, I see that the queue is highest on the
> Primary Node and the cluster nodes see very little work.
>
>
>
> My question is: will I increase throughput to the cluster nodes if I
> increase the nifi.cluster.node.protocol.threads from 2 to something higher?
> What effect does nifi.cluster.node.protocol.threads have on the nodes?
>
>
>
> Thanks,
>
>
>
> Kevin
>

Re: Cluster setup - NCM error

[Matthew Clarke]

Igor,
   You are seeing this error because you did not configure
the nifi.web.http.host(s)= property in your nifi.properties file on your
Node.  The heartbeat message is reporting the API address as "localhost".
So the is trying to send requests to localhost on 8082 which is not really
your node and it fails.

  Also see you are not configured for site-to-site.  You will wont to
configure the host and port for that as well if you wish to be able to use
RemoteProcessGroups.

Thanks,
Matt

On Thu, May 19, 2016 at 3:30 PM, Igor Kravzov 
wrote:

> Joe, you were right. It was configuration issue. I set
> nifi.cluster.node.protocol.port property instead
> of nifi.cluster.manager.protocol.port
> But now I am getting
>
> An unexpected error has occurred
> No nodes were able to process this request.
>
> Here is log:
>
> 2016-05-19 15:21:37,253 WARN [NiFi Web Server-78]
> o.a.n.c.m.impl.HttpRequestReplicatorImpl Node request for
> [id=15c5e29c-b576-4ca7-b980-0be395461796, apiAddress=localhost,
> apiPort=8082, socketAddress=myhost.mydomain, socketPort=1044,
> siteToSiteAddress=localhost, siteToSitePort=null] encountered exception:
> java.util.concurrent.ExecutionException:
> com.sun.jersey.api.client.ClientHandlerException:
> java.net.SocketTimeoutException: Read timed out
> 2016-05-19 15:21:37,254 INFO [NiFi Web Server-78]
> o.a.n.c.m.e.NoConnectedNodesException
> org.apache.nifi.cluster.manager.exception.NoResponseFromNodesException: No
> nodes were able to process this request.. Returning Conflict response.
>
>
>
> On Thu, May 19, 2016 at 3:09 PM, Joe Witt  wrote:
>
>> Hello
>>
>> While it is clearly a very poor error/log it appears that some
>> property was not set in the nifi.properties file.  Can you share as
>> much of the properties file you can - specifically around cluster
>> configuration?
>>
>> Thanks
>> Joe
>>
>> On Thu, May 19, 2016 at 3:00 PM, Igor Kravzov 
>> wrote:
>> > NiFi 0.6.1
>> > I followed simple cluster setup guide. But when i start NCM it shuts
>> down
>> > and this is what I found in application log. What can be wrong?
>> >
>> >
>> > 2016-05-19 14:41:56,703 WARN [Thread-1]
>> > o.e.j.util.component.AbstractLifeCycle FAILED
>> > org.eclipse.jetty.server.Server@37b2e53f:
>> > org.springframework.beans.factory.BeanCreationException: Error creating
>> bean
>> > with name 'niFiWebApiSecurityConfiguration': Injection of autowired
>> > dependencies failed; nested exception is
>> > org.springframework.beans.factory.BeanCreationException: Could not
>> autowire
>> > method: public void
>> >
>> org.apache.nifi.web.NiFiWebApiSecurityConfiguration.setJwtService(org.apache.nifi.web.security.jwt.JwtService);
>> > nested exception is
>> org.springframework.beans.factory.BeanCreationException:
>> > Error creating bean with name 'jwtService' defined in class path
>> resource
>> > [nifi-web-security-context.xml]: Cannot resolve reference to bean
>> > 'userService' while setting constructor argument; nested exception is
>> > org.springframework.beans.factory.BeanCreationException: Error creating
>> bean
>> > with name 'userService' defined in class path resource
>> > [nifi-administration-context.xml]: Cannot resolve reference to bean
>> > 'userTransactionBuilder' while setting bean property
>> 'transactionBuilder';
>> > nested exception is
>> org.springframework.beans.factory.BeanCreationException:
>> > Error creating bean with name 'userTransactionBuilder' defined in class
>> path
>> > resource [nifi-administration-context.xml]: Cannot resolve reference to
>> bean
>> > 'authorityProvider' while setting bean property 'authorityProvider';
>> nested
>> > exception is org.springframework.beans.factory.BeanCreationException:
>> Error
>> > creating bean with name 'clusterManager': FactoryBean threw exception on
>> > object creation; nested exception is
>> > org.springframework.beans.factory.BeanCreationException: Error creating
>> bean
>> > with name 'clusterManagerProtocolSenderListener' defined in class path
>> > resource [nifi-cluster-protocol-context.xml]: Cannot resolve reference
>> to
>> > bean 'clusterManagerProtocolListener' while setting constructor
>> argument;
>> > nested exception is
>> > org.springframework.beans.factory.UnsatisfiedDependencyException: Error
>> > creating bean with name 'clusterManagerProtocolListener' defined in
>> class
>> > path resource [nifi-cluster-protocol-context.xml]: Unsatisfied
>> dependency
>> > expressed through constructor argument with index 1 of type [int]:
>> Could not
>> > convert constructor argument value of type [null] to required type
>> [int]:
>> > Failed to convert value of type [null] to required type [int]; nested
>> > exception is java.lang.IllegalArgumentException: Cannot convert value of
>> > type [null] to required type [int]: PropertyEditor
>> > [org.springframework.beans.propertyeditors.CustomNumberEditor] returned
>> > inappropriate value of type [null]
>> > 

Re: OutOfMemory / PermGen space

[Matthew Clarke]

Dmitry,
   What version of Java are you running?   My guess is you are running
Java 7 and in that case I suggest you uncomment out the java related lines
in the bootstrap.conf file:

java.arg.7=-XX:ReservedCodeCacheSize=256m
java.arg.8=-XX:CodeCacheFlushingMinimumFreeSpace=10m
java.arg.9=-XX:+UseCodeCacheFlushing
java.arg.11=-XX:PermSize=128M
java.arg.12=-XX:MaxPermSize=128M

 Restart you NiFi after making these changes.

  If you running Java 8, PermGen memory space just uses heap space
unlike Java 7.  so in that case try increasing the min and max
sizeconfigured for your heap in the same bootstrap.conf file:

java.arg.2=-Xms4096m
java.arg.3=-Xmx4096m

   Again any changes made to all but teh logback.xml config file require a
restart to take affect.

Matt

On Tue, Mar 29, 2016 at 2:31 PM, Dmitry Goldenberg  wrote:

> Hi,
>
> Started getting OOM once I've added PutHDFS to a flow.
>
> I'm running on a CentOS VM with 2GB of memory allocated. I've munged the
> nifi startup script to do -Xms1024m -Xmx1024m".   Any recommendations on
> this?
>
> Thanks,
> - Dmitry
>
> 2016-03-29 11:25:54,200 ERROR [Flow Service Tasks Thread-2]
> o.a.nifi.controller.StandardFlowService Unable to save flow controller
> configuration due to: java.lang.OutOfMemoryError: PermGen space
> java.lang.OutOfMemoryError: PermGen space
> at java.lang.ClassLoader.findBootstrapClass(Native Method) ~[na:1.7.0_67]
> at java.lang.ClassLoader.findBootstrapClassOrNull(ClassLoader.java:1070)
> ~[na:1.7.0_67]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:414) ~[na:1.7.0_67]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:412) ~[na:1.7.0_67]
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> ~[na:1.7.0_67]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ~[na:1.7.0_67]
> at
> org.apache.nifi.nar.NarThreadContextClassLoader.loadClass(NarThreadContextClassLoader.java:96)
> ~[nifi-nar-utils-0.5.1.jar:0.5.1]
> at
> javax.xml.transform.FactoryFinder.getProviderClass(FactoryFinder.java:114)
> ~[na:1.7.0_67]
> at javax.xml.transform.FactoryFinder.newInstance(FactoryFinder.java:182)
> ~[na:1.7.0_67]
> at javax.xml.transform.FactoryFinder.find(FactoryFinder.java:296)
> ~[na:1.7.0_67]
> at
> javax.xml.transform.TransformerFactory.newInstance(TransformerFactory.java:101)
> ~[na:1.7.0_67]
> at
> org.apache.nifi.controller.StandardFlowSerializer.serialize(StandardFlowSerializer.java:102)
> ~[na:na]
> at
> org.apache.nifi.controller.FlowController.serialize(FlowController.java:1262)
> ~[na:na]
> at
> org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.save(StandardXMLFlowConfigurationDAO.java:128)
> ~[na:na]
> at
> org.apache.nifi.controller.StandardFlowService$SaveReportingTask.run(StandardFlowService.java:842)
> ~[na:na]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> [na:1.7.0_67]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
> [na:1.7.0_67]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> [na:1.7.0_67]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> [na:1.7.0_67]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_67]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_67]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
>
>

Re: Routing based on conditions in NiFi - ?

[Matthew Clarke]

Dmitry,
   You will want to use the RouteOnAttribute processor with a "routing
strategy" of "route to property name" and two added custom properties.

   That one added property would be used to identify the FlowFiles that a
mime type of text/xml.
   For example:
   property:  value:
   type-xml   ${'mime.type':equals('text/xml')}
   type-html  ${'mime.type':equals('text/html')}

  The routeOnAttribute will then have three attributes (type-xml,
type-html,  and unmatched).
  How are you identifying the mime.type and setting that attribute on your
FlowFiles prior to this RouteOnAttribute?

Matt


On Thu, Mar 24, 2016 at 5:33 PM, Dmitry Goldenberg  wrote:

> Thanks, Pierre.
>
> It doesn't look like NiFi would create 2 relationships, one per condition.
> It only seems to allow for 'matched' and 'unmatched' relationships.
> Therefore, I believe I'll need:
>
>- a RouteOnAttribute with 'matched' that points to subflow for
>processing ofproperty=is_xml, value=mimetype equals 'text/xml'
>- the 'unmatched' on it would lead to the next RouteOnAttribute where
>'matched' would point to subflow for processing ofproperty=is_html,
>value=mimetype equals 'text/html'
>- the 'unmatched' on the second RouteOnAttribute would lead to the
>subflow for processing all other cases
>
> (It'll probably be OK to use the same boolean prop for 'is_xml' and
> 'is_html', something like 'is_match')
>
> It'd be quite great to have a visual representation of this if/then/else
> type of construct but the chained set of RouteOnAttribute's should work.
>
> - Dmitry
>
>
>
> On Thu, Mar 24, 2016 at 5:15 PM, Pierre Villard <
> pierre.villard...@gmail.com> wrote:
>
>> Hi Dmitry,
>>
>> I'd suggest to add RouteOnAttribute processor and add a custom property
>> by expected condition :
>> one property with name textxml (for example) and with value the EL you
>> suggested (it supposed that you have the mime type in one of the attribute
>> of your flow file)
>> property with name texthml (for example) and with value the EL
>> ${mimetype:equals('text/html')}
>>
>> then you will have two automaticcaly created relationship and one
>> "unmatched". It will answer your need.
>>
>> Hope this helps,
>> Pierre
>>
>> 2016-03-24 21:29 GMT+01:00 Dmitry Goldenberg :
>>
>>> Hi,
>>>
>>> I'm looking for a 'best practice' to set up a conditional within a
>>> dataflow, based on some attribute(s).
>>>
>>> I want to express something like this:
>>>
>>> if ("mimetype" equals "text/xml") {
>>> route FlowFile to XmlProcessingSubflow
>>> } else if ("mimetype" equals "text/html") {
>>> route FlowFile to HtmlProcessingSubflow
>>> } else {
>>> route FlowFile to DefaultDocProcessingSubflow
>>> }
>>>
>>> How can I do this in NiFi?  I'm looking at RouteOnAttribute but I can't
>>> quite grok the Routing Strategy:
>>>
>>> Route to Property name
>>> A copy of the FlowFile will be routed to each relationship whose
>>> corresponding expression evaluates to “true”.
>>>
>>> Route to ‘matched’ if all match
>>> Requires that all user-defined expressions evaluate to true for the
>>> FlowFile to be considered a match
>>>
>>> Route to ‘matched’ if any matches
>>> Requires that at least one user-defined expression evaluate to true for
>>> the FlowFile to be considered a match.
>>>
>>> The first option I'm quite confused about. Route to property? Where are
>>> the relationships I'd be routing to?
>>>
>>> The second option seems to be what I need. Presumably I can add a "New
>>> property" with the name "mimetype"; the value -- not sure. Perhaps this
>>> is ${mimetype:equals('text/xml')}?  But I already have "mimetype" in
>>> the property name.  Should the expression be somewhat different?
>>>
>>> Subsequently, if this allows me to route to one subflow based on one
>>> value, how do I route based on a different value of the mimetype, e.g.
>>> "text/html", to another subflow?
>>>
>>> Thanks.
>>> - Dmitry
>>>
>>>
>>>
>>>
>>
>

Re: List Files

[Matthew Clarke]

The expression language can also be used to return values from both JVM
properties set in the NiFi bootstrap.conf and system environment variables
for the user running NiFi.
On Mar 4, 2016 11:00 AM, "Joe Percivall"  wrote:

> Hello,
>
> ListFile is a source processor so this behavior is expected. It supports
> expression language so that it can be configured to utilize certains
> methods. For example, some people may want to get a list of files from a
> rotating directory that gets created every hour. To do that they would need
> to use the date functions included in Expression Language.
>
> Does forbidding input hinder a specific use-case you have?
>
> Joe
>
> - - - - - -
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
>
>
>
> On Friday, March 4, 2016 10:38 AM, Charlie Frasure <
> charliefras...@gmail.com> wrote:
>
>
>
> I'm using the 0.5.1 build and having what I think is odd behavior for
> ListFile.  The processor supports expression language in the Input
> Directory property, however I can't figure out how to configure an
> attribute as input.
>
> I tried using UpdateAttribute prior to ListFile, but ListFile doesn't seem
> to allow incoming connections.  I also tried creating an attribute within
> ListFile, but it doesn't seem to be available for referencing within the
> same processor.
>
> Is this expected?  If so, what is the intended purpose of allowing
> expression language in this attribute?
>

Re: splitText output appears to be getting dropped

[Matthew Clarke]

Conrad,
 The mergeContent processor will bin files based upon the configuration
you have configured.  Since it is taking multiple files and creating one
output file from them, that output file cannot have multiple filenames.
MergeContent will use the filename of the first file in the bin as the
filename of the output file.  As far as the rest of the attributes go from
the numerous source files, the 'Attribute Strategy' property in
MergeContent determines how they are applied to the new output file.

Matt

On Fri, Feb 19, 2016 at 11:25 AM, Conrad Crampton <
conrad.cramp...@secdata.com> wrote:

> Hi,
> Perfect!
> I tried \n for linefeed – didn’t think of shift+enter!
>
> The reason I was updating filename early on in my flow file was just
> because I already had UpdateAttributes that was a handy place to do so. I
> can put it just before the PutFile though so no major issue, just wondered
> why this was happening and if it was be design (feature) or bug.
>
> Thanks
> Conrad
>
> From: Bryan Bende <bbe...@gmail.com>
> Reply-To: "users@nifi.apache.org" <users@nifi.apache.org>
> Date: Friday, 19 February 2016 at 16:16
> To: "users@nifi.apache.org" <users@nifi.apache.org>
> Subject: Re: splitText output appears to be getting dropped
>
> Hello,
>
> MergeContent has properties for header, demarcator, and footer, and also
> has a strategy property which specifies whether these values come from a
> file or inline text.
>
> If you do inline text and specify a demarcator of a new line (shift +
> enter in the demarcator value) then binary concatenation will get you all
> of the lines merged together with new lines between them.
>
> As far as the file naming, can you just wait until after RouteContent to
> rename them? They just need be renamed before the PutFile, but it doesn't
> necessarily have to be before RouteOnContent.
>
> Let us know if that helps.
>
> Thanks,
>
> Bryan
>
>
> On Fri, Feb 19, 2016 at 11:01 AM, Conrad Crampton <
> conrad.cramp...@secdata.com> wrote:
>
>> Hi,
>> Sorry to piggy back on this thread, but I have pretty much the same issue
>> – I am splitting log files -> routeoncontent (various paths) two of these
>> paths (including unmatched), basically need to just get farmed off into a
>> directory just in case they are needed later.
>> These go into a MergeContent processor where I would like to merge into
>> one file – each flowfile content as a line in the file delimited by line
>> feed (as like the original file), whichever way I try this though doesn’t
>> quite do what I want. If I try BinaryConcatenation the file ends up as one
>> long line, if TAR each Flowfile is a separate file in a TAR (not
>> unsurprisingly). There doesn’t seem to be anyway of merging flow file
>> content into one file (that ideally has similar functions to be able to
>> compress, specify number of files etc.)
>>
>> Another related question to the answer below (really helped me out with
>> same issue), however if I rename the filename early on in my process flow,
>> it appears to be changed back to its original at MergeContent processor
>> time so I have to put another UpdateAttributes step in after the Merge to
>> rename the filename.
>> The flow is
>>
>> UpdateAttributes -> RouteOnContent -> UpdateAttribute -> MergeContent ->
>> PutFile
>>  ^   ^^ ^
>>  |  | ||
>> Filename changed samesame reverted
>>
>> If I put an extra UpdateAttribute before PutFile then fine. Logging at
>> each of the above points shows filename updated to ${uuid}-${filename}, but
>> at reverted is back at filename.
>>
>> Any suggestions on particularly the first question??
>>
>> Thanks
>> Conrad
>>
>>
>>
>> From: Jeff Lord <jeffrey.l...@gmail.com>
>> Reply-To: "users@nifi.apache.org" <users@nifi.apache.org>
>> Date: Friday, 19 February 2016 at 03:22
>> To: "users@nifi.apache.org" <users@nifi.apache.org>
>> Subject: Re: splitText output appears to be getting dropped
>>
>> Matt,
>>
>> Thanks a bunch!
>> That did the trick.
>> Is there a better way to handle this out of curiosity? Than writing out a
>> single line into multiple files.
>> Each file contains a single string that will be used to build a url.
>>
>> -Jeff
>>
>> On Thu, Feb 18, 2016 at 6:00 PM, Matthew Clarke <
>> matt.clarke@gmail.com> wrote:
>>
>>> Jeff,
>>>   It appears you files are being dropped because your are
>>> auto-terminating the failure relationship on y

Re: splitText output appears to be getting dropped

[Matthew Clarke]

Jeff,
  It appears you files are being dropped because your are
auto-terminating the failure relationship on your putFile processor. When
the splitText processor splits the file by lines every new file has the
same filename as the original it came from. My guess is the first file is
being worked to disk and all others are failing because a file of the same
name already exists in target dir. Try adding an UpdateAttribute processor
after the splitText to rename all the files. Easiest way is to append the
files uuid to its filename.  I also do not recommend auto-terminating
failure relationships except in rare cases.

Matt
On Feb 18, 2016 8:36 PM, "Jeff Lord"  wrote:

> I have a pretty simple flow where I query for a list of ids using
> executeProcess and than pass that list along to splitText where I am trying
> to split on each line to than dynamically build a url further down the line
> using updateAttribute and so on.
>
> executeProcess -> splitText -> putFile
>
> For some reason I am only getting one file written with one line.
> I would expect something more like 100 files each with one line.
> Using the provenance reporter it appears that some of my items are being
> dropped.
>
> Time02/18/2016 17:13:46.145 PST
> Event DurationNo value set
> Lineage Duration00:00:12.187
> TypeDROP
> FlowFile Uuid7fa42367-490d-4b54-a32f-d062a885474a
> File Size14 bytes
> Component Id3b37a828-ba2c-4047-ba7a-578fd0684ce6
> Component NamePutFile
> Component TypePutFile
> DetailsAuto-Terminated by failure Relationship
>
> Any ideas on what I need to change here?
>
> Thanks in advance,
>
> Jeff
>

Re: SSL configuration on cluster

[Matthew Clarke]

Now that you have all the keys you need for the systems in your cluster,
you will need to create some keys for your users to load into their web
browsers in order to securely access your NiFi. This is done in much of the
same way as you created your server keys:

1. openssl genrsa -out .key 2048

2. openssl req -new -key .key -out .csr

3. openssl x509 -req -in .csr -CA rootCA.pem -CAkey rootCA.key
-CAcreateserial -out .crt -days 800

4. openssl pkcs12 -export -out .p12 -inkey .key -in
.crt -certfile rootCA.srl



Now you have a p12 file for user1. they can load this in to their browser
certs to use to authenticate against your secure NiFi.



Now remember you must manually add that first "ROLE_ADMIN" user to the
authorized-users.xml file. So you will need the DN from the user key you
created for this Admin user and add it in to your Authorized-users.xml
file.



Here is an example of what it may look like:

dn="EMAILADDRESS=n...@none.com, CN=, OU=NiFi, O=someplace,
L=Baltimore, ST=Maryland, C=US"



If you have the DN format wrong in your authorized-users.xml file, rather
then gaining access to the NiFi you will get prompted to "request access”.
Do *Not* click the request access link. You must instead go fix the DN in
the authorized-users.xml file.  You need to create that first admin account
that can approve those requests.  If you click request access , you will
need to stop your NiFi and delete the user db, otherwise, even fixing your
authorized-usesr.xml file will not gain you access because your account
will be stuck in a pending auth state.



You can look at the request that came in in the nifi-users.log to get the
exact DN pattern to fix your authorized-usesr.xml file entry:

You should see something that looks like this:



INFO [NiFi Web Server-58023] o.a.n.w.s.x509.X509AuthenticationFilter
Attempting request for 

Re: ListenHTTP processor

[Matthew Clarke]

NiFi only needs to be running as root if you are trying to use ports 1024
or below. Ports above 1024 do not require privileged access on Linux. Try
using port 8081 instead for your listenHTTP processor. This will allow you
to run your NiFi as a non root user.
On Jan 20, 2016 1:23 AM, "sudeep mishra"  wrote:

> Thanks Joe.
>
> One doubt though... maybe some Linux changes required but do I have to run
> NiFi as root user only to make use of ListenHTTP processor?
>
> On Tue, Jan 19, 2016 at 7:06 PM, Joe Witt  wrote:
>
>> Hello
>>
>> Sounds like you're on track now.  Just wanted to correct one comment made
>>
>>   ListenHTTP by default does not set an input port and instead it is a
>> required input.
>>
>> Thanks
>> Joe
>>
>> On Tue, Jan 19, 2016 at 7:49 AM, sudeep mishra 
>> wrote:
>> > Listening on port 81 on Linux RHEL 6.5
>> >
>> > On Tue, Jan 19, 2016 at 6:04 PM,  wrote:
>> >>
>> >> Hi Sudeep,
>> >>
>> >> On which port are you listening ?
>> >> Also which OS are you using ?
>> >>
>> >> Best regards,
>> >> Louis-Etienne
>> >>
>> >> > On Jan 19, 2016, at 2:11 AM, sudeep mishra > >
>> >> > wrote:
>> >> >
>> >> > Hi,
>> >> >
>> >> > I am getting an error as 'failed to invoke @OnSchedule method due to
>> >> > java.net.SocketException:Permission denied' when using 'ListenHTTP
>> >> > processor'.
>> >> >
>> >> > Please suggest how to resolve the issue.
>> >> >
>> >> > Thanks & Regards,
>> >> >
>> >> > Sudeep Shekhar Mishra
>> >> >
>> >
>> >
>> >
>> >
>> > --
>> > Thanks & Regards,
>> >
>> > Sudeep Shekhar Mishra
>> >
>> >
>>
>
>
>
> --
> Thanks & Regards,
>
> Sudeep Shekhar Mishra
>
>
>

Re: One-time Processors

[Matthew Clarke]

Lars,
If start a processor and then stop it, the thread does not get killed.
The processor will continue to run until that thread completes its job. The
processor will then remain stopped and not execute again until manually
started again.

Matt
On Jan 18, 2016 10:54 AM, "Joe Percivall"  wrote:

> Joe and Lars,
>
> I created a Jira ticket for this issue:
> https://issues.apache.org/jira/browse/NIFI-1407
>
> We can continue the conversation there instead two different email threads.
>
> Joe
> - - - - - -
> *Joseph Percivall*
> linkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
>
>
>
> On Sunday, January 10, 2016 4:03 PM, Lars Francke 
> wrote:
>
>
> Joe,
>
> thanks for the welcome and the (really quick) reply.
>
> My use-case (and again I may use NiFi for things it wasn't intended to) is
> for a one-time/initial-import scenario. I have a Processor that I'd like to
> start that then calls a Web service and pages through all results creating
> FlowFiles and transferring them onwards. When it's reached the end I'd like
> it to Stop automatically without being rescheduled.
>
> I'd like to do it this way to maximise code-reuse with other flows (that
> for example do incremental imports, a reason I'm excited about the upcoming
> State Management) and get the benefits of monitoring etc.
>
> So the feature proposed would work for me (run X times). Has a JIRA for
> this been filed that I can track (looked but I couldn't find one)?
> If I understand correctly there's currently no way from within a Processor
> to stop itself?
>
> Cheers,
> Lars
>
>
>
> On Sun, Jan 10, 2016 at 9:37 PM, Joe Witt  wrote:
>
> Lars,
>
> First, welcome to NiFi.
>
> So you'll initiate the flow from NiFi through some manual trigger and
> that part makes sense/sounds straightforward.  But then the question
> is 'once the flow is done' I'd like it to shut down.  What are the
> processors/things that begin the flow?  We've recently had someone
> suggest adding a way to tell a processor to only run 'X' times before
> being unscheduled and thus requiring manual start.  Sounds like that
> might help you with your case as well.
>
> There is little impact to having all the processors after the first
> one running.  If there is no work to do they'll be quite efficient.
>
> Do you think if a processor had a 'max executions' concept that it
> would address your need?
>
> Thanks
> Joe
>
> On Sun, Jan 10, 2016 at 3:32 PM, Lars Francke 
> wrote:
> > Hi,
> >
> > I'm just getting started with Nifi and understand that I probably still
> have
> > some misconceptions :)
> >
> > For various reasons I'd love to use NiFi for one-time actions/processors
> > which I'd like to trigger as needed from the UI. I've seen the various
> > scheduling strategies and the closest I've come is to have a really long
> run
> > schedule or a specific cron schedule.
> >
> > What I'd really like is a way for a Processor to gracefully stop itself
> when
> > it's done processing.
> >
> > Is there any way to achieve this?
> >
> > Thanks for your help
>
>
>
>
>

Re: trouble with remote process groups.

[Matthew Clarke]

Russel,

 What version of NiFi are your running?

Thanks,
Matt



On Fri, Jan 15, 2016 at 9:32 AM, Ritenour, Russell C. <
russell.c.riten...@leidos.com> wrote:

> Matt;
>
>
>
> Thanks for suggestion.   The meesage is definitely related to when the
> flow contains the remote process group (not necessarily the site-to-site
> communication).   When I remove the RPG from the flow, the message stops
> appearing.   Adding it back, generates the message.
>
>
>
> I took your suggestion and made sure the
> niagarafiles.remote.input.socket.host value was set on dev1 and restarted
> the nifi instance on both dev1 and dev2.   Upon reboot, I see a message in
> dev1's log with the message:
>
>
>
> ... Unable to determine whether or not site-to-site communications  with
> peer should be secure
>
>
>
> I'm guessing that validates your thought on the network (and/or the SSL
> setup) is incorrect on these machines.  Is there a good way to test how
> NIFI is determining the IP?   Note, I tried using the IP directly but got
> error messages in the log that the HTTPS hostname is wrong.
>
>
> --
> *From:* Matthew Clarke [matt.clarke@gmail.com]
> *Sent:* Thursday, January 14, 2016 5:42 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: trouble with remote process groups.
>
> Russ,
>   The message you provided had nothing to do with your site-to-site not
> working between hosts. The likely problem is caused by unknown host
> problems. In your NiFi.properties configuration file under the site-to-site
> section there are 3 properties (host, port, and secure). My guess is you
> configured a port and change secure to false but left host blank. It if
> likely your host is resolving to localhost and sending that in the
> connection URL for S2S communication. Try populating host with a hostname
> that the other NiFi can resolve our an IP and restart. Let us know if that
> solves your S2S problem.
>
> Thanks,
> Matt
> On Jan 14, 2016 3:47 PM, "Ritenour, Russell C." <
> russell.c.riten...@leidos.com> wrote:
>
>> To whom it may concern;
>>
>>
>>
>> I'm attempting to create a remote process group and send data to it from
>> a number of hosts.   First, I set up a couple of input ports and verified
>> that I could create a remote process group connection from the same host.
>> When I try to repeat the remote access from another host, I cannot see any
>> of the input ports defined and I see entries in the sending hosts log with
>> the message:
>>
>>
>>
>> "A message body reader for Java class
>> niagarafiles.web.api.entity.ControllerEntity, and Java type class
>> niagarafiles.web.api.entityt.ControllerEntity and MIME media type
>> application/json was not found"
>>
>>
>>
>> Afterwards, there is a list of compatible MIME media types...
>>
>>
>>
>> So, 2 questions:
>>
>> 1) why does this work locally and not remotely
>>
>> 2) what configuration might I be missing in the remote configuration
>>
>>
>>
>> Thank you in advance for your time and effort.
>>
>>
>>
>> Russ
>>
>

Re: nifi 0.4.1 does not start

[Matthew Clarke]

I'd this a brand new install or are you upgrading and using a flow.xml.gz
file from a previous install?  If it is a brand new install, some Linux
install do not provide enough open file handles by default I allow NiFi to
even start. Try increasing the available file handles. You can see the
current limits by running the ulimit -a command.  The stack trace in the
logs Andy was asking for should indicate the specific class it is failing
to find. If this is an upgrade, dig your previous NiFi have any custom
processors?

Thanks, Matt
On Jan 14, 2016 5:32 PM, "Andy LoPresto"  wrote:

> Hi Frank,
>
> Sorry to hear you are having trouble. Can you please provide a full stack
> trace? It should help us narrow down the issue.
>
> Andy LoPresto
> alopresto.apa...@gmail.com
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Jan 14, 2016, at 2:23 PM, Frank Martínez  wrote:
>
> Hi Guys,
>
> I have just downloaded apache nifi 0.4.1, extracted and run bin/nifi.sh
> start
> But it does not start. It fails immediatly with:
>
> Caused by: java.lang.ClassNotFoundException:
> org.apache.nifi.distributed.cache.client.Serializer
>
> Any help will be appreciated.
>
> Thanks,
>
> Frank Martínez.
>
> PS. I am using Linux Mint, and JDK 1.7.0_67
>
> --
> Frank D. Martínez M.
>
>
>

Re: trouble with remote process groups.

[Matthew Clarke]

Russ,
  The message you provided had nothing to do with your site-to-site not
working between hosts. The likely problem is caused by unknown host
problems. In your NiFi.properties configuration file under the site-to-site
section there are 3 properties (host, port, and secure). My guess is you
configured a port and change secure to false but left host blank. It if
likely your host is resolving to localhost and sending that in the
connection URL for S2S communication. Try populating host with a hostname
that the other NiFi can resolve our an IP and restart. Let us know if that
solves your S2S problem.

Thanks,
Matt
On Jan 14, 2016 3:47 PM, "Ritenour, Russell C." <
russell.c.riten...@leidos.com> wrote:

> To whom it may concern;
>
>
>
> I'm attempting to create a remote process group and send data to it from a
> number of hosts.   First, I set up a couple of input ports and verified
> that I could create a remote process group connection from the same host.
> When I try to repeat the remote access from another host, I cannot see any
> of the input ports defined and I see entries in the sending hosts log with
> the message:
>
>
>
> "A message body reader for Java class
> niagarafiles.web.api.entity.ControllerEntity, and Java type class
> niagarafiles.web.api.entityt.ControllerEntity and MIME media type
> application/json was not found"
>
>
>
> Afterwards, there is a list of compatible MIME media types...
>
>
>
> So, 2 questions:
>
> 1) why does this work locally and not remotely
>
> 2) what configuration might I be missing in the remote configuration
>
>
>
> Thank you in advance for your time and effort.
>
>
>
> Russ
>

Re: PutDistributedMapCache

[Matthew Clarke]

Sudeep,
   I was a little off on my second scenario.  The detectduplicate
processor uses the distributedcache service all on its own.. Files that are
route through it are loaded into the cache if they do not already exist in
the cache.  if they do already exist they are routed to duplicate.  The
putDistributedCache processor was a community contribution to which there
are no processor that make use of the info that it caches.

   We should probably build a processor that would make use of the data
that can be loaded by the putDistributeCache processor.  Is there a
particular use case you are trying to solve where this would be applicable?

Thanks,
Matt

On Tue, Jan 12, 2016 at 8:11 AM, Matthew Clarke <matt.clarke@gmail.com>
wrote:

> Sudeep,
> The DistributedMapCache is typically used to prevent the consumption
> of duplicate data by some of the ingest type processors (GetHBASE,
> ListHDFS, and ListSFTP).  NiFi uses the service to keep a listing of what
> has been consumed so the same files are not consumed multiple times. The
> Service can also be used to detect if duplicate data already exists within
> a NiFi Instance or cluster. This would be the scenario where some source is
> pushing data to your NiFi and perhaps they push the same data more than
> once. You want to catch these duplicates so you can perhaps kick them out
> of your flow. For this you would use the PutDistributedCache processor to
> cache all incoming data and then use the DetectDuplicate processor to find
> those duplicates.
>
> Was there a different use case you were looking to solve using the
> Distributed cache service?
>
> Thanks,
> Matt
>
> On Tue, Jan 12, 2016 at 4:36 AM, sudeep mishra <sudeepshekh...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I can cache some data to be used in NiFi flow. I can see the
>> processor PutDistributedMapCache in the documentation which saves key-value
>> pairs in DistributedMapCache for NiFi but I do not see any processor to red
>> this data. How can I read data from DistributedMapCache in my data flow?
>>
>>
>> Thanks & Regards,
>>
>> Sudeep Shekhar Mishra
>>
>>
>

Re: PutDistributedMapCache

[Matthew Clarke]

Sudeep,
The DistributedMapCache is typically used to prevent the consumption of
duplicate data by some of the ingest type processors (GetHBASE, ListHDFS,
and ListSFTP).  NiFi uses the service to keep a listing of what has been
consumed so the same files are not consumed multiple times. The Service can
also be used to detect if duplicate data already exists within a NiFi
Instance or cluster. This would be the scenario where some source is
pushing data to your NiFi and perhaps they push the same data more than
once. You want to catch these duplicates so you can perhaps kick them out
of your flow. For this you would use the PutDistributedCache processor to
cache all incoming data and then use the DetectDuplicate processor to find
those duplicates.

Was there a different use case you were looking to solve using the
Distributed cache service?

Thanks,
Matt

On Tue, Jan 12, 2016 at 4:36 AM, sudeep mishra 
wrote:

> Hi,
>
> I can cache some data to be used in NiFi flow. I can see the
> processor PutDistributedMapCache in the documentation which saves key-value
> pairs in DistributedMapCache for NiFi but I do not see any processor to red
> this data. How can I read data from DistributedMapCache in my data flow?
>
>
> Thanks & Regards,
>
> Sudeep Shekhar Mishra
>
>

Re: Route On Attribute Processor

[Matthew Clarke]

Naveen,
You need to add new properties to that processor. Each property you add
becomes a new relationship. You can use the NiFi expression language to
construct your routing rule in each property. What attribute are you trying
to use to route with? I can help you create some rules.

Matt
On Nov 11, 2015 3:50 PM, "Madhire, Naveen" 
wrote:

> Hi,
>
> I’ve a question on RouteOnAttribute processor, I don’t see any “matched”
> relationship in the processor, it only has “unmatched” relationship.
> I’ve implemented this and I could only see the “unmatched” ones.
>
> So what happens to the flow files when the condition actually satisfies?
>
>
> Auto terminate relationships [image: Info]
> unmatched
> FlowFiles that do not match any user-define expression will be routed here
> Thanks,
> Naveen
>
> --
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

Re: site to site setup - Remore instance not configured

[Matthew Clarke]

et.ssl.SSLSocketChannel.performTasks(SSLSocketChannel.java:250)
> ~[nifi-utils-0.3.0.jar:0.3.0]
>
> at
> org.apache.nifi.remote.io.socket.ssl.SSLSocketChannel.performHandshake(SSLSocketChannel.java:236)
> ~[nifi-utils-0.3.0.jar:0.3.0]
>
> ... 3 common frames omitted
>
> 2015-10-13 16:38:41,475 ERROR [Site-to-Site Worker Thread-254] 
> o.a.nifi.remote.SocketRemoteSiteListener
> RemoteSiteListener Unable to accept connection from Socket[unconnected] due
> to javax.net.ssl.SSLException: Inbound closed before receiving peer's
> close_notify: possible truncation attack?
>
>
>
> From: Matt Clarke <matt.clarke@gmail.com>
> Reply-To: "users@nifi.apache.org" <users@nifi.apache.org>
> Date: Tuesday, October 13, 2015 at 2:51 PM
>
> To: "users@nifi.apache.org" <users@nifi.apache.org>
> Subject: Re: site to site setup - Remore instance not configured
>
> So you're "nifi flow" RPG should not have any connection lines coming out
> of it.  The input port should not have any connections coming in to it. You
> should have what looks like two separate independent flows on your graph.
> Flow 1 would consist of the listenHTTP connecting to the RPG directly.
> When draw the connection to the RPG, the connection window will ask you
> which input port you want to connect with. Flow 2 would consist of the
> input port you picked in flow 1 connecting directly to the putFile.  The
> RPG will handle the load balancing for you automatically.
>
> Thanks,
> Matt
>
> *Sent from my Verizon Wireless 4G LTE DROID*
>
>
> Chakrader Dewaragatla <chakrader.dewaraga...@lifelock.com> wrote:
>
> Sweet, we need load balancer data on other end.
> how do I make it work ? Here is the nifi canvas screenshot.
>
> *http://tinyurl.com/nq9fbqr <http://tinyurl.com/nq9fbqr>*
>
>
> From: Matthew Clarke <matt.clarke@gmail.com>
> Reply-To: "users@nifi.apache.org" <users@nifi.apache.org>
> Date: Tuesday, October 13, 2015 at 2:17 PM
> To: "users@nifi.apache.org" <users@nifi.apache.org>
> Subject: Re: site to site setup - Remore instance not configured
>
> Yes, the Remote Process Group (RPG, also known as the site-to-site) will
> talk to the NCM which will provide connected nodes status information. It
> will then load balancer data to the nodes in that cluster.
> On Oct 13, 2015 5:06 PM, "Chakrader Dewaragatla" <
> chakrader.dewaraga...@lifelock.com> wrote:
>
>> Thanks Matthew it worked, no errors this time.
>> As noted below, we would like to consume http data on primary node and
>> send it back to cluster for processing.
>>
>> I have site to site setup with http listener(on primary node) —>  Input
>> port  —> SitetoSite to NCM cluster instance  —> putfile —> S3upload .
>> Does my setup achieve the purpose?
>>
>> Data movement from http listener to input port is not working and no
>> errors reported.
>>
>> Site-to-site admin document has limited information to understand.
>>
>> Thanks,
>> -Chakri
>>
>> From: Matthew Clarke <matt.clarke@gmail.com>
>> Reply-To: "users@nifi.apache.org" <users@nifi.apache.org>
>> Date: Monday, October 12, 2015 at 4:49 PM
>> To: "users@nifi.apache.org" <users@nifi.apache.org>
>> Subject: Re: site to site setup - Remore instance not configured
>>
>> For Site to Site properties the input socket host is optional. When it is
>> configured it should be set the hostname or IP of the the system the NiFi
>> instance is running on.  you must however configure an input socket port on
>> every instance. This includes all nodes and the NCM.  If you cluster is
>> configured to run securely (HTTPS), you should also have input secure set
>> to true.  Make sure you have the ports you use open in the firewalls
>> between systems.
>> On Oct 12, 2015 7:41 PM, "Chakrader Dewaragatla" <
>> chakrader.dewaraga...@lifelock.com> wrote:
>>
>>> Hi – I have a use case to collect http post requests on a nifi-clustered
>>> setup. My cluster has three nodes.
>>>
>>> Ncm —> Slave 1 (Primary)
>>>  —> Slave 2.
>>>
>>> I would like to setup a http listener on primary node and establish site
>>> to site connection to the same cluster for further processing data. So http
>>> payload receive as follows
>>>
>>>   http post —> Slave 1 (ListenHttp )— > (Site-to-site) — > NCM (put
>>> file) and (S3 upload) (I assume this data process by two slaves nodes)
>>>
>>> I have

Re: site to site setup - Remore instance not configured

[Matthew Clarke]

Yes, the Remote Process Group (RPG, also known as the site-to-site) will
talk to the NCM which will provide connected nodes status information. It
will then load balancer data to the nodes in that cluster.
On Oct 13, 2015 5:06 PM, "Chakrader Dewaragatla" <
chakrader.dewaraga...@lifelock.com> wrote:

> Thanks Matthew it worked, no errors this time.
> As noted below, we would like to consume http data on primary node and
> send it back to cluster for processing.
>
> I have site to site setup with http listener(on primary node) —>  Input
> port  —> SitetoSite to NCM cluster instance  —> putfile —> S3upload .
> Does my setup achieve the purpose?
>
> Data movement from http listener to input port is not working and no
> errors reported.
>
> Site-to-site admin document has limited information to understand.
>
> Thanks,
> -Chakri
>
> From: Matthew Clarke <matt.clarke@gmail.com>
> Reply-To: "users@nifi.apache.org" <users@nifi.apache.org>
> Date: Monday, October 12, 2015 at 4:49 PM
> To: "users@nifi.apache.org" <users@nifi.apache.org>
> Subject: Re: site to site setup - Remore instance not configured
>
> For Site to Site properties the input socket host is optional. When it is
> configured it should be set the hostname or IP of the the system the NiFi
> instance is running on.  you must however configure an input socket port on
> every instance. This includes all nodes and the NCM.  If you cluster is
> configured to run securely (HTTPS), you should also have input secure set
> to true.  Make sure you have the ports you use open in the firewalls
> between systems.
> On Oct 12, 2015 7:41 PM, "Chakrader Dewaragatla" <
> chakrader.dewaraga...@lifelock.com> wrote:
>
>> Hi – I have a use case to collect http post requests on a nifi-clustered
>> setup. My cluster has three nodes.
>>
>> Ncm —> Slave 1 (Primary)
>>  —> Slave 2.
>>
>> I would like to setup a http listener on primary node and establish site
>> to site connection to the same cluster for further processing data. So http
>> payload receive as follows
>>
>>   http post —> Slave 1 (ListenHttp )— > (Site-to-site) — > NCM (put file)
>> and (S3 upload) (I assume this data process by two slaves nodes)
>>
>> I have following error at site-to-site setup : * Remote instance Is not
>> configured for site-to-site communications at this time.*
>>
>> I followed the admin doc to set the properties, as follows (on slaves).
>>
>> # Site to Site properties
>>
>> nifi.remote.input.socket.host=10.83.14.59(NCM ip)
>>
>> nifi.remote.input.socket.port=
>>
>> nifi.remote.input.secure=false
>>
>>
>>
>> Any thoughts?
>>
>>
>> Thanks,
>>
>> -Chakri
>> --
>> The information contained in this transmission may contain privileged and
>> confidential information. It is intended only for the use of the person(s)
>> named above. If you are not the intended recipient, you are hereby notified
>> that any review, dissemination, distribution or duplication of this
>> communication is strictly prohibited. If you are not the intended
>> recipient, please contact the sender by reply email and destroy all copies
>> of the original message.
>> --
>>
> --
> The information contained in this transmission may contain privileged and
> confidential information. It is intended only for the use of the person(s)
> named above. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
> --
>

Re: site to site setup - Remore instance not configured

[Matthew Clarke]

For Site to Site properties the input socket host is optional. When it is
configured it should be set the hostname or IP of the the system the NiFi
instance is running on.  you must however configure an input socket port on
every instance. This includes all nodes and the NCM.  If you cluster is
configured to run securely (HTTPS), you should also have input secure set
to true.  Make sure you have the ports you use open in the firewalls
between systems.
On Oct 12, 2015 7:41 PM, "Chakrader Dewaragatla" <
chakrader.dewaraga...@lifelock.com> wrote:

> Hi – I have a use case to collect http post requests on a nifi-clustered
> setup. My cluster has three nodes.
>
> Ncm —> Slave 1 (Primary)
>  —> Slave 2.
>
> I would like to setup a http listener on primary node and establish site
> to site connection to the same cluster for further processing data. So http
> payload receive as follows
>
>   http post —> Slave 1 (ListenHttp )— > (Site-to-site) — > NCM (put file)
> and (S3 upload) (I assume this data process by two slaves nodes)
>
> I have following error at site-to-site setup : * Remote instance Is not
> configured for site-to-site communications at this time.*
>
> I followed the admin doc to set the properties, as follows (on slaves).
>
> # Site to Site properties
>
> nifi.remote.input.socket.host=10.83.14.59(NCM ip)
>
> nifi.remote.input.socket.port=
>
> nifi.remote.input.secure=false
>
>
>
> Any thoughts?
>
>
> Thanks,
>
> -Chakri
> --
> The information contained in this transmission may contain privileged and
> confidential information. It is intended only for the use of the person(s)
> named above. If you are not the intended recipient, you are hereby notified
> that any review, dissemination, distribution or duplication of this
> communication is strictly prohibited. If you are not the intended
> recipient, please contact the sender by reply email and destroy all copies
> of the original message.
> --
>